Knowledge
The original meaning of the gradient is a vector (vector), which means that the directional derivative of a function at that point takes the maximum value along the direction, that is, the function At this point, it changes the fastest along this direction (the direction of this gradient), and the rate of change is the largest (modulo the gradient). I feel that it is actually the direction of the partial derivative vector, along this The vector direction can find the local extremum.
from random import randomdef gradient_down(func, part_df_func, var_num, rate=0.1, max_iter=10000, tolerance= 1e-10): """ does not rely on the third library to achieve gradient descent: param func: loss (error) function: param part_df_func: partial derivative vector of loss function: param var_num: number of variables: param rate: learning rate (parameter The amplitude of each change of the parameter): param max_iter: the maximum number of calculations: param tolerance: the accuracy of the error: return: theta, y_current: the list of weight parameter values, the minimum value of the loss function """ theta = [random() for _ in range (var_num)] # Initial value of randomly given parameters y_current = func(*theta) # Parameter unpacking for i in range(max_iter): # Calculate the gradient of the current parameter (the partial derivative derivative vector value) gradient = [f(* theta) for f in part_df_func] # Update the parameters according to the gradient theta for j in range(var_num): theta[j] -= gradient[j] * rate # [0.3, 0.6, 0.7] ==> [0.3-0.3*lr , 0.6-0.6*lr, 0.7-0.7*lr] y_current, y_predict = func(*theta), y_current print(f"The {i}th iteration is in progress, and the error accuracy is {abs(y_predict-y_current)}") if abs(y_predict-y_current)
...
.. .
In the 248th iteration, the error accuracy is 1.6640999689343516e-10
The 249th iteration is in progress, and the error accuracy is 1.5684031851037616e-10
The 250th iteration is in progress, the error accuracy is 1.478208666583214e-10
The 251st iteration is in progress, and the error accuracy is 1.3931966691416164e-10
In the 252nd iteration, the error accuracy is 1.3130829756846651e-10
The 253rd iteration is in progress, and the error accuracy is 1.2375700464417605e-10
The 254th iteration is in progress, and the error accuracy is 1.166395868779091e -10
The 255th iteration is in progress, and the error accuracy is 1.0993206345233375e-10
The 256th iteration is in progress, and the error accuracy is 1.0361000946090826e-10
The 257th iteration is in progress, error The accuracy is 9.765166453234997e-11
ok, at the 257th iteration, the convergence is ok!
The optimal solution of the function is: When theta takes: [1.0, 2.0], f(theta) takes the minimum value: 2.0
[Finished in 0.0s]