机器学习基础（4）参数学习（梯度下降算法）

由上一篇文章我们知道，当$J(\theta_{0},\theta_{1})$J(θ0​,θ1​)最小时，参数$(\theta_{0},\theta_{1})$(θ0​,θ1​)所构成的假设函数$h_{\theta}=\theta_{0}+\theta_{1}x$hθ​=θ0​+θ1​x最能够拟合样本数据.

我们以$\theta_{0}$θ0​为x轴，$\theta_{1}$θ1​为y轴，代价函数为Z轴画出$\theta_{0}$θ0​、$\theta_{1}$θ1​取不同值的图像：

We will know that we have succeeded when our cost function is at the very bottom of the pits in our graph, i.e. when its value is the minimum. The red arrows show the minimum points in the graph.

The way we do this is by taking the derivative (the tangential line to a function) of our cost function. The slope of the tangent is the derivative at that point and it will give us a direction to move towards. We make steps down the cost function in the direction with the steepest descent. The size of each step is determined by the parameter α, which is called the learning rate.

For example, the distance between each ‘star’ in the graph above represents a step determined by our parameter α. A smaller α would result in a smaller step and a larger α results in a larger step. The direction in which the step is taken is determined by the partial derivative of $J(\theta_0,\theta_1)$J(θ0​,θ1​)，Depending on where one starts on the graph, one could end up at different points. The image above shows us two different starting points that end up in two different places.

The gradient descent algorithm is:
$\theta_j:=\theta_j-\alpha*\frac{\partial{J(\theta_0,\theta_1)}}{\partial{\theta_j}}$θj​:=θj​−α∗∂θj​∂J(θ0​,θ1​)​

repeat until convergence:
where

j=0,1 represents the feature index number.

At each iteration j, one should simultaneously update the parameters$\theta_1, \theta_2,...,\theta_n$θ1​,θ2​,...,θn​
Updating a specific parameter prior to calculating another one on the $j^{(th)}$j(th) iteration would yield to a wrong implementation.