[机器学习笔记] 梯度下降法

[Machine Learning notebook by NG] Gradient Descent

---

Linear Regression Model(线性模型)
$h_\theta=\theta_0+\theta_1x$hθ​=θ0​+θ1​x
$J(\theta_0,\theta_1)=\frac{1}{2m}\sum_{i=1}^{m}(h_\theta(x^{(i)})-y^{(i)})^{2}$J(θ0​,θ1​)=2m1​∑i=1m​(hθ​(x(i))−y(i))2

---

Gradient Descent algorithm

repeat until convergence {
              $\theta_j=\theta_j-\alpha\frac{\partial}{\partial\theta_j}j(\theta_0,\theta_1)$θj​=θj​−α∂θj​∂​j(θ0​,θ1​)
              $j = 0 or 1$j=0or1
}
$\alpha$α means learning rate,that is to say:it means the size of gradient step

---

Correct: Simultaneous update
$temp0=\theta_0-\alpha\frac{\partial}{\partial\theta_0}j(\theta_0,\theta_1)$temp0=θ0​−α∂θ0​∂​j(θ0​,θ1​)
$temp1=\theta_0-\alpha\frac{\partial}{\partial\theta_1}j(\theta_0,\theta_1)$temp1=θ0​−α∂θ1​∂​j(θ0​,θ1​)
$\theta_0=temp0$θ0​=temp0
$\theta_1=temp1$θ1​=temp1

Notice that:update $\theta_0$θ0​ and $\theta_1$θ1​ simultaneously

different Start point may obtion result,as than pictures show。

---

the size of learning rate $\alpha$α

if $\alpha$α is too small,gradient descent can be slow.
if $\alpha$α is too large, gradient descent can overshoot the minimum it may fail to converge, or even diverge.