Andrew Ng 's machine learning lecture note (8)
Over Fitting Problem
Over fitting means that the hypothesis can get good predictions on the traning set but can not get good predictions outside the tranining set. The figure looks like the below second one.
In order to solve this problem , we use the algorithm regularization,
Regularization
Regularization means we penalize some theta values (make theta values small enough)
We discuss 2 situations ,
(1)Linear Regression
We redefine the cost function like this:
If we increase lambda which means we penalize theta more.Be careful ! lambda can not be too large ,which may cause under fitting.
To minimize the above cost function , we can use either gradient decent or normal equation shown below:
gradient decent
normal equation
(2)logistic regression
we redefine the cost function:
Also we can use gradient decent to solve this