Machine Learning Course 4 Selection of Model

Course 4

Average Error on testing data

Two main errors:

Error due to bias
Error due to variance

Estimator

Only god knows about the best function
$\hat f$f^​
and we can only get a function from training data called
$f^*$f∗
we say:
$f^*\quad is\quad an\quad estimator \quad of\quad \hat f$f∗isanestimatoroff^​
and the difference between f^* and f^hat comes from bias and variance

Bias and Variance of Estimator

Suppose the mean of a variable is μ, and the variance of x is σ²
$Sample \quad N \quad points:\{x^1,x^2,...,x^3\}$SampleNpoints:{x1,x2,...,x3}

$m=\frac{1}{N}\sum_n x^n\neq\mu \quad s^2=\frac 1 N \sum_n (x^n-m)^2$m=N1​n∑​xn​=μs2=N1​n∑​(xn−m)2

$E[m]=E[\frac{1}{N}\sum_n x^n]=\frac 1 N \sum_n E[x^n]=\mu\quad E[s^2]=\frac{N-1}N \sigma ^2$E[m]=E[N1​n∑​xn]=N1​n∑​E[xn]=μE[s2]=NN−1​σ2

m is a biased estimator of μ, s² is a biased estimator of σ²
$Var[m]=\frac{\sigma^2}{N}$Var[m]=Nσ2​
which shows how much m deviates μ and the variance depends on the amount of sample

and the relationship of these parameters is below:

Simple model with small variance, and complicated model with large variance since simpler model is less likely to be influenced by the sampled data.

Diagnosis

• If your model cannot even fit the training data, then you got a large bias, it is Underfitting
• If you can fit training data but got large error on testing data, then you probably got a large variance, it is Overfitting

For bias, redesign your model:

• Add more features as input
• A more complex model maybe needed

For large variance:

• More data is needed(Very effective but not always practical)
• Regularization

Model Selection

There is usually a trade-off between bias and variance

Select a model that balances two kinds of error to minimize total error

Cross Validation could a possible way to make balance:

And an advanced method called N-fold Cross Validation can be used