Gaussian Discriminative Analysis 高斯判别分析 GDA

Multidimensional Gaussian Model

$z \sim N(\vec\mu,\Sigma)$z∼N(μ​,Σ)
$z \in R^n,\vec\mu \in R^n, \Sigma \in R^{n*n}$z∈Rn,μ​∈Rn,Σ∈Rn∗n
$z$z – variable
$\vec\mu = \begin{bmatrix} \mu_1\\ \mu_2 \\ ... \\ \mu_n \end{bmatrix}$μ​=⎣⎢⎢⎡​μ1​μ2​...μn​​⎦⎥⎥⎤​ – mean vector
$\Sigma$Σ – covarience matrix
All the Gaussian models share one covarience matrix.

$E(z) = \vec\mu, Cov(z)=E[(x-\vec\mu)(x-\vec\mu)^T]=E(zz^T)-(E(z))(E(z))^T$E(z)=μ​,Cov(z)=E[(x−μ​)(x−μ​)T]=E(zzT)−(E(z))(E(z))T

Intro

GDA assumes:
$x|y=0 \sim N(\mu_0,\Sigma)$x∣y=0∼N(μ0​,Σ)
$x|y=1 \sim N(\mu_1,\Sigma)$x∣y=1∼N(μ1​,Σ)
$y \sim Ber(\phi), \phi = P(y=1)$y∼Ber(ϕ),ϕ=P(y=1)

GDA model(binary classification)

Multivariate Gaussian distribution:
$P(x) = \frac{1}{(2\pi)^{\frac d2}|\Sigma|^{\frac12}}exp(-\frac12(x-\mu)^T\Sigma^{-1}(x-\mu))$P(x)=(2π)2d​∣Σ∣21​1​exp(−21​(x−μ)TΣ−1(x−μ))
$|\Sigma|$∣Σ∣ is the value of determinant of $\Sigma$Σ

---

parameter: $\mu_0,\mu_1, \Sigma, \phi$μ0​,μ1​,Σ,ϕ
$P(y) = \phi^y(1-\phi)^{1-y}$P(y)=ϕy(1−ϕ)1−y
$\phi$ϕ is prior probability, and it depends on the proportion of two classes.

---

Joint likelihood:
$L(\phi, \mu_0, \mu_1, \Sigma) = \sum\limits_{i=1}^mP(x^{(i)},y^{(i)};\phi, \mu_0, \mu_1, \Sigma) = \sum\limits_{i=1}^mP(x^{(i)}|y^{(i)})P(y^{(i)})$L(ϕ,μ0​,μ1​,Σ)=i=1∑m​P(x(i),y(i);ϕ,μ0​,μ1​,Σ)=i=1∑m​P(x(i)∣y(i))P(y(i))
MLE: $\arg\max\limits_{\phi, \mu_0, \mu_1, \Sigma}l(\phi, \mu_0, \mu_1, \Sigma)$argϕ,μ0​,μ1​,Σmax​l(ϕ,μ0​,μ1​,Σ)
$\phi = \frac{\sum\limits_{i=1}^my^{(i)}}{m}=\frac{\sum\limits_{i=1}^m1\{y^{(i)}=1\}}{m}$ϕ=mi=1∑m​y(i)​=mi=1∑m​1{y(i)=1}​
$\mu_k = \frac{\sum\limits_{i=1}^m1\{y^{(i)}=k\}x^{(i)}}{\sum\limits_{i=1}^m1\{y^{(i)}=k\}},k\in \{0,1\}$μk​=i=1∑m​1{y(i)=k}i=1∑m​1{y(i)=k}x(i)​,k∈{0,1}
$\Sigma = \frac1m\sum\limits_{i=1}^m(x^{(i)}-\mu_{y^{(i)}})(x^{(i)}-\mu_{y^{(i)}})^T$Σ=m1​i=1∑m​(x(i)−μy(i)​)(x(i)−μy(i)​)T

Based on the two Gaussian models, we can draw a boundary line.
图片来源

Prediction

$\arg\max\limits_yP(y|x) = \arg\max\limits_y \frac{P(x|y)P(y)}{P(x)}=\arg\max\limits_yP(x|y)P(y)$argymax​P(y∣x)=argymax​P(x)P(x∣y)P(y)​=argymax​P(x∣y)P(y)
($P(x)$P(x) is a constant)

& Logistic Regression

图片是我的笔记

The picture shows when our data is 1D the function looks like Sigmoid function. Actually, it is Sigmoid function and it also applys to higher dimension. I won’t prove it here.
GDA is a stricter version of logistic regression because the data has to follow Gaussian distribution.
When the data follows Gaussian distribution or the data is very big(according to the central limit theorem), GDA works better than logistic regression.
Also, the data follows Gaussian distribution so the model has no local optima.