Today it is a simple question for me. But yesterday, I was bothered by it.

Problem: Let x be a random vector with mean as $\mu$ and

$var(x)=E[(x-\mu)(x-\mu)^T]$

Now prove $var(x)$ is positive semi definite.

Proof: Let $v$ be an arbitrary vector (not random vector). Then

$v^TE[(x-\mu)(x-\mu)^T]v=E[(v^T(x-\mu))^2]\ge 0$

Q.E.D.

Highlight:

• The matrix $(x-\mu)(x-\mu)^T$ is p.s.d and of rank-1. But we can’t simply say $E[(x-\mu)(x-\mu)^T]$ is p.s.d, and of course $E[(x-\mu)(x-\mu)^T]$ is not of rank-1.
• Need not to use the definition of expectation to prove, but need use the definition of positive definite matrices.
• For scalar cases, it is easy to see because $E[(x-\mu)^2]\ge 0$

Further question: when does $E[(v^T(x-\mu))^2]= 0$?? when the elements of $x$ are independent? linear or statistically independent? So if x is independent, we have the covariance positive definite instead of positive semi-definite?