Review of probability (1)
Posted on 2013-07-23 04:44:39 Machine Learning Views: 2553

A brief review of probability theory

Some concept of discrete random variables

  • discrete random variable
  • probability mass function(pmf)
  • indicator function

Fundamental rules

1. Probability of a union

union: $ P(A \vee B)= P(A) + P(B) - P(A \wedge B) $
A,B mutually exclusive:$ P(A \vee B)= P(A) + P(B) $

2. Joint probabilities
  • product rule: $ P(A,B) = P(A \wedge B ) = P(A|B)P(B) $
  • marginal distribution: $ p(A) = \sum_{b}p(A,B) = sum_{b}p(A|B=b)p(B=b) $
  • chain rule: $ p(X_{1:D}) = p(X_1)p(X_2|X_1)p(X_3|X_2,X_1)...p(X_D|X_{1:D-1}) $
3. Conditional probability
\[ p(A|B) = \frac{p(A,B)}{p(B)}(p(B)>0)  \]
4. Bayes rule[Bayes Theorem]
\[ 
p(X=x|Y=y) = \frac{p(X=x,Y=y)}{p(Y=y)}=\frac{p(X=x)p(Y=y|X=x)}{\sum_{x'} p(X=x')(Y=y|X=x')}
\]

Independence and conditional independence

X and Y are unconditionally independent or marginally independent,
$ X \bot Y \Leftrightarrow p(X,Y)=p(X)(Y) $

Continuous random variables

  • function: $ F(q)=p(X\le q), f(x)=\frac{d}{dx}F(x) $
  • cumulative distribution function: $ p(a<X\le b)=F(b)-F(a) $
  • probability density function: $ P(a < X \le b)=\int_a^b f(x)dx $

Mean and Variance

  • discrete mean: $ E[X]=\sum_{x \in X}xp(x) $
  • continuous mean: $ E[X]=\int_{X}xp(x)dx $
  • variance: $ var[X]=E[(X-\mu)^2]=\int (x-\mu)^2p(x)dx=E[X^2]-\mu^2 \Rightarrow E[X^2]=\mu^2+\sigma^2$

Reference

Machine Learning: A Probabilistic Perspective.[Kevin P. Murphy]