Understanding the Dirichlet Distribution

What Is the Dirichlet Distribution?

A sample from a Dirichlet distribution is a probability vector: its components are nonnegative and sum to one.

The vector lies in the standard simplex,

$$\Delta^{K-1}=\left\{\boldsymbol{x}\in[0,1]^K:\sum_{i=1}^K x_i=1\right\}.$$

Each such vector can serve as the probability parameters of a categorical or multinomial distribution. In this sense, the Dirichlet is a distribution over probability distributions on a fixed set of outcomes.

For $K\geq 2$ and parameters $\alpha_i>0$, its probability density function (PDF), evaluated in the interior of the simplex, is

$$f(\boldsymbol{x};\boldsymbol{\alpha}) =\frac{1}{\mathrm{B}(\boldsymbol{\alpha})} \prod_{i=1}^{K}x_i^{\alpha_i-1}, \qquad x_i>0,\quad \sum_{i=1}^{K}x_i=1.$$

The B Function

The multivariate beta function is the integral of the unnormalized density:

$$\mathrm{B}(\boldsymbol{\alpha}) =\int_{\Delta^{K-1}}\prod_{j=1}^{K}x_j^{\alpha_j-1}\,\mathrm{d}\boldsymbol{x} =\frac{\prod_{j=1}^{K}\Gamma(\alpha_j)}{\Gamma(\alpha_0)}, \qquad \alpha_0=\sum_{j=1}^{K}\alpha_j.$$

Here the integral uses the $K-1$ free coordinates: $\mathrm{d}\boldsymbol{x}=\mathrm{d}x_1\cdots\mathrm{d}x_{K-1}$, with $x_K=1-\sum_{j=1}^{K-1}x_j$. Dividing by $\mathrm{B}(\boldsymbol{\alpha})$ therefore makes the PDF integrate to one.

The gamma function $\Gamma$ is defined, for $t>0$, by

$$\Gamma(t)=\int_0^\infty u^{t-1}e^{-u}\,\mathrm{d}u.$$

We will use its recurrence $\Gamma(t+1)=t\Gamma(t)$ to simplify the moments below.

Key Properties

For a given $\boldsymbol{\alpha}$, $\mathrm{B}(\boldsymbol{\alpha})$ is a constant that makes the density integrate to one. Its shape is therefore determined by

$$f(\boldsymbol{x};\boldsymbol{\alpha})\propto \prod_{i=1}^{K}x_i^{\alpha_i-1}.$$

When $\boldsymbol{\alpha}=(1,\ldots,1)$, the product is constant, so the distribution is uniform over the simplex. A constant density need not equal one: it is $1$ for $K=2$ and $2$ for $K=3$.

Two Components

For $K=2$, $x_2=1-x_1$, so a line segment describes all possible vectors. The density of $x_1$ is the Beta distribution. Adjust $\alpha_1$ and $\alpha_2$ below, or choose Uniform to see the constant density.

Three Components

For $K=3$, the simplex is a triangle. Each point represents a vector $(x_1,x_2,x_3)$; the color or height shows its density. Try Uniform, then change the parameters to see how the shape changes. Density is not point probability and can exceed one.

Mean and Variance

Let $\boldsymbol{X}\sim\operatorname{Dirichlet}(\boldsymbol{\alpha})$, with total concentration

$$\alpha_0=\sum_{j=1}^{K}\alpha_j.$$

For each component, the mean and variance are

$$\begin{aligned} \mathbb{E}[X_i]&=\frac{\alpha_i}{\alpha_0},\\ \operatorname{Var}(X_i)&= \frac{\alpha_i(\alpha_0-\alpha_i)}{\alpha_0^2(\alpha_0+1)}. \end{aligned}$$

The proofs use parameter shifts in the B function.

Deriving the Mean

Let $\boldsymbol{e}_i$ be the vector whose $i$th component is one and whose other components are zero. Multiplying by $x_i$ raises its exponent from $\alpha_i-1$ to $\alpha_i$, which replaces $\boldsymbol{\alpha}$ with $\boldsymbol{\alpha}+\boldsymbol{e}_i$:

$$\begin{aligned} \mathbb{E}[X_i] &=\frac{1}{\mathrm{B}(\boldsymbol{\alpha})} \int_{\Delta^{K-1}}x_i\prod_{j=1}^{K}x_j^{\alpha_j-1}\,\mathrm{d}\boldsymbol{x}\\ &=\frac{\mathrm{B}(\boldsymbol{\alpha}+\boldsymbol{e}_i)}{\mathrm{B}(\boldsymbol{\alpha})}\\ &=\frac{\Gamma(\alpha_i+1)}{\Gamma(\alpha_i)} \frac{\Gamma(\alpha_0)}{\Gamma(\alpha_0+1)}\\ &=\frac{\alpha_i}{\alpha_0}. \end{aligned}$$

All other gamma factors cancel; the last step uses $\Gamma(t+1)=t\Gamma(t)$.

Deriving the Variance

First compute the second moment. Multiplying by $x_i^2$ similarly increases $\alpha_i$ by two:

$$\begin{aligned} \mathbb{E}[X_i^2] &=\frac{\mathrm{B}(\boldsymbol{\alpha}+2\boldsymbol{e}_i)}{\mathrm{B}(\boldsymbol{\alpha})}\\ &=\frac{\Gamma(\alpha_i+2)}{\Gamma(\alpha_i)} \frac{\Gamma(\alpha_0)}{\Gamma(\alpha_0+2)}\\ &=\frac{\alpha_i(\alpha_i+1)}{\alpha_0(\alpha_0+1)}. \end{aligned}$$

Now subtract the square of the mean:

$$\begin{aligned} \operatorname{Var}(X_i) &=\mathbb{E}[X_i^2]-\bigl(\mathbb{E}[X_i]\bigr)^2\\ &=\frac{\alpha_i(\alpha_i+1)}{\alpha_0(\alpha_0+1)} -\frac{\alpha_i^2}{\alpha_0^2}\\ &=\frac{\alpha_i(\alpha_0-\alpha_i)}{\alpha_0^2(\alpha_0+1)}. \end{aligned}$$

Interpreting the Parameters

The relative sizes of the parameters determine the mean probability vector. If their proportions stay fixed, increasing their sum $\alpha_0$ reduces the variance and concentrates the distribution around that mean.

References: SciPy: Dirichlet distribution · Probability Distribution Explorer: moments.