Under construction :)

Resample

Use EIS

\( \mu \) =

\( \sigma \) =

\( n \) =

Suppose we would like to calculate the expected value of some function \(f\) with respect to the random variable \(X\), which is described by the probability density function \(g(x)\): $$ J[f] := \mathbb{E}_{X}[f(x)] = \int_{-\infty}^{+\infty} f(x) \, g(x) \, \textrm{d}x. $$ First, assume that while the above integral is too difficult to calculate directly, we are still able to draw samples from \(g(x)\). In this case, we can simply draw samples from the distribution and approximate the integral: $$ J[f] \approx \frac{1}{N}\sum_{i=0}^{N-1} f(u_{i}),\quad u_{i} \overset{\textrm{iid}}{\sim} X. $$ Now, suppose that it's also too difficult to sample \(X\), but that we can still evaluate \(g(x)\). In this case, we can draw samples of a different random variable, \(Y\), with distribution \(h(x)\), and we can use these samples to form an alternative approximation to \(J[f]\): $$ J[f] \approx \frac{1}{N}\sum_{i=0}^{N-1} f(u_{i}) \frac{g(u_{i})}{h(u_{i})},\quad u_{i} \overset{\textrm{iid}}{\sim} Y. $$ This technique of using one distribution to integrate with respect to another is known as importance sampling.

In the example above, $$ f(x) = 1, \quad g(x) = \frac{2/\pi}{1+4x^{2}},\quad\textrm{and}\quad h(y) = \frac{1}{\sigma\sqrt{2\pi}}\textrm{e}^{-\frac{1}{2}\big(\frac{y-\mu}{\sigma}\big)^{2}}. $$ Namely, we are using a normal distribution to approximate the integral of a Cauchy distribution over the real numbers. By adjusting \(\mu\) and \(\sigma\), we can test the performance of different sampling densities, and visually explore how they behave: gray rectangles are located along the horizontal axis according to where the red distribution generates samples, while their heights are calculated according to the ratio of \(g(x)/h(x)\). Since the true value of the integral is 1, we can easily guage whether the approximation is close.

A question that naturally arises is "which is the best sampling density?" One useful criteria is related to variance of the calculated expectation, implying that the best sampling density is the one that minimizes the sampling variance. Richard and Zhang (2005) describe a method for choosing such a sampling density, which they call Efficient Importance Sampling (EIS), and this distribution is used above when the "Use EIS" button is pressed.