The Poisson Estimator

Let’s say you want to estimate a positive quantity \(\mu\), but you have only access to unbiased estimates of its logarithm, i.e., \(\log\mu\). Can you obtain an unbiased estimate of \(\mu\)?

There used to be a quite nice blog post online for the estimator solving this problem, but it is now in a weird shape, not compiling the LaTeX code (update: it appears to be fixed now, but I keep my post). Recently I was discussing this estimator with friends and realized that, at the moment, there is no quick explanation of it on the web. So I decided to write one.

Let’s say, as said in the blurb, you want to estimate a quantity but only have access to unbiased estimates of its \(\log\). A typical situation is that you have a quantity of the form: \[\begin{equation} \mu(x) = \exp\left(-\frac1n\sum_{i=1}^n g_i(x)\right) \label{eq:mu} \end{equation}\] As can be seen in \(\eqref{eq:mu}\), computing \(\mu\) requires evaluating all the functions \(g_i\), which can be expensive if \(n\) is large. But you can get cheap unbiased estimates of \(\log\mu(x)\) by subsampling the functions and computing the average on a mini-batch. However, an unbiased estimate of \(\log\mu\) is not enough to obtain an unbiased estimate of \(\mu\). Let’s denote this unbiased estimate \(\widehat{\log\mu}\) and we naturally have \(\mathbb{E}[\widehat{\log\mu}]=\log\mu\). Next, note that

\[ \mathbb{E}\left[e^{\widehat{\log\mu}}\right] \geq e^{\mathbb{E}\left[\widehat{\log\mu}\right]} = \mu, \]

by Jensen’s inequality. Therefore, naively computing \(\widehat{\log\mu}\) and exponentiating it will not in general produce the right result. Is there a way around?

It turns out that there is—provided that you can obtain many i.i.d. unbiased estimates. So returning to the original example, let’s say you can obtain i.i.d. integrable random variables \(\lambda_1,\lambda_2,\ldots\) where \(\mathbb{E}[\lambda_j]=\log\mu\). Then, incredibly, the following estimator is an unbiased estimator of \(\mu\):

\[ \widehat{\mu} = e^\delta\prod_{j=1}^J\frac{\lambda_j}{\delta}, \qquad J\sim\operatorname{Poisson}(\delta), \]

where \(\delta>0\) is a parameter of the estimator and \(J\) is independent of the \(\lambda_j\). The product is defined to be \(1\) when \(J=0\). This estimator is called the Poisson estimator.

Why though? Let’s prove a quick and sketchy result. First note \(J\sim\operatorname{Poisson}(\delta)\) has the probability mass function:

\[ \mathbb{P}(J=k) = \frac{\delta^k e^{-\delta}}{k!}, \qquad k=0,1,2,\ldots. \]

Now if we take the conditional expectation of \(\widehat{\mu}\) over \(J\):

\[ \begin{aligned} \mathbb{E}\left[\widehat{\mu}\mid\lambda_1,\lambda_2,\ldots\right] &= \sum_{k=0}^\infty \mathbb{P}(J=k)e^\delta \prod_{j=1}^k\frac{\lambda_j}{\delta} \\ &= \sum_{k=0}^\infty \frac{\delta^k e^{-\delta}}{k!}e^\delta \prod_{j=1}^k\frac{\lambda_j}{\delta} \\ &= 1+\sum_{k=1}^\infty\frac{1}{k!}\prod_{j=1}^k\lambda_j. \end{aligned} \]

Note that \(\mathbb{E}[\widehat{\mu}\mid\lambda_1,\lambda_2,\ldots]\) is still a random quantity, since the \(\lambda_j\) are unbiased and stochastic estimates of \(\log\mu\). Therefore, taking another expectation, we now obtain:

\[ \begin{aligned} \mathbb{E}[\widehat{\mu}] &= 1+\sum_{k=1}^\infty \frac{1}{k!} \mathbb{E}\left[\prod_{j=1}^k\lambda_j\right] \\ &= 1+\sum_{k=1}^\infty\frac{\lambda^k}{k!}, \end{aligned} \]

where \(\lambda=\mathbb{E}[\lambda_j]=\log\mu\), since the \(\lambda_j\) are i.i.d. Now it suffices to observe that this is indeed the series representation of the exponential function, therefore:

\[ \mathbb{E}[\widehat{\mu}] = 1+\sum_{k=1}^\infty\frac{\lambda^k}{k!} = \exp(\lambda) = \mu. \]

Although this result is nice and produced some practical applications, this estimator can exhibit a huge variance. A possible way to reduce it is to introduce another variable \(c\in\mathbb{R}\) and define the estimator as:

\[ \widehat{\mu} = e^{\delta+c} \prod_{j=1}^J\frac{\lambda_j-c}{\delta}, \]

where \(J\sim\operatorname{Poisson}(\delta)\) independently of the \(\lambda_j\). You can check that this estimator is still unbiased. However, as mentioned in this blog post, this estimator can still exhibit a big variance, hence is not solving a lot of the problems we encounter in practice.