Information-based approaches to probabilistic approximation.

Normal approximation

Unlike most classical ways of proving the Central Limit Theorem (CLT), the information theoretic approach comes with very nice intuition. Consider the sequence of standardized sums of IID continuous random variables X_i, where the standardization is precisely that which preserves the variance. Then the entropy of the standardized sum S_n increases monotonically with n, until it reaches the maximum entropy possible under the variance constraint. This is the Gaussian entropy. In addition to giving the CLT the flavour of a "second law of thermodynamics" and motivating the appearance of the Gaussian as the limiting distribution, this approach gives convergence in the sense of relative entropy, which is stronger than the classical weak convergence.

The convergence in the entropic CLT was proved by Barron [Bar86], building on work of Linnik and Brown. Artstein, Ball, Barthe and Naor [ABBN04] obtained the monotonicity of entropy as a consequence of a new "entropy power inequality", proved using a functional analytic technique. In joint work with Andrew Barron [MB07], we gave a simpler proof of a more general inequality, elucidating along the way interesting connections to statistics and information theory.

Poisson approximation

A natural question is whether a similar compelling picture obtains for other limit laws, such as the "law of small numbers" (the Poisson limit law). Indeed it turns out that the Poisson distribution has a maximum entropy property, and that optimal-order Poisson approximation bounds can be proved for sums by developing the properties of a discrete analogue of Fisher information sometimes called the "scaled Fisher information". These results are due to Kontoyiannis, Harremoes and Johnson (see [Har01], [KHJ05] and [Joh07]).

In joint work with Ioannis Kontoyiannis and Oliver Johnson [MJK07], we added to this developing picture by demonstrating connections of the scaled Fisher information to minimum mean square estimation in the "Poisson channel" (which models certain optimal communication systems), as well as by displaying a monotonicity property for this quantity analogous to that which obtains for the CLT.

Compound Poisson approximation

In one part of my Ph.D. dissertation, I explored the question of extending the above picture to a much more general situation: approximating the distribution of a sum of random variables taking non-negative integer values using a compound Poisson distribution. As in the simple Poisson case, it is easy to get relative entropy approximation bounds that are of suboptimal order for large numbers of summands, but much harder to get bounds of the right order. For the special case of independent summands, we proposed various notions of "local information" that can play the role that Fisher information has in normal approximation, and used them to obtain approximation bounds in total variation and relative entropy. Those results have seen significant further development in collaboration with Oliver Johnson and Ioannis Kontoyiannis, and can be found in [JKM08].

References

[ABBN04]    S. Artstein, K. M. Ball, F. Barthe, and A. Naor. Solution of Shannon’s problem on the monotonicity of entropy. J. Amer. Math. Soc., 17(4):975–982, 2004.
[Bar86]    A.R. Barron. Entropy and the central limit theorem. Ann. Probab., 14:336–342, 1986.
[Har01]    P. Harremoes. Binomial and Poisson distributions as maximum entropy distributions. IEEE Trans. Inform. Theory, 47(5):2039–2041, 2001.
[JKM08]    O. Johnson, I. Kontoyiannis, and M. Madiman. Compound Poisson approximation via local information quantities. Preprint, 2008.
[Joh07]    O. Johnson. Log-concavity and the maximum entropy property of the Poisson distribution. Stochastic Process. Appl., Vol 117(6):791–802, 2007.
[KHJ05]    I. Kontoyiannis, P. Harremoes, and O. Johnson. Entropy and the law of small numbers. IEEE Trans. Inform. Theory, 51(2):466–472, February 2005.
[MB07]    M. Madiman and A.R. Barron. Generalized entropy power inequalities and monotonicity properties of information. IEEE Trans. Inform. Theory, 53(7):2317–2329, July 2007.
[MJK07]    M. Madiman, O. Johnson, and I. Kontoyiannis. Fisher information, compound Poisson approximation, and the Poisson channel. In Proc. IEEE Int. Symp. Inform. Theory, Nice, France, pp.976–980, 2007.

Read the relevant publications for more details, or go back to research summaries.