Abstract
We provide a rigorous proof that Jeffreys' prior asymptotically maximizes Shannon's mutual information between a sample of size n and the parameter. This was conjectured by Bernardo (1979) and, despite the absence of a proof, forms the basis of the reference prior method in Bayesian statistical analysis. Our proof rests on an examination of large sample decision theoretic properties associated with the relative entropy or the Kullback-Leibler distance between probability density functions for independent and identically distributed random variables. For smooth finite-dimensional parametric families we derive an asymptotic expression for the minimax risk and for the related maximin risk. As a result, we show that, among continuous positive priors, Jeffreys' prior uniquely achieves the asymptotic maximin value. In the discrete parameter case we show that, asymptotically, the Bayes risk reduces to the entropy of the prior so that the reference prior is seen to be the maximum entropy prior. We identify the physical significance of the risks by giving two information-theoretic interpretations in terms of probabilistic coding.
Original language | English (US) |
---|---|
Pages (from-to) | 37-60 |
Number of pages | 24 |
Journal | Journal of Statistical Planning and Inference |
Volume | 41 |
Issue number | 1 |
DOIs | |
State | Published - Aug 1994 |
Externally published | Yes |
Keywords
- Bayes risk
- Fisher information
- Jeffreys' prior
- Kullback-Leibler information
- Shannon's mutual information
- data compression
- minimax risk
- parametric density estimation
- reference priors, least favorable priors
ASJC Scopus subject areas
- Statistics and Probability
- Statistics, Probability and Uncertainty
- Applied Mathematics