Investigations particulaires pour l'inférence statistique et l'optimisation de plan d'expériences
Journal de la société française de statistique, Volume 149 (2008) no. 1, p. 27-51

Particle algorithms are Monte Carlo techniques that put together steps of importance sampling, bootstrap resampling, markovian rejuvenating and simulated annealing. We develop three examples of increasing complexity and explain how to implement such algorithms for maximum likelihood search, for inference of a model with latent variables and for optimal design. Since we believe that particle algorithms will soon become tools of choice for statistical practitioners, their results are compared with the known solutions of these rather common examples so as to test the algorithms' performances and to show their limits.

Les algorithmes particulaires sont des techniques de Monte-Carlo qui associent des étapes d'échantillonnage pondéré, de rééchantillonnage bootstrap, de régénérescence markovienne et de recuit simulé. Grâce à trois exemples de complexité croissante, nous décrivons leurs implémentations pour l'estimation du maximum de vraisemblance, l'évaluation de la distribution a posteriori pour un modèle à variables latentes et la recherche du plan d'expérience optimal. Les solutions de ces exemples pédagogiques illustrent les performances et les limites de ces algorithmes, promis à une place de choix dans la trousse à outils du statisticien.

Keywords: particle algorithms, Monte Carlo simulation, optimal experimental design, bayesian inference
@article{JSFS_2008__149_1_27_0,
     author = {Parent, \'Eric and Amzal, Billy and Girard, Philippe},
     title = {Investigations particulaires pour l'inf\'erence statistique et l'optimisation de plan d'exp\'eriences},
     journal = {Journal de la soci\'et\'e fran\c caise de statistique},
     publisher = {Soci\'et\'e fran\c caise de statistique},
     volume = {149},
     number = {1},
     year = {2008},
     pages = {27-51},
     language = {fr},
     url = {http://www.numdam.org/item/JSFS_2008__149_1_27_0}
}
Parent, Éric; Amzal, Billy; Girard, Philippe. Investigations particulaires pour l'inférence statistique et l'optimisation de plan d'expériences. Journal de la société française de statistique, Volume 149 (2008) no. 1, pp. 27-51. http://www.numdam.org/item/JSFS_2008__149_1_27_0/

[1] Amzal B., Bois F., Parent E., Robert C.P. 2006. Bayesian Optimal Design Via Interacting Particle Systems. Journal of the American Statistical Association, 101(474), 773-785. | MR 2281248 | Zbl 1119.62308

[2] Andrieu C., Doucet A. 2000. Simulated Annealing for Maximum a posteriori Parameter Estimation of Hidden Markov Models. IEEE Trans. Information Theory, 46(3), 994-1004. | MR 1763472 | Zbl 1009.62070

[3] Andrieu C., Doucet A., Singh S., Tadic V. 2004. Particle Methods for Change Detection, System Identification, and Control. Proceedings of the IEEE, 92(3), 423-438.

[4] Arulampalam M. S., Maskell S., Gordon N., Clapp T. 2002. A Tutorial on Particle Filters for Online Nonlinear/Non-Gaussian Bayesian Tracking. IEEE Transaction on Signal Processing, 50(2), 174-188.

[5] Barnett V. 1973. Bayesian and Decision Theoric Methods Applied to Industrial Problems. The statistician, 22(3), 199-226.

[6] Berger J. O. 1985. Statistical Decision Theory and Bayesian Analysis. Springer Verlag, New York. | MR 804611 | Zbl 0572.62008

[7] Bernier J., Parent E., Boreux J-J. 2000. Statistique de l'Environnement. Traitement Bayésien de l'Incertitude. Paris : Lavoisier.

[8] Brooks S P. 1998. Markov Chain Monte Carlo Method and its Application. The Statistician, 47, 69-100.

[9] Brooks S.P. 2003. Bayesian Computation : A Statistical Revolution. Trans. Roy. Statist. Soc., series A, 15, 2681-2697. | MR 2050672 | Zbl 1044.62029

[10] Brooks S.P., Morgan B.J.T. 1995. Optimization Using Simulated Annealing. The Statistician, 44, 241-257.

[11] Cappé O., Guilin A., Marin J.M., Robert C.P. 2004. Population Monte Carlo. Journal of Computational and Graphical Statistics, 13(4), 907-929. | MR 2109057

[12] Cappé O., Moulines E., Rydèn T. 2005. Inference in Hidden Markov Models. Springer. | MR 2159833 | Zbl 1080.62065

[13] Carlin B.P., Kadane J.B., Gelfand A.E. 1998. Approaches for Optimal Sequential Decision Analysis in Clinical Trials. Biometrics, 54(3), 964-975. | Zbl 1058.62548

[14] Chopin N. 2002. A Sequential Particle Filter Method for Static Models. Biometrika, 89, 539-552. | MR 1929161 | Zbl 1036.62062

[15] Chopin N. 2004. Central Limit Theorem for Sequential Monte Carlo Methods and its Application to Bayesian Inference. Ann. Stat., 32(6), 2385-2411. | MR 2153989 | Zbl 1079.65006

[16] Del Moral P. 2004. Feynman-Kac Formulae : Genealogical and Interacting Particle Systems with Applications. Springer-Verlag. | MR 2044973 | Zbl 1130.60003

[17] Del Moral P., Guionnet A. 1999. Central Limit Theorem for Nonlinear Filtering and Interacting Particule Systems. Ann. Appl. Prob., 155-194. | MR 1687359 | Zbl 0938.60022

[18] Del Moral P., Kallel L., Rowe J. 2001. Natural Computing Series : Theoretical Aspects of Evolutionary Computing. Springer-Verlag, Berlin. Chap. Modelling Genetic Algorithms with Interacting Particle Systems, pages 10-67.

[19] Douc R., Cappé O. 2005. Comparison of Resampling Schemes for Particle Filtering. Pages 64-69 of : Proceedings of the 4th International Symposium on Image and Signal Processing and Analysis.

[20] Douc R., Cappé O., Moulines E., Robert C.P. 2002. On the Convergence of the Monte Carlo Maximum Likelihood Method for Latent Variable Models. Scandinavian Journal of Statistics, 29(4), 615-635. | MR 1988415 | Zbl 1035.62015

[21] Douc R., Guilin A., Marin J.M., Robert C.P. 2007. Convergence of Adaptive Mixtures of Importance Sampling Schemes. Ann. Stat., 35(1), 420-448. | MR 2332281 | Zbl 1132.60022

[22] Doucet A., De Freitas N., Gordon N. 2001. Sequential Monte Carlo Methods in Practice. Springer-Verlag. | MR 1847783 | Zbl 0967.00022

[23] Doucet A., Godsill J.M., Robert C.P. 2002. Population Marginal maximum a posteriori Estimation Using Markov Chain Monte Carlo. Statistics and Computing, 12, 77-84. | MR 1877581 | Zbl 1247.62087

[24] Doucet A., Del Moral P., Peters G.W. 2004. Sequential Monte Carlo Samplers. Tech. rept. Cambridge University.

[25] Gelman A., Rubin D. B. 1992. Inference from Iterative Simulation Using Multiple Sequences. Statistical Science, 457-472.

[26] Gelman A., Carlin J. B., Stern H. S., Rubin D. B. 1995. Bayesian Data Analysis. London : Chapman and Hall. | MR 1385925 | Zbl 0914.62018

[27] Geman S., Geman D. 1984. Stochastic Relaxation, Gibbs Distribution and the Bayesian Restoration of Images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 6(6), 721-741. | Zbl 0573.62030

[28] Gilks W.R., Richardson S., Spiegelhalter D. 1996. Markov Chain Monte Carlo in Practice. London : Chapman and Hall. | MR 1397966 | Zbl 0832.00018

[29] Kass R. E., Carlin B. P., Gelman A., Neal R. M. 1998. Markov Chain Monte Carlo in Practice : A Roundtable Discussion. The American Statistician, 52(2), 93-100. | MR 1628427

[30] Kuhn E., Lavielle M. 2004. Coupling a Stochastic Approximation Version of EM with an MCMC Procedure. ESAIM, Probab. Stat., 8, 115-131. | Numdam | MR 2085610 | Zbl 1155.62420 | Zbl pre02161878

[31] Künsch H. 2001. State Space and Hidden Markov Models. Pages 109-173 of : Barndor-Nielsen, O. E., Cox, D. R., Klüppelberg, C. (eds), Complex Stochastic Systems. Chapman and Hall. | MR 1893412 | Zbl 1002.62072

[32] Künsch H. 2005. Recursive Monte Carlo Filters. Ann. Stat., 1983-2021. | Zbl 1086.62106

[33] Liu J.S., Chen R. 1998. Sequential Monte Carlo Methods for Dynamic Systems. Journal of American Statistical Association, 93(443), 113-119. | MR 1649198 | Zbl 1064.65500

[34] Mc Cullagh P., Nelder J.A. 1989. Generalized Linear Models. second edn. Chapman and Hall/CRC.

[35] Mengersen K.L., Robert C.P., Guihennec-Jouyaux C. 1999. MCMC Convergence Diagnostics : A Reviewww (with Discussion). Pages 415-440 of : Bernardo, J.M., Berger, J.O., Dawid, A.P., Smith, A.F.M. (eds), Bayesian Stat., vol. 6. Oxford University Press. | MR 1723507 | Zbl 0957.62019

[36] Metropolis N., Rosenbluth A.W., Rosenbluth M.N., Teller A.H, Teller E. 1953. Equations of State Calculations by Fast Computing Machines. J. Chem. Phys., 21, 1087-1091.

[37] Müller P. 1999. Simulation-Based Optimal Design. Pages 459-474 of : Bernardo, J.M., Berger, J.O., Dawid, A.P., Smith, A.F.M. (eds), Bayesian Stat., vol. 6. Oxford University Press. | MR 1723509 | Zbl 0974.62058

[38] Munier B., Parent E. 1998. Le Développement Récent des Sciences de la Décision : Un Regard Critique sur la Statistique Décisionnelle Bayésienne. In : Parent, E., Hubert, P., Bobée, B., Miquel, J. (eds), Bayesian Methods in Hydrol. Sci. UNESCO Publishing.

[39] Oujdane N., Ruberthaler S. 2005. Stability and Uniform Particle Approximation of Nonlinear Filters in case of non Ergodic Signals. Stochastic Analysis and Applications, 23, 421-448. | MR 2140972 | Zbl 1140.93485

[40] Parent E., Bernier J. 2007. Le Raisonnement Bayésien : Modélisation et Inférence. Paris : Springer Verlag France. | Zbl 1284.62017

[41] Parent, E., Chaouche A., Girard P. 1995. Sur l'Apport des Statistiques Bayésiennes au Contrôle de la Qualité par Attribut, partie 1 : Contrôle Simple. Rev. Statistique Appliquée, XLIII(4), 5-18. | Numdam

[42] Parent E., Lang G., Girard P. 1996. Sur l'Apport des Statistiques Bayésiennes au Contrôle de la Qualité par Attribut, partie 2 : Contrôle Séquentiel Tronqué. Rev. Statistique Appliquée, XLIV(1), 37-54. | Numdam

[43] Pitt M., Shephard N. 1999. Filtering Via Simulation : Auxiliary Particle Filters. Ecological Applications, 94(446), 590-599. | MR 1702328 | Zbl 1072.62639

[44] Reeves C.R., Wright C.C. 1996. Genetic Algorithms and the Design of Experiments. In : Proc. IMA Fall Workshop on Evolutionary Algorithms.

[45] Robert C. P. 2006. Le Choix Bayésien : Principes et Pratique. Paris : Springer Verlag France.

[46] Robert C.P., Casella G. 2004. Monte-Carlo Statistical Methods. Springer-Verlag. | MR 2080278 | Zbl 1096.62003

[47] Rubin D.B. 1988. Using the SIR Algorithm to Simulate Posterior Distributions. Pages 395-402 of : Bayesian Statistics 3, vol. 3. Oxford University Press. | Zbl 0713.62035

[48] Smith A.F.M., Gelfand A.E. 1992. Bayesian Statistics without Tears : a Sampling-Resampling Perspective. The American Statistician, 46, 84-88. | MR 1165566

[49] Tagaras G. 1986. Economic Design of Acceptance Sampling and Process Control Procedures for Quality Assurance in Complex Production Systems. Ph.D. thesis, Stanford University.

[50] Tanner M. H. 1992. Tools for Statistical Inference : Observed Data and Data Augmentation Methods. New York : Springer-Verlag. | MR 1096956 | Zbl 0724.62003

[51] Van Laarhoven P.J.M., Aarts E.H.L. 1987. Simulated Annealing : Theory and Applications. Reider Pub. and Kluwer Dordrecht, Holland. | MR 904050 | Zbl 0643.65028

[52] Von Neumann J., Morgenstern O. 1944. Theory of Games and Economic Behavior. Princeton Univ. Press, Princeton NJ. | MR 11937 | Zbl 0063.05930 | Zbl 1109.91001