Bayesian calibration of a numerical code for prediction

Carmassi, Mathieu; Barbillon, Pierre; Chiodetti, Matthieu; Keller, Merlin; Parent, Eric

Journal de la société française de statistique, Numéro spécial : analyse de mélanges, Tome 160 (2019) no. 1, pp. 1-30.

Field experiments are often difficult and expensive to carry out. To bypass these issues, industrial companies have developed computational codes. These codes are intended to be representative of the physical system, but come with a certain number of problems. Despite continuous code development, the difference between the code outputs and experiments can remain significant. Two kinds of uncertainties are observed. The first one comes from the difference between the physical phenomenon and the values recorded experimentally. The second concerns the gap between the code and the physical system. To reduce this difference, often named model bias, discrepancy, or model error, computer codes are generally complexified in order to make them more realistic. These improvements increase the computational cost of the code. Moreover, a code often depends on user-defined parameters in order to match field data as closely as possible. This estimation task is called calibration. This paper proposes a review of Bayesian calibration methods and is based on an application case which makes it possible to discuss the various methodological choices and to illustrate their divergences. This example is based on a code used to predict the power of a photovoltaic plant.

Les difficultés de mise en œuvre d’expériences de terrain ou de laboratoire, ainsi que les coûts associés, conduisent les sociétés industrielles à se tourner vers des codes numériques de calcul. Ces codes, censés être représentatifs des phénomènes physiques en jeu, entraînent néanmoins tout un cortège de problèmes. Le premier de ces problèmes provient de la volonté de prédire la réalité à partir d’un modèle informatique. En effet, le code doit être représentatif du phénomène et, par conséquent, être capable de simuler des données proches de la réalité. Or, malgré le constant développement du réalisme de ces codes, des erreurs de prédiction subsistent. Elles sont de deux natures différentes. La première provient de la différence entre le phénomène physique et les valeurs relevées expérimentalement. La deuxième concerne l’écart entre le code développé et le phénomène physique. Pour diminuer cet écart, souvent qualifié de biais ou d’erreur de modèle, les développeurs complexifient en général les codes, les rendant très chronophages dans certains cas. De plus, le code dépend de paramètres à fixer par l’utilisateur qui doivent être choisis pour correspondre au mieux aux données de terrain. L’estimation de ces paramètres propres au code s’appelle le calage. Ce papier propose une revue des méthodes de calage bayésien et s’appuie sur un cas d’application qui permet de discuter les divers choix méthodologiques et d’illustrer leurs divergences. Cet exemple s’appuie sur un code de calcul servant à prédire la puissance d’une centrale photovoltaïque.

[1] Albert, Isabelle; Donnet, Sophie; Guihenneuc-Jouyaux, Chantal; Low-Choy, Samantha; Mengersen, Kerrie; Rousseau, Judith Combining expert opinions in prior elicitation, Bayesian Analysis, Volume 7 (2012) no. 3, pp. 503-532 | Zbl

[2] Bachoc, François; Bois, Guillaume; Garnier, Josselin; Martinez, Jean-Marc Calibration and improved prediction of computer models by universal Kriging, Nuclear Science and Engineering, Volume 176 (2014) no. 1, pp. 81-97

[3] Bayarri, Maria J; Berger, James O; Paulo, Rui; Sacks, Jerry; Cafeo, John A; Cavendish, James; Lin, Chin-Hsu; Tu, Jian A framework for validation of computer models, Technometrics, Volume 49 (2007) no. 2, pp. 138-154

[4] Brynjarsdóttir, J.; O’Hagan, Anthony Learning about physical parameters: The importance of model discrepancy, Inverse Problems, Volume 30 (2014) no. 11 | DOI | Zbl

[5] Carmassi, Mathieu CaliCo: Code Calibration in a Bayesian Framework (2018) https://CRAN.R-project.org/package=CaliCo (R package version 0.1.0)

[6] Craig, Peter S; Goldstein, Michael; Rougier, Jonathan C; Seheult, Allan H Bayesian forecasting for complex systems using computer simulators, Journal of the American Statistical Association, Volume 96 (2001) no. 454, pp. 717-729 | Zbl

[7] Currin, Carla; Mitchell, Toby; Morris, Max; Ylvisaker, Don Bayesian prediction of deterministic functions, with applications to the design and analysis of computer experiments, Journal of the American Statistical Association, Volume 86 (1991) no. 416, pp. 953-963

[8] Cox, Dennis D; Park, Jeong-Soo; Singer, Clifford E A statistical method for tuning a computer code to a data base, Computational statistics & data analysis, Volume 37 (2001) no. 1, pp. 77-92 | Zbl

[9] Damblin, G Contributions statistiques au calage et à la validation des codes de calcul, PhD thesis, Université Paris Saclay (2015) (Ph. D. Thesis)

[10] Duffie, John A; Beckman, William A Solar engineering of thermal processes, John Wiley & Sons, 2013

[11] Damblin, Guillaume; Barbillon, Pierre; Keller, Merlin; Pasanisi, Alberto; Parent, Éric Adaptive numerical designs for the calibration of computer codes, SIAM/ASA Journal on Uncertainty Quantification, Volume 6 (2018) no. 1, pp. 151-179 | Zbl

[12] Damblin, Guillaume; Keller, Merlin; Barbillon, Pierre; Pasanisi, Alberto; Parent, Éric Bayesian model selection for the validation of computer codes, Quality and Reliability Engineering International, Volume 32 (2016) no. 6, pp. 2043-2054

[13] Da Veiga, Sébastien; Marrel, Amandine Gaussian process modeling with inequality constraints, Annales de la Faculté des Sciences de Toulouse, Volume 21 (2012) no. 3, pp. 529-555 | Numdam | Zbl

[14] Fang, Kai-Tai; Li, Runze; Sudjianto, Agus Design and modeling for computer experiments, CRC Press, 2005 | Zbl

[15] Gu, Mengyang; Wang, Long An improved approach to Bayesian computer model calibration and prediction, arXiv preprint arXiv:1707.08215 (2017)

[16] Hastings, W Keith Monte Carlo sampling methods using Markov chains and their applications, Biometrika, Volume 57 (1970) no. 1, pp. 97-109 | Zbl

[17] Higdon, Dave; Gattiker, James; Williams, Brian; Rightley, Maria Computer model calibration using high-dimensional output, Journal of the American Statistical Association, Volume 103 (2008) no. 482, pp. 570-583 | Zbl

[18] Higdon, Dave; Kennedy, Marc; Cavendish, James C; Cafeo, John A; Ryne, Robert D Combining field data and computer simulations for calibration and prediction, SIAM Journal on Scientific Computing, Volume 26 (2004) no. 2, pp. 448-466 | Zbl

[19] Jones, Donald R; Schonlau, Matthias; Welch, William J Efficient global optimization of expensive black-box functions, Journal of Global optimization, Volume 13 (1998) no. 4, pp. 455-492 | Zbl

[20] Kennedy, Marc C; O’Hagan, Anthony Bayesian calibration of computer models, Journal of the Royal Statistical Society: Series B (Statistical Methodology), Volume 63 (2001) no. 3, pp. 425-464 | Zbl

[21] Kennedy, M; O’Hagan, A Supplementary details on bayesian calibration of computer. Rap. tech., University of Nottingham, Statistics Section (2001)

[22] Liu, Fei; Bayarri, MJ; Berger, JO Modularization in Bayesian analysis, with emphasis on analysis of computer models, Bayesian Analysis, Volume 4 (2009) no. 1, pp. 119-150 | Zbl

[23] Luque, Antonio; Hegedus, Steven Handbook of photovoltaic science and engineering, John Wiley & Sons, 2011

[24] Morris, Max D; Mitchell, Toby J Exploratory designs for computational experiments, Journal of statistical planning and inference, Volume 43 (1995) no. 3, pp. 381-402 | Zbl

[25] Morris, Max D Factorial sampling plans for preliminary computational experiments, Technometrics, Volume 33 (1991) no. 2, pp. 161-174

[26] Martin, N; Ruiz, JM Calculation of the PV modules angular losses under field conditions by means of an analytical model, Solar Energy Materials and Solar Cells, Volume 70 (2001) no. 1, pp. 25-38

[27] Metropolis, Nicholas; Rosenbluth, Arianna W; Rosenbluth, Marshall N; Teller, Augusta H; Teller, Edward Equation of state calculations by fast computing machines, The journal of chemical physics, Volume 21 (1953) no. 6, pp. 1087-1092 | Zbl

[28] Oberkampf, William L; Sindir, M; Conlisk, AT Guide for the verification and validation of computational fluid dynamics simulations, American Institute of Aeronautics and Astronautics, Reston, VA (1998) | Zbl

[29] Plumlee, Matthew Bayesian Calibration of Inexact Computer Models, Journal of the American Statistical Association, Volume 112 (2017) no. 519, pp. 1274-1285 | DOI

[30] Pronzato, Luc; Müller, Werner G Design of computer experiments: space filling and beyond, Statistics and Computing, Volume 22 (2012) no. 3, pp. 681-701 | Zbl

[31] Roustant, Olivier; Ginsbourger, David; Deville, Yves Dicekriging, Diceoptim: Two R packages for the analysis of computer experiments by kriging-based metamodelling and optimization, Journal of Statistical Software, Volume 51 (2012) no. 1, 54 pages | DOI

[32] Roache, Patrick J Verification of codes and calculations, AIAA journal, Volume 36 (1998) no. 5, pp. 696-702

[33] Robert, Christian Méthodes de Monte Carlo par chaînes de Markov, Economica, 1996 | Zbl

[34] Rocquigny, E de Quantifying uncertainty in an industrial approach: an emerging consensus in an old epistemological debate, SAPI EN. S. Surveys and Perspectives Integrating Environment and Society (2009) no. 2.1

[35] Sacks, Jerome; Welch, William J; Mitchell, Toby J; Wynn, Henry P Design and analysis of computer experiments, Statistical science (1989), pp. 409-423 | Zbl

[36] Santner, Thomas J; Williams, Brian J; Notz, William I The design and analysis of computer experiments, Springer Science & Business Media, 2013

[37] Tuo, Rui; Wu, CF Jeff Efficient calibration for imperfect computer models, The Annals of Statistics, Volume 43 (2015) no. 6, pp. 2331-2352 | Zbl

[38] Tuo, Rui; Wu, J. A theoretical framework for calibration in computer models: parametrization, estimation and convergence properties, SIAM/ASA Journal on Uncertainty Quantification, Volume 4 (2016) no. 1, pp. 767-795 | Zbl

[39] Wong, Raymond KW; Storlie, Curtis B; Lee, Thomas A frequentist approach to computer model calibration, Journal of the Royal Statistical Society: Series B (Statistical Methodology), Volume 79 (2017) no. 2, pp. 635-648 | Zbl