Substitution de modèle et approche multifidélité en expérimentation numérique

De Lozzo, Matthias

Journal de la société française de statistique, Tome 156 (2015) no. 4, pp. 21-55.

Cet article présente une synthèse bibliographique sur la substitution de modèle en expérimentation numérique où l’objectif est d’approcher un simulateur numérique à partir de quelques unes de ses évaluations. Les principaux modèles de substitution y sont décrits : réseaux de neurones artificiels, modèles par processus gaussien, machines à vecteurs de support et polynômes de chaos. Des éléments d’apprentissage statistique sont par ailleurs exposés afin de choisir la complexité et les paramètres d’un modèle de substitution permettant une bonne approximation du simulateur numérique. Une ouverture à la modélisation multifidélité est proposée afin de tenir compte de sources d’observations complémentaires lorsque l’évaluation du simulateur est trop coûteuse.

This article presents a review of research literature on surrogate modeling in the context of computer experimentation where the goal is to approach a numerical simulator from some evaluations. The main surrogate models are described: artificial neural networks, gaussian process models, support vector machines and polynomial chaos expansions. Elements of statistical learning are expounded in order to select the complexity and the parameters of a surrogate model which assure a good approximation of the numerical simulator. An extension to multifidelity modelization is also proposed so as to take into account complementary sources of observations when the simulator evaluation is too expensive.

Mot clés : expérimentation numérique, apprentissage supervisé, modèle de substitution, multifidélité, régression hétéroscédastique, modèle par processus gaussien, synthèse bibliographique
Keywords: computer experiments, supervised learning, surrogate model, multifidelity, heteroscedastic regression, Gaussian process model, survey

[1] Akaike, Hirotugu Information theory and an extension of the maximum likelihood principle, Second international symposium on information theory, Akademinai Kiado (1973), pp. 267-281 | MR | Zbl

[2] Arlot, Sylvain; Massart, Pascal Data-driven Calibration of Penalties for Least-Squares Regression., Journal of Machine Learning Research, Volume 10 (2009), pp. 245-279 http://dblp.uni-trier.de/db/journals/jmlr/jmlr10.html#ArlotM09

[3] Attali, Jean-Gabriel; Pagès, Gilles Approximations of Functions by a Multilayer Perceptron : a New Approach, Neural Networks, Volume 10 (1997) no. 6, pp. 1069-1081

[4] Arlot, Sylvain Model selection by resampling penalization, Electronic Journal of Statistics, Volume 3 (2009), pp. 557-624 | MR | Zbl

[5] Arlot, Sylvain Choosing a penalty for model selection in heteroscedastic regression, arXiv :0812.3141 (2010)

[6] Barron, Andrew R Universal approximation bounds for superpositions of a sigmoidal function, Information Theory, IEEE Transactions on, Volume 39 (1993) no. 3, pp. 930-945 | MR | Zbl

[7] Bettebghor, D.; Bartoli, N.; Grihon, S.; Morlier, J.; Samuelides, M. Surrogate modeling approximation using a mixture of experts based on EM joint estimation, Structural and Multidisciplinary Optimization, Volume 43 (2011) no. 2, pp. 243 -259

[8] Barron, Andrew; Birgé, Lucien; Massart, Pascal Risk bounds for model selection via penalization, Probability theory and related fields, Volume 113 (1999) no. 3, pp. 301-413 | MR | Zbl

[9] Bandler, John W; Cheng, Qingsha S; Dakroury, Sameh A; Mohamed, Ahmed S; Bakr, Mohamed H; Madsen, Kaj; Sondergaard, J Space mapping : the state of the art, Microwave Theory and Techniques, IEEE Transactions on, Volume 52 (2004) no. 1, pp. 337-361

[10] Bonnans, J. Frédéric; Gilbert, Jean Charles; Lemaréchal, Claude; Sagastizábal, Claudia A. Numerical Optimization : Theoretical and Practical Aspects (Universitext), Springer-Verlag New York, Inc., Secaucus, NJ, USA, 2006 | MR | Zbl

[11] Bishop, Christopher M. Neural Networks for Pattern Recognition, Oxford University Press, 1995 | MR

[12] Broomhead, D. S.; Lowe, David Radial Basis Functions, Multi-Variable Functional Interpolation and Adaptive Networks, Complex Systems, Volume 2 (1988), pp. 321-355 | MR | Zbl

[13] Baudry, Jean-Patrick; Maugis, Cathy; Michel, Bertrand Slope heuristics : overview and implementation., Statistics and Computing, Volume 22 (2012) no. 1, pp. 455-470 | MR | Zbl

[14] Broyden, C. G. The Convergence of a Class of Double-rank Minimization Algorithms 1. General Considerations, IMA Journal of Applied Mathematics, Volume 6 (1970) no. 1, pp. 76-90 | Zbl

[15] Currin, C.; Mitchell, T.; Morris, M.; Ylvisaker, D. A Bayesian approach to the design and analysis of computer experiments (1988) no. ORNL–6498 (Technical report)

[16] Comte, F.; Rozenholc, Y. Adaptive estimation of mean and volatility functions in (auto-)regressive models, Stochastic Processes and their Applications, Volume 97 (2002) no. 1, pp. 111-145 | DOI | MR | Zbl

[17] Chen, Victoria CP; Tsui, Kwok-Leung; Barton, Russell R; Meckesheimer, Martin A review on design, modeling and applications of computer experiments, IIE transactions, Volume 38 (2006) no. 4, pp. 273-291

[18] Cybenko, George Approximations by superpositions of sigmoidal functions, Mathematics of Control, Signals, and Systems, Volume 2 (1989) no. 4, pp. 303-314 | MR | Zbl

[19] De Lozzo, Matthias Modèles de substitution spatio-temporels et multifidélité : Application à l’ingénierie thermique, Université de Toulouse - Institut National des Sciences Appliquées de Toulouse (2013) (Ph. D. Thesis)

[20] Den Hertog, Dick; Kleijnen, JPC; Siem, AYD The correct Kriging variance estimated by bootstrapping, Journal of the Operational Research Society, Volume 57 (2005) no. 4, pp. 400-409 | Zbl

[21] Damianou, Andreas C.; Lawrence, Neil D. Deep Gaussian Processes, Proceedings of the Sixteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2013, Scottsdale, AZ, USA, April 29 - May 1, 2013 (2013), pp. 207-215

[22] Dreyfus, G.; Martinez, J.-M.; Samuelides, M.; Gordon, M. B.; Thiria, S. Apprentissage statistique : Réseaux de neurones - Cartes topologiques - Machines à vecteurs supports, Eyrolles, 2008

[23] Dean, Angela; Morris, Max; Stufken, John; Bingham, Derek Handbook of Design and Analysis of Experiments, Chapman and Hall/CRC, 2015 | MR

[24] Efron, Bradley Nonparametric estimates of standard error : The jackknife, the bootstrap and other methods, Biometrika, Volume 68 (1981) no. 3, pp. 589-599 | MR | Zbl

[25] Efron, Bradley Estimating the Error Rate of a Prediction Rule : Improvement on Cross-Validation, Journal of the American Statistical Association, Volume 78 (1983) no. 382, pp. 316-331 | MR | Zbl

[26] Efron, Bradley; Tibshirani, Robert An Introduction to the Bootstrap, Chapman & Hall, 1993 | MR | Zbl

[27] Analyse de sensibilité et exploration de modèles : application aux sciences de la nature et de l’environnement (Faivre, Robert; Iooss, Bertrand; Mahévas, Stéphanie; Makowski, David; Monod, Hervé, eds.), Collection Savoir-faire, Éditions Quae, 2013

[28] Forrester, Alexander I.J.; Keane, Andy J. Recent advances in surrogate-based optimization, Progress in Aerospace Sciences, Volume 45 (2009) no. 1, pp. 50 -79

[29] Fang, K.-T.; Li, R.; Sudjianto, A. Design and modeling for computer experiments, Chapman & Hall/CRC, 2006 | MR | Zbl

[30] Friedman, Jerome H. Multivariate Adaptive Regression Splines, The Annals of Statistics, Volume 19 (1991) no. 1, pp. 1-67 | DOI | MR | Zbl

[31] Forrester, Alexander I. J.; Sobester, Andras; Keane, Andy J. Multi-fidelity optimization via surrogate modelling, Proceedings of Royal Society A, Volume 463 (2007), pp. 3251-3269 | MR | Zbl

[32] Forrester, Alexander; Sobester, Andras; Keane, Andy Engineering Design via Surrogate Modelling : A Practical Guide, Wiley, 2008

[33] Gendre, Xavier Simultaneous estimation of the mean and the variance in heteroscedastic Gaussian regression, Electron. J. Statist., Volume 2 (2008), pp. 1345-1372 | DOI | MR | Zbl

[34] Gendre, Xavier Model selection and estimation of a component in additive regression, ESAIM : Probability and Statistics (2012) | Numdam | MR | Zbl

[35] Goldstein, Michael Bayes Linear Statistics : Theory and Methods, Wiley, Chichester, 2007 | Zbl

[36] Ghanem, Roger G.; Spanos, Pol D. Stochastic finite elements : a spectral approach, Springer-Verlag, New York, 1991 | MR | Zbl

[37] Hoerl, Arthur E.; Kennard, Robert W. Ridge Regression : Biased Estimation for Nonorthogonal Problems, Technometrics, Volume 12 (1970) no. 1, pp. 55-67 | Zbl

[38] Hornik, Kurt Approximation Capabilities of Multilayer Feedforward Networks, Neural Networks, Volume 4 (1991) no. 2, pp. 251-257

[39] Hastie, Trevor; Tibshirani, Robert; Friedman, Jerome The elements of statistical learning : data mining, inference and prediction, Springer, 2009 | MR

[40] Igel, Christian; Hüsken, Michael Improving the Rprop Learning Algorithm, Proceedings of the Second International Symposium on Neural Computation, NC2000 (Press, ICSC Academic, ed.) (2000), pp. 115-121

[41] Kohavi, Ron A study of cross-validation and bootstrap for accuracy estimation and model selection, Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, Volume 2 (1995), pp. 1137-1143

[42] Kim, Hong Seok; Koç, Muammer; Ni, Jun A hybrid multi-fidelity approach to the optimal design of warm forming processes using a knowledge-based artificial neural network, International Journal of Machine Tools and Manufacture, Volume 47 (2007) no. 2, pp. 211-222

[43] Kleijnen, J.P.C. Design and Analysis of Simulation Experiments, Springer, 2015 | MR

[44] Kennedy, Marc C.; O’Hagan, Anthony Predicting the output from a complex computer code when fast approximations are available, Biometrika, Volume 87 (2000) no. 1, pp. 1-13 | MR | Zbl

[45] Krige, D. G. A statistical approach to some basic mine valuation problems on the Witwatersrand, Journal of the Chemical, Metallurgical and Mining Society, Volume 52 (1951), pp. 119-139

[46] Lasance, Clemens J. M. Ten Years of Boundary-Condition- Independent Compact Thermal Modeling of Electronic Parts : A Review, Heat Transfer Engineering, Volume 29 (2008) no. 2, pp. 149-168 | DOI

[47] Le Gratiet, Loic Bayesian analysis of hierarchical multi-fidelity codes, SIAM/ASA Journal of Uncertainty Quantification (2013), pp. 244-269 | MR | Zbl

[48] Levenberg, K. A method for the solution of certain problems in least squares, Quart. Applied Math., Volume 2 (1944), pp. 164-168 | MR | Zbl

[49] Levin, David The Approximation Power of Moving Least-squares, Math. Comput., Volume 67 (1998) no. 224, pp. 1517-1531 | DOI | MR | Zbl

[50] Le Gratiet, Loic; Garnier, Josselin Recursive co-kriging model for Design of Computer experiments with multiple levels of fidelity, International Journal for Uncertainty Quantification, Volume 5 (2014) no. 4, pp. 365-386 | MR

[51] Lebarbier, E.; Mary-Huard, T. Une introduction au critère BIC : fondements théoriques et interprétation, Journal de la SFdS, Volume 147 (2006) no. 1, pp. 39-57 | MR | Zbl

[52] Li, Junpeng; Wang, Heping Data fusion of multi-fidelity model and its application in low speed reflexed airfoil shape optimization, Artificial Intelligence, Management Science and Electronic Commerce (AIMSEC), 2011 2nd International Conference on (2011), pp. 2910 -2913

[53] Mallows, C. L. Some Comments on CP, Technometrics, Volume 15 (1973) no. 4, pp. 661-675 | Zbl

[54] Marquardt, Donald W. An Algorithm for Least-Squares Estimation of Nonlinear Parameters, SIAM Journal on Applied Mathematics, Volume 11 (1963) no. 2, pp. 431-441 | MR | Zbl

[55] Massart, Pascal Concentration Inequalities and Model Selection, Lecture Notes in Mathematics, Springer-Verlag, 2007 | MR | Zbl

[56] Massart, Pascal Sélection de modèle : de la théorie à la pratique, Journal de la SFDS 149, Volume 149 (2008) no. 4, pp. 5-28 | MR | Zbl

[57] Matheron, Georges Principles of geostatistics, Economic Geology, Volume 58 (1963) no. 8, pp. 1246-1266

[58] Matheron, Georges La théorie des variables régionalisées et ses applications, Les Cahiers du Centre de Morphologie Mathématique, Volume 5, Fontainebleau, École Nationale Supérieure des Mines de Paris, 1970

[59] Mitchell, H. B. Multi-Sensor Data Fusion : An Introduction, Springer Publishing Company, Incorporated, 2007

[60] Myers, R.H.; Montgomery, D.C.; Anderson-Cook, C.M. Response Surface Methodology : Process and Product Optimization Using Designed Experiments, Wiley Series in Probability and Statistics, Wiley, 2009 | MR

[61] Nguyen, D.; Widrow, B. Improving the learning speed of 2-layer neural networks by choosing initial values of the adaptive weights, Proceedings of the International Joint Conference on Neural Networks, Volume 3 (1990), pp. 21-26

[62] Oussar, Yacine; Dreyfus, Gérard How to be a gray box : dynamic semi-physical modeling., Neural Networks, Volume 14 (2001) no. 9, pp. 1161-1172 http://dblp.uni-trier.de/db/journals/nn/nn14.html#OussarD01

[63] Oussar, Yacine; Rivals, Isabelle; Personnaz, Léon; Dreyfus, Gérard Training wavelet networks for nonlinear dynamic input-output modeling, Neurocomputing, Volume 20 (1998) no. 1-3, pp. 173-188 | DOI | Zbl

[64] Picheny, Victor; Ginsbourger, David A Nonstationary Space-Time Gaussian Process Model for Partially Converged Simulations, SIAM/ASA Journal on Uncertainty Quantification, Volume 1 (2013) no. 1, pp. 57-78 | MR | Zbl

[65] Picheny, Victor; Ginsbourger, David; Roustant, Olivier; Haftka, Raphael T.; Kim, Nam H. Adaptive Designs of Experiments for Accurate Approximation of a Target Region, Journal of Mechanical Design, Volume 132 (2010) no. 7

[66] Qian, Peter Z. G.; Wu, C. F. Jeff Bayesian Hierarchical Modeling for Integrating Low-accuracy and High-accuracy Experiments, Technometrics, Volume 50 (2008) no. 2, pp. 192-204 | MR

[67] Riedmiller, M.; Braun, H. A Direct Adaptive Method for Faster Backpropagation Learning : The RPROP Algorithm, IEEE International Conference on Neural Networks (1993), pp. 586-591

[68] Rumelhart, David E.; Hinton, Geoffrey E.; Williams, Ronald J. Learning representations by back-propagating errors, Nature, Volume 323 (1986), pp. 533-536 | Zbl

[69] Rasmussen, Carl Edward; Williams, Christopher K. I. Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning), The MIT Press, 2005 | MR | Zbl

[70] Santner, T. J.; B., Williams; W., Notz The Design and Analysis of Computer Experiments, Springer-Verlag, 2003, 283 pages | MR | Zbl

[71] Schwarz, Gideon Estimating the Dimension of a Model, The Annals of Statistics, Volume 6 (1978) no. 2, pp. 461-464 | MR | Zbl

[72] Simon, Dan Evolutionary Optimization Algorithms, Wiley, 2013 | MR | Zbl

[73] Sun, Guangyong; Li, Guangyao; Zhou, Shiwei; Xu, Wei; Yang, Xujing; Li, Qing Multi-fidelity optimization for sheet metal forming process, Structural and Multidisciplinary Optimization, Volume 44 (2011) no. 1, pp. 111-124

[74] Simpson, T.W.; Poplinski, J.D.; Koch, P. N.; Allen, J.K. Metamodels for Computer-based Engineering Design : Survey and recommendations, Engineering with Computers, Volume 17 (2001) no. 2, pp. 129-150 | Zbl

[75] Scholkopf, Bernhard; Smola, Alexander J. Learning with Kernels : Support Vector Machines, Regularization, Optimization, and Beyond, MIT Press, Cambridge, MA, USA, 2001

[76] Stein, Michael L. Interpolation of Spatial Data : Some Theory for Kriging (Springer Series in Statistics), Springer, 1999 | MR | Zbl

[77] Sacks, Jerome; Welch, William J.; Mitchell, Toby J.; Wynn, Henry P. Design and Analysis of Computer Experiments, Statist. Sci., Volume 4 (1989) no. 4, pp. 409-423 | DOI | MR | Zbl

[78] Tikhonov, A. N.; Arsenin, V. Y. Solutions of Ill-Posed Problems, Wiley, 1977 | MR

[79] Tarantola, Albert Inverse Problem Theory and Methods for Model Parameter Estimation, Society for Industrial and Applied Mathematics, Philadelphia, PA, USA, 2004 | MR | Zbl

[80] Tibshirani, Robert Regression Shrinkage and Selection Via the Lasso, Journal of the Royal Statistical Society, Series B, Volume 58 (1994), pp. 267-288 | MR | Zbl

[81] Vapnik, Vladimir The nature of statistical learning theory, Springer, 1995 | MR | Zbl

[82] Vapnik, V.N. An overview of statistical learning theory, Neural Networks, IEEE Transactions on, Volume 10 (1999) no. 5, pp. 988-999 | DOI

[83] Vapnik, Vladimir; Golowich, Steven E.; Smola, Alexander J. Support Vector Method for Function Approximation, Regression Estimation and Signal Processing, Advances in Neural Information Processing Systems 9, NIPS, Denver, CO, USA, December 2-5, 1996 (1996), pp. 281-287

[84] Williams, Christopher K. I. Computation with infinite neural networks, Neural Comput., Volume 10 (1998) no. 5, pp. 1203-1216 | DOI

[85] Xiu, Dongbin; Karniadakis, George Em The Wiener–Askey Polynomial Chaos for Stochastic Differential Equations, SIAM J. Sci. Comput., Volume 24 (2002) no. 2, pp. 619-644 | DOI | MR | Zbl

[86] Zou, Hui; Hastie, Trevor Regularization and variable selection via the Elastic Net, Journal of the Royal Statistical Society, Series B, Volume 67 (2005), pp. 301-320 | MR | Zbl