Model evaluation in nonlinear mixed effect models, with applications to pharmacokinetics

Comets, Emmanuelle; Brendel, Karl; Mentré, France

Journal de la société française de statistique, Tome 151 (2010) no. 1, pp. 106-127.

L’évaluation est une partie importante de la construction de modèles, faisant l’objet de recommendations de la part des autorités régulant la mise sur le marché de nouveaux médicaments. Dans ce papier, nous effectuons une courte revue de métriques récemment proposées, en particulier les VPC (Visual Predictive Check), les discordances de prédictions ($\mathrm{pd}$) et les erreurs de prédiction sur la distribution ($\mathrm{npde}$). Nous illustrons ces métriques sur quelques exemples simulés. Nous montrons comment il est possible de construire des bandes de prédiction autour de la courbe des médianes (ou d’autres percentiles) des données simulées. Ces bandes de prédiction sont un outil visuel particulièrement efficace pour détecter des zones où le modèle peut être amélioré. La distribution de certaines métriques est connue et permet de proposer des tests pour compléter les graphes diagnostiques.

Model evaluation is an important part of model building, and has been the subject of regulatory guidelines in drug development. In the present paper, we illustrate the use of some recently proposed metrics on several simulated datasets. These metrics include Visual Predictive Checks (VPC), prediction discrepancies ($\mathrm{pd}$) and normalised prediction distribution errors ($\mathrm{npde}$). We illustrate them using simulated datasets. Prediction bands around selected percentiles can be obtained through repeated simulations under the model being tested, and their addition to VPC plots or plots of $\mathrm{pd}$ and $\mathrm{npde}$ versus time and predictions are useful to highlight model deficiencies. Tests for some of the metrics are also available and can be used as a complement to graphs.

Keywords: Nonlinear mixed effect models, model evaluation, VPC, npde
Mot clés : Modèles non-linéaires à effets mixtes, évaluation de modèles, VPC, npde

[1] Bayarri, M; Berger, J P values for composite null models, Journal of the American Statistical Association, Volume 95 (2000), p. 1127-42 | Zbl

[2] Brendel, Karl; Comets, Emmanuelle; Laffont, Céline; Laveille, Christian; Mentré, France Metrics for external model evaluation with an application to the population pharmacokinetics of gliclazide, Pharmaceutical Research, Volume 23 (2006), p. 2036-49

[3] Brendel, Karl; Comets, Emmanuelle; Laffont, Céline; Mentré, France Evaluation of different tests based on observations for external model evaluation of population analyses, Journal of Pharmacokinetics and Pharmacodynamics, Volume 37 (2010), pp. 49-65

[4] Brendel, Karl; Dartois, Céline; Comets, Emmanuelle; Lemenuel-Diot, Annabelle; Laveille, Christian; Tranchand, Brigitte; Girard, Pascal; Laffont, Céline; Mentré, France Are Population Pharmacokinetic and/or Pharmacodynamic Models Adequately Evaluated? A Survey of the Literature from 2002 to 2004, Clinical Pharmacokinetics, Volume 46 (2007) no. 3, pp. 221-234

[5] Bergstrand, M; Hooker, AC; Wallin, JE; Karlsson, MO Prediction Corrected Visual Predictive Checks, American Conference on Pharmacometrics, October 4-7, 2009, Mashantucket, USA (2009) http://www.go-acop.org/sites/all/assets/webform/Poster_ACoP_VPC_091002_two_page.pdf

[6] Box, G Science and Statistics, Journal of the American Statistical Association, Volume 71 (1976), p. 791-9 | Zbl

[7] Box, GEP Sampling and Bayes inference in scientific modeling and robustness, Journal of the Royal Statistical Society A, Volume 143 (1980), pp. 383-430 | Zbl

[8] Belin, TR; Rubin, DB The analysis of repeated-measures data on schizophrenic reaction times using mixture models, Statistics in Medicine, Volume 14 (1995), p. 747-68

[9] Boeckmann, A; Sheiner, L; Beal, S NONMEM Version 5.1 (1998)

[10] Committee for Medicinal Products for Human Use, European Medicines Agency Draft guideline on reporting the results of population pharmacokinetic analyses (2006) http://www.emea.eu.int/pdfs/human/ewp/18599006en.pdf

[11] Comets, Emmanuelle; Brendel, Karl; Mentré, France Computing normalised prediction distribution errors to evaluate nonlinear mixed-effect models: the npde add-on package for R, Computer Methods and Programs in Biomedicine, Volume 90 (2008), p. 154-66

[12] Davidian, M; Giltinan, D Nonlinear models for repeated measurement data, Chapman & Hall, London, 1995, pp. 145-176

[13] Ette, E; Williams, PJ Pharmacometrics: the science of quantitative pharmacology, Wiley-Interscience, Hoboken, New Jersey, 2007 | Zbl

[14] Food and Drug Administration Guidance for Industry: Population Pharmacokinetics (1999) http://www.fda.gov/cder/guidance/index.htm

[15] Girard, P; Blaschke, T; Kastrissiosr, H; Sheiner, L A Markov mixed effect regression model for drug compliance, Statistics in Medicine, Volume 17 (1998), p. 2313-33

[16] Gelman, A; Carlin, J; Stern, H.; Rubin, D Bayesian Data Analysis, Chapman & Hall, London, 1995 | Zbl

[17] Gelfand, AE; Det, DK; Chang, H Bayesian statistics (Bernardo, JM; Berger, J0; David, AP; Smith, AFM, eds.), Oxford University Press, Oxford, 1992, pp. 147-167

[18] Gelman, A; Van Mechelen, I; Verbeke, G; Heitjan, D; Meulders, M Multiple imputation for model checking: completed-data plots with missing and latent data, Biometrics, Volume 61 (2005), pp. 74-85 | Zbl

[19] Holford, N Wings for Nonmem, http://wfn.sourceforge.net, University of Auckland, Auckland, New Zealand, 2000 http://wfn.sourceforge.net

[20] Holford, N The Visual Predictive Check: superiority to Standard Diagnostic (Rorschach) Plots, 14th meeting of the Population Approach Group in Europe, Pamplona, Spain (2005)

[21] Hooker, A; Staatz, C; Karlsson, MO Conditional Weighted Residuals (CWRES): a model diagnostic for the FOCE method, Pharmaceutical Research, Volume 24 (2007), p. 2187-97

[22] Hénin, E; You, B; VanCutsem, E; Hoff, P M; Cassidy, J; Twelves, C; Zuideveld, K P; Sirzen, F; Dartois, C; Freyer, G; Tod, M; Girard, P A dynamic model of hand-and-foot syndrome in patients receiving capecitabine, Clinical Pharmacology & Therapeutics, Volume 85 (2009), p. 418-25

[23] Jonsson, E; Karlsson, M Xpose–an S-PLUS based population pharmacokinetic/pharmacodynamic model building aid for NONMEM, Computer Methods and Programs in Biomedicine, Volume 58 (1999) no. 1, pp. 51-64

[24] Karlsson, M; Holford, N A tutorial on Visual Predictive Checks, 17th meeting of the Population Approach Group in Europe, Marseille, France (2008)

[25] Kuhn, E; Lavielle, M Maximum likelihood estimation in nonlinear mixed effects models, Computational Statistics and Data Analysis, Volume 49 (2005), pp. 1020-1038 | Zbl

[26] Karlsson, MO; Savic, RM Diagnosing model diagnostics, Clinical Pharmacology & Therapeutics, Volume 82 (2007), pp. 17-20

[27] Lindstrom, M; Bates, D Nonlinear mixed effects models for repeated measures data, Biometrics, Volume 46 (1990), p. 673-87

[28] Laffont, C; Concordet, D A new exact test to globally assess a population PK and/or PD model, 18th meeting of the Population Approach Group in Europe, St-Petersburg, Russia (2009)

[29] Lemenuel-Diot, A; Laffont, C M; Jochemsen, R; Foos-Gilbert, E Evaluation of model of heart rate during exercise tolerance test with missing at random dropouts, 16th meeting of the Population Approach Group in Europe, Copenhague, Denmark (2007)

[30] Lindbom, L; Pihlgren, P; Jonsson, EN PsN-Toolkit–a collection of computer intensive statistical methods for non-linear mixed effect modeling using NONMEM, Computer Methods and Programs in Biomedicine, Volume 79 (2005), p. 241-57

[31] Mallet, A A maximum likelihood estimation method for random coefficient regression models, Biometrika, Volume 73 (1986), p. 645-56 | Zbl

[32] Mentré, F; Escolano, S Prediction discrepancies for the evaluation of nonlinear mixed-effects models, Journal of Pharmacokinetics and Biopharmaceutics, Volume 33 (2006), p. 345-67

[33] Meng, XL Posterior predictive p-values, Annals of Statistics, Volume 22 (1994), p. 1142-60 | Zbl

[34] Mesnil, F; Mentré, F; Dubruc, C; Thénot, JP; Mallet, A Population pharmacokinetics analysis of mizolastine and validation from sparse data on patients using the nonparametric maximum likelihood method, Journal of Pharmacokinetics and Biopharmaceutics, Volume 26 (1998), p. 133-61

[35] Pinheiro, J; Bates, D Approximations to the log-likelihood function in the non-linear mixed-effect models, Journal of Computational and Graphical Statistics, Volume 4 (1995), pp. 12-35

[36] Rubin, DB Bayesianly justifiable and relevant frequency calculations for the applied statistician, Annals of Statistics, Volume 12 (1984), p. 1151-72 | Zbl

[37] Robins, J; van der Vaart, A; Ventura, V Asymptotic distribution of p-values in composite null models, Journal of the American Statistical Society, Volume 95 (2000), p. 1143-56 | Zbl

[38] Savic, RM; Lavielle, M Performance in population models for count data, part II: A new SAEM algorithm, Journal of Pharmacokinetics and Pharmacodynamics, Volume 36 (2009), p. 367-79

[39] Semmar, N; Urien, S; Bruguerolle, B; Simon, N Independent-model diagnostics for a priori identification and interpretation of outliers from a full pharmacokinetic database: correspondence analysis, Mahalanobis distance and Andrews curves, Journal of Pharmacokinetics and Pharmacodynamics, Volume 35 (2008), p. 159-83

[40] Vozeh, S; Maitre, P; Stanski, D Evaluation of population (NONMEM) pharmacokinetic parameter estimates, Journal of Pharmacokinetics and Biopharmaceutics, Volume 18 (1990), p. 161-73

[41] Wilkins, J; Karlsson, M; Jonsson, N Patterns and power for the visual predictive check, 15th meeting of the Population Approach Group in Europe, Brugges, Belgium (2006)

[42] Wolfinger, R Laplace’s approximation for nonlinear mixed models, Biometrika, Volume 80 (1993), p. 791-5 | Zbl

[43] Yano, Y; Beal, S; Sheiner, LB Evaluating pharmacokinetic/pharmacodynamic models using the posterior predictive check., Journal of Pharmacokinetics and Pharmacodynamics, Volume 28 (2001) no. 2, pp. 171-192