Bellman equation and viscosity solutions for mean-field stochastic control problem

Pham, Huyên; Wei, Xiaoli

ESAIM: Control, Optimisation and Calculus of Variations, Tome 24 (2018) no. 1, pp. 437-461.

We consider the stochastic optimal control problem of McKean−Vlasov stochastic differential equation where the coefficients may depend upon the joint law of the state and control. By using feedback controls, we reformulate the problem into a deterministic control problem with only the marginal distribution of the process as controlled state variable, and prove that dynamic programming principle holds in its general form. Then, by relying on the notion of differentiability with respect to probability measures recently introduced by [P.L. Lions, Cours au Collège de France: Théorie des jeux à champ moyens, audio conference 2006−2012], and a special Itô formula for flows of probability measures, we derive the (dynamic programming) Bellman equation for mean-field stochastic control problem, and prove a verification theorem in our McKean−Vlasov framework. We give explicit solutions to the Bellman equation for the linear quadratic mean-field control problem, with applications to the mean-variance portfolio selection and a systemic risk model. We also consider a notion of lifted viscosity solutions for the Bellman equation, and show the viscosity property and uniqueness of the value function to the McKean−Vlasov control problem. Finally, we consider the case of McKean−Vlasov control problem with open-loop controls and discuss the associated dynamic programming equation that we compare with the case of closed-loop controls.

[1] N.U. Ahmed and X. Ding, Controlled McKean−Vlasov equation. Commun. Appl. Anal. 5 (2001) 183–206. | MR | Zbl

[2] L. Ambrosio, N. Gigli and G. Savaré, Gradient Flows in Metric Spaces and in the Space of Probability Measures. Lect. Math. Birkhäuser Verlag, Basel (2005). | MR | Zbl

[3] D. Andersson and B. Djehiche, A maximum principle for SDEs of mean-field type. Appl. Math. Optimiz. 63 (2010) 341–356. | DOI | MR | Zbl

[4] E. Bayraktar, A. Cosso and H. Pham, Randomized dynamic programming principle and Feynman-Kac representation for optimal control of McKean−Vlasov dynamics. Trans. Amer. Math. Soc. 370 (2018) 2115–2160. | DOI | MR | Zbl

[5] A. Bensoussan, J. Frehse and P. Yam, The Master equation in mean-field theory. J. Math. Pures Appl. 103 (2015) 1441–1474. | DOI | MR | Zbl

[6] A. Bensoussan, J. Frehse and P. Yam, On the interpretation of the Master equation. Stochastic Processes their Appl. 127 (2017) 2093–2137. | DOI | MR | Zbl

[7] A. Bensoussan, K.C. Sung, P. Yam and S.P. Yung, Linear-quadratic mean field games. J. Optimiz. Theory Appl. 169 (2016) 496–529. | DOI | MR | Zbl

[8] T. Björk, M. Khapko and A. Murgoci, On time inconsistent stochastic control in continuous time. Finance Stoch. 21 (2017) 331–360. | DOI | MR | Zbl

[9] R. Buckdahn, B. Djehiche and J. Li, A general maximum principle for SDEs of mean-field type. Appl. Math. Optimiz. 64 (2011) 197–216. | DOI | MR | Zbl

[10] R. Buckdahn, J. Li, S. Peng and C. Rainer, Mean-field stochastic differential equations and associated PDEs. Ann. probab. 45 (2017) 824–878. | DOI | MR | Zbl

[11] P. Cardaliaguet, Notes on mean field games, Notes from P.L. Lions lectures at Collège de France (2013)

[12] R. Carmona and F. Delarue, The Master equation for large population equilibriums, Proceedings in Mathematics and Statistics 100. | MR

[13] R. Carmona and F. Delarue, Forward-backward Stochastic Differential Equations and Controlled McKean Vlasov Dynamics, Ann. Probab. 43 (2015) 2647–2700. | DOI | MR | Zbl

[14] R. Carmona, F. Delarue and A. Lachapelle, Control of McKean−Vlasov dynamicsversus mean field games. Math. Financial Econ. 7 (2013) 131–166. | DOI | MR | Zbl

[15] R. Carmona, J.P. Fouque and L. Sun, Mean field games and systemic risk. Commun. Math. Sci. 13 (2015) 911–933. | DOI | MR | Zbl

[16] J.F. Chassagneux, D. Crisan and F. Delarue, A probabilistic approach to classical solutions of the master equation for large population equilibria. Preprint (2015). | arXiv | MR

[17] J.L. Doob, Measure Theory. Graduate texts Math. 143 Springer (1994). | MR | Zbl

[18] G. Fabbri, F. Gozzi and A. Swiech, Stochastic Optimal Control in Infinite Dimension: Dynamic Programming and HJB Equations with Chapter 6 by M. Fuhrman and G. Tessitore (2015). | MR

[19] J. Feng and M. Katsoulakis, A comparison principle for Hamilton-Jacobi equations related to controlled gradient flows in infinite dimensions. Archive Rat. Mech. Anal. 192 (2009) 275–310. | DOI | MR | Zbl

[20] M. Fisher and G. Livieri, Continuous time mean-variance portfolio optimization through the mean-field approach. ESAIM: PS 20 (2016) 30–44. | DOI | Numdam | MR | Zbl

[21] W.H. Fleming and H.M. Soner, Controlled Markov Processes and Viscosity Solutions, 2nd edition, Springer Verlag (2006). | MR | Zbl

[22] W. Gangbo, T. Nguyen and A. Tudorascu, Hamilton-Jacobi equations in the Wasserstein space. Methods Appl. Anal. 15 (2008) 155–184. | DOI | MR | Zbl

[23] W. Gangbo and A. Swiech, Metric viscosity solutions of Hamilton-Jacobi equations depending on local slopes. Calcul. Variat. Partial Differ. Equ. 54 (2015) 1183–1218. | DOI | MR | Zbl

[24] M. Huang, P. Caines and R. Malhamé, Large population stochastic dynamic games: closed-loop McKean−Vlasov systems and the Nash certainty equivalence principle. Commun. Infor. Syst. 6 (2006) 221–252. | DOI | MR | Zbl

[25] B. Jourdain, S. Méléard and W. Woyczynski, Nonlinear SDEs driven by Lévy processes and related PDEs. ALEA, Latin Amer. J. Probab. 4 (2008) 1–29. | MR | Zbl

[26] M. Kac, Foundations of kinetic theory, in Proceedings of the 3rd Berkeley Symposium on Mathematical Statistics and Probability 3 (1956) 171–197. | MR | Zbl

[27] J.M. Lasry and P.L. Lions, Mean-field games. Japanese J. Math. 2 (2007) 229–60. | DOI | MR | Zbl

[28] M. Laurière and O. Pironneau, Dynamic programming for mean-field type control. J. Optimiz. Theory Appl. 169 (2016) 902–924. | DOI | MR | Zbl

[29] D. Li and X.Y. Zhou, Continuous-time mean-variance portfolio selection: a stochastic LQ framework. App. Math. Optimiz. 42 (2000) 19–33. | DOI | MR | Zbl

[30] P.L. Lions, Viscosity solutions of fully nonlinear second-order equations and optimal control in infinite dimension. Part I: the case of bounded stochastic evolution. Acta Math. 161 (1988) 243–278. | DOI | MR | Zbl

[31] P.L. Lions, Viscosity solutions of fully nonlinear second-order equations and optimal control in infinite dimension. Part III: Uniqueness of viscosity solutions for general second-order equations. J. Functional Anal. 86 (1989) 1–18. | DOI | MR | Zbl

[32] P.L. Lions, Cours au Collège de France: Théorie des jeux à champ moyens, audio conference 2006–2012.

[33] H.P. Mckean, Propagation of chaos for a class of nonlinear parabolic equations. Lect. Series Differ. Equ. 7 (1967) 41–57. | MR

[34] H. Pham, Continuous-time stochastic control and applications with financial applications. Series Stochastic Modeling and Applied Probability 61. Springer (2009). | MR | Zbl

[35] D. Revuz and M. Yor, Continuous Martingales and Brownian Motion, 3rd edition. New York, Berlin: Springer (1999). | DOI | MR | Zbl

[36] A.S. Sznitman, Topics in propagation of chaos, in Lect. Notes Math. Springer 1464 (1989) 165–251. | MR | Zbl

[37] C. Villani, Optimal Transport, Old and New. Springer (2009). | DOI | Zbl

[38] J. Yong, A linear-quadratic optimal control problem for mean-field stochastic differential equations. SIAM J. Control Optimiz. 51 (2013) 2809–2838. | DOI | MR | Zbl