Backward SDEs and infinite horizon stochastic optimal control

Confortola, Fulvia; Cosso, Andrea; Fuhrman, Marco

ESAIM: Control, Optimisation and Calculus of Variations, Tome 25 (2019), article no. 31.

Suite au passage du modèle économique de la revue en S20, le texte intégral des articles des années 2019 et 2020 est accessible uniquement sur le site de l'éditeur et est réservé aux abonnés.

We study an optimal control problem on infinite horizon for a controlled stochastic differential equation driven by Brownian motion, with a discounted reward functional. The equation may have memory or delay effects in the coefficients, both with respect to state and control, and the noise can be degenerate. We prove that the value, i.e. the supremum of the reward functional over all admissible controls, can be represented by the solution of an associated backward stochastic differential equation (BSDE) driven by the Brownian motion and an auxiliary independent Poisson process and having a sign constraint on jumps.

In the Markovian case when the coefficients depend only on the present values of the state and the control, we prove that the BSDE can be used to construct the solution, in the sense of viscosity theory, to the corresponding Hamilton-Jacobi-Bellman partial differential equation of elliptic type on the whole space, so that it provides us with a Feynman-Kac representation in this fully nonlinear context.

The method of proof consists in showing that the value of the original problem is the same as the value of an auxiliary optimal control problem (called randomized), where the control process is replaced by a fixed pure jump process and maximization is taken over a class of absolutely continuous changes of measures which affect the stochastic intensity of the jump process but leave the law of the driving Brownian motion unchanged.

[1] E. Bandini, Constrained BSDEs Driven by a Non Quasi-Left-Continuous Random Measure and Optimal Control of PDMPs on Bounded Domains. Preprint (2017). | arXiv

[2] E. Bandini, Optimal control of piecewise deterministic Markov processes: a BSDE representation of the value function. ESAIM: COCV 24 (2018) 311–354. | Numdam | MR | Zbl

[3] E. Bandini and M. Fuhrman, Constrained BSDEs representation of the value function in optimal control of pure jump Markov processes. Stochastic Process. Appl. 127 (2017) 1441–1474. | DOI | MR | Zbl

[4] E. Bandini, A. Cosso, M. Fuhrman, and H. Pham, Backward SDEs for optimal control of partially observed path-dependent stochastic systems: a control randomization approach. Ann. Appl. Probab. 28 (2018) 1634–1678. | DOI | MR | Zbl

[5] A. Bensoussan, Stochastic Control by Functional Analysis Methods. Vol. 11 of Studies in Mathematics and its Applications. North-Holland Publishing Co., Amsterdam/New York (1982). | MR | Zbl

[6] B. Bouchard, A stochastic target formulation for optimal switching problems in finite horizon. Stochastics. 81 (2009) 171–197. | DOI | MR | Zbl

[7] A. Cosso, M. Fuhrman, and H. Pham, Long time asymptotics for fully nonlinear Bellman equations: a backward SDE approach. Stochastic Process. Appl. 126 (2016) 1932–1973. | DOI | MR | Zbl

[8] C. Dellacherie and P.-A. Meyer, Probabilities and Potential A. Vol. 29 of North-Holland Mathematics Studies. North-Holland Publishing Co., Amsterdam-New York (1978). | MR | Zbl

[9] R. Elie and I. Kharroubi, Adding constraints to BSDEs with jumps: an alternative to multidimensional reflections. ESAIM: PS 18 (2014) 233–250. | DOI | Numdam | MR | Zbl

[10] R. Elie and I. Kharroubi, BSDE representations for optimal switching problems with controlled volatility. Stoch. Dyn. 14 (2014) 1450003. | DOI | MR | Zbl

[11] W. H. Fleming and R. W. Rishel, Deterministic and stochastic optimal control, in Applications of Mathematics, No. 1. Springer-Verlag, Berlin-New York (1975). | MR | Zbl

[12] W. H. Fleming and H. M. Soner, Controlled Markov processes and viscosity solutions. Vol. 25 of Stochastic Modelling and Applied Probability, 2nd edn. Springer (2006). | MR | Zbl

[13] M. Fuhrman and H. Pham, Randomized and backward SDE representation for optimal control of non-Markovian SDEs. Ann. Appl. Probab. 25 (2015) 2134–2167. | DOI | MR | Zbl

[14] M. Fuhrman, H. Pham, and F. Zeni, Representation of non-Markovian optimal stopping problems by constrained BSDEs with a single jump. Electron. Commun. Probab. 21 (2016) 7. | DOI | MR | Zbl

[15] H. Ishii, On uniqueness and existence of viscosity solutions of fully nonlinear second-order elliptic PDEs. Commun. Pure Appl. Math. 42 (1989) 15–45. | DOI | MR | Zbl

[16] I. Kharroubi and H. Pham, Feynman–Kac representation for Hamilton–Jacobi–Bellman IPDE. Ann. Probab. 43 (2015) 1823–1865. | DOI | MR | Zbl

[17] I. Kharroubi, J. Ma, H. Pham, and J. Zhang, Backward SDEs with constrained jumps and quasi-variational inequalities. Ann. Probab. 38 (2010) 794–840. | DOI | MR | Zbl

[18] I. Kharroubi, N. Langrené, and H. Pham, A numerical algorithm for fully nonlinear HJB equations: an approach by control randomization. Monte Carlo Methods Appl. 20 (2014) 145–165. | DOI | MR | Zbl

[19] I. Kharroubi, N. Langrené, and H. Pham, Discrete time approximation of fully nonlinear HJB equations via BSDEs with nonpositive jumps. Ann. Appl. Probab. 25 (2015) 2301–2338. | DOI | MR | Zbl

[20] N. V. Krylov, Controlled diffusion processes. Vol. 14 of Stochastic Modelling and Applied Probability. Translated from the 1977 Russian original by A. B. Aries, Reprint of the 1980 edition. Springer-Verlag, Berlin (2009). | MR | Zbl

[21] M. Nisio, Stochastic control theory, in Dynamic Programming Principle. Vol. 72 of Probability Theory and Stochastic Modelling. Springer, Tokyo, 2nd edn. (2015). | DOI | MR | Zbl

[22] É. Pardoux, Backward stochastic differential equations and viscosity solutions of systems of semilinear parabolic and elliptic PDEs of second order, in Stochastic Analysis and Related Topics, VI (Geilo, 1996). Vol. 42 of Progr. Probab. Birkhäuser Boston, Boston, MA (1998) 79–127. | MR | Zbl

[23] É. Pardoux, BSDEs, weak convergence and homogenization of semilinear PDEs, in Nonlinear Analysis, Differential Equations and Control (Montreal, QC, 1998). Vol. 528 of NATO Sci. Ser. C Math. Phys. Sci. Kluwer Academic Publishers, Dordrecht (1999) 503–549. | MR | Zbl

[24] É. Pardoux and S. Peng, Backward stochastic differential equations and quasilinear parabolic partial differential equations, in Stochastic Partial Differential Equations and their Applications (Charlotte, NC, 1991). Vol. 176 of Lecture Notes in Control and Information Sciences. Springer, Berlin (1992) 200–217. | DOI | MR | Zbl

[25] S. Peng, Probabilistic interpretation for systems of quasilinear parabolic partial differential equations. Stoch. Stoch. Rep. 37 (1991) 61–74. | DOI | MR | Zbl

[26] S. Peng, Monotonic limit theorem of BSDE and nonlinear decomposition theorem of Doob-Meyer’s type. Probab. Theory Related Fields. 113 (1999) 473–499. | DOI | MR | Zbl

[27] S. Peng, G-expectation, G-Brownian motion and related stochastic calculus of Itô type, in Stochastic Analysis and Applications. Vol. 2 of Abel Symposium. Springer, Berlin (2007) 541–567. | DOI | MR | Zbl

[28] H. Pham, Continuous-time stochastic control and optimization with financial applications. Vol. 61 of Stochastic Modelling and Applied Probability. Springer-Verlag, Berlin (2009). | DOI | MR | Zbl

[29] D. Revuz and M. Yor, Continuous Martingales and Brownian Motion, 3rd edn. Springer, Berlin (1999). | DOI | MR | Zbl

[30] H. M. Soner, N. Touzi and J. Zhang, Wellposedness of second order backward SDEs. Probab. Theory Related Fields. 153 (2012) 149–190. | DOI | MR | Zbl

[31] S. Tang and X. Li, Necessary conditions for optimal control of stochastic systems with random jumps. SIAM J. Control Optim 32 (1994) 1447–1475. | DOI | MR | Zbl

[32] J. Yong and X. Y. Zhou, Stochastic Controls: Hamiltonian Systems and HJB Equations. Vol. 43 of Applications of Mathematics. Springer-Verlag (1999). | MR | Zbl