Optimal control of piecewise deterministic Markov processes: a BSDE representation of the value function

Bandini, Elena

ESAIM: Control, Optimisation and Calculus of Variations, Tome 24 (2018) no. 1, pp. 311-354.

We consider an infinite-horizon discounted optimal control problem for piecewise deterministic Markov processes, where a piecewise open-loop control acts continuously on the jump dynamics and on the deterministic flow. For this class of control problems, the value function can in general be characterized as the unique viscosity solution to the corresponding Hamilton−Jacobi−Bellman equation. We prove that the value function can be represented by means of a backward stochastic differential equation (BSDE) on infinite horizon, driven by a random measure and with a sign constraint on its martingale part, for which we give existence and uniqueness results. This probabilistic representation is known as nonlinear Feynman−Kac formula. Finally we show that the constrained BSDE is related to an auxiliary dominated control problem, whose value function coincides with the value function of the original non-dominated control problem.

[1] A. Almudevar, A dynamic programming algorithm for the optimal control of piecewise deterministic Markov processes. SIAM J. Control Optimiz. 40 (2000) 525–539. | DOI | MR | Zbl

[2] E. Bandini, Existence and uniqueness for backward stochastic differential equations driven by a random measure. Electr. Commun. Probab. 20 (2015) 1–13. | MR | Zbl

[3] E. Bandini and Confortola, F. Optimal control of semi-Markov processes with a backward stochastic differential equations approach. Math. Control Signals Syst. 29 (2017) 1–35. | DOI | MR | Zbl

[4] E. Bandini and M. Fuhrman, Constrained BSDEs representation of the value function in optimal control of pure jump Markov processes. Stochastic Proc. Appl. 127 (2017) 1441–1474. | DOI | MR | Zbl

[5] E. Bandini and F. Russo, Weak Dirichlet processes with jumps. Preprint arXiv:1512.06236 (2016). | MR

[6] E. Bandini and F. Russo, Special weak Dirichlet processes and BSDEs driven by a random measure. Preprint arXiv:1512.06234 (2016). To appear on Bernouilli (2017). | MR

[7] Barles, G., Buckdahn, R. and Pardoux, É. Backward stochastic differential equations and integral-partial differential equations. Stochastics Rep. 60 (1997) 57–83. | DOI | MR | Zbl

[8] N. Bauerle and U. Rieder, Optimal control of piecewise deterministic Markov processes with finite time horizon. In Modern Trends in Controlled Stochastic Processes: Theory and Applications. Luniver Press, United Kingdom (2010) 123–143.

[9] D. Becherer, Bounded solutions to backward SDE’s with jumps for utility optimization and indifference hedging. Ann. Appl. Probab. 16 (2006) 2027–2054. | DOI | MR | Zbl

[10] P. Brémaud, Point processes and queues, Martingale dynamics. Springer, Series in Statistics (1981). | DOI | MR | Zbl

[11] H. Brezis, Functional Analysis. Sobolev Spaces and Partial Differential Equations. Springer (2010). | MR | Zbl

[12] S. Choukroun and A. Cosso, Backward SDE Representation for Stochastic Control Problems with Non Dominated Controlled Intensity. Ann. Appl. Probab. 26 (2016) 1208–1259. | DOI | MR | Zbl

[13] S. Choukroun, A. Cosso and H. Pham, Reflected BSDEs with nonpositive jumps, and controller-and-stopper games. Stochastic Proc. Appl. 125 (2015) 597–633. | DOI | MR | Zbl

[14] S. Cohen and R. Elliott, Existence, uniqueness and comparisons for BSDEs in general spaces. Ann. Probab. 40 (2012) 2264–2297. | DOI | MR | Zbl

[15] F. Confortola and M. Fuhrman, Backward stochastic differential equations and optimal control of marked point processes. SIAM J. Control Optim. 51 (2013) 3592–3623. | DOI | MR | Zbl

[16] F. Confortola and M. Fuhrman, Backward stochastic differential equations associated to jump Markov processes and their applications. Stochastic Proc. Appl. 124 (2014) 289–316. | DOI | MR | Zbl

[17] F. Confortola, M. Fuhrman and J. Jacod, Backward stochastic differential equations driven by a marked point process: an elementary approach, with an application to optimal control. Ann. Appl. Probab. 26 (2016) 1743–1773. | DOI | MR | Zbl

[18] A. Cosso, M. Fuhrman and H. Pham, Long time asymptotics for fully nonlinear Bellman equations: a Backward SDE approach. Stochastic Proc. Appl. 126 (2016) 1932–1973. | DOI | MR | Zbl

[19] A. Cosso, H. Pham and H. Xing, BSDEs with diffusion constraint and viscous Hamilton−Jacobi equations with unbounded data. Ann. Inst. Henri Poincaré Probab. Stat. 53 (2017) 1528–1547. | DOI | MR | Zbl

[20] O.L. Costa and F. Dufour, Continuous Average Control of Piecewise Deterministic Markov Processes. Springer Briefs in Mathematics. Springer (2013). | MR | Zbl

[21] M.H.A. Davis, Markov models and optimization. In Vol. 49 of Monographs on Statistics and Applied Probability. Chapman and Hall, London (1993). | MR | Zbl

[22] M.H.A. Davis and M. Farid, Piecewise deterministic processes and viscosity solutions. A Volume in Honour of W. H. Fleming on Occasion of His 70th Birthday, edited by W.M. Mceneaney, Stochastic Analysis, Control Optimization and Applications. Birkhäuser (1999) 249–268. | DOI | MR | Zbl

[23] C. Dellacherie and P. Meyer, Probabilities et Potentiel, Chapters I to IV. Hermann, Paris (1978).

[24] M.A.H. Dempster, Optimal control of piecewise deterministic Markov processes. Applied stochastic analysis, Vol. 5 of Stochastics Monogr. Gordon and Breach, New York (1991) 303–325. | MR | Zbl

[25] M.A.H. Dempster and J.J. Ye, Necessary and sufficient conditions for control of piecewise deterministic processes. Stochastics and Stochastics Reports 40 (1992) 125–145. | DOI | MR | Zbl

[26] E.L. Karoui, N., S. Peng and M.C. Quenez, Backward stochastic differential equations in finance. Math. Finance 7 (1997) 1–71. | DOI | MR | Zbl

[27] R.J. Elliott, Stochastic Calculus and its Applications. Springer (1982). | MR

[28] R. Elie and I. Kharroubi, Probabilistic representation and approximation for coupled systems of variational inequalities. Statist. Probab. Lett. 80 (2010) 1388–1396. | DOI | MR | Zbl

[29] R. Elie and I. Kharroubi, BSDE representations for optimal switching problems with controlled volatility. Stoch. Dyn. 14 (2014) 1450003. | DOI | MR | Zbl

[30] M. Fuhrman and H. Pham, Randomized and Backward SDE representation for optimal control of non-markovian SDEs. Ann. Appl. Probab. 25 (2015) 2134–2167. | DOI | MR | Zbl

[31] S. He, J. Wang and J. Yan, Semimartingale theory and stochastic calculus, Science Press, Bejiing, New York (1992). | MR | Zbl

[32] J. Jacod, Multivariate point processes: predictable projection, Radon−Nikodym derivatives, representation of martingales. Z. Wahrscheinlichkeitstheorie und Verw. Gebiete 31 (1974/75) 235–253. | DOI | MR | Zbl

[33] J. Jacod, Calcul stocastique et problèmes de martingales. Vol. 714 of Lect. Notes Math. Springer, Berlin (1979). | MR | Zbl

[34] J. Jacod and P.E. Protter, Probability essentials. Springer Science and Business Media (2003). | MR | Zbl

[35] J. Jacod and A.N. Shiryaev, Limit Theorems for Stochastic Processes, 2nd edition, Springer, Berlin (2003). | DOI | MR

[36] I. Kharroubi, N. Langrené and H. Pham, A numerical algorithm for fully nonlinear HJB equations: an approach by control randomization. Monte Carlo Methods Appl. 20 (2014) 145–165. | DOI | MR | Zbl

[37] I. Kharroubi, N. Langrené and H. Pham, Discrete time approximation of fully nonlinear HJB equations via BSDEs with nonpositive jumps. Ann. Appl. Probab. 25 (2015) 2301–2338. | DOI | MR | Zbl

[38] I. Kharroubi and H. Pham, Feynman−Kac representation for Hamilton−Jacobi−Bellman IPDE. Ann. Probab. 43 (2015) 1823–1865. | MR | Zbl

[39] I. Kharroubi, Ma, J., H. Pham and J. Zhang, Backward SDEs with contrained jumps and quasi-variational inequalities. Ann. Probab. 38 (2010) 794–840. | DOI | MR | Zbl

[40] S. Peng, Monotonic limit theorem for BSDEs and non-linear Doob-Meyer decomposition. Probab. Theory Rel. 16 (2000) 225–234.

[41] D. Revuz and M. Yor, Continuous Martingales and Brownian Motion. Grundlehren der mathematischen Wissenschaften. Springer, 3rd edition (1999). | DOI | MR

[42] H.M. Soner, Optimal control with state-space constraint II. SIAM J. Control Optim. 24 (1986) 1110–1122. | DOI | MR | Zbl

[43] D.W. Stroock and S.S. Varadhan, Multidimensional diffusion processes, Springer (2007). | MR | Zbl

[44] D. Vermes, Optimal control of piecewise deterministic Markov process. Stochastics 14 (1985) 165–207. | DOI | MR | Zbl

[45] J. Xia, Backward stochastic differential equations with random measures. Acta Math. Appl. Sinica 16 (2000) 225–234. | DOI | MR | Zbl