The estimation of motion in an image sequence is a fundamental task in image processing. Frequently, the image sequence is corrupted by noise and one simultaneously asks for the underlying motion field and a restored sequence. In smoothly shaded regions of the restored image sequence the brightness constancy assumption along motion paths leads to a pointwise differential condition on the motion field. At object boundaries which are edge discontinuities both for the image intensity and for the motion field this condition is no longer well defined. In this paper a total-variation type functional is discussed for joint image restoration and motion estimation. This functional turns out not to be lower semicontinuous, and in particular fine-scale oscillations may appear around edges. By the general theory of vector valued functionals its relaxation leads to the appearance of a singular part of the energy density, which can be determined by the solution of a local minimization problem at edges. Based on bounds for the singular part of the energy and under appropriate assumptions on the local intensity variation one can exclude the existence of microstructures and obtain a model well-suited for simultaneous image restoration and motion estimation. Indeed, the relaxed model incorporates a generalized variational formulation of the brightness constancy assumption. The analytical findings are related to ambiguity problems in motion estimation such as the proper distinction between foreground and background motion at object edges.
DOI : 10.1051/m2an/2015036
Mots-clés : Optical flow, BVfunctional, relaxation, microstructures
@article{M2AN_2015__49_5_1463_0, author = {Conti, Sergio and Ginster, Janusz and Rumpf, Martin}, title = {A $BV$ functional and its relaxation for joint motion estimation and image sequence recovery}, journal = {ESAIM: Mathematical Modelling and Numerical Analysis }, pages = {1463--1487}, publisher = {EDP-Sciences}, volume = {49}, number = {5}, year = {2015}, doi = {10.1051/m2an/2015036}, zbl = {1326.49019}, language = {en}, url = {http://archive.numdam.org/articles/10.1051/m2an/2015036/} }
TY - JOUR AU - Conti, Sergio AU - Ginster, Janusz AU - Rumpf, Martin TI - A $BV$ functional and its relaxation for joint motion estimation and image sequence recovery JO - ESAIM: Mathematical Modelling and Numerical Analysis PY - 2015 SP - 1463 EP - 1487 VL - 49 IS - 5 PB - EDP-Sciences UR - http://archive.numdam.org/articles/10.1051/m2an/2015036/ DO - 10.1051/m2an/2015036 LA - en ID - M2AN_2015__49_5_1463_0 ER -
%0 Journal Article %A Conti, Sergio %A Ginster, Janusz %A Rumpf, Martin %T A $BV$ functional and its relaxation for joint motion estimation and image sequence recovery %J ESAIM: Mathematical Modelling and Numerical Analysis %D 2015 %P 1463-1487 %V 49 %N 5 %I EDP-Sciences %U http://archive.numdam.org/articles/10.1051/m2an/2015036/ %R 10.1051/m2an/2015036 %G en %F M2AN_2015__49_5_1463_0
Conti, Sergio; Ginster, Janusz; Rumpf, Martin. A $BV$ functional and its relaxation for joint motion estimation and image sequence recovery. ESAIM: Mathematical Modelling and Numerical Analysis , Tome 49 (2015) no. 5, pp. 1463-1487. doi : 10.1051/m2an/2015036. http://archive.numdam.org/articles/10.1051/m2an/2015036/
Functionals defined on partitions in sets of finite perimeter. I. Integral representation and -convergence. J. Math. Pures Appl. 69 (1990) 285–305. | Zbl
and ,Functionals defined on partitions in sets of finite perimeter. II. Semicontinuity, relaxation and homogenization. J. Math. Pures Appl. 69 (1990) 307–333. | Zbl
and ,On the relaxation in BV(Ω;Rm) of quasi-convex integrals. J. Funct. Anal. 109 (1992) 76–97. | DOI | Zbl
and ,L. Ambrosio, N. Fusco and D. Pallara, Functions of Bounded Variation and Free Discontinuity Problems. Oxford Math. Monogr. Oxford University Press, New York (2000). | Zbl
A mathematical study of the relaxed optical flow problem in the space . SIAM J. Math. Anal. 30 (1999) 1282–1308. | DOI | Zbl
and ,Computing optical flow via variational techniques. SIAM J. Appl. Math. 60 (1999) 156–182. | DOI | Zbl
, and ,Variational integrals on mappings of bounded variation and their lower semicontinuity. Arch. Ration. Mech. Anal. 115 (1991) 201–255. | DOI | Zbl
and ,J. Bigun and G.H. Granlund, Optical flow based on the inertia matrix of the frequency domain. In Proc. of SSAB Symposium on Picture Processing: Lund University, Sweden (1988) 132–135.
A global method for relaxation in and . Arch. Ration. Mech. Anal. 165 (2002) 187–242. | DOI | Zbl
, , and ,A global method for relaxation. Arch. Ration. Mech. Anal. 145 (1998) 51–98. | DOI | Zbl
, and ,T. Brox, A. Bruhn and J. Weickert, Variational motion segmentation with level sets. In Computer Vision – ECCV 2006. Edited by A.P. H. Bischof and A. Leonardis. Vol. 3951 of Lect. Notes Comput. Sci. Springer (2005) 471–483
Detection of intensity and motion edges within optical flow via multidimensional control. SIAM J. Img. Sci. 2 (2009) 1190–1210. | DOI | Zbl
, and ,Numerical solution of the double-well problem allowing for microstructure. Math. Comput. 66 (1997) 997–1026. | DOI | Zbl
and ,Snakes in movement. SIAM J. Numer. Anal. 33 (1996) 2445–2456. | DOI | Zbl
and ,A first-order primal-dual algorithm for convex problems with an applications to imaging. J. Math. Imaging and Vision 40 (2011) 120–145. | DOI | Zbl
and ,I. Cohen, Nonlinear variational method for optical flow computation. In Proc. of the 8th SCIA, June 93 (1993) 523–530.
S. Conti, A. Garroni and A. Massaccesi, Modeling of dislocations and relaxation of functionals on 1-currents with discrete multiplicity. Calc. Var. Partial Differ. Eqs. (2015) DOI: 10.1007/s00526-015-0846-x.
D. Cremers and C. Schnörr, Motion competition: Variational integration of motion segmentation and shape regularization. In Pattern Recognition − Proc. of the DAGM, edited by L. Van Gool. Vol. 2449 of Lect. Notes in Comput. Sci. (2002) 472–480. | Zbl
Statistical shape knowledge in variational motion segmentation. Image and Vision Computing 21 (2003) 77–86. | DOI
and ,Motion competition: A variational framework for piecewise parametric motion segmentation. Int. J. Comput. Vision 62 (2005) 249–265. | DOI | Zbl
and ,B. Dacorogna, Direct Methods in the Calculus of Variations. Springer-Verlag, New York (1989). | Zbl
L.C. Evans and R.F. Gariepy, Measure Theory and Fine Properties of Functions. CRC Press (1992). | Zbl
D. Fleet and Y. Weiss, Optical flow estimation. In Handbook of Mathematical Models in Computer Vision. Springer, New York (2006) 239–257.
Relaxation of quasiconvex functionals in BV(Ω,Rp) for integrands f(x,u,∇u). Arch. Ration. Mech. Anal. 123 (1993) 1–49. | DOI | Zbl
and ,A remark on Serrin’s theorem. NoDEA Nonlin. Differ. Eqs. Appl. 13 (2006) 425–433. | DOI | Zbl
, and ,Sublinear functions of measures and variational integrals. Duke Math. J. 31 (1964) 159–178. | DOI | Zbl
and ,A morphological, affine and galilean invariant scale–space for movies. IEEE Trans. Image Process. 7 (1998) 444–456. | DOI
,W. Hinterberger, O. Scherzer, C. Schnörr and J. Weickert, Analysis of optical flow models in the framework of calculus of variations. Technical Report No. 8, University of Mannheim, Germany (2001). | Zbl
Determining optical flow. Artificial Intelligence 17 (1981) 185–203. | DOI | Zbl
and ,A multiscale finite element method for elliptic problems in composite materials and porous media. J. Comput. Phys. 134 (1997) 169–189. | DOI | Zbl
and ,An optimal optical flow, SIAM J. Control Optim. 44 (2005) 728–742. | DOI | Zbl
,C. Kanglin and D.A. Lorenz, Image sequence interpolation based on optical flow, segmentation and optimal control. Image Process., IEEE Trans. 21 (2012) 1020–1030.
Image sequence analysis via partial differential equations. J. Math. Imaging Vision 11 (1999) 5–26. | DOI
, and ,Relaxation of signed integral functionals in BV. Calc. Var. Partial Differ. Equ. 37 (2010) 29–62. | DOI | Zbl
and ,A proof of convergence of the Horn–Schunck optical flow algorithm in arbitrary dimension. SIAM J. Imaging Sci. 7 (2014) 277–293. | DOI | Zbl
, , and ,B. D. Lucas and T. Kanade, An iterative image registration technique with an application to stereo vision. In Proc. of Seventh International Joint Conference on Artificial Intelligence. Vancouver, Canada (1981) 674–679.
E. Memin and P. Perez, A Multigrid Approach for Hierarchical Motion Estimation. In Proc. of ICCV (1998) 933–938.
An investigation of smoothness constraints for the estimation of displacement vector fields from image sequences. IEEE Trans. Pattern Anal. Mach. Intell. 8 (1986) 565–593. | DOI
and ,H.H. Nagel and M. Otte, Optical flow estimation: Advances and comparisons. In Proc. of the 3rd European Conference on Computer Vision. Edited by J.-O. Eklundh. Vol. 800 of Lect. Notes Comput. Sci. Springer (1994) 51–70.
Variational approach to optical flow estimation managing discontinuities. Image and Vision Comput. 11 (1993) 419–439. | DOI
,Direct incremental model-based image motion segmentation for video analysis. Signal Process. 66 (1998) 143–155. | DOI | Zbl
and ,Highly accurate optic flow computation with theoretically justified warping. Int. J. Computer Vision 67 (2006) 141–158. | DOI
, , , and ,Geodesic active contours and level sets for the detection and tracking of moving objects. IEEE Trans. Pattern Anal. Mach. Intel. 22 (2000) 266–280. | DOI
and ,TV-L1 Optical flow estimation. Image Processing On Line 3 (2013) 137–150. | DOI
, and ,Y. Rathi, N. Vaswani, A. Tannenbaum and A. Yezzi, Particle filtering for geometric active contours with application to tracking moving and deforming objects. In CVPR IEEE Computer Society (2005) 1–8.
General theorems on semicontinuity and convergence with functionals. Sibirsk. Mat. Ž. 8 (1967) 1051–1069. | Zbl
,Lower semicontinuity and Young measures in BV without Alberti’s rank-one theorem. Adv. Calc. Var. 5 (2012) 127–159. | DOI | Zbl
,Nonlinear total variation based noise removal algorithms. Physica D 60 (1992) 259–268. | DOI | Zbl
, and ,M. Tristarelli, Computation of coherent optical flow by using multiple constraints. In Proc. of International Conference on Computer Vision. IEEE Computer Society Press (1995) 263–268.
A. Wedel, T. Pock, C. Zach, H. Bischof and D. Cremers, An improved algorithm for TV-L1 optical flow. In Statistical and Geometrical Approaches to Visual Motion Analysis, edited by D. Cremers, B. Rosenhahn, A.L. Yuille and F.R. Schmidt. Vol. 5604 of Lect. Notes Comput. Sci. Springer Berlin Heidelberg (2009) 23–45.
Lucas/Kanade meets Horn/Schunck: Combining local and global optic flow methods. International. J. Computer Vision 61 (2005) 211–231. | Zbl
, and ,Variational optic flow computation with a spatio-temporal smoothness constraint. J. Math. Imaging Vision 14 (2001) 245–255. | DOI | Zbl
and ,Analysis of the heterogeneous multiscale method for elliptic homogenization problems. J. Amer. Math. Soc. 18 (2005) 121–156. | Zbl
, and ,Simultaneous higher-order optical flow estimation and decomposition. SIAM J. Sci. Comput. 29 (2007) 2283–2304. | DOI | Zbl
, and ,Cité par Sources :