We consider the problem of estimating an unknown regression function when the design is random with values in ${\mathbb{R}}^{k}$. Our estimation procedure is based on model selection and does not rely on any prior information on the target function. We start with a collection of linear functional spaces and build, on a data selected space among this collection, the least-squares estimator. We study the performance of an estimator which is obtained by modifying this least-squares estimator on a set of small probability. For the so-defined estimator, we establish nonasymptotic risk bounds that can be related to oracle inequalities. As a consequence of these, we show that our estimator possesses adaptive properties in the minimax sense over large families of Besov balls ${\mathcal{B}}_{\alpha ,l,\infty}\left(R\right)$ with $R>0$, $l\ge 1$ and $\alpha >{\alpha}_{l}$ where ${\alpha}_{l}$ is a positive number satisfying $1/l-1/2\le {\alpha}_{l}<1/l$. We also study the particular case where the regression function is additive and then obtain an additive estimator which converges at the same rate as it does when $k=1$.

Keywords: nonparametric regression, least-squares estimators, penalized criteria, minimax rates, Besov spaces, model selection, adaptive estimation

@article{PS_2002__6__127_0, author = {Baraud, Yannick}, title = {Model selection for regression on a random design}, journal = {ESAIM: Probability and Statistics}, pages = {127--146}, publisher = {EDP-Sciences}, volume = {6}, year = {2002}, doi = {10.1051/ps:2002007}, mrnumber = {1918295}, zbl = {1059.62038}, language = {en}, url = {http://archive.numdam.org/articles/10.1051/ps:2002007/} }

Baraud, Yannick. Model selection for regression on a random design. ESAIM: Probability and Statistics, Volume 6 (2002), pp. 127-146. doi : 10.1051/ps:2002007. http://archive.numdam.org/articles/10.1051/ps:2002007/

[1] Model selection for regression on a fixed design. Probab. Theory Related Fields 117 (2000) 467-493. | MR | Zbl

,[2] Risk bounds for model selection via penalization. Probab. Theory Related Fields 113 (1999) 301-413. | MR | Zbl

, and ,[3] Minimum complexity density estimation. IEEE Trans. Inform. Theory 37 (1991) 1738. | MR | Zbl

and ,[4] An adaptive compression algorithm in Besov spaces. Constr. Approx. 16 (2000) 1-36. | MR | Zbl

and ,[5] Minimum contrast estimators on sieves: Exponential bounds and rates of convergence. Bernoulli 4 (1998) 329-375. | MR | Zbl

and ,[6] Gaussian model selection. JEMS 3 (2001) 203-268. | EuDML | MR | Zbl

and ,[7] ${C}_{p}$ criterion for Gaussian model selection, Technical Report. University Paris 6, PMA-647 (2001).

and Massart, A generalized[8] How many bins should be put in a regular histogram, Technical Report. University Paris 6, PMA-721 (2002).

and ,[9] Statistical learning theory and stochastic optimization, in École d'été de probabilités de Saint-Flour. Springer (2001). | Zbl

,[10] Wavelet and fast wavelet transform on an interval. Appl. Comp. Harmon. Anal. 1 (1993) 54-81. | MR | Zbl

, and ,[11] Ten lectures on wavelets. SIAM: Philadelphia (1992). | MR | Zbl

,[12] Constructive approximation. Springer-Verlag, Berlin (1993). | MR | Zbl

and ,[13] Ideal spatial adaptation via wavelet shrinkage. Biometrika 81 (1994) 425-455. | MR | Zbl

and ,[14] Minimax estimation via wavelet shrinkage. Ann. Statist. 26 (1998) 879-921. | MR | Zbl

and ,[15] Inequalities for uniform deviations of averages from expectations with applications to nonparametric regression. J. Statist. Plann. Inference 89 (2000) 1-23. | MR | Zbl

,[16] Nonparametric regression function estimation using interaction least square splines and complexity regularization. Metrika 47 (1998) 147-163. | MR | Zbl

,[17] Minimax theory of image reconstruction. Springer-Verlag, New York NY, Lecture Notes in Statis. (1993). | MR | Zbl

and ,[18] Additive regression and other nonparametric models. Ann. Statist. 13 (1985) 689-705. | MR | Zbl

,[19] Model selection in non-parametric regression, Preprint. Yale University (2000). | Zbl

,[20] Model selection for nonparametric regression. Statist. Sinica 9 (1999) 475-499. | MR | Zbl

,[21] Combining different procedures for adaptive regression. J. Multivariate Anal. 74 (2000) 135-161. | MR | Zbl

,[22] Information-Theoretic determination of minimax rates of convergence. Ann. Statist. 27 (1999) 1564-1599. | MR | Zbl

and ,*Cited by Sources: *