Utilization of Low Dimensional Structure to Improve the Performance of Nonparametric Estimation in High Dimensions

When the number of covariates is small, nonparametric regression methods serve a number of useful purposes. In this setting, nonparametric regression methods often demonstrate better predictive performance than parametric models. This is because nonparametric methods have the virtue of being able to detect nonlinear structure and complex interactions. In settings where the sample size is small or the level of noise is high, it may be the case that parametric models outperform nonparametric methods. However, even in this setting, nonparametric methods can be useful for diagnosing problems of model misspecification. Unfortunately, when the number of covariates is large, the curse of dimensionality, in its many forms, renders many of the most commonly used nonparametric regression meth- ods unstable and prone to overfitting. We have developed two methods that, in some sense, overcome the curse of dimensionality. Both methods implicitly assume the existence of lower dimensional structure. First, we have developed a variant of random forests, called fuzzy forests. Fuzzy forests reduce the bias observed in random forest variable importance measures by clustering covariates into distinct groups such that the correlation of covariates within a group is high and the cor- relation between groups is low. Fuzzy forests is expected to work well when the true regression function exhibits an additive structure. Second, we have extended a machine learning method called metric learning to right-censored survival out- comes. If the true regression function is multi-index, we have shown that a closely related metric learning estimator achieves a rate of convergence dependent on the number of indices rather than the number of covariates.