17
$\begingroup$

There seems to be an obsession in machine learning as well as traditional regression modeling for feature selection. The majority of analysts do not seem to understand that feature selection is not a required part of modeling and can often stand in opposition to optimal prediction. How can we better get this general point across?

$\endgroup$
8
$\begingroup$

TL;DR: Frank and Scortchi have a point for prediction - but the question Scortchi proposes doesn't really address people's concerns about assessing a variable's relevance. I think this is a pertinent question to which we don't yet have a good answer. Apart from point 9 in gung's answer here, which says that no technique will save the analyst from having to actually think about his data.


I think there may be an unstated philosophical/epistemological/teleological point here: WHY do people actually want to model?

If they want to model for prediction, then things are easier. I fully agree with Frank and Scortchi here.

However, often people model data to "understand a predictor's relevance", to use terms that are not loaded with technical meaning. This usually takes the form of inferential statistics (and can lead to an unhealthy obsession with p-values), but things like variable importance in random forests aim in the same direction. As do formal causal models.

I would say that this is a legitimate aim of data analysis, both in an academic and an applied context. However, understanding a predictor's relevance runs into problems that we don't see if all we aim for is a good overall response prediction.

In this second situation, we could argue for "tapering effect sizes" in the sense of Burnham & Anderson: everything has an impact, it's just that effect sizes taper off to insignificance. This would argue for keeping everything in the model... except for the bias-variance tradeoff (see @Scortchi's suggestion to link to this question) and non-identifiability. So we need some kind of dimension reduction or regularisation.

Unfortunately, dimension reduction mangles predictors beyond recognition, so assessing a variable's relevance after, say, PCA, will be hard. And regularization can introduce an element of arbitrariness that can be hard to defend - if a predictor's relevance depends on your Bayesian priors, this is a weak point in your argumentation. Optimal choice of lasso or elastic net parameters can at least remove this element of arbitrariness, and then we can look at standardized parameter estimates in an elastic net, so this is something I like, although I don't know of any actual research on assessing variable relevance via elastic nets (pointers would be most welcome).

Unfortunately (again), this seems like we would need to push people towards one very specific kind of model: elastic net-based regularization. But what do we answer those people that are using some other kind of model, for whatever reason, and find that it does not work well with the full set of predictors?

Alternatively, we can argue that you only have understood a process well if you can predict it well. So, to assess a variable's importance, build prediction models with and without that variable (and any transforms of the variable you want), test the models out-of-sample, and see whether prediction accuracy goes down. (Diebold-Mariano type tests can be used for disciplines that can't get away from p-values.) This is actually my preferred position.

However, there is the problem that nominally out-of-sample tests actually aren't, because people will run this sequence iteratively in building their models. In addition, it is hard to get an approach like this accepted in disciplines that traditionally don't tick like this, like psychology. Not to speak of the fascinating discussions you can have about how to actually measure a prediction's accuracy, where the arbitrariness discussed above creeps in again.

$\endgroup$
  • $\begingroup$ Thanks for this thoughtful answer. I wasn't thinking about prediction especially, however. If "assessing a variable's relevance" means making point & interval estimates for coefficients of terms involving it, you don't want to drop it during feature selection, & you don't want the biased estimates got with regularization. (Note that when you've a couple of highly collinear predictors, LASSO's just as flighty in which it excludes as stepwise procedure.) If it means testing how well you can predict with it vs without it, then do just that, as you say. So I wouldn't be pushing people towards ... $\endgroup$ – Scortchi - Reinstate Monica Oct 5 '15 at 13:04
  • $\begingroup$ ... elastic net regularization in either case. If someone had a model that didn't work well (wide confidence intervals for coefficient estimates, badly over-fitting) & wanted to make inferences about all predictors, I'm not sure what I'd suggest except to get more data. (There might be needles-in-a-haystack type problems where feature selection could help by setting low coefficient estimates to zero - examining its consistency through resampling would be advisable.) But I think this question is more concerned with people who've got perfectly good models that they unfortunately feel they ... $\endgroup$ – Scortchi - Reinstate Monica Oct 5 '15 at 13:08
  • $\begingroup$ ... ought to muck about with even though they needn't $\endgroup$ – Scortchi - Reinstate Monica Oct 5 '15 at 13:08
1
$\begingroup$

One way to communicate feature selection is not a required part of modeling would be to present non-trivial use cases wherein feature selection would be detrimental, despite its utility in the assessment of variable importance as well as interpretation.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .