Glen_b, I like this idea a lot. Wedged between the ever-rising tide of questions asking about how to use R, there's a bounty of great statistical content on CV.
I don't know how much of that content is well-suited for introductory texts, though. There are a plethora of introductory probability and statistics textbooks (with a high variance in quality). But to my knowledge, there are not a great number of textbooks directly addressing how to do good predictive modeling with machine learning methods. The textbooks which do exist tend to fall along the lines of "a first course in predictive modeling." These fill a niche and are certainly necessary, but after completion, there are not great subsequent resources. It's usually in this space where the difficulty of the material suddenly jumps like it found a spider under the toilet seat. People get stuck in the local minimum of looking at models with some particular flavor (trees, Bayes, kernels, neural nets), but being a good practitioner means knowing how to use appropriate tools for the problem at hand, and that point seems to get swept under the rug of the oft-mentioned-but-rarely-examined "domain knowledge."
I suppose I'm imagining a book with the subtitle "What is good decision-support, and how do we accomplish it?"
I think the best form a book could take on would be a tour of the "black arts" of predictive classification, i.e. the topics that are incredibly helpful to practitioners but are rarely discussed in a single location.
- Presently I'm putting together a presentation on the use and abuse of
different performance metrics. I was surprised to find that the
material I'm drawing from is scattered across a diversity of snippets
from different articles, but not any single resource. And my interactions with Prof. Harrell underscore just how subtle the question is.
- My highly-upvoted post on Euclidean distance high-dimensional spaces is actually borrowed from an article on subtle-but-important points in the machine learning tradition. IIRC, that article was published 15 years ago, but its points remain just as obscure today as they were then... at least, in my anecdotal experience.
- If I had an up-vote for every time I spoke to an engineer who wanted to use random forest to do feature selection for a linear model... well, actually, I just might.
- What are the use cases for uncalibrated models? When is using ROC ACU obviously and provably the wrong choice?
- "Why does everyone at the office get snippy when I mention gradient descent?"
- Machine learning tools tend to work best with atomic units. What are the options for working with data that is non-atomic? Networked, geospatial, time-series, clustered, hierarchical?
- This is probably neither here nor there, but nothing I ever learned in a linear algebra classroom has ever helped me actually do linear algebra in practice. Knowing how to set up your pipeline of functions and analysis so that you have convenient factorizations stored for future use in computing inverses is huge!
- Hyperparameter optimization: most people use grid searches, but there are more intelligent ways to go about this, such as using surrogate models.
But these are just my observations from my very narrow view into modern applied statistics. My perspective, like anyone else's, is constrained by context, so perhaps I'm just making a big deal out of things that are important to my role.
I guess what I'm getting at is there's considerable value to the scattered uncommon insights that the specialists at CV have touched on.
Also, I might just be cranky.