I was reviewing first post questions and came across one seeming totally off-topic tagged data-preprocessing. Curious, I found a lot of ML questions invoking this tag. My gut says "data-preprocessing" is fundamentally off-topic as it's not statistical. Statistics relies on data being already "cleaned": free of typos, formatted correctly, etc. However, distributional properties of data aren't and shouldn't be modified to conform to models, rather it should be vice versa. (Cue the very famous Tukey Quote): "The exact answer to the approximate question is far less valuable than an approximate answer to the exact question."
$\begingroup$
$\endgroup$
12
1 Answer
$\begingroup$
$\endgroup$
2
It is on-topic. Because every comment so far could be an interesting answer to this question I am pasting them here:
The tag or term data-preprocessing isn't universal across statistical science (minimally, it's not a term I meet around the place or in literature), but whether and how to deal with outliers, work on transformed scales, etc. seem utterly and completely on-topic. Let the tag be used by anyone to whom it's standard jargon.
gung:
It seems to me that "data-preprocessing" sounds a lot like data-cleaning. I have argued that this is "within the scope of statistics writ large, and although I recognize good people can disagree, I think such questions can be on-topic". I would say the same here. It may be that the tags should be made synonyms, but I'd have to investigate their usages first to be sure.
Only textbooks, and maybe the rare academician, can rely on data already being cleaned and perfectly processed. Many people report that most of their time working as statisticians consists of processing and cleaning data. I wouldn't like to exclude as off-topic the activities that most occupy people who practice statistics. Having said that, I admit to closing many posts that ask only about the details of executing a particular data processing step with specific software, such as how to combine columns in an R dataframe, etc. I have felt uncomfortable even about closing those.
Since data collection is on-topic it would seem odd for the step in between collection and analysis to be off topic.
You wrote "Statistics relies on data being already "cleaned": free of typos, formatted correctly, etc". In my experience, this is incorrect. Indeed, cleaning data is a huge part of my job and is often highly statistical. I remember reading some famous statistician (it might have been Box) saying he had never had a data set entered correctly.
And my two cents are: I have been working with databases specially treating and organizing data for map visualization, and it is always the most difficult step to eliminate inconsistencies from large datasets. One technique I use to visualize 'wrong' patterns is quickly plotting data in different types of charts, then deciding what to do to tackle the problem (sometimes I can see an inconsistency, but find difficult to know what to do with it).
Of course, "pre-processing" questions here would be judged the same as other types of questions, in a case by case. They need to be clear, not broad, not primarily opinion-based, having something to due with statistics (and not only how to manipulate data using software x, as mentioned by whube), etc.
Perhaps an idea would be to collect some questions in this subject that are on topic, and some that are not, then post them as an answer here. Tag this question as faq and include in our on topic help page 'data cleaning' (or 'data pre-processing').
answered Jan 5, 2018 at 12:47
-
$\begingroup$ I like the idea of collecting questions and classifying them as on-topic/off-topic. However (and partly my rationale for even asking the question here), I notice a large proportion of questions have the deadly mixture of extremely high-level jargon and lack of a clear problem statement/question. I also find a great deal of terminology largely conflated with programming, so that if it is even a pure programming question, one cannot tell. Here, a user grasps LOCF but is basically asking for software to do this (I think): stats.stackexchange.com/questions/262324/data-synchronisation $\endgroup$– AdamOJan 8, 2018 at 14:53
-
$\begingroup$ Agree it is not easy to make this menu of on-topic and off-topic questions. I'd start excluding questions which should be closed for another reason than ontopicness. For example: "However (...), I notice a large proportion of questions have the deadly mixture of extremely high-level jargon and lack of a clear problem statement/question." --> those would be unclear questions. And the question you linked last is also unclear; note the OP did not answer last whuber's comment. But, if edited could be an on-topic question related to interpolation, time-series, data-cleaning, etc. (@AdamO). $\endgroup$ Jan 8, 2018 at 15:37
data-preprocessingisn't universal across statistical science (minimally, it's not a term I meet around the place or in literature), but whether and how to deal with outliers, work on transformed scales, etc. seem utterly and completely on-topic. Let the tag be used by anyone to whom it's standard jargon. $\endgroup$Rdataframe, etc. I have felt uncomfortable even about closing those. $\endgroup$data-preprocessingis just computer science/machine learning vernacular for cleaning and transforming data. If one is doing knn, they should standardize their data before analysis. This is preprocessing, and it is wholly relevant to statistical issues. Just as taking the log of a variable before entering it into a regression equation is preprocessing for traditional statistical techniques. SO is for, "What is wrong with my preprocessing code?" while CV is for, "How should I preprocess my data for valid inferences and predictions?" $\endgroup$