12
$\begingroup$

I was reviewing first post questions and came across one seeming totally off-topic tagged . Curious, I found a lot of ML questions invoking this tag. My gut says "data-preprocessing" is fundamentally off-topic as it's not statistical. Statistics relies on data being already "cleaned": free of typos, formatted correctly, etc. However, distributional properties of data aren't and shouldn't be modified to conform to models, rather it should be vice versa. (Cue the very famous Tukey Quote): "The exact answer to the approximate question is far less valuable than an approximate answer to the exact question."

$\endgroup$
12
  • 10
    $\begingroup$ The tag or term data-preprocessing isn't universal across statistical science (minimally, it's not a term I meet around the place or in literature), but whether and how to deal with outliers, work on transformed scales, etc. seem utterly and completely on-topic. Let the tag be used by anyone to whom it's standard jargon. $\endgroup$ – Nick Cox Jan 4 '18 at 19:35
  • 13
    $\begingroup$ It seems to me that "data-preprocessing" sounds a lot like data-cleaning (which we have a tag for). I have argued that this is "within the scope of statistics writ large, & although I recognize good people can disagree, I think such questions can be on-topic". I would say the same here. It may be that the tags should be made synonyms, but I'd have to investigate their usages first to be sure. $\endgroup$ – gung - Reinstate Monica Jan 4 '18 at 20:17
  • 20
    $\begingroup$ Only textbooks, and maybe the rare academician, can rely on data already being cleaned and perfectly processed. Many people report that that most of their time working as statisticians consists of processing and cleaning data. I wouldn't like to exclude as off-topic the activities that most occupy people who practice statistics. Having said that, I admit to closing many posts that ask only about the details of executing a particular data processing step with specific software, such as how to combine columns in an R dataframe, etc. I have felt uncomfortable even about closing those. $\endgroup$ – whuber♦ Jan 4 '18 at 20:33
  • 16
    $\begingroup$ Since data collection is on-topic it would seem odd for the step in between collection and analysis to be off topic. $\endgroup$ – Glen_b Jan 5 '18 at 2:15
  • 7
    $\begingroup$ You wrote "Statistics relies on data being already "cleaned": free of typos, formatted correctly, etc". In my experience, this is incorrect. Indeed, cleaning data is a huge part of my job and is often highly statistical. I remember reading some famous statistician (it might have been Box) saying he had never had a data set entered correctly. $\endgroup$ – Peter Flom Jan 5 '18 at 11:53
  • 7
    $\begingroup$ data-preprocessing is just computer science/machine learning vernacular for cleaning and transforming data. If one is doing knn, they should standardize their data before analysis. This is preprocessing, and it is wholly relevant to statistical issues. Just as taking the log of a variable before entering it into a regression equation is preprocessing for traditional statistical techniques. SO is for, "What is wrong with my preprocessing code?" while CV is for, "How should I preprocess my data for valid inferences and predictions?" $\endgroup$ – Mark White Jan 5 '18 at 15:45
  • 2
    $\begingroup$ @PeterFlom I partly agree. I'll cite one anecdote I use when people suggest that an abnormal range test alone justifies deleting (an) observation(s): "If I randomly sample 100 people and one happens to be Bill Gates, then as far as the data tell me, Bill Gates is representative of 1/100th of the population." Statistically, the Bill Gates case is indistinguishable from data entry errors. My concern is that "data cleaning" and its analogues seem to support such a practice ("make the data normal", "delete the outliers", etc.). $\endgroup$ – AdamO Jan 5 '18 at 15:51
  • 4
    $\begingroup$ I agree, without having looked closely at how they're actually used, that data-preprocessing and data-cleaning should probably be made synonyms. "Preprocessing" more implies to me rescaling / changing featurizations / etc and "cleaning" more about dropping outliers / etc, but the line is very blurry. $\endgroup$ – Danica Jan 5 '18 at 20:04
  • 1
    $\begingroup$ @AdamO that is a primary reason why the topic should be on this site: To answer questions getting at what are appropriate methods for cleaning and processing. $\endgroup$ – Mark White Jan 5 '18 at 20:04
  • 5
    $\begingroup$ I definitely think questions that relate to practices like removing outliers, transforming data and so forth are on topic. I've answered hundreds and I only answer a tiny fraction of the ones that have been posted. $\endgroup$ – Glen_b Jan 8 '18 at 2:43
  • 2
    $\begingroup$ Is the box-cox transform a pre-processing? If so then this process, named for two top-tier statisticians, Geoge Box and David Cox, should be considered part of what Box and Cox consider to be statistics. Missing data imputation is pre-processing before the model fit process, so again, pre-processing is part of stats. $\endgroup$ – EngrStudent Jan 8 '18 at 2:54
  • $\begingroup$ Statistical theory and concepts are useful not just in the analysis stage but earlier stages as well. What happens at the data collection, cleaning, and processing stages have implications for subsequent statistical analysis, hence a statistician has something to say about them. Perhaps similarly, a cook cares about what happens to ingredients before they enter the pot. On topic at cooking.stackexchange.com isn't just cooking narrowly defined but also "food handling and storage" and "ingredient selection and use." $\endgroup$ – Matthew Gunn Jan 10 '18 at 15:21
23
$\begingroup$

It is on-topic. Because every comment so far could be an interesting answer to this question I am pasting them here:

Nick Cox:

The tag or term data-preprocessing isn't universal across statistical science (minimally, it's not a term I meet around the place or in literature), but whether and how to deal with outliers, work on transformed scales, etc. seem utterly and completely on-topic. Let the tag be used by anyone to whom it's standard jargon.

gung:

It seems to me that "data-preprocessing" sounds a lot like . I have argued that this is "within the scope of statistics writ large, and although I recognize good people can disagree, I think such questions can be on-topic". I would say the same here. It may be that the tags should be made synonyms, but I'd have to investigate their usages first to be sure.

whuber:

Only textbooks, and maybe the rare academician, can rely on data already being cleaned and perfectly processed. Many people report that most of their time working as statisticians consists of processing and cleaning data. I wouldn't like to exclude as off-topic the activities that most occupy people who practice statistics. Having said that, I admit to closing many posts that ask only about the details of executing a particular data processing step with specific software, such as how to combine columns in an R dataframe, etc. I have felt uncomfortable even about closing those.

Glen_b:

Since data collection is on-topic it would seem odd for the step in between collection and analysis to be off topic.

Peter Flom:

You wrote "Statistics relies on data being already "cleaned": free of typos, formatted correctly, etc". In my experience, this is incorrect. Indeed, cleaning data is a huge part of my job and is often highly statistical. I remember reading some famous statistician (it might have been Box) saying he had never had a data set entered correctly.

And my two cents are: I have been working with databases specially treating and organizing data for map visualization, and it is always the most difficult step to eliminate inconsistencies from large datasets. One technique I use to visualize 'wrong' patterns is quickly plotting data in different types of charts, then deciding what to do to tackle the problem (sometimes I can see an inconsistency, but find difficult to know what to do with it).

Of course, "pre-processing" questions here would be judged the same as other types of questions, in a case by case. They need to be clear, not broad, not primarily opinion-based, having something to due with statistics (and not only how to manipulate data using software x, as mentioned by whube), etc.

Perhaps an idea would be to collect some questions in this subject that are on topic, and some that are not, then post them as an answer here. Tag this question as and include in our on topic help page 'data cleaning' (or 'data pre-processing').

$\endgroup$
2
  • $\begingroup$ I like the idea of collecting questions and classifying them as on-topic/off-topic. However (and partly my rationale for even asking the question here), I notice a large proportion of questions have the deadly mixture of extremely high-level jargon and lack of a clear problem statement/question. I also find a great deal of terminology largely conflated with programming, so that if it is even a pure programming question, one cannot tell. Here, a user grasps LOCF but is basically asking for software to do this (I think): stats.stackexchange.com/questions/262324/data-synchronisation $\endgroup$ – AdamO Jan 8 '18 at 14:53
  • $\begingroup$ Agree it is not easy to make this menu of on-topic and off-topic questions. I'd start excluding questions which should be closed for another reason than ontopicness. For example: "However (...), I notice a large proportion of questions have the deadly mixture of extremely high-level jargon and lack of a clear problem statement/question." --> those would be unclear questions. And the question you linked last is also unclear; note the OP did not answer last whuber's comment. But, if edited could be an on-topic question related to interpolation, time-series, data-cleaning, etc. (@AdamO). $\endgroup$ – Andre Silva Jan 8 '18 at 15:37

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .