# Data Science SE

I noticed the ad on the main site today for the data science proposal, which evidently is 77% committed. Some of the people committed to the proposal are users on CV. Most of the questions listed look like they would be on-topic on CV, or even might be duplicates of existing questions. A minority of the questions are more programming oriented, but I would guess they could fit on one or another of the various SE programming sites. Of course, people can form a new site if they want, but I'm wondering what the value-add is? Is there something we're lacking, or should we be more welcoming of something? Moreover, what do people think of the proposal?

• Just looking at the first page of example questions near half are about efficient storage or retrieval of data - which I totally agree are very important for an analyst job. These are not currently on topic on SO (and definitely not on topic here). They are maybe on topic at the database site, but my impressions are the goals of the DB administrators are so different they would not be a good fit (see existing tags on the database site). – Andy W Apr 13 '14 at 13:17
• It seems like a reboot of a failed Big data, and reboots are rarely successful... – user88 Apr 13 '14 at 20:54
• Maybe they should start with defining the "data science" term. I still don't understand what it is. I have seen many people who call themselves "data scientists". It's hard to find a pattern to describe them, they are so diverse. – Aksakal Apr 14 '14 at 13:35
• @Aksakal As someone once flippantly put it "Data science is statistics done on a Mac." – Fomite Apr 23 '14 at 23:08
• Just as an update: The site now reached 100% commitment, so we'll see how it goes in the beta. – Momo Apr 24 '14 at 15:25

Someone raised this issue in Area 51, Overlap with existing sites, with only a little discussion.

I would think Cross Validated would be suitable for most Data Science topics, but apparently there is a perception that CV is for theoretical stats questions. The CV description is "a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization" which seems aligned with Data Science.

As far as the question of what can be done to avoid the overlap goes, we can

• participate in the Area 51 discussion
• start some more organized effort of "try CV first"
• add "data science" to the CV description
• be accepting of less statistical data science questions (data cleaning, storage, ...)

As a point of comparison, there are two math sites:

• Mathematics "for people studying math at any level and professionals in related fields"
• MathOverflow "for professional mathematicians"

Spoiler: a highly opinionated answer follows.

I am here with Peter Flom. I think questions on how to handle stuff in statistical packages deserve a site that is separate from CV and separate from SO. I would be the first to admit that I am behind on any of the Data Science issues involving Hadoop or Mahout -- I don't get to see the limits of my existing tools in the work that I have, and however much I would love to learn Python and such, I don't have the need for the applications that I am an expert in, and thus have no pressure to learn them.

Here's how it settles in my head (which is the sample of size 1, and there are 4000 heads on CV meta, most of which are undoubtedly better than mine). To me, programming is figuring out how compilers work, and what they allow you to do, almost regardless of the input data (and a good program will protect itself from poor data, anyway). Most statistical packages (excluding probably R) lack the syntax features to express these levels of abstraction from the data. On the contrary, in data analysis tasks, we usually have to take data for granted, and dance around it, may be with some reshaping or contracting of the data or construction of some new variables to better explicate the existing data structures (and the vast majority of the real life data sets represent a single rectangle -- medical patients and their vitals, say; sometimes two or three rectangles -- patients plus hospitals plus insurance companies). (Going back to my ignorance comment in the first paragraph -- this is the kind of data that I deal with, not the jagged, multiple sites twitter threads that many people fancy to mine.) So R or SAS or Stata questions on SO have looked out of place to me all the way along.

There has been a very recent discussion about this on Statalist, the support forum for Stata software. Due to some planned hardware and personnel retirement, this whole list is being transplanted from a text-only email version to a forum version. My wishful thinking was, though, that the mass of Stata users (several hundred, I am guessing) could move to StackExchange, for the mutual benefit. The Statalist users will get a functional platform that works well in improving answers and collecting knowledge. CV will get a group of users with high technical expertise (although some of the most active ones are already here; you can easily identify them in the appropriate tag activity).

As far as duplication of the content goes, there has been discussion of how to split mathematical statistics questions between CV and math, and what to do with education questions. Duplication is probably unavoidable; however in case of the overlap with math, at least math people are aware that theoretical statistics is present on CV. Whether the data science people are aware that A/B tests have been invented some hundred years ago is an open question to me; some are, but many probably aren't. What to do with the data science movement is an open question for professional statistical societies, but that's a wave that statisticians cannot steer even the slightest. Extrapolating this experience here, I would expect that the CV users won't be able to stop the Data Science Area 51 from whatever trajectory it is on (although comparisons with Big Data proposal that never took off the ground are very obvious), especially given a much greater mass of programmers-proper in the StackExchange system as a whole compared to the mass of statisticians-proper on CV, and a much lower reputation of whuber compared to Jon Skeet. If anybody will have the patience to sift through the Data Science questions and point to the appropriate duplicates on CV, I guess we should give them a platinum badge of a saint.

• +1, I appreciate your thoughts. There has also been an occasional push to have an R-centered SE site. I'm not sure what I think of it, but the idea of a (general) statistical software site has merit. – gung - Reinstate Monica Apr 15 '14 at 14:49
• Factual comment: number of subscribers to Statalist as a listserver exceeded 5000. – Nick Cox Apr 15 '14 at 15:02
• Another solution suggested by these (very well considered thoughts) is that by being only slightly more accepting of software-related questions here, we would be suitable for all needs: the data science people (which already have more than 50% overlap with us in their highest-voted example questions) and Statalist (and R, SPSS, SAS, etc). The question that arises is whether that would make such a substantial change in our site that it would drive people elsewhere. – whuber Apr 16 '14 at 18:43
• +1. I also appreciate your thoughts! – Tim Apr 17 '14 at 22:27
• @whuber I do think there's something to be said for being somewhat more accepting of software-related questions; there's clear demand for them but on their own they're not likely to support a new SE site any time in the near future. We could at least look at accepting more of the ones that are not obviously on topic elsewhere. – Glen_b Jun 15 '14 at 5:34

Update September 18, 2014 (data are for public Beta)

\begin{array}{| r | r | r |} \hline & \text {CV} & \text{Data Science} & \text{Data Science}\\ \hline \hline \text {Metric} & \text {end of Beta} & \text{80 days in Beta} & \text{128 days in Beta}\\ \hline \text {Q per day} & 7.6 & 2.6 & 2\\ \text{% answered} & \text {96%} & \text {90%} & \text {82%}\\ \text{avid users} & 138 & 47 & 55\\ \text {total users} & 1,763 & 1,773 & 2,311\\ \text {answer ratio} & 3 & 2.5 & 2.4\\ \text {Visits/day} & 1,159 & 230 & 179\\ \hline \end{array}

Looks like a decline. The interesting fact is that the only metric doing really good is "total users" (at $+550$ in $48$ days, this is $11$-$12$ new accounts per day). This means that there is an interest for the site "out there", but those interested are passively waiting to get knowledge by "other peoples' questions and answers"... But sometimes everybody is "other people", as indicated by the decline of the "Q per day" metric, which in turns makes the site poor and leads to lower visits by the (increasing number of) total users. Also, one could conjecture that the interested users do not feel knowledgeable or confident enough to answer questions, as indicated by the falling "% answered" and "answer ratio" metrics (which sounds a reasonable conjecture given that the field is essentially nascent).

Update August 2, 2014 (data are for public Beta)

\begin{array}{| r | r | r |} \hline & \text {CV} & \text{Data Science} \\ \hline \hline \text {Metric} & \text {end of Beta} & \text{80 days in Beta} \\ \hline \text {Q per day} & 7.6 & 2.6 \\ \text{% answered} & \text {96%} & \text {90%} \\ \text{avid users} & 138 & 47 \\ \text {total users} & 1,763 & 1,773 \\ \text {answer ratio} & 3 & 2.5 \\ \text {Visits/day} & 1,159 & 230 \\ \hline \end{array}

• Thanks for the update. I'm guessing those numbers may not be strong enough. – gung - Reinstate Monica Aug 2 '14 at 1:49
• The first and the last are particularly the issue, from what I can understand. They have to improve one of the two to remain in beta. Check the interesting case of "Reverse Engineering", area51.stackexchange.com/proposals/49551/reverse-engineering, which is in Beta for 500 days (why? I guess because it is doing really good on all fronts, even in visits, but has a really low Q per day metric -so it is neither shut down nor does it graduate). – Alecos Papadopoulos Aug 2 '14 at 1:54

Edit:

The site has now launched.

Old post:

I lurked around there and want to share my preliminary, biased impression:

Judging from the first few questions there is a lot of overlap and redundancy with SO and CV. Case in point: the five most recent questions were R vs Python, sentiment analysis, calculating AUC, extract rows in R, number of layers in neural networks. All these questions have been answered on SO or on CV. Voting behavior tends to be a bit better and currently the fraction of unanswered questions is extremely low. I am not yet comfortable with giving a qualified judgement of the quality of the answers and questions, but they are missing the "knights in shiny armor" that CV undoubtedly has. Preliminarily it appears to me to not be too high, a lot of opinion and hand-waving, less SE like and more discussion forum-ish. But perhaps the "CV-way" is too engrained for me to be fair.

Authentic "data science" questions (i.e. not suitable for CV or SO) seem to be rare and the site seems to tie in mostly with what data science has come to mean nowadays: Data analysis for engineers who have a very narrow (and imo misinformed*) definition of "statistics". In that it is telling that the highest voted question is The difference between DS and Stats and the top answer pretty much sums that (imo misguided view) up (we older people would call it "computational statistics" and what is listed as specific DS tasks are what we did in grad school since the early 90ies in our statistics curriculum).

*This is certainly also statistics' own fault, especially with respect to how it is traditionally taught.

• +1, thanks for keeping up with this! – gung - Reinstate Monica Jun 12 '14 at 13:54
• +1. All SE sites--including ours--have better voting and more active, engaged participation during the first month or three: they have something to prove and several thresholds to meet if they are to survive. With an average of just 4 questions per day (6 recently), it is evident the Data Science site will not meet those thresholds. It has attracted some useful kinds of new questions concerning how to manage very large datasets, which I would be pleased to migrate to CV when DS meets its demise. The extent to which statistics is misunderstood and mis-characterized there is distressing, though. – whuber Jun 12 '14 at 17:05
• Yes, this is also my prediction. I also think that summary statistics in this early stage can be misleading, after all its easy to have 100% answered questions, when you get only 4 qs per day. Couldn't agree more with the comment on misunderstanding statistics, it really makes me sad and angry on equal measures. – Momo Jun 12 '14 at 18:28
• Guess I was wrong. – Momo Aug 14 '17 at 20:18

I am not sure what about "data science" would not be covered either here or at StackOverflow, but one thing I can think of is programming questions about R and SAS and the like.

• The questions didn't seem to be about how to get a regression model in R, eg, though. – gung - Reinstate Monica Apr 13 '14 at 14:07
• FWIW, here's Yavar's example question: "How can I train Baum Welch Algorithm used in Hidden Markov Model in Mahout in parallel?" Parallel processing in Mahout is probably off-topic here, but there seem to be several related topics on SO...I suspect they're not entirely sure either, as was the case with the Mental Fitness beta that closed recently. – Nick Stauner Apr 13 '14 at 20:33
• I agree with you. +1 – Tim Apr 17 '14 at 22:26
• There are more than 50,000 questions tagged r on StackOverflow and almost 6000 on CrossValidated – Henry Apr 19 '14 at 18:56
• @Henry that's a testament to the volume of questions and confusion as to what warrants an r tag here. I think the site would be improved greatly to remove it altogether, but that's been discussed before and I'm in the minority. An r tag would be important on a data science SE since R is pretty lousy with large data management tasks. – AdamO Sep 19 '14 at 16:42