I would like to reopen the discussion regarding CV and the Data Science beta. This question is related to this previous one: Data Science SE but now with a better view of where Data Science seems to be going. I was inspired to make this post because of https://stats.stackexchange.com/questions/126403/crossvalidated-vs-datascience-what-is-different.

The difference between CV and data science appears to be that CV focuses on data analysis theory (statistics, machine learning and math to a lesser extent) while data science focuses on (big) data analysis in practice (software frameworks, databases, languages). At least on paper.

CV's mission statement:

Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization.

Data Science's mission statement:

Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field.

These mission statements are pretty vague but already there one can immediately see tremendous overlap. I think historically the need for Data Science arose because CV rejected 'implementation-related' questions. That may have been a mistake.


I am not convinced that these should be separate, especially considering the evolution of data science. Maybe CV is focusing on independence of statistics too much.

Data Science is getting a lot of theoretical questions which should probably have ended up here (most are in fact already answered here), some examples:

The list goes on. If we consider the list given at Data Science meta to be used, such questions would fit on both venues. The idea that (a non-trivial amount of) questions may well end up on either site is in direct contradiction to the overall mission of StackExchange sites (e.g. to provide a single place to answer certain questions that cannot be found in other places).

On CV we are (fairly) consistently closing questions that belong on data science while they appear to not be doing the same. Essentially this boils down to 'when in doubt, ask at Data Science'. This is just an observation, don't consider this to be a complaint or accusation. It seems to me that CV needs better PR at least.


My question: is having two small, heavily intertwined sites better than one large one related to 'data analysis' in all its forms? StackOverflow has shown that a single go-to point for programming stuff has worked tremendously well, so maybe the equivalent for data analysis has its merits? From a new user's perspective, it would make a lot more sense.

  • 5
    (1) Might also be useful to pick some examples of the kind of questions one large site would be accepting that CV currently doesn't, (& consider which of those aren't suitable for SO). (2) Are we closing many Data Science questions? The last I can see migrated to DS was asked on 19 Sep, two and a half months ago. – Scortchi Dec 3 '14 at 11:42
  • "It seems to me that CV needs better PR at least.". What is PR? – Andre Silva Dec 3 '14 at 15:36
  • 2
    @AndreSilva public relations, e.g. a 'better' image. – Marc Claesen Dec 3 '14 at 15:59
  • 13
    This topic is excellent, well-posed, and well worth discussion. It looks pretty clear, though, that the data science site (in its current "unhealthy" condition) is doomed, so I wouldn't worry overmuch about the overlap. (The rate at which new questions appear on the DS site currently is just one-quarter of the rate at which we are closing questions here!) – whuber Dec 3 '14 at 16:23
  • 3
    @whuber: If it's doomed should we be thinking whether we'd want the Data Science questions that are currently off-topic on both CV & on Stack Overflow? – Scortchi Dec 5 '14 at 11:13
  • 1
  • 2
    @StephanKolassa: Taking the highest-voted answer to that question at face value, the problems tackled by data science - data collection, manipulation, scale, mining, & communication - are not obviously off-topic on CV. – Scortchi Dec 5 '14 at 14:31
  • 2
    I think the community agree we should expand the scope of Cross Validated to incorporate (some) topics of Data Science. Next step maybe would be a question asking for opinions about the scope, in which there would be separate answers containing the Data Science topics so people could vote on (e.g. treatment of large data, computational issues, etc). Of course we would need to wait to see that site's future (if it will work or not). cc/ @conjectures. – Andre Silva Dec 8 '14 at 13:31
  • 1
    @Andre None of those things have been considered off-topic here, so it's not evident that any expansion of our scope is needed. The DS help is useless because it doesn't even state what is on-topic there! A beta post about the potential overlap reflects a dismaying amount of misunderstanding of what CV is about, but I don't think that's our fault: it looks like willful ignorance on the part of the respondent there (who mischaracterizes CV although he has never even joined it!) – whuber Dec 14 '14 at 22:48
  • 2
    @Andre (and everyone else): some thoughtful replies have appeared in a related DS meta thread. They provide good reasons for DS to exist (eg, "It is about how the engineering and stats should work together when you need to build a solution to a problem"), as well as pointing to reasons why it is failing (too many low-quality questions and answers). – whuber Dec 14 '14 at 23:10
  • Another example from today: datascience.stackexchange.com/q/3760/1156 – shadowtalker Dec 26 '14 at 16:25
  • @whuber the answer you link to summarizes CV as "regression and R." That is a brutal assessment of our site, even if it has truth in it. – shadowtalker Jan 12 '15 at 6:04
  • This question should be closed as answers here will be biased. A neutral place to discuss this is meta.stackexchange.com/q/266067/158075 – Martin Thoma Jan 27 '16 at 8:53
  • 3
    @moose: Though most answers are "No", the different reasons given are interesting. I suspect questions on the Stack Exchange Meta risk being overlooked by most participants on both CV & DS Metas, & can't see any harm in CV members discussing a matter of interest to them on their own Meta. Still, linking to similar discussions on DS Meta & elsewhere is a good idea, & of course we'd welcome hearing contrary views from people who've been more active on DS than CV (the rep. requirement to post here is only 5). – Scortchi Jan 27 '16 at 12:01
  • 1
    @moose You're welcome. And thanks for bringing up the matter - I've added an answer to make the links to where this question's been discussed elsewhere more prominent. If you know of others please add them. – Scortchi Jan 27 '16 at 12:23

10 Answers 10

My question: is having two small, heavily intertwined sites better than one large one related to 'data analysis' in all its forms?

For what it's worth, my answer is 'no'. If anything, I think Data Science should be merged into Cross Validated.

I can respect that some people would want to keep the engineering and theory separate (I'll let them make that case); however:

  1. It is possible to have a single site which deals with different aspects of the same area. Tags have a role to play.
  2. It's kind of annoying having to check two sites, have two sets of rep etc.
  3. There will be wasteful duplication and dilution of conversations.
  4. Computational tractability is often driving the choice of theories to pursue. IMO there is a complementarity between theoretical questions and engineering issues.
  5. It's going to get dull seeing 'is this one for data science?' in comments fields.
  6. One could argue that it is inconsistent that on CV we will sometimes provide R/matlab code, but seem to shy away from larger systems engineering questions or less familiar software.

Many of the good questions on DS.SE would be well-suited for CrossValidated. The rest are either a) suited just fine for either StackOverflow or SciComp, or b) just plain bad questions anyway.

Fact (read: strong opinion) is, the data science site shouldn't exist as a separate entity. The way I see it, its existence can affect CrossValidated somewhere along a continuum between two outcomes:

  1. It could have a positive effect on CrossValidated by attracting more users to the StackExchange network who would have otherwise never signed up at all. These users might then sign up for CV and start contributing here. It would be especially positive if these users had somewhat different backgrounds from the existing CV user base and could offer a different and expanded range of expertise.

  2. It could have a negative effect by dilution of CV's user base and concentration of CV's purview. This ties back to what I said above. It is entirely possible that every user who signs up for DS.SE is a user who would otherwise have signed up for CV, but now didn't. That potentially means fewer good questions and, of particular concern lately, fewer good answers from people that might have some real expertise to contribute. At the same time, the subject matter will also be split between CV and DS, making both sites less relevant, not to mention engendering confusion and duplication of question-answering effort (of which I suspect there is a limited maximum flow over time).

The same two points could be made about StackOverflow, except that StackOverflow is a juggernaut and is far too big to be affected by anything other than its own half-decade of inertia. Indeed, the fact that it can coexist with Ubuntu, Linux, Programmers, ServerFault, Databases, Code Review, Theoretical CS, and a host of other specialized software-specific sites is a testament to this. CrossValidated is not StackOverflow.

Based on what I see over at DS, it has in fact attracted some new expertise, and I doubt that expertise has really been "poached" from CrossValidated's would-be user base. But the crossover has not happened, and I think this is to the detriment of both groups of users. The data scientists in academia and industry have already discovered that there has been an artificial gap between the computer scientists and the statisticians (and the economists and political scientists and sociologists and linguists and neuroscientists and business analysts and marketing researchers......). Over the last year they've rushed to close it, because mixing of expertise and exchange of ideas between two mature fields is possibly the best recipe for rapid advancement. The same benefits apply, I think, to a website that exists purely to facilitate the sharing of expertise.

As far as CV's rejection of "implementation" questions goes, I think SciComp will take care of that problem (and here's to its success) in the same way that Open Data should take care of our rejection of "dataset request" questions. DS questions are either statistics questions, programming questions, or numerical/high-performance computing questions. Those that fall under more than one category can be split up and asked piecemeal on different sites. That is how the rest of StackExchange works, it seems to work well, and I'm a big fan of the model. There is no reason why it can't be applied to handle a new breed of questions.

  • 1
    "... I think SciComp will take care of that ... " The problem is that there are interactions between the stats models used (Stats.SE), the coding (SO.SE), the implementation (SciComp.SE) and to be asking aspects of what is actually one question in multiple places is not going to get the best answer. I would expect that DS.SE would be able to suggest where on the various layers of the solution you should be working on the problem and making a change. Im still undecided about one/two sites question though. – Marcus D Mar 22 '16 at 10:20

I agree with @ssdecontrol. I am generally skeptical of "data science" as the big new thing. I do see that there is a place for the discussion / development of some new issues that have arisen in the internet age (pertaining to how to implement analyses when the amount of data is so large that it cannot be fit on a computer), but much of the buzz about DS seems to be marketing hype.

As far as having a separate SE site for DS, I suppose people can do what they like within SE's rules, but I don't really see a need for it. There is the fact that it does not seem to be getting off the ground. In addition, I gather that their best threads duplicate material that already exists here or on SO, that doesn't seem to bode well either. I hope that users new to SE on DS will come over here to contribute.

On the other hand, I don't see the need for CV to expand so that it overlaps more with other existing SE sites. Practical questions about coding and implementation do very well on SO and theoretical questions about coding and implementation can be well handled on Theoretical Computer Science. Although I always use the word "statistics" as a catch-all, questions on the machine learning and data mining end of the spectrum are part of our mandate and have always been welcome here. I think what we need isn't an expanded mandate, but just a larger number of such questions and users answering them to reach a better critical mass. But I don't think more 'how do I make a neural network in MATLAB?' threads will benefit CV in that way and swapping Hadoop for MATLAB won't make any difference.


Let me address some specific points:

  • I don't know why people felt motivated to start a new SE rather than contribute here and at other existing SE sites.
  • It is true that our purview is more clearly defined and theirs is more vague, and that we tend to close threads outside our purview but they do so less. However, CV seems to be growing and DS seems to be dying (I don't mean for that to come off as snotty). It isn't clear that our strategy is failing.
  • Better PR sounds great; at a minimum, it wouldn't hurt anything.
  • I'm a fan of the SE policy of having different topics covered by different sites.
  • 4
    I believe we should be more opened to practical questions on machine learning and data mining. If 'how do I do a neural network in matlab' requires knowledge that is not only programming, than it is ok. – Andre Silva Dec 16 '14 at 20:09
  • 1
    @AndreSilva in my experience, we are open to those questions, but as amoeba mentioned in the comments to Gala's answer we might lack expertise in that area. – shadowtalker Dec 20 '14 at 6:31
  • 3
    "I don't know why people felt motivated to start a new SE rather than contribute here and at other existing SE sites." -- what kind of opportunities are there for us to "recruit" DS users to CV? Is there some kind of framework for this, especially if DS does in fact fail? – shadowtalker Dec 20 '14 at 6:32
  • 1
    @ssdecontrol, if someone has an account over there, or wants to start one, they could raise the issue (eg, on meta.DS) of what to do if the site gets closed. They could suggest that people contribute to existing sites like CV, SO, & TCS. You'd have to be very diplomatic, though. – gung Dec 20 '14 at 15:04
  • 1
    @gung, as a (new) user of both DS and Stats, my reading suggests the reason for DS creation was the closing as off topic of a number of questions which had elements of both stats AND server architecture. So either Stats SE broaden its remit to include valid questions within DS or accept that DS.SE should exist ... its a hard one ... and emotive too! – Marcus D Mar 22 '16 at 10:13

I also think having another closely related site creates more problems than it solves and that the data science site is an ill-conceived project. Others have already outlined the main reasons for that.

But this is at least the third attempt at creating another site (two “machine learning” attempts and now “data science”) in spite of warnings that the distinction was not clear and that much of the material would be welcome here anyway.

I suspect it also has a lot to do with “identity issues” and it's not clear that there is a lot we can do about it. I fully expect that once the new “Data science” fails and closes, someone will show up on Area51 and create yet another proposal along the same lines.

  • 5
    Apart from identity issues, there is a lot less expertise in machine learning here on CV than on classical statistics. Questions e.g. on t-tests and binomial confidence intervals get amazingly thorough answers here, but questions on deep belief or convolutional networks will usually get superficial answers at best. I have no idea if the situation is any better on Data Science (probably not), but I think I can understand why somebody from machine learning would rather try to open another site than join this one. If only to avoid dealing with hundreds of layman questions on hypothesis testing. – amoeba Dec 19 '14 at 10:23
  • 4
    @amoeba I don't disagree with your observations but I am not sure I see how they fit together. The same could probably be said for specific sub-fields of statistics and of course for less popular programming languages on SO, yet I don't think splitting is generally seen as a good solution in those cases. And of course there is not much expertise of any kind on a fledging site. So there is no reason why creating a new one would seem so attractive as a solution to these problems. – Gala Dec 20 '14 at 0:14

I'm relatively new to CV, and have very little history with SE, but as a practitioner with feet in both statistics and data science (you can put those in quotes if you want, I usually do since among practitioners both can be so broad you could have "statisticians" talking better about SQL than they do about MLE and "data scientists" rabid about Hadoop and clueless about a t-test). I talk with a lot of industry folks, I work in federal government and contract for private firms. I'm not an expert at anything, so this is just my opinion.

I think our role should be to try and moderate the divide that is happening between "traditional" statistics and "fast-paced" data science. It may be that in other areas, statisticians have angst that data scientists get their glory and give none back, and data scientists see statisticians as slow and inflexible, but we should not be perpetuating that myth. To the extent that we can be welcoming and cross-over, and I think we would be doing the entire field (to include statistics and data science) a huge favor that may not be recognized for a while. I recognize there are aspects of data science (managing Big Data) that do not belong at CV - but every analytical tool, even visualization stuff that is very basic analysis, should be addressed at CV. And we should be careful to not be overly condescending when a poster comes at us with limited theoretical knowledge but has a good handle on working with data - that is the type of person who could run from a statistician.

Just my newbie opinion.

I had not seen that the new Data Science site had been created. This is a huge mistake and should never have happened. This will cause major confusion among a huge number of users and will make me question my participation. Can this error be corrected before more damage is done? How do we reverse course?

Data Science is at least 1/2 Statistics. It would make sense to create a site such as Computational Science or Computer Data Science but not Data Science.

  • 7
    "Can this error be corrected before more damage is done? How do we reverse course?" We can't. This is within SE rules. However, the site seems to not get off the ground, so it might fail soon. One way or other, it would be very sad if you'd stop participating. I think we need you and the Data Science people to participate here. – Momo Jan 2 '15 at 23:26
  • 1
    Thanks for your comment. I will continue to participate in stats.stackexchange but for "Data Science" it may be best to post comments referring statistics-related questions to this site. – Frank Harrell Jan 3 '15 at 15:33
  • 3
    This is a bit hyperbole IMO. By looking at the front page, we have gotten around 40 questions in the prior 24 hours. Data Science is averaging slightly over 2 per day. (I agree what they discuss there is in large parts on topic here - I'm just disagreeing with the assertion that it will have a noticeable impact on the site here.) I slightly question the presumption that the neighboring sites are a zero sum game though in terms of Q's and A's. – Andy W Jan 3 '15 at 15:35
  • @AndyW I tried (ineloquently) to discuss those ideas in my answer – shadowtalker Jan 12 '15 at 5:58

I think my answer may have the advantage of seeing how things have played out in comparison to others who have answered earlier, but...

I think the data science site is very necessary. It's true that there's plenty of overlap between statistics questions and data science questions. There's also plenty of overlap between statistics questions and mathematics, but I don't think anyone believes this means that CV should be merged into Mathematics anytime soon.

And as such, the questions data scientists have are not necessarily the same as traditional statisticians. I think most of us would agree that "how do you run neural networks on a cluster efficiently?" doesn't exactly belong here, but it doesn't exactly belong on SO either. It very naturally belongs on DS.

Lots of statisticians feel like data scientist is just a sexier name for statistician or data analyst. But I would say the distribution of challenges faced on a daily basis by people with the title data scientist is fairly different than the distribution of challenges faced by people with the title statistician. Data science is just as much, if not more so, about handling data as it is about analyzing. Given that they do have a unique cross section of problems, it would make sense that they have their own discussion groups.

Plus, I don't want to have wade through any more questions about whether we could use gradient descent in place of maximum likelihood estimation (50% jk).

  • 2
    I don't disagree that there are questions that could be on topic at DS, but off topic here & on SO. The question is whether there are enough of them to sustain the site. My impression is that most of their Qs are better asked elsewhere (& in fact are often duplicates); despite that, however, DS seems to be not getting enough action to stay alive. – gung Sep 26 '15 at 17:36
  • 2
    @gung I had this doubt when I was a newbie at both the sites, so I posted a question at (meta.stackexchange.com/questions/266067/…); and after spending time in both, I have self-answered(meta.stackexchange.com/questions/266067/…) it from my experiences and answers from here. And coming to the activity part, I do agree with you that it is moving towards a slow death. – Dawny33 Sep 28 '15 at 13:02
  • 3
    @gung re:duplicates -- It is the proud tradition of data science to expend much time and effort recreating results from other fields. – Sycorax Feb 3 '16 at 19:28

This question & related ones have also been discussed on the Data Science & Stack Exchange Meta sites, & on Area 51:

  • 3
    I'm adding this an an answer because links away from CV don't show in the "Linked" list at the top right, & I think these deserve more prominence than they'd have buried in comments. – Scortchi Jan 27 '16 at 12:14

Data science and statistics is multi- / inter- / cross-disciplinary. The image below from https://www.r-bloggers.com/how-to-replace-a-pie-chart/ shows this well

The amount of time spent on various tasks by surveyed managers in data-science positions

bar-chart


Type of questions on datascience.stackexchange

Questions on data science are like (picking the latest 9):

I will agree that this is debatable, however the overall picture and feel of the data science website is much different to me (much closer to computer science, informatics, computational science).

It is not so easy to put exactly into words how they are different from the questions at cross validated, but when you look at it then I would say it is not difficult to see that they are different from the questions at cross validated.


Performance of datascience.stackexchange

The datascience.stackexchange seems to do well (At least the quantity goes well like the number of questions and answers is growing well and seems to be adding a larger user base to the entire network. I would very much agree that quality is a lot more debatable).

Adding to this that the questions and answers over there look to me much different than at Cross Validated (not necessarily different based on topic or discipline, but more like a different orientation and a style that is less strongly based on rigorous, purely mathematical, foundations. It is along the lines of the difference between technology vs science even though it is called data-"science").

Based on that the site has some good properties to stay.


Analogous case 'bioinformatics'

Apparently there is also a beta website for bioinformatics. This is another stackexchange site where the differences might be more clear and it shows a bit of an analogy to my point here about the data science site. Many similar arguments exist that it should be contained or split up to one or more of SO, CV, DataScience or Biology. However I think that arguments about similarities do not have much ground. Bio-informatics/-statistics is the same but different as well


Analogous case 'difference between languages'

Since this answer seems to attract a lot of negative votes, I would wish to add one more analogy.

I hope that this will clarify a bit more the intention of this answer.

  • I would not like to give so much a clear answer in the sense that this answer is clearly in favour of differentiating data-science and statistics, and especially in the practical case whether they deserve to be different topics on the stackexchange network (which involves many more arguments than just the principle whether they are different or not)
  • However, in the other answers I sense little acknowledgement for the difference between datascience and statistics. I would like this answer to show that there is at least an alternative viewpoint. We can see, in principle and from particular viewpoints, datascience and statistics as different.

So the analogy that I have in mind is the Limburgian language. Limburg is a region in the southeast-tip of the Netherlands and borders Germany and France. The people over there speak a different language than the people in the rest of the Netherlands. It is constantly being debated whether this language should be considered a dialect or a language on it's own (linguists do not really make the differentiation between languages and dialects and speak more about a continuum, but the same happens here where we do not really have a difference between a rank of disciplines, like statistics vs datascience, and deal with a continuum).

Limburgian is often considered, by many Dutch people, as just a variation of the Dutch language. However, Limburgian is officialy recognized as a regional language. And interestingly an important argument (from the linguists) is that the Limburgian language has more in common with neighboring French and German, and because of that should be considered as clearly distinct from the Dutch language (even when most Dutch people believe otherwise or do not recognize/acknowledge the differences).

  • 2
    The neural networks questions (number of CNN filters & dropout) seem much like question we have under conv-neural-network and dropout. – Sycorax Sep 13 at 22:21
  • 2
    Six of those questions seem perfectly on-topic here, two on-topic at Stack Overflow, & one perhaps better suited for Computational Science; I can't see much difference in the style of asking. Two (on circular predictors & on evaluation of classifiers) I recognize as already having been asked here, with less irrelevant & more comprehensive answers respectively. – Scortchi Sep 14 at 7:11
  • 3
    Note that "time-stamps in ridge regression" is explicitly asking how to represent time of day as a predictor (so that a minute to midnight is as close to a minute past midnight as a minute to three is to a minute past three). If it had been asked on CV, I don't doubt it would have been quickly identified & closed as a duplicate, whereas it's still awaiting an adequate answer on DS. If there's a subset of questions that are on-topic on both sites, & if, as your sample suggests, they make up over half of the DS questions, then the question has to be asked - What good's that doing anyone? – Scortchi Sep 15 at 20:51
  • 1
    @Scortchi The obvious conclusion is that OP asked their circular-statistics question on datascience.SE because OP wanted a data science answer, not a statistics answer. ;-) – Sycorax Sep 15 at 22:47
  • @Sycorax: Quite. My hope's that with time DS will get more picky about what kind of questions it accepts, rather like CV has got. After all, what Data Science actually is may be obscure, but there seems to be at least general agreement that it can be represented by a Venn diagram, in which its scope is represented by the intersection of other fields, not their union. – Scortchi Sep 18 at 14:44

After almost 4 years, we can draw some conclusions. There is no clear distinction for users. Topics and questions overlap and there is lots of cross-linking between two sites.

I mean, stats is a different topic than data science, but the reality points us in another direction. I would like to see these two merged for the sake of simplicity at least.

I would like data science to stay and stats go, even though stats is way more popular. The reason is stats has tons of questions not directly related to statistics, falling into a broader genre of data science.

  • 4
    (-1) Other than a reaction of No!!! I'd like to underline that you can think privately of CV as "stats" if you like but its public focus is explicit as "statistics, machine learning, data analysis, data mining, and data visualization", so statistics in a very broad and generous sense. – Nick Cox Sep 14 at 14:12
  • 3
    While I disagree with this answer, it is helpful because it illustrates that there is a misunderstanding about what is on-topic here among some people. – Sycorax Sep 14 at 14:39

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .