12
$\begingroup$

I just flagged: R solution for high performance and large datasets for closing since it's quite obviously off-topic (I can't really see as instructions for setting up a Spark server from R Studio could be considered a statistical question).

However, I'm not sure if this can be considered a programming question either (it doesn't concern a specific language, but more correctly specific programs/cloud platforms, and I'm not sure how one would go around providing a sample test case).

Among the various options of the close flag, one can propose a new site to which the question would be migrated. Currently, three sites are allowed, Math Overflow, Stack Overflow and Math Exchange. Of course it wouldn't make sense to include all possible SE sites, but what about adding just Data Science SE? Questions like the one I just flagged look to me exactly the kind of question which would be on-topic there.

In my new position I work on and/or collaborate with people which work on problems very similar to that described by the OP, and they're quite correctly (IMO) considered to be Data Science in the private sector.

EDIT: I see that the same question has been flagged by other users. Regardless of where it will be migrated (or closed), I'm glad to see I'm not the only one who doesn't believe it's a valid CV question.

EDIT2: in meantime, here is a question which has been (correctly, IMO) migrated to DS

https://datascience.stackexchange.com/questions/22591/what-are-the-downsides-to-microsoft-azure-vs-scikit-learn-r

I wasn't involved in the migration, and surely a single case doesn't justify adding DS to the quick list.

$\endgroup$
9
  • 1
    $\begingroup$ I think that makes sense. $\endgroup$
    – Peter Flom Mod
    Commented Aug 22, 2017 at 11:53
  • 4
    $\begingroup$ Isn't it Cross Validated Meta rather than Math Overflow? BTW I think we can have up to five on the list - meta.stackexchange.com/q/77264/225179. For rare migration paths you can flag the post for a moderator to migrate. $\endgroup$
    – Scortchi Mod
    Commented Aug 22, 2017 at 12:56
  • 4
    $\begingroup$ It isn't clear to me how often we really need to migrate to DS, even accepting your argument that this particular thread should be. $\endgroup$
    – gung - Reinstate Monica
    Commented Aug 22, 2017 at 13:54
  • $\begingroup$ @Scortchi, it could be CV Meta instead than MO: I haven't tried flagging another post just to verify that :) yes, I know I could suggest the moderator to migrate the question to DS, but I think this migration path may become more common in the future. See my response to gung's comment. $\endgroup$
    – DeltaIV
    Commented Aug 22, 2017 at 21:03
  • 2
    $\begingroup$ @gung I think migrating to DS may become more common in the future. Deep Learning is attracting a lot of users, and many DL-related problems are at a cross-road, so to speak. You don't just have to use Keras (for example) to build your DNN: questions of this kind could be safely migrated to SO). But you will also need to bring your data to the cloud, munge the data, use Amazon Web Services to train the DNN because you're not Google and you don't have 450 GPUs in your company, etc. I've already seen a few ML questions on CV which cannot be considered statistical questions... $\endgroup$
    – DeltaIV
    Commented Aug 22, 2017 at 21:11
  • 1
    $\begingroup$ ...and I think the trend will become more common. Of course I may be wrong: " making predictions is hard, expecially about the future". $\endgroup$
    – DeltaIV
    Commented Aug 22, 2017 at 21:12
  • 1
    $\begingroup$ These two are the only others I can find migrated to DS from CV this year: Boolean classification on strings & Why it is taking more training time on one machine comparatively to similar another?. I did think How to generate distance variable using street addresses might be on-topic at DS, but was told not when I asked on their chat. $\endgroup$
    – Scortchi Mod
    Commented Aug 27, 2017 at 10:57
  • 1
    $\begingroup$ In fact I find it difficult to work out just what's supposed to be on-topic at DS - the help page gives five examples of "questions [scil. question titles] that are likely to be on-topic" (four of which seem entirely appropriate for CV), but no explanation. $\endgroup$
    – Scortchi Mod
    Commented Aug 27, 2017 at 13:42
  • $\begingroup$ @Scortchi your numbers match those by whuber. Clearly for now the answer to my question is no ("it doesn't make sense to add DS to the list of suggested sites [...]"). Maybe next year it will be different :-) Anyway, it's not like I'm looking forward to migrating questions from CV - I like this site, I owe you guys a lot and I'll likely keep pestering people with my questions. It's just that some questions I'm interested in, are not the right fit for this site, so I'm interested in migrating them the best way. $\endgroup$
    – DeltaIV
    Commented Aug 27, 2017 at 17:10

2 Answers 2

Reset to default
13
$\begingroup$

The migration dialog provides a short list of target sites. These are the sites all users may choose as likely targets. Because it's important to get along with other sites, it is crucial that all our users with vote-to-migrate privileges well understand the on-topic criteria at those sites. For years the only options have been SO, Math, and our own Meta site (the one you're on now). But we can change that.

Until DS (Data Science) graduated from Beta status last month, it wasn't possible to include it on the target list: beta sites aren't allowed. This discussion therefore is timely.

This year we have migrated approximately 370 posts to other SE sites. Of those, only three were to DS. We migrate (slightly) more frequently to Quantitative Finance and Open Data. Other sites we migrate to (rarely) include Signal Processing, Physics, Academia, Economics, and Computer Science. If DS is in the discussion, then so should these be.

Perhaps the best way to utilize those five default target options in the migration dialog would be periodically to review whither we migrate questions and put the top five targets into the default list. (Roland, in a comment, makes an astute observation about "confirmation bias." The implication is it's likely that whatever five targets we first include will become a self-perpetuating set of top five targets because the ease of migrating to them will automatically make them the most popular.)

$\endgroup$
5
  • $\begingroup$ Based on anectodal evidence, i.e., a few posts I saw, which I thought could be migrated to DS instead than closed, as well as on a trend I see on other social media (Twitter, Medium, LinkedIn, etc.), I thought that questions related to "Data Science Carpentry", so to speak, could become more prevalent in the future. But of course I don't have at my disposal the quantitative data you have. If, based on data, you believe other site are more worthy of addition to the list than DS, then by all means please add them instead than DS. Statistical analysis (nearly) always trumps anectodal evidence :) $\endgroup$
    – DeltaIV
    Commented Aug 22, 2017 at 21:18
  • 2
    $\begingroup$ You are making a highly suspect implicit assumption that the growth of interest in data science ought to lead to a growth in questions that would be appropriate for the DataScience SE site but not here on CV! Arguably, it could go either way. For instance, as data science becomes better known, people learning it might have a greater tendency to see it as part of statistics and machine learning, thereby decreasing the number of (already rare) questions that are on-topic only on the DS site. $\endgroup$
    – whuber Mod
    Commented Aug 22, 2017 at 21:22
  • $\begingroup$ Wait, wait! First of all, I said I agree with your analysis: you have data (and you are a statistician of proven worth), I don't have data (and I'm not a professional statistician), so how could I disagree? Secondly, I have never said that the growth of interest in Data Science will not lead to an increase to the number of questions here on CV, and I could have never said that, because I am sure the number of questions on CV will increase (not sure if the number of answers will also increase, since many questions go unanswered on CV, but I digress). For example, some data... $\endgroup$
    – DeltaIV
    Commented Aug 22, 2017 at 21:43
  • 2
    $\begingroup$ ...scientists who contacted me on a social media were baffled by the Freedman's paradox. These guys could sure use some help here on CV. And hopefully the number of data scientists who understand they need to learn Statistics better, to be successful at their jobs, will grow. At the same time, though, there are questions which pertain to Data Science practice and which don't belong here. Since this site has much more traffic than DS, I think that the growing interest in Data Science will lead to an increase of both in-topic and off-topic questions here. But again, I might be wrong. $\endgroup$
    – DeltaIV
    Commented Aug 22, 2017 at 21:53
  • 4
    $\begingroup$ I'm not sure that "put the top five targets into the default list" is a good approach. At least, my experience from Stack Overflow is that the defaults make migration so much easier that you get a type of confirmation bias. $\endgroup$
    – Roland
    Commented Sep 12, 2017 at 10:18
1
$\begingroup$

No.

The particular question that you refer to should belong to either R or to the databases. The reason why it's not a generic high performance question is because the capabilities to deal with large data sets are very different in different platforms. What may work for R will not necessarily work fro SAS. This depends on the platform architecture, availability of drivers, native vs. ODBC/JDBC etc.

If there was not "R" in question then this would belong to databases or other IT domain, perhaps, SO.

$\endgroup$
3
  • $\begingroup$ Neither Spark or Azure HDInsight are simply databases. They are cloud computing environments with significant support for large data analysis, distributed computing and machine learning. They are much more than an SQL or an Historian database. Even if you remove the reference to R, the question doesn't appear to be about databases, though the OP admits that she/he doesn't have a clear idea of what she/he wants. $\endgroup$
    – DeltaIV
    Commented Aug 22, 2017 at 22:01
  • 2
    $\begingroup$ Spark is just the distributed computing framework. Any questions about it are better covered in its own forums. In our firm it's IT who deals with it. $\endgroup$
    – Aksakal
    Commented Aug 22, 2017 at 22:03
  • $\begingroup$ Also R has "its own forum" (the R-help mailing list), but this definitely doesn't make that forum the best place to ask any questions about R. Anyway, it looks like there are many questions on Spark and R on SO: stackoverflow.com/search?q=Spark+rstudio I still suspect that since the OP asks for a comparison with Azure HDinsights, this won't make her/his question well-received on SO, but I'm not sure. $\endgroup$
    – DeltaIV
    Commented Aug 23, 2017 at 8:58

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .