6
$\begingroup$

Questions which are problems faced when applying stats techniques to large(or big) data.

A similar question for reference

I know that theoretical questions about Big Data are on-topic here, as they deal with applications of stats and visualization techniques and algorithms on large data.

But, the problems like: How should I use < this algorithm > when the data is large? on-topic here?

Or do they need to be migrated to the DataScience SE?

$\endgroup$
5
  • 7
    $\begingroup$ What is especially interesting about the link you supply is that it has nothing at all to do with big data! According to one answer, that question deals with "small data." Although here is not the place to debate what "big data" means, you ought to consider differentiating between "big" in the sense of (a) large enough to create computational difficulties and (b) large enough to suggest using non-standard, sophisticated, or detailed statistical procedures for exploration and analysis. $\endgroup$ – whuber♦ Sep 22 '15 at 12:31
  • $\begingroup$ @whuber That question might not be about Big Data(the data might look humungous for the OP, but in general it's not). But, it's just an example I cited for explaining the question, as it's wordings seem to be close to my doubt. I am asking about such type of questions. Just wondering, is my question clear? $\endgroup$ – Dawny33 Sep 22 '15 at 12:36
  • 2
    $\begingroup$ It's not clear to me because I still don't understand what you mean by "type," especially because you have supplied an example that does not seem to be an example at all. It does show that the phrase "data is large" is subjective and arbitrary, which rules it out as an effective criterion for determining whether a question is on topic. $\endgroup$ – whuber♦ Sep 22 '15 at 13:23
  • 4
    $\begingroup$ I think that a question e.g. like "How can I apply PCA to a 100gb data table that does not fit into my RAM?" would be absolutely on topic, even though 100gb is not really "Big Data". In fact, we do have similar questions and they have useful answers and are not closed (see e.g. this question and the ones linked from there). $\endgroup$ – amoeba Sep 22 '15 at 14:26
  • $\begingroup$ @whuber Seems like Andre have managed to cite the examples which I wanted to cite in the question. $\endgroup$ – Dawny33 Sep 23 '15 at 5:52
8
$\begingroup$

It seems to be on-topic, as long as the question is well worded/asked.

For example, based on this search, there are some examples which I think relate to what you are asking:

Clustering algorithm for my situation?

What is a good algorithm for estimating the median of a huge read-once data set?

Online algorithm for mean absolute deviation and large data set

$\endgroup$
1
  • 1
    $\begingroup$ Thanks for clarifying. Seems like these are the exact examples I wanted to cite, but somehow ended up putting a wrong one. $\endgroup$ – Dawny33 Sep 22 '15 at 17:40
6
$\begingroup$

I agree with @amoeba. I see no problem with the question, "How should I use < this algorithm > when the data is large?" If you are asking for code / packages, etc., that would be off topic. But how having so much data it won't fit on your computer affects the usage of a machine learning algorithm seems well within our mandate.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .