# Are questions about solving practical data science problems (memory etc.) on topic?

Are questions like How to resolve Error: cannot allocate vector of size 1.1 GB for glmnet in R on topic? On one hand, it asks about software, on another, this is about solving a practical data science problem and has general, software-agnostic, solutions (algorithms working in online mode etc.). Or maybe we should migrate it to datascience.stackexchange.com ?

• I think if questions like this stats.stackexchange.com/questions/2806 are on topic then this one can also be considered on topic. It would benefit from some editing though which I tried to do now. – amoeba says Reinstate Monica Feb 21 '18 at 12:13
• To me questions on general statistical computing are usually fine; questions on specific code usually not. A rule of thumb is to imagine the question without specific code references. If little or nothing is left, it is often off-topic. If something of wide interest is left, it is on-topic. A related test is whether a question focusing on software X should be almost equally interesting to people who use Y or Z instead. Why do different programs give different results is a kind of question to which the answer is often statistical. – Nick Cox Feb 21 '18 at 14:12
• I imagine that this question has different angles and is in the gray area (so I'd say it is on topic). Somebody is looking for a hack to allocate some vector. This might be done in two different ways, considering mathematical/computational tricks, or considering software engineering tricks. In the discussion we see both (discussion about memory allocation and about changing the strategy to get to the solution). – Sextus Empiricus Feb 21 '18 at 14:50
• That question is very clearly off topic--but for a completely different reason: it doesn't supply enough information to permit a diagnosis. We couldn't send it to any other SE site because of that. For example, if this is a Gaussian response with linear link, there are potential ways to reduce RAM requirements by working only with the sums-of-squares matrix, but if it is for a GLM (say, a logistic regression) then such an option is off the table. – whuber Feb 21 '18 at 15:54
• @whuber This information has been provided by the OP in an edit. – amoeba says Reinstate Monica Feb 22 '18 at 8:40
• If you think the problem can be resolved in software agnostic manner, then you can help OP to reformulate the question in such a way that makes it software agnostic instead of outright closing or porting the question to SE – Aksakal Feb 28 '18 at 18:55

I think that question, as currently framed, should be considered off topic.

The way I think about this is really pretty simple: I ask myself what the OP needs explained. If the answer is something about code / software, then it's off topic; if it's something about statistics / machine learning, then it's on topic, even if the question is asked in terms of some particular software (and may not even be askable without reference to software). It is true that the latter condition can mean that a given question cannot be answered by people who don't use that software, and may not even be understandable by them, but because the answer is an explanation of a statistical concept, it is ultimately a statistical question, and thus on-topic.

It's true that the specific thread at issue includes the line, "Or if you can recommend another way to proceed with the variable selection that would also work for me?", which may have been intended in a software-neutral way, but this seems like an afterthought. If the question were rephrased to ask about general computational strategies for large datasets, or about variable selection, then it would become on-topic, in my opinion. It would be fine if the question still makes reference to software as part of providing informative context for their situation. But the OP needs to show that they are clear that they are asking a statistical / software-neutral question, and that they will be happy with a sufficiently clear statistical answer, even if it were illustrated with, say, minitab. If the OP would not be OK with that, then they are not asking a question that is appropriate for this site.

With regard to migrating the thread to datascience.SE, I suppose that would be possible, but it probably isn't something that I would do. It is true that DS handles questions concerning topics related to the domains of CV and SO, but which would be off-topic on those sites for various reasons. In that light, the thread may be OK there, so I'm not necessarily opposed. It just isn't something that would even occur to me, to be honest.

• +1. Do you think the current version of this question (after several edits) can be considered on-topic? – amoeba says Reinstate Monica Feb 22 '18 at 9:08
• @amoeba, IMO no. It is still asking about R code, how it works, how to use it to get around the error. It does not seem to be asking about the related statistical issues, nor does it seem to be asking about a computational algorithm (to be implemented later in some barely mentioned language) as w/ the PCA thread you linked to above. – gung - Reinstate Monica Feb 22 '18 at 12:26
• Hmm. OK. I edited it again (only slightly) - can you check if it's okay now? – amoeba says Reinstate Monica Feb 22 '18 at 12:44
• @amoeba, from the body of the Q, the main question seems to be, "Is there some package that I can use...". Ie, no, this is still off-topic from my point of view. As I discuss above, my definition is based on what the OP needs explained, & they seem to be after R code / packages, or more generally an R based solution. Another way to test whether a Q is on topic (which is imperfect, but often clarifying) is to imagine someone answers w/ an example coded in software to which the OP does not have access, would that be a sufficient answer for the OP? If not, the Q is off topic. – gung - Reinstate Monica Feb 22 '18 at 13:09
• In this case, 'try package y in software z' is the kind of answer the OP seems to want, but will only be of value if the OP has access to software z. That's b/c the OP is asking about code, not statistics. – gung - Reinstate Monica Feb 22 '18 at 13:10
• I can see your logic but I'd say it's too restrictive. Somebody wants to run logistic regression with elastic net penalty -- a very clear & meaningful statistical problem. The dataset is too large and the standard tool (glmnet) fails to work -- a very clear obstacle that other people can later have too. This person comes to our forum and asks "What do do?!" -- and we are telling them: Sorry, off-topic. This just looks strange to me. – amoeba says Reinstate Monica Feb 22 '18 at 13:17
• @amoeba, I think these kinds of clear boundaries are the only thing that's workable in the long run. People who need tech support for their software need to use the software's native facilities. – gung - Reinstate Monica Feb 22 '18 at 13:20
• They can ask, 'what can I do?', but 'what code / package do I use?' is off topic. There is a difference between those. It may be subtle, but it's clear to me. – gung - Reinstate Monica Feb 22 '18 at 13:24
• I could edit the question to make it about 'what can I do?', but it would be a large edit. It isn't my place to change the nature of the OP's question. Moreover, I think the answer to that question would not be an answer the OP would find helpful, b/c the OP is clearly after, 'library(z); mod <- y(data); summary(y)'. – gung - Reinstate Monica Feb 22 '18 at 13:33
• Perhaps the clearest strategy would be to remove all reference to software, packages, error messages, etc. Then state that you want to know what strategies or algorithms are available for penalized LR when the matrix is too large to fit in a reasonable computer's memory. That is more drastic than necessary, but it does lay out a clear path forward. If the OP would not be interested in the answer to that kind of Q, the Q they are asking is off topic here. – gung - Reinstate Monica Feb 22 '18 at 13:40
• OK but if this Q had not even mentioned glmnet, then my first comment would be along the lines of "Have you already tried glmnet? Did it not work?" It seems strange to require to edit this useful information out. – amoeba says Reinstate Monica Feb 22 '18 at 13:50
• I'm with @amoeba on this one. It's an interesting Statistics problem, which just can't be solved without a computer, and which would be very hard to solve without using a dedicated software tool. The OP is not asking us to debug her code (as a matter of fact, she provides very little code at all), and there are at least two answers which work in R. The simplest answer is to broadly describe the implementation of the methodology which solves her problem, and then point her at the right package. The hardest, less useful answer is to describe the implementation in minute detail, don't mention 1/ – DeltaIV Feb 23 '18 at 8:42
• 2/ the existing package and expecting her to reinvent the wheel. Then there are other approaches based on free cloud computing resources. I think it's an interesting, answerable question of general interest, and closing it is a pity. But of course that's just my opinion. – DeltaIV Feb 23 '18 at 8:44
• Thanks @DeltaIV. I suggest you vote to reopen. I cannot because I already did and the Q did get reopened but then was closed by gung again; the system does not allow me to vote again. One of us should probably also post an answer here explaining our disagreement with gung. – amoeba says Reinstate Monica Feb 23 '18 at 8:58
• @DeltaIV Well I could not resist a procrastination opportunity and did post an answer here. – amoeba says Reinstate Monica Feb 23 '18 at 9:12

I have initially thought that I agree with @gung (and even upvoted his answer) but the discussion in the comments showed that I actually disagree.

I think this question (after the clarifying edits) is on-topic.

@Gung writes:

If the question were rephrased to ask about general computational strategies for large datasets, or about variable selection, then it would become on-topic, in my opinion. It would be fine if the question still makes reference to software as part of providing informative context for their situation.

I think this is what already happened. The OP wants to run logistic regression with elastic net penalty -- a very clear & meaningful statistical setup. The dataset is too large and the standard tool (glmnet) fails to work -- a very clear obstacle. This person comes to our forum and essentially asks: "What can I do?". This must be on-topic.

@Gung replies that

In this case, 'try package y in software z' is the kind of answer the OP seems to want

and I agree but I do not think this makes it off-topic. Imagine there is a more efficient algorithm of fitting this model that does not require so much memory. An answer describing this algorithm without mentioning the software package is pretty much useless because nobody wants to re-implement stuff like that from scratch. Everybody would want to have a working implementation.

Of course an answer "try package y in software z" and nothing else would be a bad answer. But a possibility of bad answer does not make the question off-topic. A description of the methodology plus a package recommendation would be a fine answer.

• It's still off-topic to me, "I have tried to use sparse matrix and it reduces the size from 2.2 to 1.1 GB, but I am still not able to run the function. Is there some package that I can use - I saw bigmemory but am not really sure how to apply it? Or if you can recommend another way to proceed with the variable selection that would also work for me?". But I think software/package recommendations have their place here on CV, so I also do not fully agree with gung. – Firebug Feb 23 '18 at 14:00
• @Firebug Can you clarify what aspect makes it off-topic and what kind of edit could make it on-topic? If OP cuts out the first two sentences of the part you quoted, will it become any better? – amoeba says Reinstate Monica Feb 23 '18 at 14:02
• I think a discussion that illuminates the statistical & computational issues that led to the OPs result, & a "description of the methodology" that addresses the above, would be a great answer. Of course, there is no problem w/ pairing such an answer with an illustration (that would obviously entail code). If you read through some of my answers, you will note that that pattern is one I use often. I have no problem with any of that. – gung - Reinstate Monica Feb 23 '18 at 14:46
• Instead, the issue WRT this specific Q is that I don't think the OP is after that. One thing to note is that all the clarifying edits that serve to move the Q somewhat away from 'what library & code do I use?' were made by you, not the OP. Even the edited Q still implies that the answer they are after is 'library(z); mod <- y(data); summary(mod)'. If you had both, my guess is that the OP would skip the "technical and wordy" stuff & just look at the code. – gung - Reinstate Monica Feb 23 '18 at 14:51
• As I said above, another conceptual test that I use to think through whether a Q is on topic, is what if the issue were illustrated, but with software the OP doesn't have? In this case, I think the answer we are describing would be an absolute $0$: The part about statistics is of no interest to the OP, & they would not be able to run the code. So the whole thing would be viewed as a waste. – gung - Reinstate Monica Feb 23 '18 at 14:54
• By way of contrast, here is an answer that follows the general pattern, describing a plot, & illustrating it, but w/ software to which I don't have access (& can't even read). Nonetheless, I thought it was quite helpful to me. – gung - Reinstate Monica Feb 23 '18 at 14:57
• @gung This OP never posted on CV before and is obviously unfamiliar with our rules. When I see a suboptimal Q that I consider potentially interesting (also for others and for future readers), I'd rather edit it to make it comply with our guidelines, than close as not complying. This was my intention here. Just trying to "save" a potentially interesting thread. – amoeba says Reinstate Monica Feb 23 '18 at 15:00
• I'm not blaming you for trying, @amoeba. I hope I didn't give you that impression. – gung - Reinstate Monica Feb 23 '18 at 15:01
• @gung whoa, it must have taken some ...guts to define your nice answer about the difference btw pp-plots and qq-plots "technical & wordy". There's literally not a single equation in it. – DeltaIV Feb 23 '18 at 15:27
• @gung one last comment and then I'll refrain from intervening on this topic anymore: you and amoeba have already discussed it at length, and you're users with way more experience than me. I tend to agree with gung that the OP seems to be interested in code only, but a "method + a little code" answer would be useful to other users. Does this make a difference in deciding if the Q is on-topic or off-topic? Not a rhetorical question. – DeltaIV Feb 23 '18 at 15:29
• Exactly my point @DeltaIV: I often tend to think about other users more than about OP. – amoeba says Reinstate Monica Feb 23 '18 at 15:30
• That's an excellent question, @DeltaIV. It has kept coming up in my thinking as amoeba & I have been discussing this. I'm not sure what the answer is (although implicitly I've been saying 'no' by not reopening). It may be worth a separate meta thread to hash out. The underlying issue goes farther than this thread. What is 1 answer will be better for the asker, but a different answer will be better for posterity, eg? – gung - Reinstate Monica Feb 23 '18 at 15:32
• Ok, I'll ask it. I don't have time now to write a nice question (and I actually need to write two of them), so it will be short, but hopefully good. – DeltaIV Feb 23 '18 at 15:38