Shouldn't CV be a valid resource on how to implement non-trivial statistical models in different software?

Question

Currently the Help Center > Asking states that questions are on-topic if they're about "statistical analysis, applied or theoretical", and whenever it's about programming "if it needs statistical expertise to understand or answer". This also is the topic of this meta discussion.

Shouldn't therefore questions concerning the implementation of a particular statistical model in different software be on-topic on Cross Validated (CV)?

What happened: A few days ago I encountered the problem of how a specific statistical "far from trivial" model can be implemented in R rather than in Stata. Google's results were rare and one pointed me straight to this question on CV which gave me valuable hints, but the solution was not provided. I had to do quite an amount of extra research and after I found a solution, and after I've convinced myself that the same statistical method is being used, I wrote an answer to share my insights with the OP and the community.

Just after I've posted my answer, the question was closed as off-topic. I am wondering why, since the OP has written his question in a "general format" and elucidates "underlying statistical methods" as demanded in this meta discussion.

Rather, perhaps my answer is somewhat off-topic, as it does not contain much statistical explanation. The question hasn't been migrated to Stack Overflow (SO), since it might be a "borderline question" in terms of @Shog9. IMO it therefore shouldn't have been closed on CV.

My point is that on CV answers to questions about how a certain statistical model X can be implemented in software Y (perhaps with an unknown package Z) are often invaluable. I personally doubt to get statistically valid answers on Software Recommendations or r-help - as some suggest where to ask such questions - when things enter non-trivial statistical areas. I would most likely trust answers or consider them significant if they are right here on CV. There are several other similar questions which are not closed, e.g.: 1, 2, 3, 4, 5. I have found several of them to be really useful. Some such questions would be off-topic on SO since they are "too broad", for example this question, or doesn't contain a MCVE, for example this question. Further asking to "recommend or find a book, tool, software library, tutorial or other off-site resource" is off-topic on SO as also noted in @Tim's CV meta question.

I consider CV as an invaluable resource of such model implementation questions in different software and I would be disappointed to see that to change.

I'm aware that this topic has its history. Maybe my question contributes to some further differentiation (from e.g. pure programming R questions). This answer doesn't fit to the type of questions since it doesn't cover model implementation in different software.

Also this community wiki isn't suitable to the question since it is not purely software-related and needs a statistically educated brain to give an adequate answer which can't be a entry on a simple package list.

Answer 1

I agree with many of the points gung makes (and several of his answers to subquestions) but I disagree with the conclusion in relation to the post in question.

I also agree that it has been litigated many times, and the help/on-topic hasn't changed across many iterations of this sort of question... indicating that its contents are as intended (or presumably they would have been changed, or at least a solid call for change would be made).

If we read the help/on-topic, it's on topic if it requires "statistical expertise to understand or answer". I think the indicated question qualifies on that basis. [I don't think every such question does, but there's certainly a subset that do]

Here's a challenge: Let's imagine we find 100 random expert programmers who lack any statistical expertise, and give them full access to the R manuals, language definition, etc. Now get them to answer the original question. How many will successfully do it? (Maybe a couple?)

So now if a somewhat larger proportion of expert statisticians would be able to do it correctly as a result of their statistical background, then there must be something in their statistical expertise that is needed to have a good chance at answering it.

There's room for disagreement about whether expertise is necessary in particular cases (until we actually perform such a test I suppose), and if it's not pretty clear that it is required, a degree of conservatism isn't a bad approach -- but my reading of the question suggests that a considerable degree of statistical expertise is certainly needed to correctly understand what the question is even asking about, and consequently some is required to be confident you have the right answer. (If some of our random expert programmers happened to locate a good solution, would they really know if they had it? Or would they be saying something nearer to "uh, maybe see if this is the sort of thing you want, I don't really know")

In the particular instance here, I strongly doubt that I could easily produce a correct answer without substantial effort, but it wouldn't be a lack of knowledge of R that would be the barrier on this one (I believe my knowledge of R is more than sufficient) -- it would be my lack of substantial familiarity with the statistical problem in question - my own lack of statistical expertise. If I was to try to solve this question, I'd begin by learning more about this particular area of stats, then I'd start searching based on that knowledge. If that doesn't indicate a case where 'statistical expertise is required', I don't know what does.

I think the question at issue here is an example of the sort of question the particular phrase in the help was intended to encompass. Perhaps we should explore (in a new meta question) whether that phrase should be clarified or tightened/rewritten to be more restrictive but as things stand I think it includes the question-that-was-closed as being within the scope of on topic.

Answer 2

As far as I can tell, the example question should be squarely on-topic at Stack Overflow (& the answer a fine one). Both https://stackoverflow.com/help/on-topic & the close reason

Questions asking us to recommend or find a book, tool, software library, tutorial or other off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.

link to What exactly is a recommendation question, where a highly up-voted answer draws the distinction between describing a problem & "shopping". It would also be suitable for the R-help mailing list. While I agree with @gung that there's no onus on CV to ensure that every type of question can be found a home somewhere on SE or the internet; I think we can, & should, give the existence of other places to ask a given type some weight when considering our ambit. In particular, more overlap between SE sites than needed to avoid impractically tortuous delimitation of their scopes doesn't help anyone.

On the other hand, the example question's certainly off-topic here according to our current practice of closing questions that focus on "performing routine operations within a statistical computing platform". Perhaps our help/on-topic should be clarified. Or perhaps we should embrace such questions—but I find it hard to imagine how we'd draw a line, even roughly, between the trivial & the non-trivial or between the commonplace & the recondite. Would we need to? "Does not show research effort" & "not useful" are already reasons to down-vote, & we could each vote in individual cases as we see fit rather than collectively struggle with legislation & litigation.

Answer 3

This issue has been litigated many times on meta.CV, and the specific thread referenced is clearly off topic. The question is only asking about how R works (what package / function, and an error message), not the underlying statistical issues. In addition, the explicit questions (the places where there is a "?"), are the title, "How to... in R... ?", and the final sentence / paragraph "Any suggestions?" (which is obviously too broad). The answer is only code and does not contain any statistical explanation, as you note.

---

Let's address some of the specific issues:

Should CV offer code help and/or tutorials as long as the models discussed are "far from trivial"?

No. This is not training or tech support for any software, including R. We also don't do tutorials. Software should make resources available (documentation, books, courses, etc.) to provide adequate assistance to users. Moreover, "far from trivial" is in the eye of the beholder. I teach introductory statistics classes, and many of my students would consider a simple logistic regression model to be "far from trivial" (I'm guessing you would not—at least, I do not).

Shouldn't we allow such questions here, since there doesn't seem to be a better place to ask elsewhere on the internet?

No. The universe is not required to provide a free, convenient answer somewhere. And our scope is not determined by the scope of other SE sites, or other sites generally. Whether or not any suggestion for where a question can be asked can be given, our scope is the same. To see the illogic in this suggestion, consider applying it elsewhere, say Seasoned Advice, SE's cooking Q&A site. Shouldn't you be allowed to ask how to fit a bivariate probit model there, since it isn't on topic elsewhere? No, because it is outside their scope. That may not seem fair, since that is a cooking site, and this is a statistics site. But that's the point, this site is for questions about statistics, not about software (even statistical software) and how to use it.

Shouldn't we allow this question, since it appears other questions have been allowed?

No. This is a common retort by people who want answers and believe CV should be obliged to give them. There are several issues: First, the site's standards and scope have evolved over time. Second, there are many questions asked every day. No one has read them all. I don't read all the new questions every day anymore. If I had seen the thread at issue here when it was asked, I would have closed it immediately. I have not yet investigated the threads you link, but it is possible they just slipped by and should be closed now. Third, you need to consider not only the few possible exceptions to the rule, but the many, many threads that have been closed in accordance with our policies. Moreover, whether or not any other off-topic thread remains open has little bearing on whether this off-topic thread should be closed.

---

I am very sympathetic to the fact that there are people who have questions and need help. (I have answered over 1.5k questions here, after all.) I am also sympathetic to the fact that people can find R inscrutable at times and really want help with that—I've been that person myself more than once. People commonly say they want to ask here because this is the best site with the highest quality information. Despite my sympathies, I am also responsible to maintaining the standards of this site, in terms of both boundaries and quality. In fact, part of the reason the site has high quality information is because we are strict about what we do and how we do it.

So how do I determine if something is on-topic, or if a seeming code question really "takes statistical expertise to answer"? It's fairly simple really. I do not ask if only someone with statistical training would have heard of the test / model at issue, instead, I ask myself what the OP needs explained. If what needs to be explained is a statistical idea, it's on-topic, if it's not, it's off-topic. For example, if the question had asked how seemingly unrelated bivariate probit regression works, or how any software could estimate it 'under the hood', that would have been on-topic. Sometimes people show that what appears to be the same model / test gets different answers from different software. The issue is always that the software are making different default assumptions or using different estimators. Other times the way the code is used or how it can go wrong is related to subtle and conceptually difficult aspects of the underlying procedures. Those are all statistical issues to be explained.

---

In conjunction with my statement that we sometimes miss questions that should be closed, but that their remaining open is not a valid argument that another off-topic question should be allowed, it may be worth laying out the history here. I became aware of this thread because the answer was flagged for deletion as not an answer by CV's standards. I agreed with that, but nonetheless I did not delete the answer. The problem is that the question is clearly off topic and no post that actually answered the question would be a valid CV answer. On the other hand, I recognize that the answer posted probably does help the OP (and future readers, since the thread won't disappear, even if closed). So I left the answer, but closed the question.