13
$\begingroup$

Currently the Help Center > Asking states that questions are on-topic if they're about "statistical analysis, applied or theoretical", and whenever it's about programming "if it needs statistical expertise to understand or answer". This also is the topic of this meta discussion.

Shouldn't therefore questions concerning the implementation of a particular statistical model in different software be on-topic on Cross Validated (CV)?

What happened: A few days ago I encountered the problem of how a specific statistical "far from trivial" model can be implemented in R rather than in Stata. Google's results were rare and one pointed me straight to this question on CV which gave me valuable hints, but the solution was not provided. I had to do quite an amount of extra research and after I found a solution, and after I've convinced myself that the same statistical method is being used, I wrote an answer to share my insights with the OP and the community.

Just after I've posted my answer, the question was closed as off-topic. I am wondering why, since the OP has written his question in a "general format" and elucidates "underlying statistical methods" as demanded in this meta discussion.

Rather, perhaps my answer is somewhat off-topic, as it does not contain much statistical explanation. The question hasn't been migrated to Stack Overflow (SO), since it might be a "borderline question" in terms of @Shog9. IMO it therefore shouldn't have been closed on CV.

My point is that on CV answers to questions about how a certain statistical model X can be implemented in software Y (perhaps with an unknown package Z) are often invaluable. I personally doubt to get statistically valid answers on Software Recommendations or r-help - as some suggest where to ask such questions - when things enter non-trivial statistical areas. I would most likely trust answers or consider them significant if they are right here on CV. There are several other similar questions which are not closed, e.g.: 1, 2, 3, 4, 5. I have found several of them to be really useful. Some such questions would be off-topic on SO since they are "too broad", for example this question, or doesn't contain a MCVE, for example this question. Further asking to "recommend or find a book, tool, software library, tutorial or other off-site resource" is off-topic on SO as also noted in @Tim's CV meta question.

I consider CV as an invaluable resource of such model implementation questions in different software and I would be disappointed to see that to change.

I'm aware that this topic has its history. Maybe my question contributes to some further differentiation (from e.g. pure programming R questions). This answer doesn't fit to the type of questions since it doesn't cover model implementation in different software.

Also this community wiki isn't suitable to the question since it is not purely software-related and needs a statistically educated brain to give an adequate answer which can't be a entry on a simple package list.

$\endgroup$
11
$\begingroup$

I agree with many of the points gung makes (and several of his answers to subquestions) but I disagree with the conclusion in relation to the post in question.

I also agree that it has been litigated many times, and the help/on-topic hasn't changed across many iterations of this sort of question... indicating that its contents are as intended (or presumably they would have been changed, or at least a solid call for change would be made).

If we read the help/on-topic, it's on topic if it requires "statistical expertise to understand or answer". I think the indicated question qualifies on that basis. [I don't think every such question does, but there's certainly a subset that do]

Here's a challenge: Let's imagine we find 100 random expert programmers who lack any statistical expertise, and give them full access to the R manuals, language definition, etc. Now get them to answer the original question. How many will successfully do it? (Maybe a couple?)

So now if a somewhat larger proportion of expert statisticians would be able to do it correctly as a result of their statistical background, then there must be something in their statistical expertise that is needed to have a good chance at answering it.

There's room for disagreement about whether expertise is necessary in particular cases (until we actually perform such a test I suppose), and if it's not pretty clear that it is required, a degree of conservatism isn't a bad approach -- but my reading of the question suggests that a considerable degree of statistical expertise is certainly needed to correctly understand what the question is even asking about, and consequently some is required to be confident you have the right answer. (If some of our random expert programmers happened to locate a good solution, would they really know if they had it? Or would they be saying something nearer to "uh, maybe see if this is the sort of thing you want, I don't really know")

In the particular instance here, I strongly doubt that I could easily produce a correct answer without substantial effort, but it wouldn't be a lack of knowledge of R that would be the barrier on this one (I believe my knowledge of R is more than sufficient) -- it would be my lack of substantial familiarity with the statistical problem in question - my own lack of statistical expertise. If I was to try to solve this question, I'd begin by learning more about this particular area of stats, then I'd start searching based on that knowledge. If that doesn't indicate a case where 'statistical expertise is required', I don't know what does.

I think the question at issue here is an example of the sort of question the particular phrase in the help was intended to encompass. Perhaps we should explore (in a new meta question) whether that phrase should be clarified or tightened/rewritten to be more restrictive but as things stand I think it includes the question-that-was-closed as being within the scope of on topic.

$\endgroup$
  • $\begingroup$ I think this answer touches more the core of my question. There are often a bunch of packages seemingly doing the same, which confronts a scientist with the agony of choice. E.g. CRAN Task views is great list to get an overview, but the descriptions are also extremely brief, which is also to be expected from a simple list. I find cross validated answers extremely helpful in such cases and CV should remain this resource. $\endgroup$ – jay.sf Apr 15 at 8:34
  • 5
    $\begingroup$ @jay.sf If R and Stata-based questions and answers are allowed, we need to allow them in any software whatsoever. A simple criterion is whether a question or answer is of use or interest to people using quite different software. Although it's often thought puzzling or unsatisfactory by individuals, in principle and in practice there are questions that don't belong on either CV or SO or any other SE site, because "we" don't want them, and neither do "they". (I write as someone active on SO too.) $\endgroup$ – Nick Cox Apr 15 at 8:53
  • 1
    $\begingroup$ +1. I think you are making a very compelling case. I had a similar feeling upon seeing this question, but did not manage to my finger on why exactly I think this question could be considered on-topic. $\endgroup$ – amoeba Apr 15 at 13:29
  • 2
    $\begingroup$ Whether expertise is necessary is not really different from whether the model is "far from trivial". If we're stipulating programmers without any statistical expertise, I doubt they would know what simple probit regression is either. What the question is about (in the sense of what needs to be explained to answer it) is much more straightforwardly assessable than the amount of expertise someone may need. What is difficult for one is trivial for another. Moreover, the "programmers" answering R questions on SO, or r-help, are much more statistically savvy than you're giving them credit for. $\endgroup$ – gung Apr 15 at 13:44
  • $\begingroup$ My claim is that expertise is necessary in the current instance. (Edit: I am not sure I get the point you're trying to make about my post on that first thing; sorry to be dense about it). In relation to programmers on SO, I know for sure many of them are statistically expert; I made no claim about lack of expertise there -- instead I was trying to specify that we consider a collection of programmers who did happen to lack statistical expertise (because if they had it, we would not be considering the effect of adding statistical expertise - which was the main point of the hypothetical) ...ctd $\endgroup$ – Glen_b Apr 15 at 13:50
  • $\begingroup$ ctd... . I also didn't assume they were expert R programmers (otherwise there would be no need to supply R documentation). It feels like you're responding to things I didn't actually write there, though certainly it's possible I have been ambiguous in conveying my intent. [Edit; If I mistake what you're saying in your comments, I apologize] $\endgroup$ – Glen_b Apr 15 at 13:54
  • $\begingroup$ @Glen_b, the OP asserts that the thread should remain open because the model discussed is "far from trivial". That's the subject of this meta.CV thread. I contest that in my answer. As I understand your answer, you think the thread discussed should stay open because it requires statistical expertise to answer. What I am saying here is that this conception of 'requiring statistical expertise' is ultimately the same as the model being 'far from trivial', and thus the same argument I made addresses both. $\endgroup$ – gung Apr 15 at 14:11
  • 1
    $\begingroup$ Thanks @gung ... I see what you're trying to say in that part now. I completely misunderstood at first $\endgroup$ – Glen_b Apr 15 at 14:12
  • 1
    $\begingroup$ I'm not sure of the implication though -- are you thereby also saying that your comments in your answer apply to the wording in the help/on-topic? If so are you proposing that it be fixed? $\endgroup$ – Glen_b Apr 15 at 14:16
  • 4
    $\begingroup$ I have not (explicitly) suggested that the wording of the help/on-topic be changed, but I have thought that a number of times over the years, & I wonder about it again now. I'm arguing for an interpretation of what "takes statistical expertise to answer" means. There is an interpretation of the phrase implicit in your answer (which is quite reasonable, BTW), but I argue that that should not be taken as the correct interpretation b/c the resulting rule isn't ultimately coherent. The interpretation I prefer leads to a simple & coherent rule. $\endgroup$ – gung Apr 15 at 14:26
  • 2
    $\begingroup$ @jay.sf, discussions like this typically need some examples to ground them, otherwise they become too vague & abstract for the discussion to go anywhere & people just talk past each other. You choose that question to motivate the discussion, which was perfectly fine. As Nick Cox commented, I find both the specific example & the idea it was intended to illustrate unconvincing. If you find a different example, my position may differ, but it is relevant here to note that that Q is clearly off-topic. $\endgroup$ – gung Apr 16 at 13:41
  • 1
    $\begingroup$ @jay.sf I suggest not to mark any answer as accepted until some sort of consensus is reached. $\endgroup$ – amoeba Apr 16 at 19:01
  • 2
    $\begingroup$ @jay.sf Suggesting a precise rule is constructive but ultimately not practicable if it just arises out of individual or subgroup experience. Yours, for example, as well as a reference to Angrist as if a familiar name, suggests that you are an econometrician or economist using econometrics. The difficulty is that the site extends in all sorts of directions and covers all kinds of expertise. $\endgroup$ – Nick Cox Apr 17 at 8:50
  • 2
    $\begingroup$ [ctd] Wanting say some one who is expert in machine learning to decide whether the question they are looking at is as easy as OLS implemented in R or as difficult as OLS with robust SEs not implemented in R is unfortunately a non-starter. It's possible to be highly competent and to have enough reputation here to vote to close without having any idea of what that means. This is a grey area, has been for many years, and even moderators with very high reputation such as Glen_b and gung can draw the line in different places. $\endgroup$ – Nick Cox Apr 17 at 8:53
  • 1
    $\begingroup$ [ctd] The main point is that contentious cases can always be discussed on their merits, which is where we started. $\endgroup$ – Nick Cox Apr 17 at 8:56
5
$\begingroup$

As far as I can tell, the example question should be squarely on-topic at Stack Overflow (& the answer a fine one). Both https://stackoverflow.com/help/on-topic & the close reason

Questions asking us to recommend or find a book, tool, software library, tutorial or other off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.

link to What exactly is a recommendation question, where a highly up-voted answer draws the distinction between describing a problem & "shopping". It would also be suitable for the R-help mailing list. While I agree with @gung that there's no onus on CV to ensure that every type of question can be found a home somewhere on SE or the internet; I think we can, & should, give the existence of other places to ask a given type some weight when considering our ambit. In particular, more overlap between SE sites than needed to avoid impractically tortuous delimitation of their scopes doesn't help anyone.

On the other hand, the example question's certainly off-topic here according to our current practice of closing questions that focus on "performing routine operations within a statistical computing platform". Perhaps our help/on-topic should be clarified. Or perhaps we should embrace such questions—but I find it hard to imagine how we'd draw a line, even roughly, between the trivial & the non-trivial or between the commonplace & the recondite. Would we need to? "Does not show research effort" & "not useful" are already reasons to down-vote, & we could each vote in individual cases as we see fit rather than collectively struggle with legislation & litigation.

$\endgroup$
  • $\begingroup$ It may also be a matter of interpretation. May I point at my last comment to @Glen_b s answer, where I state kind of a vision. $\endgroup$ – jay.sf Apr 17 at 8:15
  • 3
    $\begingroup$ I agree with @Nick Cox's reply to that comment. Though in fact we can postulate someone familiar both with using robust standard errors around OLS fits, & with $X$, the subject of the question, & still wonder if their line-drawing won't be forced & capricious. Let $X$ range over Poisson regression, negative binomial regression, k-means clustering, CHAID, Fisher's Exact Test, linear discriminant analysis ... I've no idea how to apply your rule in any of these cases. $\endgroup$ – Scortchi Apr 17 at 10:35
4
$\begingroup$

This issue has been litigated many times on meta.CV, and the specific thread referenced is clearly off topic. The question is only asking about how R works (what package / function, and an error message), not the underlying statistical issues. In addition, the explicit questions (the places where there is a "?"), are the title, "How to... in R... ?", and the final sentence / paragraph "Any suggestions?" (which is obviously too broad). The answer is only code and does not contain any statistical explanation, as you note.


Let's address some of the specific issues:

Should CV offer code help and/or tutorials as long as the models discussed are "far from trivial"?

No. This is not training or tech support for any software, including R. We also don't do tutorials. Software should make resources available (documentation, books, courses, etc.) to provide adequate assistance to users. Moreover, "far from trivial" is in the eye of the beholder. I teach introductory statistics classes, and many of my students would consider a simple logistic regression model to be "far from trivial" (I'm guessing you would not—at least, I do not).

Shouldn't we allow such questions here, since there doesn't seem to be a better place to ask elsewhere on the internet?

No. The universe is not required to provide a free, convenient answer somewhere. And our scope is not determined by the scope of other SE sites, or other sites generally. Whether or not any suggestion for where a question can be asked can be given, our scope is the same. To see the illogic in this suggestion, consider applying it elsewhere, say Seasoned Advice, SE's cooking Q&A site. Shouldn't you be allowed to ask how to fit a bivariate probit model there, since it isn't on topic elsewhere? No, because it is outside their scope. That may not seem fair, since that is a cooking site, and this is a statistics site. But that's the point, this site is for questions about statistics, not about software (even statistical software) and how to use it.

Shouldn't we allow this question, since it appears other questions have been allowed?

No. This is a common retort by people who want answers and believe CV should be obliged to give them. There are several issues: First, the site's standards and scope have evolved over time. Second, there are many questions asked every day. No one has read them all. I don't read all the new questions every day anymore. If I had seen the thread at issue here when it was asked, I would have closed it immediately. I have not yet investigated the threads you link, but it is possible they just slipped by and should be closed now. Third, you need to consider not only the few possible exceptions to the rule, but the many, many threads that have been closed in accordance with our policies. Moreover, whether or not any other off-topic thread remains open has little bearing on whether this off-topic thread should be closed.


I am very sympathetic to the fact that there are people who have questions and need help. (I have answered over 1.5k questions here, after all.) I am also sympathetic to the fact that people can find R inscrutable at times and really want help with that—I've been that person myself more than once. People commonly say they want to ask here because this is the best site with the highest quality information. Despite my sympathies, I am also responsible to maintaining the standards of this site, in terms of both boundaries and quality. In fact, part of the reason the site has high quality information is because we are strict about what we do and how we do it.

So how do I determine if something is on-topic, or if a seeming code question really "takes statistical expertise to answer"? It's fairly simple really. I do not ask if only someone with statistical training would have heard of the test / model at issue, instead, I ask myself what the OP needs explained. If what needs to be explained is a statistical idea, it's on-topic, if it's not, it's off-topic. For example, if the question had asked how seemingly unrelated bivariate probit regression works, or how any software could estimate it 'under the hood', that would have been on-topic. Sometimes people show that what appears to be the same model / test gets different answers from different software. The issue is always that the software are making different default assumptions or using different estimators. Other times the way the code is used or how it can go wrong is related to subtle and conceptually difficult aspects of the underlying procedures. Those are all statistical issues to be explained.


In conjunction with my statement that we sometimes miss questions that should be closed, but that their remaining open is not a valid argument that another off-topic question should be allowed, it may be worth laying out the history here. I became aware of this thread because the answer was flagged for deletion as not an answer by CV's standards. I agreed with that, but nonetheless I did not delete the answer. The problem is that the question is clearly off topic and no post that actually answered the question would be a valid CV answer. On the other hand, I recognize that the answer posted probably does help the OP (and future readers, since the thread won't disappear, even if closed). So I left the answer, but closed the question.

$\endgroup$
  • 2
    $\begingroup$ I am with @gung here. The thread in question is off-topic and the answer doesn't make it less so. I really don't want to have to judge in deciding whether to vote to close on what is a "non-trivial" model; there's no Platonic standard for that. $\endgroup$ – Nick Cox Apr 15 at 8:30
  • $\begingroup$ It is not necessarily about this particular question, nor is it a questioning of closing this very question. I didn't mean to give this impression, otherwise I'd have written "Why was this question closed?". I think @Glen_b's answer captures a little more the point I intended to make. $\endgroup$ – jay.sf Apr 15 at 8:54
  • 2
    $\begingroup$ Sure, it's an example, but it's intended to represent your viewpoint. I find the example as well as the general case unconvincing. (I'm highly supportive of questions about statistical software and have been posting on such in various places for $>$ 20 years -- but also of each site or forum setting its own policies.) $\endgroup$ – Nick Cox Apr 15 at 9:02
  • 4
    $\begingroup$ I think Glen_b makes a very compelling case in his answer. I agree with him (and hence disagree with this answer). $\endgroup$ – amoeba Apr 15 at 13:27
  • $\begingroup$ @amoeba, how much "statistical expertise" is required is just as fuzzy as how "non-trivial" the model is. What's difficult for one person is easy for another. $\endgroup$ – gung Apr 15 at 13:46
  • 2
    $\begingroup$ @gung I agree; this means there will always be some gray zone. Our rule is that questions that "take statistical expertise to answer" are on-topic. What we are doing here is to debate where to draw the line. $\endgroup$ – amoeba Apr 15 at 14:11
  • 1
    $\begingroup$ @amoeba, I'm not so much debating where to draw the line, but how to draw the line. Ie, what should "requires statistical expertise" be understood to mean? My contention is that a question "takes statistical expertise to answer" because the real answer to the question is explaining a statistical concept, not providing the code for a model that someone unfamiliar w/ statistics would not have heard of. There are many people who are unfamiliar w/ statistics who would not have heard of logistic regression. $\endgroup$ – gung Apr 15 at 14:19
  • 3
    $\begingroup$ So under that definition, the question cannot be understood (& thus answered) by someone without statistical expertise, & a question asking for the R code to fit a logistic regression model would be on-topic here. I'm saying no. The actual way to understand this is much simpler: is the needed answer explaining a statistical idea (even if the question is asked in terms of code on the surface)? If so, it's on-topic, if not, it is not on-topic. $\endgroup$ – gung Apr 15 at 14:21
  • $\begingroup$ I can see your point better now. May I suggest that you try to come up with a better wording for the help/on-topic and we discuss it here (or possibly start another Meta thread to discuss it)? I feel that the most natural interpretation of the current wording is the one developed in the Glen's answer. If we were to follow your interpretation then we should change that wording, but I'm not sure how exactly. $\endgroup$ – amoeba Apr 18 at 22:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .