# Should questions about comparing the output from different statistical software be on topic here?

For instance, this question asks about why R and SAS give different results in a multi-level model. I think this sort of question is more likely to get an answer here than elsewhere. R list users may not know SAS; SAS users may not know R; programmers may not know about multilevel models.

What do others think?

• I voted for that linked question to "stay open". The criterium is simple: it needs statistical expertise to answer. – kjetil b halvorsen Mar 29 '18 at 20:31

I would lean towards allowing such questions. For example, I answered such a question here: Difference in output between SAS's proc genmod and R's glm. My general position is that whether a question should be viewed as off topic should be determined by what the OP needs explained: if the required explanation is statistical, it's best here, but if the required explanation is the code or about how the software works, it should be considered off-topic here. This rule usually leads to an unambiguous decision (although it may be opaque if you aren't familiar with the software or the statistical concepts at issue). There can be a gray area, though. When statistical software is designed, default choices are made by the developers that are statistically important. Often users will use the defaults without much thought or even being aware of them. In some sense, the answer could be, 'read the documentation for the two systems and notice that they make some different default choices'. But in another sense, to answer such questions well really does require statistical expertise to understand or answer, and constitutes a teachable moment. So this is an area where I will typically leave such questions open if they aren't egregious.

• (+1) "Egregious"? I suppose we should expect some archaism from ancient Chinese philosophers, but it may confuse some readers more used to the modern sense. – Scortchi - Reinstate Monica Mar 28 '18 at 15:35
• That's a typo, @Scortchi, it was supposed to have been "aren't egregious". – gung - Reinstate Monica Mar 28 '18 at 16:03
• Oh! Never crossed my mind that you'd make a typo, so I googled the word: en.oxforddictionaries.com/definition/egregious – Scortchi - Reinstate Monica Mar 28 '18 at 16:06
• I recall a question on why two different programs gave kurtosis values that differed by something like 2.999988. The answer was "Look at the documentation: one uses a formula with 3 subtracted and the other doesn't." The question, however, was on-topic because the issue at root was statistical, how quantities are defined. – Nick Cox Mar 28 '18 at 18:38
• @Scortchi "Egregious" is really not that uncommon, and it's a great word! :) – Alexis Mar 28 '18 at 19:31
• I think there are substantial gray areas in "how the software works". If it's about how the software works for programming (e.g. about macro variables in SAS; about the [] subset thing in R) then it's surely off topic. But what if it's about how a particular program does something statistical? – Peter Flom Mar 28 '18 at 19:31
• It's hard to say, @PeterFlom, it depends on exactly what "how a particular program does something statistical" means in each context. I've seen several questions that amount to 'why did the developers pick this default vs that one?', but that isn't really a question we can answer. The proper response is, 'go ask them why they made that choice'. The idea to consider the proper answer to a Q (ie, if it's a statistical explanation, on topic here, if not, not) works really well 99% of the time. – gung - Reinstate Monica Mar 28 '18 at 19:57
• @Alexis: Indeed; but it was its apparent use to mean "outstandingly good" that seemed to me to confirm gung's antiquity. Exactly the sort of slip you'd expect from St Germain, Orlando, Ayesha, & the like. Still not sure I really believe that was just a typo ... – Scortchi - Reinstate Monica Mar 29 '18 at 8:18

I'd say that if it is about understanding the output, as in the quoted example, it would be on-topic. On another hand, if the question asked

• where can be the $p$-value of some test be found on multi-page SPSS report,
• or, how to extract the statistic value from R object returned by some method, etc.

• I definitely agree with your two bullets. – Peter Flom Mar 28 '18 at 19:28

I think this is a good question.

Sometimes such questions and their answers can be illuminating, as when someone asked What is the difference between various Kruskal-Wallis post-hoc tests?, and, since I am the author of one of the four packages mentioned, I was able to provide some insight as to the statistical differences underlying the different results the OP was finding in my answer.

When might comparisons of output given by two or more pieces of software1 be inappropriate? Do any of the following circumstances not fit our remit? What others might likewise not be a good fit?

• When the comparisons are shopping for software recommendations?

• When the comparisons are about ease of use or implementation?

• When the comparisons are not asking about numerical differences, but instead layout and scope of output? (E.g., Stata's regress presents sums of squares and degrees of freedom details that are not output by, for example a summary of R's lm.)

1 Why does software come in "pieces?!"

• That's a good answer; I upvoted that long ago. Note however, that the 1st sentence is: "Understanding how these tests differ requires understanding the actual test statistics themselves." Ie, the explanation is statistical in nature. – gung - Reinstate Monica Mar 28 '18 at 20:01

+1 to all the answers provided. I think that all of them make good points and follow a general consensus that such questions appear "generally" on-topic.

I would like to add that from personal experience, I think that such questions are very educational. It is "trivial" for a learner to say for an LME model: "Yeah, obviously: $y \sim N(X\beta +Z\gamma, \sigma^2 I)$" or for ridge regression "simple L.A. stuff: $\hat{\beta} = (X^TX+ \lambda I)^{-1}X^Ty$". Nevertheless when going ahead and comparing solutions from different software implementation, small subtle differences not only become apparent, but also become appreciated for the impact they might have. Even something trivial like why Python's numpy.std and R's sd do not give the same result highlight differences that can be educational and meaningful.

Definitely yes.

If results can be reproduced across multiple platforms, we have a methodology. If all platforms give different answers, we at best have implementations... and we don't know which one is the right one, if any.

@gung suggested that the expertise required to answer questions like that needs to be statistical. Well... this is where statistics meets computation. A theoretical statistician would not be able to say why a logistic regression converges in stats::glm() but logit throws an error in Stata... hint: the likelihood is the same, but the treatment of infinite slopes and quasi-separation/perfect prediction is different. I believe statisticians would generally benefit from a greater exposure to computing issues, and that would help them understand the tools that they use.

Finally, near monopoly of R is all great for statisticians on this site to talk in common language. But hey, how many of us would be happy if your cell service were a monopoly??? I use R and Stata in my work, and SAS occasionally, and the crucial check for me that I am doing things right is when I can compare the results and see that they match.

• I really like this answer because it emphasises that the "stats/computing" dichotomy that we often use as a rule-of-thumb for what's on-topic can sometimes be misleading. Sometimes understanding the computing side is actually important in its own right, even from the statistician's point of view. – Silverfish Mar 31 '18 at 16:57