For instance, this question asks about why R and SAS give different results in a multi-level model. I think this sort of question is more likely to get an answer here than elsewhere. R list users may not know SAS; SAS users may not know R; programmers may not know about multilevel models.
What do others think?
I would lean towards allowing such questions. For example, I answered such a question here: Difference in output between SAS's proc genmod and R's glm. My general position is that whether a question should be viewed as off topic should be determined by what the OP needs explained: if the required explanation is statistical, it's best here, but if the required explanation is the code or about how the software works, it should be considered off-topic here. This rule usually leads to an unambiguous decision (although it may be opaque if you aren't familiar with the software or the statistical concepts at issue). There can be a gray area, though. When statistical software is designed, default choices are made by the developers that are statistically important. Often users will use the defaults without much thought or even being aware of them. In some sense, the answer could be, 'read the documentation for the two systems and notice that they make some different default choices'. But in another sense, to answer such questions well really does require statistical expertise to understand or answer, and constitutes a teachable moment. So this is an area where I will typically leave such questions open if they aren't egregious.
I'd say that if it is about understanding the output, as in the quoted example, it would be on-topic. On another hand, if the question asked
then this would be about the software and reading the documentation, not about statistics.
Definitely yes.
If results can be reproduced across multiple platforms, we have a methodology. If all platforms give different answers, we at best have implementations... and we don't know which one is the right one, if any.
@gung suggested that the expertise required to answer questions like that needs to be statistical. Well... this is where statistics meets computation. A theoretical statistician would not be able to say why a logistic regression converges in stats::glm() but logit throws an error in Stata... hint: the likelihood is the same, but the treatment of infinite slopes and quasi-separation/perfect prediction is different. I believe statisticians would generally benefit from a greater exposure to computing issues, and that would help them understand the tools that they use.
Finally, near monopoly of R is all great for statisticians on this site to talk in common language. But hey, how many of us would be happy if your cell service were a monopoly??? I use R and Stata in my work, and SAS occasionally, and the crucial check for me that I am doing things right is when I can compare the results and see that they match.
I think this is a good question.
Sometimes such questions and their answers can be illuminating, as when someone asked What is the difference between various Kruskal-Wallis post-hoc tests?, and, since I am the author of one of the four packages mentioned, I was able to provide some insight as to the statistical differences underlying the different results the OP was finding in my answer.
When might comparisons of output given by two or more pieces of software1 be inappropriate? Do any of the following circumstances not fit our remit? What others might likewise not be a good fit?
When the comparisons are shopping for software recommendations?
When the comparisons are about ease of use or implementation?
When the comparisons are not asking about numerical differences, but instead layout and scope of output? (E.g., Stata's regress presents sums of squares and degrees of freedom details that are not output by, for example a summary of R's lm.)
1 Why does software come in "pieces?!"
+1 to all the answers provided. I think that all of them make good points and follow a general consensus that such questions appear "generally" on-topic.
I would like to add that from personal experience, I think that such questions are very educational. It is "trivial" for a learner to say for an LME model: "Yeah, obviously: $y \sim N(X\beta +Z\gamma, \sigma^2 I)$" or for ridge regression "simple L.A. stuff: $\hat{\beta} = (X^TX+ \lambda I)^{-1}X^Ty$". Nevertheless when going ahead and comparing solutions from different software implementation, small subtle differences not only become apparent, but also become appreciated for the impact they might have. Even something trivial like why Python's numpy.std and R's sd do not give the same result highlight differences that can be educational and meaningful.
