I'm reviewing a paper for publication at the moment and I find the statistical methods in it quite questionable. But I've only got limited experience and don't want to call them out on something that isn't actually wrong either. So I'm wondering about the best way to ask for help on the main site. That is,(a) is it even ok to ask the equivalent of "Am I right that you can't analyse X using Y?", (b) how do I best go about keeping confidentiality, and (c) are there any other tips for how to best deal with this sort of problem on CV?
Although @cardinal raises some good points, I think some questions could be germane to CrossValidated, as long as things are handled appropriately.
Please please please do not post an entire manuscript, or even big excepts from a manuscript, and ask for critiques. I think this is Cardinal's major concern and it is a very valid one. Among other things:
- It is ethically dubious and violates your agreement with the authors/journal.
- It might 'out' you as the reviewer (if you work in a field with anonymous reviews)
- It is unlikely to work anyway: I doubt many people here would be willing to spend hours plowing through a manuscript in order to write your review for you.
All that said, I think it would probably be fine to post specific and general-interest questions, particularly if done in a way that doesn't reveal too much about the original manuscript. For example, I think a question like this would be okay:
I am reading a paper where the authors use a False Discovery Rate procedure to compensate for a large number of multiple comparisons. I was under the impression that the Benjamini–Hochberg procedure only worked for independent tests, but there are some arbitrary (i.e., positive and negative) dependencies between their tests.
Can this procedure be used on dependent tests? If not, are there any similar alternatives?
This question is answerable (in fact, it has already been answered here!), it is not too localized, and it probably would not violate any confidentiality agreement you have with the journal.
All that said, I think this a somewhat thorny issue and I hope others weigh in. Thank you for trying to be as conscientious a reviewer as possible! I'll buy you a drink if we're ever at the same meeting!
With some regrets, I think this should be strongly discouraged on this, or any other, site.
First the regrets: An obviously conscientious reviewer taking the time to get feedback on a paper up for publication is probably much more likely to post a well-expressed, interesting statistical question. From a selfish perspective, I'd much rather read and think about such questions and they'd almost certainly increase overall site quality. I'm pleased that you would consider this venue and equally pleased that you took the time to ask about it here on meta first. There should be more reviewers like you in this world.
Now for the discouragement
As you know, reviewers are given a big responsibility and, as a form of compensation, get continuous first contact with new research advances. Confidentiality is one of their biggest and most important responsibilities. I think doing anything to potentially breach that, even unintentionally, is treading on very thin ice.
Here are my suggestions.
- Know your journal's policies. Many (maybe most) journals have strict policies on sharing manuscripts under consideration, even in private. While often these are "softly" ignored (some reviewers may pass off certain MS under review to graduate students), you should know the policies and be comfortable that whatever action you take meets them.
- Don't overburden yourself. Double check with the associate editor handling the paper to see if there is a separate statistical reviewer. In many fields, particularly those allied to medicine, this is common practice and lightens some of this burden placed on other reviewers. That said, if you have concerns about the statistical methodology and can clearly enunciate them, they will still be taken into account when the AE makes their decision.
- Think globally, act locally. If there is no statistical reviewer, or you simply want to do so, seek out some assistance at the local level. If in (or near) academia and you have access to a statistics department or a statistical consulting service, visit them or send an email to people working in relevant areas. This lets you carefully control access to the manuscript and you can even have them sign something more formal to protect confidentiality, if needed. If in industry, presumably you have contact with statistical practitioners that might be able to help in a similar manner.
- Use the network, but not the (inter)net. If there is a contributor here that you believe you can trust, you might contact them privately to see if they would be able to help. You should work out, in advance, what if any attribution should occur.
- As a very last resort, play the anonymous interested reader, with extreme(!) care and caution. If and only if an identical copy of the manuscript under consideration is posted to a public permanent archive (like the arXiv) or similar, then formulate your question as one of an interested reader unsure about some portion of the paper. This is still fraught with complications. You will have to wrestle with how to incorporate the feedback you get in a way that (a) meets your ethical thresholds, (b) meets the standards of required attribution as detailed by this site's copyright and content-sharing license and (c) will not subject you to the possibility that the author will see your report and then later happen to find the associated question on CV and be able to identify you. If you have any doubts, you should run it by the AE first. My opinion is that this is a minefield that is best avoided.
NB: I have consciously written this post employing a rather strong tone, but it merely reflects my opinion and shouldn't be taken as authoritative. Hopefully some of the thoughts will be helpful as you think this through.