20
$\begingroup$

I'm reviewing a paper for publication at the moment and I find the statistical methods in it quite questionable. But I've only got limited experience and don't want to call them out on something that isn't actually wrong either. So I'm wondering about the best way to ask for help on the main site. That is,(a) is it even ok to ask the equivalent of "Am I right that you can't analyse X using Y?", (b) how do I best go about keeping confidentiality, and (c) are there any other tips for how to best deal with this sort of problem on CV?

$\endgroup$
  • 7
    $\begingroup$ (+1) To a great question. I wouldn't normally suggest this, but you might consider posting a modified version of this question to the academia.SE main site. You might make it slightly less SE-centric (though mentioning it as a special case). There are some very thoughtful CS researchers active on that site, so you might even get a more specialized response. If you do post such a question, please provide cross-links in both places. $\endgroup$ – cardinal Mar 9 '13 at 18:42
  • $\begingroup$ I have very little experience in reviewing papers. Would it be inappropriate to just say to the editor something like "I have doubts about the validity of X, but my knowledge of statistics is insufficient to say for sure"? $\endgroup$ – markseeto Mar 13 '13 at 8:31
  • $\begingroup$ @mark999 I don't think it would be inappropriate, but I would be wary the concerns might just be brushed aside and, if they were justified, the paper be published with incorrect and misleading stats. Most stats in my area of CS are woefully inadequate, as neither authors, not reviewers, nor editors, seem to have the necessary statistical background to do them right. Therefore, unless you can back up your concerns with cold, hard facts, I wouldn't expect the editor to reject a paper because one reviewer thinks they might have gotten something wrong. $\endgroup$ – ThomasH Mar 13 '13 at 17:59
  • $\begingroup$ @ThomasH Thanks for your reply. $\endgroup$ – markseeto Mar 13 '13 at 19:30
  • $\begingroup$ Here is a question that does what you propose: stats.stackexchange.com/questions/14416/… $\endgroup$ – David LeBauer Apr 8 '14 at 21:53
18
$\begingroup$

Although @cardinal raises some good points, I think some questions could be germane to CrossValidated, as long as things are handled appropriately.

Please please please do not post an entire manuscript, or even big excepts from a manuscript, and ask for critiques. I think this is Cardinal's major concern and it is a very valid one. Among other things:

  • It is ethically dubious and violates your agreement with the authors/journal.
  • It might 'out' you as the reviewer (if you work in a field with anonymous reviews)
  • It is unlikely to work anyway: I doubt many people here would be willing to spend hours plowing through a manuscript in order to write your review for you.

All that said, I think it would probably be fine to post specific and general-interest questions, particularly if done in a way that doesn't reveal too much about the original manuscript. For example, I think a question like this would be okay:

I am reading a paper where the authors use a False Discovery Rate procedure to compensate for a large number of multiple comparisons. I was under the impression that the Benjamini–Hochberg procedure only worked for independent tests, but there are some arbitrary (i.e., positive and negative) dependencies between their tests.

Can this procedure be used on dependent tests? If not, are there any similar alternatives?

This question is answerable (in fact, it has already been answered here!), it is not too localized, and it probably would not violate any confidentiality agreement you have with the journal.

All that said, I think this a somewhat thorny issue and I hope others weigh in. Thank you for trying to be as conscientious a reviewer as possible! I'll buy you a drink if we're ever at the same meeting!

$\endgroup$
  • 4
    $\begingroup$ (+1) I wholeheartedly agree that generally applicable questions like the one you give as an example are acceptable, on-topic and interesting. If it can be done in this manner, that's great. I suspect many such questions would be hard to abstract out appropriately without losing important context, though. $\endgroup$ – cardinal Mar 9 '13 at 18:48
  • 1
    $\begingroup$ @Matt Your example is precisely the kind of question I had in mind - general applicability of statistical procedures in certain contexts. As it is, I had to already submit the review and noted my statistical concerns, which I had confirmed through general web searches. But, in this kind of situation in general, I would like to substantiate my review with a statistical reference as, in my area (general computer science), proper statistical knowledge is usually missing, even among reviewers, and I would rather point authors in the right direction than just telling them they're wrong. $\endgroup$ – ThomasH Mar 9 '13 at 21:02
  • $\begingroup$ @Cardinal: Glad you agree! I'm not sure about the question quality. On the one hand, I'd hope that people far enough in their career to referee manuscripts could also put together a solid question (vs, say, a college freshman asking for help with Stats 101). On the other hand, the peer review process is so rarely full of pleasant surprises :-/ $\endgroup$ – Matt Krause Mar 11 '13 at 6:57
8
$\begingroup$

With some regrets, I think this should be strongly discouraged on this, or any other, site.

First the regrets: An obviously conscientious reviewer taking the time to get feedback on a paper up for publication is probably much more likely to post a well-expressed, interesting statistical question. From a selfish perspective, I'd much rather read and think about such questions and they'd almost certainly increase overall site quality. I'm pleased that you would consider this venue and equally pleased that you took the time to ask about it here on meta first. There should be more reviewers like you in this world.

Now for the discouragement

As you know, reviewers are given a big responsibility and, as a form of compensation, get continuous first contact with new research advances. Confidentiality is one of their biggest and most important responsibilities. I think doing anything to potentially breach that, even unintentionally, is treading on very thin ice.

Here are my suggestions.

  1. Know your journal's policies. Many (maybe most) journals have strict policies on sharing manuscripts under consideration, even in private. While often these are "softly" ignored (some reviewers may pass off certain MS under review to graduate students), you should know the policies and be comfortable that whatever action you take meets them.
  2. Don't overburden yourself. Double check with the associate editor handling the paper to see if there is a separate statistical reviewer. In many fields, particularly those allied to medicine, this is common practice and lightens some of this burden placed on other reviewers. That said, if you have concerns about the statistical methodology and can clearly enunciate them, they will still be taken into account when the AE makes their decision.
  3. Think globally, act locally. If there is no statistical reviewer, or you simply want to do so, seek out some assistance at the local level. If in (or near) academia and you have access to a statistics department or a statistical consulting service, visit them or send an email to people working in relevant areas. This lets you carefully control access to the manuscript and you can even have them sign something more formal to protect confidentiality, if needed. If in industry, presumably you have contact with statistical practitioners that might be able to help in a similar manner.
  4. Use the network, but not the (inter)net. If there is a contributor here that you believe you can trust, you might contact them privately to see if they would be able to help. You should work out, in advance, what if any attribution should occur.
  5. As a very last resort, play the anonymous interested reader, with extreme(!) care and caution. If and only if an identical copy of the manuscript under consideration is posted to a public permanent archive (like the arXiv) or similar, then formulate your question as one of an interested reader unsure about some portion of the paper. This is still fraught with complications. You will have to wrestle with how to incorporate the feedback you get in a way that (a) meets your ethical thresholds, (b) meets the standards of required attribution as detailed by this site's copyright and content-sharing license and (c) will not subject you to the possibility that the author will see your report and then later happen to find the associated question on CV and be able to identify you. If you have any doubts, you should run it by the AE first. My opinion is that this is a minefield that is best avoided.

NB: I have consciously written this post employing a rather strong tone, but it merely reflects my opinion and shouldn't be taken as authoritative. Hopefully some of the thoughts will be helpful as you think this through.

$\endgroup$
  • 2
    $\begingroup$ Thanks for your insights and suggestions. Confidentiality is my main concern in this. I was only thinking of posing a general question (e.g. "I thought test X is inappropriate with a statistical design Y") but since I am trying to "ask in the negative", so to speak, I wasn't sure if that type of question is ok on CV without giving some motivation as to why I'd be asking. $\endgroup$ – ThomasH Mar 9 '13 at 21:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .