12
$\begingroup$

First, I considered posting this on the academia SE, but since this is more a question of statistical validity, I thought it more appropriate to post here. If there are similar questions/answers here or there, please direct me to them, as I've searched for a while and not yet found anything useful.

A colleague with the same background as me (Ecology) is now working in a different research field and has started a project. Both of us have concerns about the validity of tests and results used in similar studies (potential pseudo-replication, wrong choice of hypothesis test, not correcting for multiple comparisons), yet it seems pervasive in the field. Relevant journals have decent impact factors, and I assume that this field has resolved any issues that we perceive as problematic.

I recognize there are often differences in how data or results are presented from one discipline to another, and I may simply be overlooking something that is implied or established knowledge in a field different from mine. However, the 'Statistical Analysis' sub-heading within Methods often does not describe any actual methods; rather, it states (in a single sentence or two) the software used, and relegates the actual hypothesis test used and associated stats/metrics to Figure captions. This baffles me.

Journals typically require detailed statistical analysis methodology explaining the reasoning for choosing a particular test, that assumptions were checked and data transformed if necessary, type of post-hoc analysis, etc. However, I am encountering articles that lack this detail and present figures that (to me) look like results of ANOVA, but state that t-tests were used. Thus, I'd like to ask some questions here in hopes that the community can help me understand if/why these particular analyses are correct.

It is not my goal to find errors or call out publications, just to understand the analyses and address any gaps in my own knowledge. My question is:

Am I allowed to refer to a particular article(s) with the analysis I have a question about, or do I need to create a similar reproducible example with relevant context?

$\endgroup$
2
  • 8
    $\begingroup$ "Relevant journals have decent impact factors, and I assume that this field has resolved any issues that we perceive as problematic." Judging from my limited experience in two unrelated fields (psychology and forecasting), I would say this is a heroic assumption. $\endgroup$ Commented Aug 29, 2022 at 9:10
  • 2
    $\begingroup$ @E.Moore did you ask the question now? I am curious. $\endgroup$ Commented Sep 7, 2022 at 14:34

1 Answer 1

17
$\begingroup$

I think pointing to specific articles is fine. It's fairly common here for readers to quote sections of textbooks and ask for explanations of concepts that aren't clear. Questions calling out bad statistical arguments from scientists are not common but also accepted, so I think you're probably on safe ground asking to understand/critique methods from a paper.

That said, while quoting a paragraph or three should be fine, it would probably go too far if that text wasn't sufficiently comprehensive for readers to understand your question. If after reading your question, readers still need to refer to the paper or ask for more context to understand, it may be a poor fit for SE.

But some of us here are ecologists! So there may be members who have the background to answer even without a detailed explanation of field-specific concepts (though minimising or explaining such jargon is a good idea).

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .