5
$\begingroup$

I'm teaching myself how to do meta-analysis, and as a beginner to the subject I would like to read journal article examples of meta-analysis done well.

It's easy enough to find examples of meta-analysis, but I don't have the expertise to assess the quality of those examples. It would be helpful for me in my effort to self teach to read meta-analysis articles done well as well as explanations of what is being done rignt in those articles. The flip would also be useful (that is, to read articles where the meta-analysis could be greatly improved and an explanation of where they could be improved).

Is Cross Validated an appropriate place to ask for this information?

--

EDIT1: I tried to put this in a comment but ended up being too verbose for the comments section.

I do see how the question wouldn’t be a good fit for the website (too subjective, insufficiently finite). That said, I do think there is value in seeing examples of things done well. Guidelines are helpful, but to see everything pulled together in an example offers a different kind of insight. A list of anatomy drawing guidelines is helpful in learning how to draw (e.g., the head is one-eighth of the length of the body), but it’s helpful in a different way than actually seeing a figure drawing.

Now with figure drawings, most people can assess whether or not something looks right. With meta-analysis, that’s not so, and of the reviews of meta-analyses I’ve read (meta-meta-analyses?), it seems many meta-analyses fail to meet basic guidelines even though they passed the peer review process. I would guess that’s because the peer reviewers had subject matter expertise but not meta-analysis expertise, and I would think that the authors of those “needs improvement” meta-analyses would have been well served by seeing acceptable examples of meta analyses. I know I would.

Regarding the point that people do things differently. No doubt there’s variation in how things can be done acceptably, but I don’t think that makes individual examples of acceptably done meta-analysis useless. Some journals share example abstracts, figures, tables, etc. in their writer’s guides, and I think that’s helpful even if there are other ways to do things acceptably. I think it would be the same for seeing an entire example of a meta-analysis written up, but I am more than happy to agree to disagree on this point.

||||||
$\endgroup$
  • 4
    $\begingroup$ You can ask for recommended references either way with brief annotation. Such threads exist on CV and are often declared Community Wiki. A good pattern is to ask for nominations from each person who answers of one excellent article (or the opposite), or more if the spirit moves anyone. Here, and indeed elsewhere, it's pertinent to ask how long do you expect people to spend on answers? Thinking about that implies to me that expectations, or hopes, of a detailed tutorial or review are unlikely to be met. CV works best with very specific questions (that aren't trivial). $\endgroup$ – Nick Cox Sep 16 '19 at 18:06
  • 4
    $\begingroup$ There is some potential for such a question to be considered primarily opinion-based (which is a reason for closing a question). Who gets to decide what constitutes "done well" & by what criteria (other than their own opinions)? @NickCox is right that we sometimes let such things slide by making them CW, & that might happen here. Bear in mind that any published paper has passed peer review, which is the typical standard of sufficient quality (grumbles aside). Applied statistics textbooks typically discuss examples & you could look for a cited example the author thinks highly of. $\endgroup$ – gung - Reinstate Monica Sep 16 '19 at 18:29
  • 5
    $\begingroup$ You can also read some papers that have made suggestions (here's one, it cites MA's it finds well done). You could check out the PRISMA guidelines, MOOSE guidelines, &/or AMSTAR checklist. You might also want to check out the Cochrane reviews. $\endgroup$ – gung - Reinstate Monica Sep 16 '19 at 18:44
  • $\begingroup$ Sounds like the question would only be tenuously appropriate if at all appropriate. However, the links provided by @gung meet my needs quite nicely. Many thanks! $\endgroup$ – Angela Sep 17 '19 at 0:29
  • 4
    $\begingroup$ Questions which invite answers where each answerer's personal opinion may differ but where each is a valid answer - which sort of question will tend to generate "big list" answers - are usually not suitable questions for StackExchange sites and they normally close as too broad. With very careful phrasing (especially if there's some sufficiently specific metric of good), in some situations such questions can be made specific enough, but usually the criteria cannot be made specific. $\endgroup$ – Glen_b -Reinstate Monica Sep 17 '19 at 2:05
  • 2
    $\begingroup$ @Glen_b, has provided a more nuanced version of what I was getting at, Angela. My comment wasn't intended to be a firm no. It's just that it's a fine needle to thread. $\endgroup$ – gung - Reinstate Monica Sep 17 '19 at 2:18
4
$\begingroup$

Like the true academic I shall answer a slightly different question.

If you had asked the question and we (the relatively small collection of people who answer on the meta-analysis tag) had answered with our favourite examples would you be any further forward? My feeling is that you would not. It is of course perfectly true that the underlying mathematical theory of meta-analysis is identical across disciplines but custom and practice differ. Part of this is for good reasons. In the field where I work, health, there are often quite few studies and each gives rise to a single effect size. In other areas there may be many hundreds of studies or each study may give rise to dozens of effect sizes. The range of examples needed to cover all the bases would be large.

A further issue is that the field advances but practitioners are still taught what was current some time, possibly years ago. The advice to look at guidelines offered by @gung in a comment should at least avoid some of the pitfalls.

Although I have written this specifically about meta-analysis I believe that the issues would be quite similar for other fields of applied statistics and underpins the reluctance to have list answers on these sites.

||||||
$\endgroup$
  • 3
    $\begingroup$ Compare say regression where the spectrum of opinion runs from say econometrics which is in many ways an entire field centred on regression to the late David Freedman who asserted that he couldn't find really good examples of regression working as claimed. Or P-values, where experienced opinion ranges from wanting them banned to regarding them as central to statistical practice. (+1) $\endgroup$ – Nick Cox Sep 19 '19 at 10:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .