11
$\begingroup$

This question is rather related to an earlier question by Chris C about "learning from other people's reviews, and somewhat related to my previous post on how best to use the review queue.

I tend to be fastidious when reviewing, so only apply "No Action Needed" on about 20% of reviews (excluding skipped reviews). I wondered whether this was normal and having look at other reviewers, I found some apply "No Action Needed" to 80%+ of reviews.

I find it hard to imagine 80%+ of posts really need no action, based on what comes my way in the review queue. Of course, it's possible they apply the reverse strategy to me: skipping many posts with clear problems (maybe especially those that look hard to fix) and NANing the decent ones, whereas I tend to skip posts that look okay, superficially, but which I don't feel I can certify as "acceptable" since their content lies beyond my expertise. Or perhaps they see First Post review as little more than triage to weed out clearly poor or inappropriate content, so (if we imagine different "layers" of review we could perform) they may not take the review to quite the same level of depth:

Absolutely minimal

  • Appropriately flag
    • spam
    • content in a foreign language or patent nonsense
    • "I am having this problem too" or "link only" answers
  • Appropriately handle copy-and-pasted homework questions (if necessary inform about self-study tag, tell OP to show us where they are stuck, possibly vote to close in worst cases)

Clearly none of this material should survive First Post review, and all of this can be caught without subject knowledge expertise or even carefully reading through the post.

Requires effort but minimal expertise

  • Remove salutations, thanks and signatures
  • Fix spelling and grammar errors; formatting (particularly $\LaTeX$ and code formatting and indentation)
  • Make title clearer and more specific if necessary
  • Vote to close as unclear if a question is written in such poor English as to be indecipherable

This at least requires superficial engagement to read through the post's text, but not substantive expertise. I could perform all these actions without knowing anything about the topic of the post. Editing the title requires thought, but since OPs often pick one that's too generic, it might be improved simply by summarising the final point of their question.

Requires more expertise and/or more effort

  • Adding relevant tags — so long as I can recognise the subtopic, then even if I know little about the area I can usually improve on the OP's tags: they rarely know which tags we have available. On questions I don't substantively understand, sometimes I can't even pin down the topic well enough to find a relevant tag (this is most frustrating when the one tag the question has is clearly irrelevant, and really should be replaced by something else).
  • Checking for a duplicate question. This takes both effort (searching) and judgment (two questions might be presented and worded quite differently, but with the same core concept; it may take a degree of familiarity with the subject matter to identify that this is the case). I have a good memory for duplicates but rarely actively search for dupes unless I suspect one exists; perhaps I should be searching as a matter of course? In topics outside my area of expertise, I would struggle to identify whether two questions are duplicate or simply related (though this can be resolved by posting a link in a comment and letting more expert users identify whether this counts as a duplicate or not).
  • Check a question contains enough information to answer it — this is often possible even if I wouldn't want to venture an answer. The simplest case is "could you include (a sample of) your data", while with probability questions we may need to clarify "are the variables intended to be independent" or similar. If the question clearly doesn't contain enough information, it may be necessary to vote to close it as unclear. But it is often the case that, outside my area of expertise, I have little idea whether a topic expert would regard the question as sufficiently clear to be answerable.
  • Check an answer substantively addresses the question — even if I don't fully understand the answer (or indeed the question), it's generally possible, with a little familiarity with the topic, to ascertain whether an answer is at least germane to the question, and proposes some kind of solution. If not, it may be necessary to flag the question for moderator attention. If the answer is good, it should be upvoted, but I don't like doing this unless I understood (at least well enough to strongly suspect it's correct).

All of these require reading a question or answer closely and thinking in a critical and evaluative way about its content. The further its content lies from my sphere of knowledge, the less capable I am of performing this. Particularly when it comes to evaluating questions, perhaps these tasks are best handled by potential answerers, rather than reviewers.


Clearly reviews are multilayered, and we can't expect every reviewer to put total effort into every review, or only to review questions they are technically qualified to answer. On the other hand, CV is very fortunate that none of our review queues are Sisyphean tasks. If there is a post that lies outside our area of expertise, and there is no obvious issue to deal with, there seems no very strong reason (other than it adding one to our review queue score!) to say "superficially it looks okay, no action needed", if we could instead skip the review and have it passed on to a user with more (even if not masterful) topic expertise.

My philosophy generally has been "do what you can" — fix, flag or vote (up/down/close) — and if there is nothing (further) I can see to do, opt for "No Action Needed" (or "I'm Done") if I'm confident that the post is now good to go, but "Skip" if I think it would benefit from fresh eyes (perhaps even after my fixes). In many situations this means I get through the "requires effort but minimal expertise" before pressing Skip, and I will generally only NAN (or "I'm Done") if I reached the "requires more expertise and/or more effort" part of the checklist (though may not have done a full search for duplicates, completely understood an answer, etc). Is this an overly cautious approach? Should I feel I've "done enough" at a more minimal level of effort, and perhaps leave the rest for potential answerers to decide on? It certainly seems pointless (more work for everyone) to skip a post with no superficial flaws in the hope that the review will be "passed on" to someone with more subject expertise, if that post is then reviewed by someone who is only looking for superficial issues anyway. In this sense it would help if we were all reviewing to a somewhat comparable depth. Do my checklists leave anything important out? What's a realistic expectation for the depth of a review, particularly when you hit a post on the edge or even outside of your expertise?

$\endgroup$
  • 7
    $\begingroup$ +1. This question provides a very convincing illustration of how fastidious you indeed are :) $\endgroup$ – amoeba says Reinstate Monica Oct 19 '15 at 23:30
7
$\begingroup$

Some people certainly put more effort in than others.

A number of our regular reviewers (you among them) put in effort above the average (one regular reviewer who I won't name stands out above the rest in terms of time per review and on a number of other indicators of both effort and quality); the dedication of our careful reviewers and editors is a big part of why CV works as well as it does.

A few reviewers do cause me particular concern because their average review time is astoundingly short, especially on the first post queue. I am a pretty fast reader but I could not even read a post in the time some reviewers manage to click "Looks Okay" in, let alone ponder editing the post or adding tags, consider whether to make a comment to encourage improvements or request more information -- and that's not everything that should be considered for a first post.

Those same "fastest gun" reviewers also seem to miss many important problems, and that concerns me even more, because it defeats the entire purpose of the review.

Clicking "looks okay" on posts that are obviously not okay makes more effort down the track for others. Better to leave it in the queue.

[When I see someone new posting and their first post is in bad shape, but isn't in the queue any longer, you can bet I go and look who did that. And then start checking what else they did recently. Everyone makes the odd mistake and new reviewers should be expected to make more ... but there are limits to what's plausibly mere error and then we're getting into deliberate choice to let extremely poor posts through without so much as a comment.]

Moderators also get access to summary statistics on people's reviews. We can, if we choose, see information about review queues and how people perform. (It's not something I want to use to police the queues, I'd much rather trust our high reputation members to treat reviewing seriously.)

So I would ask the rapid-fire reviewers out there -- if you're not feeling like you want to put some effort into improving first-posts one day, just avoid the first post review queue at that time. If you just aren't sure about a post, it's okay to click 'skip'.

Sometimes posts are both brief and perfectly fine and you really can just click "Looks Okay" fairly quickly and move on, but it looks to me that more often than not a first post needs some improvement. If you're regularly letting 80% of new posts through as okay, you're certainly missing things - and some of them matter*.

My philosophy generally has been "do what you can"

That's exactly how it's supposed to work.

With some posts where I want to make some fixes but realize there's more to be checked, I have (on rare occasions) clicked "skip" in the queue, but opened the post in another tab to make the changes I want. Maybe I can fix the spelling, title and tags, add a comment for the poster but leave much of the body content for someone with more knowledge than me.

The first post queues needn't be perfect new user filters, but there are certainly times when they could work a little better.

I broadly agree with your points in your post; I am not sure there's any way to really assess how much effort we're "meant to" put in to the reviews beyond the advice we already have about the purpose of the queue and what seems broadly to work well.


*I have once or twice seriously considered whether friendly warnings or even more serious action (such as review suspensions) might be warranted in some cases. I'd prefer not to have to do that.

$\endgroup$
  • 3
    $\begingroup$ What stops you from a friendly warning in such cases? I don't review much myself (so this particular issue can hardly apply to me), but I don't think I would feel offended if a moderator pointed out to me that one or another aspect of my behaviour here can and perhaps should be improved; on the contrary, I think I would appreciate it. $\endgroup$ – amoeba says Reinstate Monica Oct 20 '15 at 0:32
  • 5
    $\begingroup$ Several things: (i) I find the process in issuing a warning rather daunting and quite limiting (you're enjoined from saying much of anything direct; often the central point is buried in polite platitudes); (ii) the great ease with which people who are otherwise good contributors can be offended; this is also a concern with private chat rooms (another process whose setup I find relatively daunting, especially if they're not a user of chat already). It's easy to give offense, and I've seen good contributors leave, or greatly reduce their contributions over what seem to be fairly mild comments... $\endgroup$ – Glen_b -Reinstate Monica Oct 20 '15 at 0:56
  • 5
    $\begingroup$ (ctd) ... from moderators or other high reputation users. It can be very difficult to judge exactly what might give offense in these situations. Lastly, I think it's important not to be too heavy-handed, and it can be hard to judge whether there's likely to be real benefit in acting rather than (say) discussing things here. I have complained (in general terms) about similar things here at least once. $\endgroup$ – Glen_b -Reinstate Monica Oct 20 '15 at 0:58
  • 2
    $\begingroup$ I tend to hang about on the main forum and voting whether to close much more than I look at first posts. (I don't know why; I do have an impulse to improve posts, as I edit quite a lot.) That gives me a biased perspective as I see many poor questions that are off-topic or very badly presented. So, how did they get through this process? I know many are not first posts. On the other hand, I am quite possibly a fast reviewer, but when you are checking whether something belongs and there are other votes and comments, that is often evident very quickly. $\endgroup$ – Nick Cox Oct 20 '15 at 10:44
  • 1
    $\begingroup$ It's a feature when people who react strongly to peer comment decide to leave the system without a big fuss. We do seem to have lost some valuable contributors that way, but no one promises to stay forever. $\endgroup$ – Nick Cox Oct 20 '15 at 10:48
  • $\begingroup$ Re "average time on a review" metric: in many cases, things that come up in the review queue are things I've seen before and already spent a bit of time thinking about (at least in the back of my mind): for example, because they are on the active posts page, or because I saw something on the review queue before but closed the tab (or skipped, then went back to it using my review history with "show skipped reviews" ticked). Sometimes I will clear a whole batch of reviews in a "couple of minutes" but it's usually been longer than that! Wouldn't surprise me if other people work in a similar way. $\endgroup$ – Silverfish Oct 24 '15 at 18:13
  • 1
    $\begingroup$ @Silverfish Certainly people do things like that -- and that will move the averages about... but then you won't have a bunch of reviews that have let through posts that anyone who even gives the quickest of glances to the posts would realize they needed to be acted on. "Short review time" isn't of itself a problem, if you're not letting through a lot of bad posts. $\endgroup$ – Glen_b -Reinstate Monica Oct 25 '15 at 0:06

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .