We have a very large & widening gap between questions and answers. How do we fix it?

Question

Cross-validated is a question-and-answer site. I think it's worthwhile to ask "How well does this website serve people who ask (on-topic, answerable) questions?"

To answer this question in a quantifiable way, we can look at the gap between the number of (non-closed) questions asked in a month in total and the number of (non-closed) questions with answers. I have a query that does this. Behold! The chasm between the number of questions asked in a month and the number of answered questions yawns!

This plot clearly shows that, since 2016, the gulf between questions and answers is more than 1000 unanswered questions per month. For recent years, it appears that roughly half of questions are not answered. Moreover, the number of answers is on a gentle decline.

---

If we amend my query to break out questions with the [tag:neural-networks] tag vs those without that tag, we can see that the answer gap does not solely arise from neural networks. (It was easier to just do the post-processing in R.)

---

Glen_b asked a related question several years ago, Are we seeing a dramatic drop in answers per question? But I think that this is not the best way to measure how we are doing. Instead, I think we should ask "If a user asks an on-topic, answerable question, will that user get an answer?" Additionally, I think we can decisively answer Glen_b's titular question in the affirmative (look at the plot!), since a bunch of smart statisticians have all made their own graphs and measured their own quantities of interest over the past several years. Now that we have evidence that there is a problem, I think we should start looking for solutions.

Answer 1

We need more answers, and therefore answer-writers

There's no escaping that more answers to on-topic questions are required to close the gap.

More active participation in closure queues is a partial solution to our increasing volume of questions. However, a particularly distressing trend in this plot is that the number of answered questions per month is flat. Even if we more aggressively moderate poor questions and close duplicate questions, the plot in my post shows that the number of questions asked per month is linear in time, but the number of answers is basically the same. Moreover, even if we can identify more duplicates and close marginal and off-topic questions, that does nothing to address the fact that solid, on-topic questions often lack answers.

One of the major take-aways from Are we seeing a dramatic drop in answers per question? is that we need more "journeyman statistician" participation. I think this is a great idea, but I don't know how we get there!

• At the broadest level, Stack-exchange is undertaking to make the entire suite of websites a more accessible and friendlier place (discussed here We'd like your feedback on our new Code of Conduct!).

• Making SE more friendly seems like a good, long-term way to accumulate new users to answer questions to all sites. But is there anything that we, as statisticians and machine learners specifically, can do to gather more of our kind to answer questions?

If someone answers a question in a comment, copy the comment in to an answer

I'm as guilty of leaving answers-as-comments as the next person, but if a question is fully and completely answered by a comment, you might as well leave that same text as an answer.

Moreover, if you see this happen, you can just copy-paste the comment to an answer, with attribution to the original commenter. I have a boilerplate explanation that I append whenever I do this.

 I've copied @____'s comment as an answer because the comment is, more or less, an answer to this question. We have a dramatic gap between answers and questions. At least part of the problem is that some questions are answered in comments: if comments which answered the question were answers instead, we would have fewer unanswered questions.


 - https://stats.meta.stackexchange.com/questions/2242/are-we-seeing-a-dramatic-drop-in-answers-per-question/5320#5320

 - https://stats.meta.stackexchange.com/questions/832/comments-that-are-actually-answers

 - https://stats.meta.stackexchange.com/questions/5325/we-have-a-very-large-widening-gap-between-questions-and-answers-how-do-we-fix

If you don't particularly care about the rep, you could make your answer community wiki.

Don't let the perfect be the enemy of the good

Sometimes we get a question which is completely answered with two or three trite sentences. "How do I know whether weight decay improves my neural network?" is fully and completely answered by pointing out that the question is an empirical one, and a good way to gather empirical information is by conducting an experiment. I'd like to emphasize that the question "How do I know whether weight decay improves my neural network?" is wholly on-topic, (it's about a core topic in neural networks!), so closing it is not the right solution. Instead of closing the question, write the (short, clear, obvious) answer.

Another question asked how to label images for a neural network. The poster had taken photographs of objects of interest and had seen code examples of data loading functions which yield image data and image labels. The user didn't know that, at some point in time, a human had undertaken the tedious effort of telling a computer "this image is a cat" and "this image is a car."

Both of these are on-topic questions with obvious answers. Not all answers need to be, or can be, multi-paragraph efforts with figures, citations, graphs and equations.

If you can completely answer a question with a few clear sentences, do it!

More aggressively close overly broad, unclear and marginal questions

I think that a large portion of the problem is that there are quite a few poor questions, or marginally useful questions, or questions for which there is no known answer, or questions which ask the answerer to outline an entire statistical analysis project. We could do more to aggressively close these kinds of questions.

I specifically want to call out "marginal" as a genre of question. This is a question that asks a question that is, purely in the abstract, answerable. The dullest version of this question is "what's the best model for my data?" followed by a long explanation of how a database is configured, or how a website tracks user behavior. There surely is an answer -- but no one here is equipped to provide that answer without carrying out the actual analysis. This genre of question is the machine learning cousin to "What test should I run?"

Moreover, the strategy of writing a single, canonical answer to address a large audience of future readers is a losing proposition in the face of a rising tide of question for which the interest is entirely parochial. We need to close marginal questions.

As an aside, if anyone's looking for a quick way to get a lot of rep, we could use a canonical answer about back-propagation. I'm willing to award a princely bounty for such an answer (if I don't write one first). (Or, if one exists, we need to use that existing answer as a canonical duplicate target.)

(This seems to be a good place to note that I think we do a good job of directing questions which are purely about programming, math or obtaining data to their respective host sites, so good job team!)

We have a "moderator blind spot" on modern machine learning; we need a specialist

I don't want this to sound like criticism or disparagement. We have an exceptionally knowledgable and experienced group of moderators. However, I think there is a "blind spot" among them with respect to modern machine learning topics such as machine-learning, xgboost and neural-networks. I've adopted these tags as pet projects of mine, but I do notice that VTC efforts often get stuck at 4 votes, rather than the required 5.

I think that stats.SE would benefit from adding at least one moderator who specializes in these topics. (But machine learning is a large field! We might need more than one!)

Duplicate Patrol

Likewise, duplicate questions are a problem. It can be hard to passively identify and close duplicate questions without a near-omniscient knowledge of the thousands of question on CV. I often find that I can recall an answer to a question exists somewhere, but struggle to find the relevant thread.

To solve this, I've developed a strategy which I (lamely) call "duplicate patrol." It starts by looking at questions bearing some tag in which I am interested & feel that I am competent, and looking to identify "veins" of unanswered questions in need of an answer. If an answered version of that question can't be found, answering one of these question gives dupe targets for the others. When composing an answer, the goal is to identify common "themes" of questions, so as to address an issue which is of broad interest (and close several threads at once).

Answer 2

Downvote

Here is the mouseover text on the downvote button:

The question does not show any research effort; it is unclear or not useful

I'd recommend we be more liberal with actually clicking on that button. If a question is "not useful", downvote it! If someone else finds it "useful", they can always upvote it.

Why? This is why:

The system will automatically delete unlocked, unanswered questions that have negative score that are at least 30 days old.

Vegetation needs to be pruned so healthy growth can resume.

If you are concerned about the reputation loss from downvoting, remember that you get your rep back when the question is deleted.

Related from the same thread:

The system will automatically delete unlocked, unanswered questions on main (non-meta) sites with score of zero (or one, if the owner is deleted), fewer than 1.5 views per day on average, and fewer than two comments that are at least 365 days old.

(So: if you find an unclear and un-upvoted question that already has one comment, e.g., a request for clarification, think twice before adding another comment. (Or: just downvote per above.))

Answer 3

We do not have a problem with unanswered questions. We have a problem with low quality questions.

See the below image from this sql script which gives for the monthly answers and questions the 10-log of the average score.

• The overall decline for both questions and answers makes somewhat sense, since it takes time to acquire votes. (It does make the voting system somewhat biased for older questions and answers, but I can not imagine that this is very problematic)
• The relative decline shows that the questions are now being less valued (relatively) than the answers (this is true for other stackexchange sites as well). This may indicate that the questions are becoming, on average, less good/valuable (at least in comparison to the average quality of the answers)

So I believe we should not try to work on getting more questions answered, but instead on getting a better fraction of good questions.

---

Sidenote: We see that answers are getting relatively higher scored than questions, but this is not necessarily due to 'a change in the questions' (although I believe it is a factor). It might also be caused by answers getting better.

A possible mechanism is a selection effect. The users of stackexchange are being trained (by the voting system) not to give answers when they are likely not gonna attract votes.

People answering questions are more influenced by the voting system than people asking questions. The result of the voting system, aiming for bounties and hot questions, is that less attractive questions are also less likely gonna get an answer. That low score, less popular, questions are not getting answers is (nearly) an intentional part of the stackexchange formula.

Total/Unanswered votes for questions above some score: It is interesting to check out this SQL script, which looks at total questions vs unanswered questions considering questions with some minimal score. For questions with a score 2 or higher the gap starts to become smaller and questions with a score 5 or higher are nearly always answered.

answer rates for questions above some score The graph from this script indicates an interesting drop in answer rate for questions with zero votes. Questions with $\geq0$ and $\geq1$ score have most of the time very similar answer rates except for a sudden shift in the last year where questions with $\geq0$ score have a particular low answer rate.

This shift is not something due to a change in the behavior, but the questions with 0 score and 0 answers get removed by community after one year.

Questions with score >1 Thus the drop in answer rate has always been there and is an effect of the community requiring time to answer questions or remove bad questions (and hence you will see that more recent questions have a worse answer rate).

You can see these bumps in all of the graphs so these "trends" should not been seen as solely a change in behavior. Some better way of looking at it would be to select only the posts with score>0, in which case you still see a gap, but it is not increasing a lot in time (the absolute numbers might be even decreasing).

---

Update

When we look again at the script that compares the answered fraction as a function of the number of votes for the question, then we see that for specific categories (questions with 2 or more votes), the rate has been going up slightly.

Answer 4

If a user asks an on-topic, answerable question, will that user get an answer?

Most likely yes (currently ~60% of all questions are answered).

The question graph is biased because the unanswered question series assumes these are 'on-topic and answerable questions' which they are not. There are thousands of questions among that total which are duplicate, unclear, off topic, too broad, etc (i.e. questions that should already be closed).

I think there could be a rank of actions to be taken which consequently would make the next action in the rank easier to achieve.

1. Quickly/immediately closing questions is fundamental.

Let's set up the asking quality bar at an average high level. Let's use the SE system to our favour, not have it as an obstacle. One of the greatest advantages of SE in comparison to other types of 'forums' is that we don't need to answer the same and same types of questions again and again.

We need more users with the close vote privilege. Moreover, +3k rep users should not defer to immediately close questions which are duplicates, off-topic, unclear, broad, opinion-based. Even if the question is 'clear' to you, but someone requested clarifications in comments, if you judge the request is valid, then also vote to close it. Help the person who is seeking information to answer the question.

Let the following do their job:

• close reason notice messages;
• feedback in comments;
• the tour and help center pages; and

• the many faq posts we have on Meta CV;
  

  It is important we don't treat new users differently regarding this aspect. Everybody (who speaks English), if they want to, can improve their question with all the guidance currently available in the Stack Exchange network. A new [Ask Question Wizard] which will facilitate even more the task of asking is coming.

  We don't need to be impolite to (new) users to carry on closing questions firmly. We need to give priority to build a high quality repository Q/A (which interests to coletivity) over the single user experience.
  

2. Vote.

Currently, we have ~23k answers with zero score (most of them has zero votes). We don't need more specialists to vote on many of these. Let's vote for them!!

Compare how such types of answers increased from a previous related post: Criterion for "Unanswered" question - should it change?.

3. Help attract new answerers.

Alright, I would then agree with Xavier and Sycorax.

---

About advices (like the one below) to convert comments to answers and make them community wiki:

If someone answers a question in a comment, copy the comment in to a community wiki answer.

Please, dO convert the comment into an answer, but do not make it community wiki. Give the proper credit to the commentator in the answer, but don't make it community wiki just because of that. Many reasons to do so:

• You are already doing the community a favour by answering the question. Answers should always be provided as answers (agree with this or not, it is how it works). If you take the time to expand an answer based on a comment (or even copy it from a comment), and that information is useful, you should share the credits for it. The community will benefit from more users having privileges which are based on reputation (voting, casting close/reopen privilege, accessing moderators tools, etc). Give the proper credit to the commentator in the answer, but do not make it CW.
• If you make it CW you are inviting anyone to edit/own that answer as well. Not always this is wanted, because it might happen someone will add content you don't agree, and by making it CW it is more difficult to make your own point to prevail.
• That is not how community wiki is supposed to work. It is not a quick fix.

I won't blame users who answer in comments (despite I don't encourage this). I've read some people arguing sometimes they don't have the time to write a full answer, so they make comments to help OP (fair point). In other situations, the answer is simple and people don't want to have in their answer history a simple answer (so, they just comment to help OP).

Because such users decided to comment, instead of posting an answer they don't have the right to complain if someone else decides later to convert their comment into answers and earn reputation from it. If someone gets upset with this, the solution is very easy, post the comment as an answer next time or just don't comment. I doubt most users would be upset about this, though.

Remember that converting a comment to an answer is not a zero cost activity. You need to read everything, post and format the answer, and sometimes expand it. The casual reputation is deserved.

Answer 5

We need more answers, and therefore answer-writers There's no escaping that more answers to on-topic questions are required to close the gap.

In my opinion this is the single most important point, and all other ideas, while very good, will only have a marginal effect as they are not addressing the fundamental structural problem the site is facing. With that out of the way here are a few controversial ideas ranked from most impactful to least impactful in my opinion.

1) A paid, targeted advertising campaign

• At statistical conferences, in events, awards
• In specialized journals, magazines, books
• On related websites, forums

2) A free advertising campaign

• Volunteer SE users giving access to university groups, forums, events
• Links with non profit and teaching organizations
• Partnerships (e.g. with Khan academy, Datacamp or others)

3) A banner on the homepage

why note, Jimmy Wales does it all the time

4) A banner on questions with zero answers

5) An email campaign to less active past users

6) An official guideline for word of mouth campaigning

7) Unofficial guerrilla campaign

• Ask the top 100 active users to promote an organized, unofficial guerrilla campaign on campus, at work, during conferences.
• Ask professors and TA to discuss SE and its usefulness with their students in class

8) New bonuses, badges and recognition awards for new users

You guys had it easy answering questions on the usefulness of $R^2$ back in 2011 and getting 250 votes for a 15 line answer ! Now days getting 1000 points is a battle.

Since SE is an educational and most importantly very useful website for students and professionals alike, I don't see why we shouldn't aim to attract new contributors in any way we can.

Answer 6

I don't think the problem is solvable just by closing more questions or downvoting. I think all users start at a point where they just ask questions, then some of them become knowledgeable enough to start answering questions, and get hooked in the process, so the number of answers/time increases. Then, as in all human endeavours, they reach a point where they start losing interest and/or they have less time to spend in this activity, thus the rate of answers decreases (save for a few exceptions). Since new askers keep coming, but just a fraction of the askers become answerers, I think the only solution is to find a way to increase the number of answerers (maybe encourage short answers, by upvoting them) or to just accept that the gap will increase.

As Nick Cox correctly noted when commenting on the OP, "[..](downvoting, cleaning up, etc.) is as exciting as tidying up an untidy cupboard/closet when the alternative is watching one of your favourite movies or TV shows". Thus if we can't manage to increase the number of answerers, a fortiori I don't think we can manage to increase the number of downvoters/closers/etc. It's true that the last two activities take less time than writing good answers, but it's also true that they're harder to pitch to users.

Another possibility is that the gap is mostly due to an increase in low quality questions: Martijn Weterings' answer provides (now stronger) evidence in that sense. However, given that it takes little time to write crappy questions, and that there is a surge of interest for some topics related to Statistics & Machine Learning currently, I don't think the rate of crappy questions will go down easily.

UPDATE: with this query I retrieved the number of total & unanswered questions per month, for each of the 10 tags with more unanswered questions (today).

The result suggests that r, regression, machine-learning and time-series are the main drivers behind the growing chasm. Thus, concentrating the VTC & "recruit new answerers" efforts on questions with one or more of these tags should provide the most benefits. I remain skeptical of a sustained solution, i.e., I predict that answerers/closers will get bored of engaging in these endeavours soon before the question rate will drop. But let's see...

Answer 7

So what? My dear friends, are we perhaps worrying about a statistic without establishing that it has any particular meaning? One should be aware that we can often satisfy an OP with comments, and so the question becomes not if something has a formal answer that wouldn't really be an answer, but whether the OP got what was needed from the asking. So, one appropriate thing to look at may be total responses, because not every response, even the definitive ones, are "answers." Now there are multiple reasons for commenting rather than sticking out one's neck with an answer that may be discussed to death, downvoted on for all the worst reasons, picked apart mercilessly despite the irrelevance to the OP of the endless discussion, when the whole idea was to give some preliminary guidance to the OP.

On a different site, someone actually answered my question in a comment, which was nice, I then requested that that very highly ranked person convert that into an answer so that I can accept and upvote it. That person thanked me, did as requested and I did what I said I would. I found the thank-you to be most gratifying, that a high reputation person behaved in a genteel fashion made my day. Think about the mechanics of that for a moment and then perhaps consider which statistic actually means something. I think please and thank-you matters a lot. It sometimes is so difficult to get an answer out, and explain it, with such drawn out quibbles about various and sundry, and then have it deleted by people who just don't get where I am trying to go, and with emotions raging, no less, that the process is too often not wholesome. Perhaps we should give some thought to people trying to change answers to get the point across more frequently, like coauthors usually do, rather than competing for chits: "Mine--all mine---my precious." If we could somehow encourage more of that we would all feel better about actually answering, I think.