Are we seeing a dramatic drop in answers per question?

Question

From time to time I look at CV site-related stats of various kinds in various places.

I was alarmed to look at this query on Stack Exchange Data Explorer (SEDE), Which shows a dramatic drop in the answers per question and what seems to be an equally dramatic rise in unanswered questions (this query is in reverse time order - the third and fourth columns are what I'm commenting on). Edit: the last couple of months is what concerns me. In the most recent month (November), the zero-answer questions will go down a little in the next few days of course, but I fear not a great deal.

Looking at other queries shows that total answers per month are nearly flat*, but questions per month are going up fast.

*(there's a few days lag in the data so the small recent drop in total answers might not signify anything at all)

Prior to October, in a month we'd never averaged more than about 58 questions per day; in November we averaged 70 questions per day. That's not a small change. It's also about 65% higher than last November, while the number of answers grew about 17.5% in the same period.

Now the number of answers a question gets does rise over time, but generally speaking if a question isn't answered within about a week or so of posting, it's likely to remain unanswered.

If the site is to remain relevant we'll need to find a way to reduce the rise in proportion of unanswered questions.

To that end, I have some thoughts about what things might help:

short term:

• Avoiding "answering in comments", or at least going back later and making the comment into an answer if one isn't forthcoming. When all that's needed is a short answer, and the OP responds with "thanks! I see now!" ... it's unlikely to ever get any more action. We should try to reduce these (I've already been trying to do it less myself, but it can become a habit).

• Closing more questions that are unlikely to generate answers because they're poor, and the OP isn't going to fix them

long term:

• Ultimately, I think we need more people answering questions - many times more. While more experts would be great, I think we actually need more ... journeyman statisticians, and even in some cases relative novices -- and ways to encourage them to answer a few of the questions within their capabilities.

So my question is in two parts:

1. We have lots of users - how do we encourage greater participation from a lot more of them to answer questions?

2. Are there other important things we can do to lift the proportion of questions that get answers?

---

Edit: I managed to find some time to do some analysis of the questions and answers per month (I looked at September 2010 to November 2014, obtaining updated figures to be sure all of November was included), to try some analysis more or less along the lines suggested by whuber (I don't expect I fitted quite what he had in mind, though).

I fitted a simple model (quasi Poisson glm) with exponential trend (i.e. constant annual growth rate) and (multiplicative) monthly seasonal factors to each series. I totally ignored any serial correlation (but I wasn't looking at any significance tests so this shouldn't be a major issue - my aim was to capture the main effects).

I then plotted the question/answer ratio and the ratio of fitted values from the two models (since that would hopefully pick up the regular seasonal variation in the ratio). The fit is reasonably good, and I can only conclude that whuber's take on it is right - almost all of what we see in the Answer-per-Question relationship (marked in blue in the plot above) is just due to the combination of the relative difference in growth rates and seasonal effects.

The recent few points are a tiny bit worse than the overall trend but there's no suggestion whatever that this is anything more than typical variation about the model (not that I believe the model - but if we have that such a simple model does adequately account for what initially seems to be surprising, it is enough for me to discount the impression that anything new is happening without much better evidence).

That's not to say there's nothing to worry about -- the Answer per Question clearly is getting worse, and at a fairly rapid rate (it has dropped by half in a little less than 3 years). It's just that there's nothing that makes the more recent values substantively different from the existing trend. The answer to the question in the title seems to be "there's no particular reason to think so", but the questions in the body (what to do about the decline) remain.

---

A small update: looking at posted questions/answers over the last year, it seems that some growth in the ratio has continued, but fortunately at nothing close to the rate I initially feared we may have been undergoing. Currently looking back over the last year that rate seems to mostly in the region of 1.5-ish, but varies between about 1.3 and 1.65. (It's a number I wish was below 1, however.)

Answer 1

I am not sure how responsible the following is for the recent drop in answers per questions ratio (possibly not at all), but here is one thing I noticed and would like to bring it up in the context of this discussion.

I am following the [pca] tag and am trying to make sure that all of the questions related to PCA are either answered or closed. As there are not so many questions asked about PCA (maybe 1-2 per day), sometimes I feel like I have some time to go over the old unanswered PCA questions. I even have this silly dream of one day bringing the number of unanswered PCA questions to zero (from currently ~250).

Many of these questions are not very interesting. Many older questions are also quite specialized (e.g. asking about kernel PCA) and not interesting. What happens then, is that I write an answer which I think does quite a decent job in addressing the question and in the ideal world should be several times upvoted and accepted. But I often find that it remains with 0 upvotes (or 1 if I am lucky; probably also with ~0 views) and un-accepted (e.g. because the OP is not active anymore).

This is extremely frustrating. Why should I spend time answering these questions if nobody cares? Apart from me trying to empty the unanswered PCA queue, which sounds rather like a symptomatic internet addiction problem than a noble enterprise.

Especially frustrating is when the answer does not get a single upvote, so the question even remains officially "unanswered" even though I did answer it with the single purpose of making it answered!

I think by now I have some understanding of which threads are more likely to attract some attention, and the temptation is to answer only those. Update to clarify: My hypothesis is that something like that is happening at the population level: recent increase in the number of questions is due to an increased number of not-so-interesting questions, and potential answerers feel that such threads are simply not worth the effort. End of update.

P.S. On a related note, what about old uninteresting and possibly unclear questions from non-active users with zero upvotes (that probably constitute the bulk of old unanswered questions). Is it a good practice to go through them and flag to close? Is it a good practice to downvote them silently, hoping that if they remain with -1 votes, they will eventually be automatically deleted?

---

Update. After writing this, I stumbled upon this older answer by @whuber where he explicitly encourages to spend time writing few "high-quality, well-crafted" answers as opposed to many short ones. First, this directly goes against answering lots of questions just to have them answered and suggested carefully selecting only interesting questions. Second, if the number of not-so-interesting questions steadily increases, then this advice will inevitably lead to more and more unanswered questions.

Answer 2

As a very "journeyman" statistician myself, I feel somewhat qualified to post an answer expressing skepticism about the power of the less academically high-powered to make a serious dent in the "unanswered questions" issue.

I scored 99.5% in my undergraduate stats exams, but despite a fair amount of self-study since then, I've never taken a Masters, had further professional training, or sought membership/certification with a professional body. My own ability to contribute at CV is probably at the high end of what could be expected for a good undergraduate or recent graduate.

I find that there are enough things for me to do here that I can perform some useful services. I am confident that most of my answers represent "low-hanging fruit" that would have received a perfectly good answer without my intervention. In that respect I am simply saving someone else some work - though even this isn't entirely clear-cut, as I have seldom been the only person to answer a question.

Where I fall down compared to a full-time industrial or academic statistician, is the range of questions I feel capable to answer. I can't really comment about the Machine Learning as I have little experience of that field, and can't judge how many of those questions are essentially at entry-level (this may be large, if anecdotes about how many people started doing it right after taking an introductory MOOC are correct, but the plural of anecdote isn't data). Of the traditionally statistical questions, fairly few seem (a) clear, (b) accessible to someone with an undergraduate background, (c) sitting around for a long time without an answer.

There are a fair number of basic questions that don't stray beyond undergraduate level but are nevertheless hard to answer, because the OP is themselves confused about either what they want to achieve, or the statistical methods which they want to use. On close examination, many of the questions seem to be posted here because the OP has fundamental misunderstanding (sometimes conceptual, sometimes simply of notation or terminology) and once that is cleared up, most likely after discussion in the comments, there is "no question to answer". (I'm skeptical whether some of these questions should be allowed to fester after the OP's issue is resolved - in the best cases, the Q&A provides a good resource that tackles a common misunderstanding. Other times I suspect folk who arrive here via search engine because they were confused about a topic in the first place, are only going to leave more confused. This is a parenthetical issue, but still an important one, because it hints that the costs of the "delete really poor questions" strategy that @Glen_b mentions above may not be too high.)

I do some freelance statistical consultancy, and the sort of problems I encounter there rarely match even the "easy end" of CV questions. The most common outcome is that people end up paying me to explain what a t-test, or equivalent, is and what the results mean for their data (or simply what a "test" is, at all). Only on very rare occasions, either where they are doing something very important or unusually complex, have I felt it appropriate to suggest a potential client instead consults a chartered statistician (who would roughly charge what I do but with a zero on the end). In my experience most novices only want very basic answers to basic questions, and as a "journeyman" statistician I am more than qualified to help them. But because their knowledge is so limited, they're very rarely able to formulate the question in the first place. It can take considerable discussion to work out what they are trying to achieve, and only at this point am I able to suggest some possible approaches that could meet their needs. This initial two-way communication stage can be pretty hard work, and even with "comments", is not well-suited to an internet Q&A site. This is probably a good thing - CV would otherwise be deluged with "novice data analysis" questions. But it does mean that those which do end up here, are often unclear, poorly suited for the format, and hence hard to answer.

On harder questions, I'm aware that even experts here rarely answer beyond a couple of fairly specialised fields. I wonder if unanswered questions represent a gap between tastes or skill-sets of answerers and the requirements of posters? If so, more journeymen seem unlikely to make much impact. It may also be due in part, as CV becomes better known, to an increase in poorly phrased "novice questions" where the gap is of the patience of answerers to unpick what has been misunderstood, or what information the questioner is really trying to ascertain. These options aren't mutually exclusive. They are also testable hypotheses - take a sample of answerers and ask them to classify questions based on such things as whether they are in their field/out of field (perhaps with some finer distinctions such as "slightly out of my field but could probably answer if I did a bit of research"), clarity of the question, expected effort it would take to answer, and whether they would be likely to submit an answer (or at least a tentative or clarifying comment).

Answer 3

I should be talking about @Glen_b's data (this is a statistics forum, isn't it?), but I have little experience with forum reports and find that I want to see a much longer run to comment. So instead I will expatiate in a miscellaneous way on what I take to be the main underlying question here, how to make Cross Validated work better?

Optimists might look at the large volume of excellent threads here (and I note that the lively discussions can be as helpful as the many definitive answers) and want to claim success. Pessimists might look at the large number of questions here that never really got off the ground, including some distressingly broad or vague or confused questions that seemed doomed to get no response that would satisfy the poster, or anyone else, and want to claim failure. Both are right. In fact any dialogue is not so much between persons as between moods.

My first suggestion is that ways of dealing with the internet seem to be evolving very quickly and this may be behind some patterns of change that we are seeing. Users here vary from those who seem to have joined at about the start and remained committed ever since, through those like myself who have joined much more recently, to users who joined a few minutes ago to ask their question (and who in many cases will not return).

It's already getting hard for me to re-create my initial puzzlement at how CV worked. For some time between learning about this site and actually joining it I was shy because it looked so very complicated (why all those badges, for example? that just looked childish). Looking again at the home page and trying to see the site from a beginner's eyes, it seems more obvious to me that you can ask a question than how you can search for an answer. As often remarked, there is a tension between the short-term and individual focus "Here is my question: What's the answer?" and the long-term and collective goal of building up a repository or resource that can be consulted. We don't usually hear from those who found something interesting or useful on our pages and just benefited from that. We do usually hear and see those who are struggling even to ask a question that triggers the right response. Either way, making it more evident that there is a site to be searched and how to search it seems a priority.

Both here and elsewhere I feel that there is a steady drift upwards in the fraction of questions based on the assumption that we are an oracle capable of answering even cryptic questions. This is tied up with deep-seated educational and cultural attitudes towards statistics, even among people with backgrounds that otherwise are reasonably quantitative and/or mathematical. Some symptomology that is recurrent includes

I know little or nothing about statistics, so you need to explain in terms I can understand.

I need to know the right way to analyse this dataset, already in my hands.

I tried Googling this, but got nowhere.

I need to know what to say in my Master's research proposal.

By the way, I should stress that rudeness or discourtesy remain exceptional here; it's a credit to our moderators and more active users that it wouldn't be tolerated.

In blunt terms, it seems that we are often expected to compensate for those who never received much of a statistical education, or whose statistical education to date has failed to inculcate much understanding. And this is increasing. The syndrome is that the internet will give me an answer and therefore that it's the job of CV to provide that answer. Naturally, in broad terms that is what we do offer, but the small print doesn't include promises that we always can satisfy those who have not tried much to help themselves.

Do other fields experience the same expectations? Forgetting statistics for a moment, let me imagine that I want to learn some physics or economics or whatever, some subject where my knowledge is very slim. I would certainly put most emphasis on reading some books; it's not practical for me to take courses but if I were a student it should be; I would use what I could find on the internet. I would feel that I had to work long and hard before bothering strangers in a forum.

Incidentally, @Silverfish's answer had an interesting autobiographical flavour. I won't quite follow suit, but so that this is not interpreted as another patronising put-down by a statistician of those struggling with statistics, I stress that I am not a statistician by training or by (any!) qualifications. I am a primitive life-form, a geographer who uses statistical methods, but a lot. Low-hanging fruit? That's the best.

Answer 4

Using this simpler query I obtained the number of new questions and answers per calendar month for the whole time of CV existence. Here it goes (vertical dashed lines separate years):

The last two months do look alarming, but I would say that at the moment it is a bit hard to say if the number of questions is going to stabilize again or to continue to grow rapidly. I am afraid that @Glen_b is rightly worried, and it will grow fast. Note that as recently as in 2013 the Q/A ratio was still around 1.

Answer 5

Here are a couple quick thoughts:

1. I used to read down the 'recent badges' along the right side of the main page every day. I would click through to all the tumbleweeds and see if they could be answered, edited (& bumped), clarified through comments, or submitted for closure. I haven't done this much since the recent badges were moved to a new tab in place of 'hot questions' from other SE sites and empty space. (As you can tell, I don't care for this change.) Nonetheless, people can go to the badges tab and look through the recent tumbleweeds as a service to the site, and a way to help address this issue.
2. It is also easy to search the site for recent unanswered questions by clicking on the UNANSWERED tab and sorting by the newest subtab. Also searching the site with the syntax, say, answers:0 created:2014-11.. will yield the unanswered questions that were asked on or after November 1st, 2014. (I am surprised to see the number is 1,216 between then and today--December 1st, 2014.)

---

3. Regarding the issue of answers in comments, I think people should feel free (after a suitable time) to answer the question, even if it is essentially the same as what already exists in the comments. There are a couple of possibilities:

   • You can add a comment to the comment-answerer to convert their comment to an answer;
   • You can add your own answer with similar material if you feel comfortable (e.g., you knew that was the answer on reading the question and before seeing the comment);
   • Or, if you don't feel comfortable doing that, you can copy the comment into the answer, with a brief note explaining what you are doing, and make the answer CW. An example where I personally have done the latter is here: How to interpret regression coefficients in a log-log model?

Answer 6

Lately I have used a lot of my spare time to answer questions on CV. I do get a little frustrated when my answers receive no attention at all, such as here, here, here and here. Now I'm happy to concede that these answers aren't as detailed or high-quality as some, but I hope that I at least gave some help to the OPs and that they may be of some help to others in a similar position in future. In some instances I spend a really long time trying to work out what the OP is actually asking. If the answers are seriously lacking then I would hope for a comment from the OP or someone else in the community. But just to get nothing deters me from answering similar questions again.

Answer 7

User @amoeba pointed out to me that data about stats.SE is queryable from a simple API.

One way to re-frame this question is "Are more questions asked in a month than are answered?"

This plot answers that question resoundingly: "yes, answers are not keeping pace with questions." The difference between the yellow and blue lines reflects the number of unanswered questions in a month. For some recent years, that gap comprises more than a thousand questions.

You can see that the number of questions in a month continues to increase, while the number of questions with 1 or more answers in a month remains basically level. (Only non-closed questions are reported here; see previous revisions of this post for other data.)

For my part, I am taking an interest in the neural-networks, xgboost and random-forest questions & attempting to write canonical answers which can be targets for duplicates.

Answer 8

Besides the subjects already mentioned in the question, I can think of some suggestions to improve the site's statistics (and perhaps indirectly increase the number of participants too):

short term:

• support this proposal, which asks for votes not being deleted when a user leaves the site.
• once in a while, pay attention in the active tab, to questions bumped up by the community user. They are questions with 0 score answers and maybe the answer(s) did not receive enough attention. This query written by Raff also clip such questions.
• go forward to the announcer, booster and publicist badges. They help to improve the site's visibility and maybe it will attract more active participants.

Answer 9

Ultimately, I think we need more people answering questions - many times more. While more experts would be great, I think we actually need more ... journeyman statisticians, and even in some cases relative novices -- and ways to encourage them to answer a few of the questions within their capabilities.

I fit this description quite well as I have taken a sabbatical to study statistics and machine learning. I have undergraduate level in statistics and probability, and slowly building some insights into more advanced topics. My feeling is that there are quite a few people around here doing the same as me.

As a result I would like to share my experience and provide some answers to your question

How do we encourage greater participation from a lot more of them to answer questions?

Highlight "accessible" answers

• My view is that since the site has been around for a while now, a lot of the "easy" questions have been asked, and a lot of the "easy" answers have been given. When I say "easy" I mean from the perspective of someone with undergraduate level statistics or a self-learner

Here is an example, which is 4 years old: - Basic question about Fisher Information matrix and relationship to Hessian and standard errors

• On the other hand: I struggle to understand most of the new questions that are asked, let alone think of an answer. So I rarely go beyond the first page of the unanswered questions feed. There must still be some easier questions out there but they are lost to me.

An idea - create a tag / flag... for "easy" and "good first answers"

... and leave those to the relative novice.

A good example of this is the Sklearn Github issues page which has two dedicated tags: