41
$\begingroup$

From time to time I look at CV site-related stats of various kinds in various places.

I was alarmed to look at this query on Stack Exchange Data Explorer (SEDE), Which shows a dramatic drop in the answers per question and what seems to be an equally dramatic rise in unanswered questions (this query is in reverse time order - the third and fourth columns are what I'm commenting on). Edit: the last couple of months is what concerns me. In the most recent month (November), the zero-answer questions will go down a little in the next few days of course, but I fear not a great deal.

enter image description here

Looking at other queries shows that total answers per month are nearly flat*, but questions per month are going up fast.

*(there's a few days lag in the data so the small recent drop in total answers might not signify anything at all)

Prior to October, in a month we'd never averaged more than about 58 questions per day; in November we averaged 70 questions per day. That's not a small change. It's also about 65% higher than last November, while the number of answers grew about 17.5% in the same period.

Now the number of answers a question gets does rise over time, but generally speaking if a question isn't answered within about a week or so of posting, it's likely to remain unanswered.

If the site is to remain relevant we'll need to find a way to reduce the rise in proportion of unanswered questions.

To that end, I have some thoughts about what things might help:

short term:

  • Avoiding "answering in comments", or at least going back later and making the comment into an answer if one isn't forthcoming. When all that's needed is a short answer, and the OP responds with "thanks! I see now!" ... it's unlikely to ever get any more action. We should try to reduce these (I've already been trying to do it less myself, but it can become a habit).

  • Closing more questions that are unlikely to generate answers because they're poor, and the OP isn't going to fix them

long term:

  • Ultimately, I think we need more people answering questions - many times more. While more experts would be great, I think we actually need more ... journeyman statisticians, and even in some cases relative novices -- and ways to encourage them to answer a few of the questions within their capabilities.

So my question is in two parts:

  1. We have lots of users - how do we encourage greater participation from a lot more of them to answer questions?

  2. Are there other important things we can do to lift the proportion of questions that get answers?


Edit: I managed to find some time to do some analysis of the questions and answers per month (I looked at September 2010 to November 2014, obtaining updated figures to be sure all of November was included), to try some analysis more or less along the lines suggested by whuber (I don't expect I fitted quite what he had in mind, though).

I fitted a simple model (quasi Poisson glm) with exponential trend (i.e. constant annual growth rate) and (multiplicative) monthly seasonal factors to each series. I totally ignored any serial correlation (but I wasn't looking at any significance tests so this shouldn't be a major issue - my aim was to capture the main effects).

I then plotted the question/answer ratio and the ratio of fitted values from the two models (since that would hopefully pick up the regular seasonal variation in the ratio). The fit is reasonably good, and I can only conclude that whuber's take on it is right - almost all of what we see in the Answer-per-Question relationship (marked in blue in the plot above) is just due to the combination of the relative difference in growth rates and seasonal effects.

enter image description here

The recent few points are a tiny bit worse than the overall trend but there's no suggestion whatever that this is anything more than typical variation about the model (not that I believe the model - but if we have that such a simple model does adequately account for what initially seems to be surprising, it is enough for me to discount the impression that anything new is happening without much better evidence).

That's not to say there's nothing to worry about -- the Answer per Question clearly is getting worse, and at a fairly rapid rate (it has dropped by half in a little less than 3 years). It's just that there's nothing that makes the more recent values substantively different from the existing trend. The answer to the question in the title seems to be "there's no particular reason to think so", but the questions in the body (what to do about the decline) remain.


A small update: looking at posted questions/answers over the last year, it seems that some growth in the ratio has continued, but fortunately at nothing close to the rate I initially feared we may have been undergoing. Currently looking back over the last year that rate seems to mostly in the region of 1.5-ish, but varies between about 1.3 and 1.65. (It's a number I wish was below 1, however.)

$\endgroup$
25
  • 5
    $\begingroup$ (+1) Actually, long-term stats do not show a dramatic drop in answering rate: they show a long-term, consistent drop. (A couple years ago we dipped below the 1:1 A:Q ratio.) Recent stats show a moderately sharp acceleration in closed and deleted questions, which may go a long way towards helping us understand why a larger proportion of the (non-closed) questions remain without answers. Note that our site exhibits a seasonal pattern in which the rate of questions (and proportion of bad ones) peaks during exam time (early December, late April). $\endgroup$
    – whuber Mod
    Dec 1, 2014 at 15:21
  • 1
    $\begingroup$ @whuber, maybe I'm missing something. Why would an acceleration in closed Qs explain why a larger proportion of non-closed Qs go w/o answers? $\endgroup$ Dec 1, 2014 at 16:12
  • 4
    $\begingroup$ @gung The closed/not-closed dichotomy masks a distribution of question quality: having a larger proportion of closed questions is an indication of a shift of the distribution of question quality towards lower quality overall, suggesting that a larger proportion of the questions that are not closed are nevertheless of such low quality that few people, if any, care to answer them. $\endgroup$
    – whuber Mod
    Dec 1, 2014 at 16:23
  • 3
    $\begingroup$ @Andre "Journeyman" refers to someone who has completed an apprenticeship in a trade, intends to become a master someday, and is actively working towards that status by assisting a master. This was the pattern of the medieval European guild structure (which persists today in such professions as engineering, medicine, law, and teaching). The word is often used loosely to refer to anyone between the novice and expert stages of learning who has some capability of carrying out independent work. $\endgroup$
    – whuber Mod
    Dec 1, 2014 at 16:28
  • $\begingroup$ @whuber thanks, yes, that's what was intended by the word. I wanted to convey someone beyond novice, not yet expert (to be honest, I'm not even sure I quite count as an expert yet), but I didn't want to get into being pointlessly prescriptive about whether that meant holding a degree or having a certain amount of experience; journeyman seemed to cover the general sense I needed. $\endgroup$
    – Glen_b
    Dec 1, 2014 at 22:00
  • 3
    $\begingroup$ Re "Answering in the comments". Comments are great for communication/clarification, and perhaps to drop a relevant hyperlink or hint. But how far complete does an "answer" have to be, before it's worth putting as an "answer"? I tend to build my CV answers up over time by editing - laying down a basic idea, then coming back to flesh it out if I feel it deserves it. Bit of a waste if the OP has already got the point, but better for those who arrive by search engine later. How welcome is this on CV? On SO, perhaps because it's higher traffic, I've had criticism for such "incremental answering". $\endgroup$
    – Silverfish
    Dec 1, 2014 at 23:36
  • $\begingroup$ In SO incremental answering can be a problem because there's so much competition to answer, and a benefit to 'going first'. Here on CV I'd actually encourage it, for three reasons. First, he number of questions that get more than one answer is pretty modest; better to encourage a reasonable answer that becomes a very good answer later than discourage an answer. Second, answers in SO tend to be "contained" much more so than here - often the question is specific and a complete answer can be brief. Here a brief answer can be helpful but leave out important caveats ... (ctd) $\endgroup$
    – Glen_b
    Dec 1, 2014 at 23:59
  • 1
    $\begingroup$ @Glen_b: It would be great if you could update your post with a long-term figure showing amount of new questions and new answers per month over last several years. Would be very interesting to see (and relevant for the discussion), and it seems that you have all the scripts ready to produce it easily. $\endgroup$
    – amoeba
    Dec 4, 2014 at 10:07
  • 1
    $\begingroup$ @amoeba Well, I don't have any scripts of my own; it's more a matter of finding someone else's script on data.stackexchange and using that. I have once or twice managed to edit a script to do something else, but I have no real idea what I'm doing when I do that. I do intend to try to do it, though. $\endgroup$
    – Glen_b
    Dec 4, 2014 at 10:48
  • 1
    $\begingroup$ Although moderators have access to graphics of this sort, we are specifically enjoined from sharing them with others. We can, however, summarize what they show: that is the basis of the remarks in my first comment. $\endgroup$
    – whuber Mod
    Dec 4, 2014 at 13:45
  • 2
    $\begingroup$ @whuber: Not that it matters a lot, but in the figure I posted below (covering all 4.5 years) I don't see any sign of regular upward swings in the number of questions asked every November... I wonder why. $\endgroup$
    – amoeba
    Dec 4, 2014 at 22:18
  • 2
    $\begingroup$ @amoeba It's hard to see such patterns when there is so much going on. Fit a trend and look at the residuals. It might also be a little clearer in the weekly data, because the onsets and ends of the swings cross monthly boundaries and are affected by the holidays. I admit I'm eyeballing this myself, because all I have are the graphics, not the data--but they do show more detail than the one you have posted. $\endgroup$
    – whuber Mod
    Dec 4, 2014 at 22:50
  • 3
    $\begingroup$ @whuber I've done some analysis, and I am forced to agree with you - the differing long term trend and differing monthly seasonality in Q's and A'a account for most of what we see (and the residual isn't nearly large enough to conclude something new might be going on). I have summarized what model I fitted to what data above, and the conclusions I draw from it in an edit to my question (since it doesn't really respond to what to do about worsening Answers per Question, I figured it was better in the question). $\endgroup$
    – Glen_b
    Dec 8, 2014 at 4:06
  • 2
    $\begingroup$ Another factor to contemplate, Glen, is revealed by data on closed questions. Until this year there were only the barest upward wiggles in the trend during end-of-semester frenzies (Mar-Apr, Nov-Dec). This year the changes during those periods are substantial: +3.5 extra closed questions daily in early April, +4.5 extra during the last month. My interpretation: perhaps a small number of classrooms are being induced to post garbage. (At the moment, we are closing more questions each day than we were receiving in total three years ago.) $\endgroup$
    – whuber Mod
    Dec 11, 2014 at 18:45
  • 2
    $\begingroup$ Answer in comments: we need to impose a requirement that nobody can be tenured in any department of statistics unless they generate a reputation of at least 2K per year. $\endgroup$
    – StasK
    Dec 11, 2014 at 23:21

9 Answers 9

34
$\begingroup$

I am not sure how responsible the following is for the recent drop in answers per questions ratio (possibly not at all), but here is one thing I noticed and would like to bring it up in the context of this discussion.

I am following the [pca] tag and am trying to make sure that all of the questions related to PCA are either answered or closed. As there are not so many questions asked about PCA (maybe 1-2 per day), sometimes I feel like I have some time to go over the old unanswered PCA questions. I even have this silly dream of one day bringing the number of unanswered PCA questions to zero (from currently ~250).

Many of these questions are not very interesting. Many older questions are also quite specialized (e.g. asking about kernel PCA) and not interesting. What happens then, is that I write an answer which I think does quite a decent job in addressing the question and in the ideal world should be several times upvoted and accepted. But I often find that it remains with 0 upvotes (or 1 if I am lucky; probably also with ~0 views) and un-accepted (e.g. because the OP is not active anymore).

This is extremely frustrating. Why should I spend time answering these questions if nobody cares? Apart from me trying to empty the unanswered PCA queue, which sounds rather like a symptomatic internet addiction problem than a noble enterprise.

Especially frustrating is when the answer does not get a single upvote, so the question even remains officially "unanswered" even though I did answer it with the single purpose of making it answered!

I think by now I have some understanding of which threads are more likely to attract some attention, and the temptation is to answer only those. Update to clarify: My hypothesis is that something like that is happening at the population level: recent increase in the number of questions is due to an increased number of not-so-interesting questions, and potential answerers feel that such threads are simply not worth the effort. End of update.

P.S. On a related note, what about old uninteresting and possibly unclear questions from non-active users with zero upvotes (that probably constitute the bulk of old unanswered questions). Is it a good practice to go through them and flag to close? Is it a good practice to downvote them silently, hoping that if they remain with -1 votes, they will eventually be automatically deleted?


Update. After writing this, I stumbled upon this older answer by @whuber where he explicitly encourages to spend time writing few "high-quality, well-crafted" answers as opposed to many short ones. First, this directly goes against answering lots of questions just to have them answered and suggested carefully selecting only interesting questions. Second, if the number of not-so-interesting questions steadily increases, then this advice will inevitably lead to more and more unanswered questions.

$\endgroup$
13
  • 1
    $\begingroup$ One thing I find interesting (and positive) about this post is that it suggests a mechanism (in this case, tags) for matching skills of answerers with needs of questioners is performing its purpose, at least in this area. ` $\endgroup$
    – Silverfish
    Dec 3, 2014 at 12:56
  • 2
    $\begingroup$ I generally try to write for a dual audience: the original poster (in "drive-by" cases, possibly long gone) and search engine arrivals. In numerical terms, the latter are probably more important, which suggests answers should be written with a broad scope rather than overly specific to (though still attempting to meet) the particular needs of the OP. This aligns with whuber's comment. I'd argue the same applies to whether a question is "interesting" - the OP thought it interesting enough to ask, search engine arrivals thought it interesting enough to search, even if potential answerers didn't. $\endgroup$
    – Silverfish
    Dec 3, 2014 at 13:01
  • 6
    $\begingroup$ "Uninteresting" isn't a listed close reason, & with good reason. But there are certainly unanswered questions that are unclear, too broad, duplicates, &c. - these should be flagged if they weren't at the time they were asked. Placing a bounty on them is an option for interesting ones you don't want to answer yourself. $\endgroup$ Dec 3, 2014 at 13:04
  • 4
    $\begingroup$ @Scortchi: It is indeed not a close reason, and I agree that it should not be, however Stack Exchange network does automatically delete questions that post factum turned out to be uninteresting to the community. Under a current system, when I look at an old question with 0 upvotes and 0 answers (that I personally find uninteresting), I can more or less certainly make it disappear by downvoting it and waiting 30 days. I am wondering what community thinks about such an action, but maybe I should ask it as a separate meta question. $\endgroup$
    – amoeba
    Dec 3, 2014 at 15:19
  • $\begingroup$ Bullet number two from my answer partially address some of the stated concerns. $\endgroup$ Dec 3, 2014 at 15:31
  • 1
    $\begingroup$ +1, I'm going to vote for "noble enterprise" over "internet addiction". It may interest you to know that there is, in fact, an unanswered question on meta.CV here: Should the standards for closing questions be relaxed somewhat for old questions?, which was intended to get at some of these issues. If you were to provide an answer of comparable quality, I predict it would get an upvote ;-). $\endgroup$ Dec 3, 2014 at 15:53
  • 4
    $\begingroup$ I've been puzzled by that post by @whuber. Naturally, I'd write a spectacularly good answer rather than a mediocre one given a straight choice between the two. But there are many questions that won't interest many people and even competent, concise, clear answers won't get more than a few upvotes. I don't think we're expected to ignore such questions. $\endgroup$
    – Nick Cox
    Dec 3, 2014 at 16:25
  • 3
    $\begingroup$ @NickCox: I think this reflects the difference in attitudes of different people. Here is a top 100 list of users sorted by average answer score among the users with over 50 answers. Cardinal is leading with a whopping average of 13. Whuber is pretty close to the top with 7. But the OP Glen_b, who is approaching whuber in total rep, has average of only 3.6, showing that he answers a lot, often getting only a few upvotes, if at all. Yours is 3.5, mine is 2.8. If everybody acts like whuber suggests, there will be a lot of unanswered questions. $\endgroup$
    – amoeba
    Dec 3, 2014 at 17:07
  • 4
    $\begingroup$ That's a interesting table. Thanks for adding data to the discussion! @Glen_b certainly gets a noble prize for helping lame dogs and lost cats. $\endgroup$
    – Nick Cox
    Dec 3, 2014 at 17:39
  • $\begingroup$ If we look to the tag's badge criteria, two of them are: i) minimum upvotes in the tag, ii) minimum number of posts in the tag. So users won't earn them with few outstanding answers nor with a lot of poor answers. So, I think the SE site wants a mix of quantity and quality (outstanding answers, but also good/ /short/objective answers). Because if it was only about quantity, SE would be like a yahoo answers, and if it was only about quality, perhaps people would prefer to invest time on a different subject, like publishing papers, etc. $\endgroup$ Dec 4, 2014 at 15:42
  • 7
    $\begingroup$ Re the update: That is an excellent point about quality vs quantity. Perhaps it comes down to whether we're here to keep answering variations of the same old questions or to generate some uniquely high-quality posts and tackle original, interesting, difficult questions. Although we would like to do all those things, it might come down to making a choice. One way to help accomplish both is to identify the duplicates of more questions and close them as duplicates rather than waste energy on answering them again, and again, and ... . $\endgroup$
    – whuber Mod
    Dec 5, 2014 at 2:01
  • $\begingroup$ @whuber, completely agree. I tried to make a similar statement in this answer. $\endgroup$ Dec 11, 2014 at 19:50
  • $\begingroup$ As a user of statistics, I came across lots of answers I dont understand the first sentence. I really cant decide whether to upvote those answers. I guess knowledge gap between askers and answerers is huge in this site. How many guys fit neural network understand the math behind it? $\endgroup$ Jul 11, 2018 at 7:47
20
$\begingroup$

As a very "journeyman" statistician myself, I feel somewhat qualified to post an answer expressing skepticism about the power of the less academically high-powered to make a serious dent in the "unanswered questions" issue.

I scored 99.5% in my undergraduate stats exams, but despite a fair amount of self-study since then, I've never taken a Masters, had further professional training, or sought membership/certification with a professional body. My own ability to contribute at CV is probably at the high end of what could be expected for a good undergraduate or recent graduate.

I find that there are enough things for me to do here that I can perform some useful services. I am confident that most of my answers represent "low-hanging fruit" that would have received a perfectly good answer without my intervention. In that respect I am simply saving someone else some work - though even this isn't entirely clear-cut, as I have seldom been the only person to answer a question.

Where I fall down compared to a full-time industrial or academic statistician, is the range of questions I feel capable to answer. I can't really comment about the Machine Learning as I have little experience of that field, and can't judge how many of those questions are essentially at entry-level (this may be large, if anecdotes about how many people started doing it right after taking an introductory MOOC are correct, but the plural of anecdote isn't data). Of the traditionally statistical questions, fairly few seem (a) clear, (b) accessible to someone with an undergraduate background, (c) sitting around for a long time without an answer.

There are a fair number of basic questions that don't stray beyond undergraduate level but are nevertheless hard to answer, because the OP is themselves confused about either what they want to achieve, or the statistical methods which they want to use. On close examination, many of the questions seem to be posted here because the OP has fundamental misunderstanding (sometimes conceptual, sometimes simply of notation or terminology) and once that is cleared up, most likely after discussion in the comments, there is "no question to answer". (I'm skeptical whether some of these questions should be allowed to fester after the OP's issue is resolved - in the best cases, the Q&A provides a good resource that tackles a common misunderstanding. Other times I suspect folk who arrive here via search engine because they were confused about a topic in the first place, are only going to leave more confused. This is a parenthetical issue, but still an important one, because it hints that the costs of the "delete really poor questions" strategy that @Glen_b mentions above may not be too high.)

I do some freelance statistical consultancy, and the sort of problems I encounter there rarely match even the "easy end" of CV questions. The most common outcome is that people end up paying me to explain what a t-test, or equivalent, is and what the results mean for their data (or simply what a "test" is, at all). Only on very rare occasions, either where they are doing something very important or unusually complex, have I felt it appropriate to suggest a potential client instead consults a chartered statistician (who would roughly charge what I do but with a zero on the end). In my experience most novices only want very basic answers to basic questions, and as a "journeyman" statistician I am more than qualified to help them. But because their knowledge is so limited, they're very rarely able to formulate the question in the first place. It can take considerable discussion to work out what they are trying to achieve, and only at this point am I able to suggest some possible approaches that could meet their needs. This initial two-way communication stage can be pretty hard work, and even with "comments", is not well-suited to an internet Q&A site. This is probably a good thing - CV would otherwise be deluged with "novice data analysis" questions. But it does mean that those which do end up here, are often unclear, poorly suited for the format, and hence hard to answer.

On harder questions, I'm aware that even experts here rarely answer beyond a couple of fairly specialised fields. I wonder if unanswered questions represent a gap between tastes or skill-sets of answerers and the requirements of posters? If so, more journeymen seem unlikely to make much impact. It may also be due in part, as CV becomes better known, to an increase in poorly phrased "novice questions" where the gap is of the patience of answerers to unpick what has been misunderstood, or what information the questioner is really trying to ascertain. These options aren't mutually exclusive. They are also testable hypotheses - take a sample of answerers and ask them to classify questions based on such things as whether they are in their field/out of field (perhaps with some finer distinctions such as "slightly out of my field but could probably answer if I did a bit of research"), clarity of the question, expected effort it would take to answer, and whether they would be likely to submit an answer (or at least a tentative or clarifying comment).

$\endgroup$
12
  • 7
    $\begingroup$ Wait, what? I've seen a large number of very high quality answers from you -- some way above the level I felt I was describing. We get many self study questions, for example, that are easily at a lower level than the questions you seem to be able to give excellent contributions on. $\endgroup$
    – Glen_b
    Dec 1, 2014 at 22:05
  • 4
    $\begingroup$ The problem of people not knowing what it is they really want - at least not in a way that they can formulate or express - is a very common one in statistics (and, no doubt, in many other fields). I think helping people with that is part of the skill set statisticians need to develop. I've occasionally driven people asking "just a quick question" to distraction with what seem to be endless naive questions (to figure out what the underlying problem is, how their data were collected, what their variables actually measure) ... before we even begin to discuss what kind of analyses may be needed. $\endgroup$
    – Glen_b
    Dec 1, 2014 at 22:13
  • $\begingroup$ If it's undergraduate-level statistics (and well-removed from machine learning) my understanding is very good indeed. Graduate level stuff (asymptotics, measure theory, anything beyond basic Bayesian approaches, some of the heavier regression stuff especially shrinkage) I will read with interest on CV and elsewhere but won't attempt to make any contribution on. That sharp cut-offs means my contributions probably represent the maximal extent that someone with strong undergraduate-level background can add. Definitely non-negligible but rarely much use on the "hard" unanswered questions. $\endgroup$
    – Silverfish
    Dec 1, 2014 at 22:16
  • 1
    $\begingroup$ Of "easy" questions, "self-study" seem easiest to construct an answer to as the question was usually posed, clearly, by a lecturer. (Can be hard to write enough of a hint, without answering the whole homework!) The "easy" questions that turn out to be "hard work" quite often seem to be when the OP doesn't really know what they want. This is familiar from my sideline at the bottom-end of the consultancy market: I've had people offer to fly me across the country to basically do a $\chi^2$ test, but if they'd known enough to know that's what they wanted, they'd never have called me to start with! $\endgroup$
    – Silverfish
    Dec 1, 2014 at 22:26
  • $\begingroup$ Your penultimate comment also isn't at all far off describing my own limitations on questions (I could maybe add a few topics to yours ... but a couple of those I've picked up basic knowledge of as a result of answering questions on CV). $\endgroup$
    – Glen_b
    Dec 1, 2014 at 23:22
  • $\begingroup$ @Glen_b The distinction between "expert" and "novice" seems to be "all things are relative"! Looking through the recent "tumbleweed" activity, I see lots of machine-learning, some hard (grad level) stats, quite a few questions I couldn't even decipher (maybe garbage, maybe just very technical and tersely worded) and some "novice data analysis" questions that could only have been resolved with lots of communication/clarification from the OP (which with "drive-by" Qs presumably needed to happen earlier). I saw no high school/undergrad self-study Qs but my sample may not have been representative. $\endgroup$
    – Silverfish
    Dec 1, 2014 at 23:45
  • $\begingroup$ Try also the "no answers"-unanswered questions (since views, comments and votes may keep questions out of Tumbleweed status, but still unanswered). You may want to scroll a bit. Also the recent-unanswered (which includes questions with answers not yet upvoted) may be a good spot to either give an answer or upvote one. If we could get a couple of dozen more people nearly as good as you, each answering 1 or 2 questions a day, I think things would improve quite noticeably (and not just in terms of answered questions). $\endgroup$
    – Glen_b
    Dec 2, 2014 at 1:15
  • 1
    $\begingroup$ A bit off-topic, but nevertheless: I have had zero encounters with statistical consultancy, but what you describe (in your answer and in the comments) sounds amazing. Who are your clients? I cannot even imagine. Students? Academic researchers in, say, biology? People from industry? What industry? Who needs to run a $t$-, or a $\chi^2$ test so badly as to pay somebody flight tickets and fees, and why? Sorry for probably very naive questions. $\endgroup$
    – amoeba
    Dec 2, 2014 at 22:38
  • $\begingroup$ @amoeba: I'll try to address that while keeping on-topic! There seems to be a substantial pool of people and organisations (eg small businesses) who have generated some data for some purpose, need to do something fairly basic with it, but either don't know what, don't know how, or don't have time (or importantly, software licenses) so want to outsource it. Data from market research, training scheme evaluations, academic work - quite diverse sources. It's all got to be analysed, and when managerial/academic deadlines loom, plenty of data users are prepared to pay for help. $\endgroup$
    – Silverfish
    Dec 2, 2014 at 23:30
  • 1
    $\begingroup$ @amoeba I think it's relevant to this discussion as my experience suggests there are many people with very basic, but often urgent, stats needs. Nick Cox makes the excellent point that folk expect the Web to provide answers these days. And even better, for free! If the type of people who seek out low-level statistical advice from the likes of me were all to discover SE, I suspect this site would be utterly deluged. Moreover they'd be very difficult to help, as they find it hard to formulate their own needs. Such a change in composition of questioners could result in (or explain) lower answer % $\endgroup$
    – Silverfish
    Dec 2, 2014 at 23:42
  • $\begingroup$ Another "hard to help" issue is lack of technical skills. Most of my clients use Excel as a giant table, maybe with pretty shading and borders, but have yet to discover the joys of =SUM(). The better-skilled got used to pressing buttons on student editions of SPSS at uni, but with no licence at work, can't run the analyses they used to. (For many folk, stats is SPSS - this often shows with new SPSS-tag Qs on CV.) Saying "download R and install this package" is no "solution" in such cases. Fortunately most "novice" questioners on CV still exhibit (to me) surprisingly high technical skills. $\endgroup$
    – Silverfish
    Dec 3, 2014 at 0:04
  • $\begingroup$ @Silverfish re SPSS users... seeing/hearing phrases like "run a correlation" (or "run-an-X" more generally) seems to be a dead giveaway for a lot of fairly novice SPSS users. I have definitely seen the tendency to equate SPSS with statistical analysis among some such users. $\endgroup$
    – Glen_b
    Jun 21, 2016 at 2:39
15
$\begingroup$

I should be talking about @Glen_b's data (this is a statistics forum, isn't it?), but I have little experience with forum reports and find that I want to see a much longer run to comment. So instead I will expatiate in a miscellaneous way on what I take to be the main underlying question here, how to make Cross Validated work better?

Optimists might look at the large volume of excellent threads here (and I note that the lively discussions can be as helpful as the many definitive answers) and want to claim success. Pessimists might look at the large number of questions here that never really got off the ground, including some distressingly broad or vague or confused questions that seemed doomed to get no response that would satisfy the poster, or anyone else, and want to claim failure. Both are right. In fact any dialogue is not so much between persons as between moods.

My first suggestion is that ways of dealing with the internet seem to be evolving very quickly and this may be behind some patterns of change that we are seeing. Users here vary from those who seem to have joined at about the start and remained committed ever since, through those like myself who have joined much more recently, to users who joined a few minutes ago to ask their question (and who in many cases will not return).

It's already getting hard for me to re-create my initial puzzlement at how CV worked. For some time between learning about this site and actually joining it I was shy because it looked so very complicated (why all those badges, for example? that just looked childish). Looking again at the home page and trying to see the site from a beginner's eyes, it seems more obvious to me that you can ask a question than how you can search for an answer. As often remarked, there is a tension between the short-term and individual focus "Here is my question: What's the answer?" and the long-term and collective goal of building up a repository or resource that can be consulted. We don't usually hear from those who found something interesting or useful on our pages and just benefited from that. We do usually hear and see those who are struggling even to ask a question that triggers the right response. Either way, making it more evident that there is a site to be searched and how to search it seems a priority.

Both here and elsewhere I feel that there is a steady drift upwards in the fraction of questions based on the assumption that we are an oracle capable of answering even cryptic questions. This is tied up with deep-seated educational and cultural attitudes towards statistics, even among people with backgrounds that otherwise are reasonably quantitative and/or mathematical. Some symptomology that is recurrent includes

I know little or nothing about statistics, so you need to explain in terms I can understand.

I need to know the right way to analyse this dataset, already in my hands.

I tried Googling this, but got nowhere.

I need to know what to say in my Master's research proposal.

By the way, I should stress that rudeness or discourtesy remain exceptional here; it's a credit to our moderators and more active users that it wouldn't be tolerated.

In blunt terms, it seems that we are often expected to compensate for those who never received much of a statistical education, or whose statistical education to date has failed to inculcate much understanding. And this is increasing. The syndrome is that the internet will give me an answer and therefore that it's the job of CV to provide that answer. Naturally, in broad terms that is what we do offer, but the small print doesn't include promises that we always can satisfy those who have not tried much to help themselves.

Do other fields experience the same expectations? Forgetting statistics for a moment, let me imagine that I want to learn some physics or economics or whatever, some subject where my knowledge is very slim. I would certainly put most emphasis on reading some books; it's not practical for me to take courses but if I were a student it should be; I would use what I could find on the internet. I would feel that I had to work long and hard before bothering strangers in a forum.

Incidentally, @Silverfish's answer had an interesting autobiographical flavour. I won't quite follow suit, but so that this is not interpreted as another patronising put-down by a statistician of those struggling with statistics, I stress that I am not a statistician by training or by (any!) qualifications. I am a primitive life-form, a geographer who uses statistical methods, but a lot. Low-hanging fruit? That's the best.

$\endgroup$
5
  • 1
    $\begingroup$ (+1) Regardless of qualifications you are clearly "expert" at a professional level. I suspect there is a strong link between your extensive professional practice, and your many excellent contributions on CV: STATA, data-vis and regression are presumably regular tools of your trade. In contrast, I tend to use CV to keep my skills sharp in those areas that I don't see in my (more limited) professional practice! Perhaps your academic background means your stats skills are "narrow but deep", but that still puts fruit in reach that someone with shallow (even if wider) skills can't. $\endgroup$
    – Silverfish
    Dec 2, 2014 at 19:49
  • $\begingroup$ Thanks for the compliments. I do have 23 years' experience of using Stata intensively. I am strongly interested in graphics and use regression-type models a lot. In terms of following questions, I mostly tend to look at what's new first and don't follow particular tags. If a link throws up something older, sometimes there is evidently work to be done. $\endgroup$
    – Nick Cox
    Dec 2, 2014 at 20:00
  • 1
    $\begingroup$ That does make me return to pondering whether unanswered "good" questions reflect skillset mismatches between askers and answerers, or perhaps a weakness in the filtering system by which answerers can find skill-matching questions (I guess this is the point of the tags, but I'm not sure how well they work). I think the thrust of your piece is a very sound explanation of the unanswered "bad" questions. I have seen suggestions on metaSO to emphasise the "possibly matching answers": this discussion reflects similar concerns to yours here. $\endgroup$
    – Silverfish
    Dec 2, 2014 at 20:01
  • 7
    $\begingroup$ Statistics has always been a subject largely made up of people from other areas. Wilcoxon was a chemist, Spearman a psychologist, Tukey a topologist, and so on. When I was a student I heard it said "nobody goes to university intending to be a statistician". [It wasn't quite true then, and it's somewhat less true now, but it may be so to a greater degree than almost any other profession.] $\endgroup$
    – Glen_b
    Dec 2, 2014 at 20:31
  • 4
    $\begingroup$ The CV mechanism works great for situations where there's an unambiguous answer. Most of my LaTeX Google queries land on TeX.SE, which provide exact answers to my questions like "How do I set this stupid table up". I am happy to upvote both the question and the answer to indicate their relevance. It looks like in statistics, the answers may be not be as clear or straightforward. As far as I can tell, TeX.SE runs at about 1 A/Q, but with twice the votes of the top questions. I can't believe that there are more people who know LaTeX than those who analyze data. $\endgroup$
    – StasK
    Dec 12, 2014 at 0:01
14
$\begingroup$

Using this simpler query I obtained the number of new questions and answers per calendar month for the whole time of CV existence. Here it goes (vertical dashed lines separate years):

enter image description here

The last two months do look alarming, but I would say that at the moment it is a bit hard to say if the number of questions is going to stabilize again or to continue to grow rapidly. I am afraid that @Glen_b is rightly worried, and it will grow fast. Note that as recently as in 2013 the Q/A ratio was still around 1.

$\endgroup$
9
  • $\begingroup$ +1 Thanks for the extended data. $\endgroup$
    – Glen_b
    Dec 4, 2014 at 14:35
  • 9
    $\begingroup$ I'd guess at two key trends. As time passes, it becomes harder to ask a really interesting new kind of question. Also, as CV grows and becomes more widely known, it attracts more lower-level questions which are less interesting to active people here. The first lame dog is cute; the 100th lame dog you've seen before. $\endgroup$
    – Nick Cox
    Dec 4, 2014 at 15:13
  • $\begingroup$ This is a guess, but I suspect if you were to subtract off the numbers of closed and deleted questions, then that alarming-looking upswing at the end might look more moderate. You also ought to be plotting these data on a vertical log scale, because the growth in most metrics on SE sites is exponential. By highlighting the summer months (Jun-Jul-Aug-Sep) and holidays (late Dec through lat Jan) you will also reveal a clear seasonal cycle in the questions asked. $\endgroup$
    – whuber Mod
    Dec 4, 2014 at 22:56
  • $\begingroup$ I would be interested in seeing an updated version of this chart. $\endgroup$
    – Sycorax Mod
    Jul 2, 2018 at 19:58
  • $\begingroup$ @Sycorax Good idea! Actually you can see a version of it directly here data.stackexchange.com/stats/query/59302/… $\endgroup$
    – amoeba
    Jul 2, 2018 at 20:17
  • $\begingroup$ @amoeba Thank you for pointing that out! I had no idea such a thing had already been "canned," although it obviously makes sense that it ought to be. $\endgroup$
    – Sycorax Mod
    Jul 2, 2018 at 20:25
  • $\begingroup$ @Sycorax I am a bit confused though because I don't see Q>A in 2014 on the graph at data.SE.com, in disagreement with the plot I posted here back in 2014. $\endgroup$
    – amoeba
    Jul 2, 2018 at 21:17
  • $\begingroup$ I’m wondering if there’s a difference between the two, perhaps one includes closed questions? Or something else? $\endgroup$
    – Sycorax Mod
    Jul 2, 2018 at 22:55
  • $\begingroup$ @amoeba I've updated my answer to include several refinements upon my original plot/the original query that you furnished to me. $\endgroup$
    – Sycorax Mod
    Jul 3, 2018 at 18:48
9
$\begingroup$

Here are a couple quick thoughts:

  1. I used to read down the 'recent badges' along the right side of the main page every day. I would click through to all the tumbleweeds and see if they could be answered, edited (& bumped), clarified through comments, or submitted for closure. I haven't done this much since the recent badges were moved to a new tab in place of 'hot questions' from other SE sites and empty space. (As you can tell, I don't care for this change.) Nonetheless, people can go to the badges tab and look through the recent tumbleweeds as a service to the site, and a way to help address this issue.
  2. It is also easy to search the site for recent unanswered questions by clicking on the UNANSWERED tab and sorting by the newest subtab. Also searching the site with the syntax, say, answers:0 created:2014-11.. will yield the unanswered questions that were asked on or after November 1st, 2014. (I am surprised to see the number is 1,216 between then and today--December 1st, 2014.)

  1. Regarding the issue of answers in comments, I think people should feel free (after a suitable time) to answer the question, even if it is essentially the same as what already exists in the comments. There are a couple of possibilities:

    • You can add a comment to the comment-answerer to convert their comment to an answer;
    • You can add your own answer with similar material if you feel comfortable (e.g., you knew that was the answer on reading the question and before seeing the comment);
    • Or, if you don't feel comfortable doing that, you can copy the comment into the answer, with a brief note explaining what you are doing, and make the answer CW. An example where I personally have done the latter is here: How to interpret regression coefficients in a log-log model?
$\endgroup$
6
$\begingroup$

Lately I have used a lot of my spare time to answer questions on CV. I do get a little frustrated when my answers receive no attention at all, such as here, here, here and here. Now I'm happy to concede that these answers aren't as detailed or high-quality as some, but I hope that I at least gave some help to the OPs and that they may be of some help to others in a similar position in future. In some instances I spend a really long time trying to work out what the OP is actually asking. If the answers are seriously lacking then I would hope for a comment from the OP or someone else in the community. But just to get nothing deters me from answering similar questions again.

$\endgroup$
2
  • 1
    $\begingroup$ Sympathies, and I suspect most of the active people here experience this some of the time, but there is no obvious solution here. I hadn't noticed any of these questions: we're all doing this in our "spare time" and pick and choose threads to study capriciously. But if there's a common element it's OPs new to the site being clueless and/or casual about closing threads and accepting questions. Short of superhero(ine) powers to freeze their computers until and unless they show more courtesy, what can be done? $\endgroup$
    – Nick Cox
    Jul 10, 2016 at 10:48
  • 7
    $\begingroup$ One day you will log on Robert and find you have been awarded the stats.stackexchange.com/help/badges/50/tenacious tenacious badge. If your luck holds you will qualify for the stats.stackexchange.com/help/badges/51/unsung-hero unsung hero badge. You can then join the International CV unsung heroes association which holds its convention in a dimly lit cellar where to a thernody on an alto saxophone mouldy cheese is eaten whil the bread is salted with tears. $\endgroup$
    – mdewey
    Jul 10, 2016 at 11:06
6
$\begingroup$

User @amoeba pointed out to me that data about stats.SE is queryable from a simple API.

One way to re-frame this question is "Are more questions asked in a month than are answered?"

This plot answers that question resoundingly: "yes, answers are not keeping pace with questions." The difference between the yellow and blue lines reflects the number of unanswered questions in a month. For some recent years, that gap comprises more than a thousand questions.

You can see that the number of questions in a month continues to increase, while the number of questions with 1 or more answers in a month remains basically level. (Only non-closed questions are reported here; see previous revisions of this post for other data.)

enter image description here

For my part, I am taking an interest in the , and questions & attempting to write canonical answers which can be targets for duplicates.

$\endgroup$
14
  • 1
    $\begingroup$ There's a distinct levelling off in the answers there; perhaps even a slight drop off. $\endgroup$
    – Glen_b
    Jul 3, 2018 at 0:51
  • $\begingroup$ Are the questions in these statistics also including the closed/deleted questions? $\endgroup$ Jul 3, 2018 at 7:22
  • 1
    $\begingroup$ @MartijnWeterings I haven't conducted a thorough analysis. I conjecture that each graph presented in this thread uses a slightly different query. I haven't studied the problem enough to say either way which query is measuring the most relevant statistics. Your hypothesis is reasonable -- we have more questions, but also a larger portion of closed questions. $\endgroup$
    – Sycorax Mod
    Jul 3, 2018 at 14:16
  • $\begingroup$ @Sycorax I find this topic intriguing. But it is difficult. This is related to an earlier topic on which I gave an answer (stats.meta.stackexchange.com/a/5101/164061). That answer was very incomplete. Later on, I found out that time was an extremely import parameter in determining the responses (explaining the mixture of time-dependent distributions resulting into the time-independent Whuber-Amoeba-Polarization-Effect, I did not add this insight to that answer). The orientation of Q&A has changed (differnet orientation rarding the use of formulae). $\endgroup$ Jul 3, 2018 at 14:50
  • $\begingroup$ You recent change to the post, with the update including 'non-closed' post shows more strongly (basically the image is the same but now we now it is independent from bad/closed questions) that the answerrate has decreased (bit absolute and relative). To be honest, hoping that all is untrue, I am still looking into models that might apply to the controverse (e.g. when questions get answered with a delay than naturally the number of answers is less than the number of questions, when the number of questions is increasing. I would wish to state this some day using the database, but I have no time) $\endgroup$ Jul 3, 2018 at 14:56
  • $\begingroup$ @MartijnWeterings I'll have to review that post -- I recall that I skimmed it, but that was some time ago. In the meantime, I've updated my answer to add what I think is a more helpful view into the question: how many total questions are there, and how many answered questions are there? $\endgroup$
    – Sycorax Mod
    Jul 3, 2018 at 14:59
  • 1
    $\begingroup$ @Glen_b Upon reflection, I do not believe that the first plot reflected the actual phenomenon of interest. The third plot seems to more sharply emphasize the gap between questions and answers. $\endgroup$
    – Sycorax Mod
    Jul 3, 2018 at 17:10
  • 1
    $\begingroup$ Yep. So linear growth in questions, flat, possibly even decreasing answers. We would seem to need either need a lot more closures or a lot more answers. I wonder if we have any particular tags that are especially short of answers or if it's a progression across the board (I'd anticipate that machine learning questions could be an issue, because of the rapidly growing interest but without a nearly so rapidly growing pool of answerers). $\endgroup$
    – Glen_b
    Jul 3, 2018 at 23:25
  • 1
    $\begingroup$ @Glen_b +1 I've been on a bit of a tear in recent days to (1) close unanswerable/poorly framed/exact duplicates and (2) answer unanswered questions. Often, these closure votes become stymied at 4 votes. But if we look at root causes, I think that you're right: we need more people with machine learning competence to dive in and start answering questions, and more people with VTC privileges to more aggressively moderate these questions. I've noticed that there is a tendency to allow marginal, unanswerable questions or questions for which the answer is unknown or unknowable to remain open... $\endgroup$
    – Sycorax Mod
    Jul 3, 2018 at 23:30
  • $\begingroup$ ... @Glen_b But I think we should move just a little bit towards closing questions for which the answer is unknowable ("I've tried estimating many models and none of them are good. Why?") or ridiculous ("I fit a logistic regression and random forest. The results are different. Why?"). I think that we have a decided need for a moderator with deeper machine learning knowledge would help to tame our unanswered question problem by VTCing poor questions. $\endgroup$
    – Sycorax Mod
    Jul 3, 2018 at 23:44
  • $\begingroup$ It doesn't help when I exhaust my VTC allotment with more than 12 hours left in the day. $\endgroup$
    – Sycorax Mod
    Jul 3, 2018 at 23:49
  • 1
    $\begingroup$ I agree completely on the value of having a moderator with a focus in that area. I've been thinking the same thing for some time. $\endgroup$
    – Glen_b
    Jul 3, 2018 at 23:54
  • $\begingroup$ @Glen_b Do you know when the next moderator election will be held? $\endgroup$
    – Sycorax Mod
    Jul 4, 2018 at 16:50
  • 2
    $\begingroup$ There's no set time. Various things might lead us to ask for one (such as the mods feel like the work has grown a lot, or some mods become inactive or less active for whatever reason, or community consensus as expressed here indicates a desire for one, among other possibilities). If there's been a substantial time gap sometimes the community moderators ("the Stackexchange team") will ask us if we want one. Typically they've been held roughly every two years here but there's no particular requirement that it be two years. (This question predates my time as a moderator.) $\endgroup$
    – Glen_b
    Jul 5, 2018 at 2:59
5
$\begingroup$

Besides the subjects already mentioned in the question, I can think of some suggestions to improve the site's statistics (and perhaps indirectly increase the number of participants too):

short term:

  • support this proposal, which asks for votes not being deleted when a user leaves the site.
  • once in a while, pay attention in the active tab, to questions bumped up by the community user. They are questions with 0 score answers and maybe the answer(s) did not receive enough attention. This query written by Raff also clip such questions.
  • go forward to the announcer, booster and publicist badges. They help to improve the site's visibility and maybe it will attract more active participants.
$\endgroup$
2
  • 2
    $\begingroup$ Second point is fine. The first point seems a red herring here. Discussion of temporary loss of reputation, a trivial problem for all and a transient problem for individuals, is a distraction from the larger aims of the forum. On the third point: I have nothing against publicity but my own experience is that only participation makes clear the attractions of moderate or high contributions here. $\endgroup$
    – Nick Cox
    Dec 2, 2014 at 14:39
  • $\begingroup$ It's just that I think this feature hearts some fair stats here (but, agree it is red herring, so won't extend on this). About the third bullet: I hypothesized that people who search for help, frequently find CV with Google, but people willing to help (e.g. journeyman statisticians) could hear more about CV through other media channels. (@NickCox). $\endgroup$ Dec 2, 2014 at 15:05
5
$\begingroup$

Ultimately, I think we need more people answering questions - many times more. While more experts would be great, I think we actually need more ... journeyman statisticians, and even in some cases relative novices -- and ways to encourage them to answer a few of the questions within their capabilities.

I fit this description quite well as I have taken a sabbatical to study statistics and machine learning. I have undergraduate level in statistics and probability, and slowly building some insights into more advanced topics. My feeling is that there are quite a few people around here doing the same as me.

As a result I would like to share my experience and provide some answers to your question

How do we encourage greater participation from a lot more of them to answer questions?

Highlight "accessible" answers

  • My view is that since the site has been around for a while now, a lot of the "easy" questions have been asked, and a lot of the "easy" answers have been given. When I say "easy" I mean from the perspective of someone with undergraduate level statistics or a self-learner

Here is an example, which is 4 years old: - Basic question about Fisher Information matrix and relationship to Hessian and standard errors

  • On the other hand: I struggle to understand most of the new questions that are asked, let alone think of an answer. So I rarely go beyond the first page of the unanswered questions feed. There must still be some easier questions out there but they are lost to me.

An idea - create a tag / flag... for "easy" and "good first answers"

... and leave those to the relative novice.

A good example of this is the Sklearn Github issues page which has two dedicated tags: enter image description here

$\endgroup$
2
  • 3
    $\begingroup$ Welcome to the site, Xavier, I've appreciated your answers. In fact, we did used to have a tag, [basic-concepts], that was similar to what you suggest. We eventually deleted it, however. Such tags are considered meta tags & are verboten in the SE system. 1 piece of advice I can give you: the unanswered tab lists Qs by votes; thew ones you see first are often old & hard. Instead, seatch on your favorite tag using the keyword, answers:0, & use the newest tab. $\endgroup$ Jun 17, 2018 at 12:24
  • $\begingroup$ Great i ll have a look! $\endgroup$ Jun 17, 2018 at 12:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .