32
$\begingroup$

Are there any ideas / proposed explanations about why CV stands out regarding the percentage of questions that have zero answers? (I have excluded questions that have answers, but none of them are upvoted/accepted).

\begin{array}{| r | r | r |} \hline \rm June\; 22, 2014 & \rm Total\ Qs & \text{Qs with no As} & \text{%}\\ \hline \rm Physics & 35,372 & 4,331 & 12.2 \\ \rm Cross \;Validated & 38,582 & 11,468 & \mathbf {29.7} \\ \rm Math \; Overflow & 51,364 & 9,859 & 19.2 \\ \rm Math.SE & 296,538 & 39,727 & 13.4 \\ \rm Stack \;Overflow & 7,509,663 & 801,136 & 10.7 \\ \hline \end{array}

For the comparison, I purposefully chose other SE sites which deal in fields that require a mathematical background.

I have the feeling this large difference cannot be solely attributed to the fact that sometimes questions are answered through comments (that in any case needs to be combined with the additional argument that on CV, we tend to do that more than on the other sites).

ADDENDUM
CV-meta link 14 months back... Why is our answer rate so low?

$\endgroup$
6
  • $\begingroup$ Have we data on number of "regular users" - say, number who are on a certain number of days/week or have a certain number of stars or something - on each site? $\endgroup$
    – Peter Flom
    Commented Jun 25, 2014 at 22:29
  • $\begingroup$ @Peter Flom I wouldn't know how to retrieve them Peter. I guess since in the user's profile there are the "visits" data (member for, visited, last seen), I guess such kind of data can be gathered. $\endgroup$
    – Alecos Papadopoulos
    Commented Jun 25, 2014 at 22:43
  • $\begingroup$ @Alecos The 'visited' data on profiles is not publicly visible - you can see yours, but presumably since I can't see yours, you won't see mine (you're welcome to it though - 783 days, 541 consecutive), so it's possible data.stats.SE might not give the information. $\endgroup$
    – Glen_b
    Commented Jun 27, 2014 at 2:23
  • $\begingroup$ @Peter what are stars? Are you referring to favorited questions or something else? $\endgroup$
    – Glen_b
    Commented Jun 27, 2014 at 2:25
  • $\begingroup$ I meant those awards they give us for doing things. Now I realize they are little squares. The bronze, silver and gold things (not sure why I thought of them as stars). $\endgroup$
    – Peter Flom
    Commented Jun 27, 2014 at 10:37
  • $\begingroup$ By any chance, do you have data on bounties? Is it possible that fewer bounties are being started on CV? (Note: I realize that bounties are a very small fraction of total questions--it was just something I was wondering about). $\endgroup$
    – Steve S
    Commented Jul 20, 2014 at 7:35

5 Answers 5

Reset to default
34
$\begingroup$

My comments expanded enough to be some kind of answer.

I think there are a number of reasons why the rate might be different - some have been discussed before. On SO or math.SE (which I participate a little on, mostly on R and stats questions respectively), questions tend to be much more specific and focused, with direct, short answers. ('How do I do this?' ... may often answered with 'call this function' or with a couple of lines of algebra.) - those are not the norm here; often the asker doesn't even know what they want, let alone how to express it.

In particular, I think a lot of questions on SO or math.SE could be answered by a competent undergraduate or enthusiastic amateur - and there seem to be plenty of both around those places.

Apart from homework-like questions, there aren't so many of either here. Many questions are subtle and the breadth of knowledge required to tackle more than a tiny subset of them takes a long time to accumulate. I think I have answers in more tags than anyone else now (about 450 I think), yet I can really only competently answer a very small fraction of the questions we get.

In the course of answering questions over a few days, I often find myself doing algebra I've never quite attempted before, running simulations I've never run before (and writing and debugging code to do them!), suggesting novel or tweaked test statistics and exploring their properties, comparing the properties of several approaches to a problem, coming up with slightly novel way to visualize some data, reading papers to follow the history of some little technique, reading more papers to even figure out what a person is asking about ... and so on. That is, a lot of questions here take actual research effort. Sometimes hours of it.

We have a pretty dedicated but small group of people who answer a lot of questions and an even smaller dedicated group who put effort into clarifying questions and getting posters to improve/focus their questions enough to be answerable (anyone who is reading this is likely to be contributing in both ways). Those need to expand.

We do quite well on some things - all our review queues are empty, almost all the time. SO would love to have that statistic. Nothing stays in a review queue for long. The survival time in the suggested edit queue often seems to be a couple of minutes at best.

I am not sure how to get more people answering questions here. I sometimes do a bit of pointing to good answers here while I am in other forums, and that sometimes brings a few people in. I do talk to people about the site, but I seem to be singularly unsuccessful in convincing other people to actually come and participate.

$\endgroup$
10
  • 7
    $\begingroup$ This seems to accord with something I was thinking -that Statistics questions are inherently more complex at all levels, because apart perhaps from purely "mathematical statistics" questions, in all other cases they directly or indirectly ask "how can I "better" approximate the real world phenomenon under study?" -and in order to give some precise and case-specific meaning to "better", we have to understand the real world phenomenon under study. This strengthens whuber's recommendation: if the going always gets tough here, you need tough to get going in greater numbers than usual. $\endgroup$
    – Alecos Papadopoulos
    Commented Jun 23, 2014 at 1:16
  • 4
    $\begingroup$ But how do you get more people? I probably fall in the 2nd category that Glen describes as for most I participate in the reviewing and editing processes. Whilst I also frequently answer questions, my knowledge is limited to econometrics which only makes a small fraction of the questions here. If I now ask some of my friends from the PhD to join CV they will be just like me. Not being able to answer the majority of questions can be quite daunting in the beginning and discourage further use of the site as an answerer. $\endgroup$
    – Andy
    Commented Jun 23, 2014 at 9:48
  • 2
    $\begingroup$ @Andy - I'd have counted you in both categories. I usually don't answer econometrics questions because I figure other people will know more about those than me (I do answer some, but leave most); unless the kind of questions you can answer are nearly all answered, I think a lot of value would be added by having many more people like you. I find answering questions is also a good way to learn more. $\endgroup$
    – Glen_b
    Commented Jun 23, 2014 at 10:07
  • 5
    $\begingroup$ @Andy There are a couple of hundreds of questions with zero answers in tags that may fall into the knowledge of an econometrics PhD ("econometrics", "regression", "least squares", "panel data", "time series", "maximum likelihood" etc). So my simple suggestion to you would be to tell your friends to join CV and attack older unanswered questions. Some of the most personally satisfying answers I posted here on CV (because they helped me also solidifying and expanding my own knowledge, as Glen_b explains), was by digging this queue. $\endgroup$
    – Alecos Papadopoulos
    Commented Jun 23, 2014 at 13:30
  • $\begingroup$ I would agree about the diversity of statistics but it would take more than 600 characters for me to justify persuasively. Given that this is true, then generally speaking improvements in tags and improvement in features for engaging with them should have a more significant impact on CV answer rates than from other improvements with the exception of increasing the number of experts. $\endgroup$
    – Meadowlark Bradsher
    Commented Jun 24, 2014 at 17:23
  • 1
    $\begingroup$ I would add to this that if questions are specialized then it increases the chances that answers are as well. Under those circumstances fewer users should feel inclined to upvote an answer because it is not in their area of expertise to verify the conclusion. That should weaken the motivation of some proportion of users to answer many such questions. $\endgroup$
    – Meadowlark Bradsher
    Commented Jun 24, 2014 at 17:36
  • $\begingroup$ @Meadowlar While I believe that often answering questions requires somewhat specialized knowledge, the issue I was trying to get at in my answer is that quite often a single question spans many areas of specialized knowledge. People often have real data analysis problems, and those almost never fit neatly into a single tag or two; one may need to deal with panel data, multivariate responses, count data, nonparametric relationships between variables, missing observations, robustness, the impact of model selection, generation of prediction intervals, and more all in the same question... (ctd) $\endgroup$
    – Glen_b
    Commented Jun 24, 2014 at 22:31
  • $\begingroup$ (ctd) Worse, you often can't disentangle the issues - each can impact how you deal with the others. It's no good being handy with (say) multivariate Poisson regression and time series (already quite specialized) if you know relatively little about how model selection affects inference and prediction - the advice will quite likely be astray. $\endgroup$
    – Glen_b
    Commented Jun 24, 2014 at 22:32
  • $\begingroup$ @Glen_b I see. I'm pretty new here but I do recall at least wondering whether I should advise someone to break their question into parts and if other community members made recommendations in this line. $\endgroup$
    – Meadowlark Bradsher
    Commented Jun 25, 2014 at 1:19
  • $\begingroup$ @MeadowlarkBradsher I frequently advise people to split questions up if the questions are reasonably independent (questions can link to other questions for context so mild dependence is not a problem). Sometimes they certainly are independent enough to split up, and when they are it's good to do so. It depends on the context. "How do I analyze this data" are often the complex ones. $\endgroup$
    – Glen_b
    Commented Jun 25, 2014 at 1:37
30
$\begingroup$

Some facts

The rate at which questions are answered here has been dropping for years. When we left beta in early 2011 the answer:question ratio was around 1.6:1. It has steadily declined: in November 2012 it dropped below 1:1 and at present it is around 0.75:1. I believe this is due to a greater rate of increase in the rate at which questions are asked compared to the increase in community members who are active and qualified to answer questions. During this time the answering rate has tripled but the asking rate has gone up sixfold.

Nuances in the comparisons

Although comparing the rate of unanswered questions to the rates at other SE sites looks disturbing, it has benign explanations. SO fields programming questions that tend to be routine and solvable. Math fields a very large number of (easily answered) homework questions. Here, we get a great number of questions that not only are related to research, but cannot immediately be answered by anybody: they would require considerable work to answer. In that regard we are perhaps most comparable to MO (19.2% unanswered) or TCS (21.6% unanswered), which focus on research-level questions. Another informative comparison to make is with GIS, which is a kind of hybrid of SO (most questions nowadays are about programming) and CV (because a huge variety of specialized platforms are covered on GIS and very few qualified people are around to answer many of them). On GIS there are 23.3% unanswered questions.

What we can do

Notwithstanding such comparisons, it would be good to leave fewer questions unanswered here. The only significant way that can be addressed is to grow the number of people who are (a) qualified to give good answers and (b) excited enough about our site to be regularly active. Thus, what I would suggest is that once we feel we understand why our rate of unanswered questions is so high, we should turn to discussing what to do about it--and I think that conversation ought to look for ideas on how to attract active, qualified people into our community.

$\endgroup$
10
  • 7
    $\begingroup$ Take a look at the unanswered upvoted questions, Alecos. Then, consider that about half our unanswered questions have no upvotes: those should be evaluated separately. Some of them are pretty bad but not so bad they were closed, but apparently the rest have no upvotes only because nobody has any interest in (or understanding of) them. $\endgroup$
    – whuber Mod
    Commented Jun 22, 2014 at 17:37
  • 2
    $\begingroup$ (re-post after accidental deletion) Do you think analyzing unanswered questions by tag would help to identify the areas of Statistics that seem to go relatively unanswered? It would give us a more concrete idea about what kind of expertise CV appears to lack, and so make the attempts to bring in new qualified answerers more targeted. We could then even "officially" invite expert academics and professionals, supplementing our invitation with a series of existing questions that fall in their expertise. This would show research effort from our part, adding respect and weight to our invitation. $\endgroup$
    – Alecos Papadopoulos
    Commented Jun 23, 2014 at 1:18
  • $\begingroup$ Here is one question of mine I am surprised get no interest: stats.stackexchange.com/questions/103834/… $\endgroup$
    – kjetil b halvorsen Mod
    Commented Jun 26, 2014 at 7:32
  • $\begingroup$ @kjetil Please see our help concerning this issue: stats.stackexchange.com/help/no-one-answers. In the meantime, I apologize that I did not notice your post and I have added a comment seeking clarification. $\endgroup$
    – whuber Mod
    Commented Jun 26, 2014 at 14:05
  • 1
    $\begingroup$ Not suggesting the policy should change, but we also do not answer many of the questions we could answer because they often happen to be the rare ones that are not vague: homework questions. I infer that other sites are less reluctant in that respect. $\endgroup$
    – Christoph Hanck
    Commented Feb 26, 2016 at 15:54
  • $\begingroup$ Given the questions are hard to answer, what should the policy be on answering a question once somebody has altered it multiple times to match a commenter's wishes? Is it not good faith to then at least attempt an answer, or state why one can't be given? What should a poster do after they've altered their question and still have no answer? Should they try asking again in a different way? $\endgroup$
    – Connor
    Commented Mar 30, 2023 at 21:10
  • 1
    $\begingroup$ Please visit our help, @Connor, and focus on the section at stats.stackexchange.com/help/asking. The pattern I have noticed in your recent questions is that they propose methods that don't look suitable for the underlying problems. That puts us in a quandary, because we know that answering your explicit question very well could just put you further down a frustrating path. The questions aren't necessarily hard to answer: they're merely unanswerable in this sense. $\endgroup$
    – whuber Mod
    Commented Mar 30, 2023 at 21:13
  • $\begingroup$ @whuber Thank you, I get that I might be wrong. But no one has suggested anything different. If it's clear that what I want to do isn't right, why doesn't someone just say that plainly, and explain why. The alternative is that I assume no one knows and I continue doing what I'm doing anyway. $\endgroup$
    – Connor
    Commented Mar 30, 2023 at 21:17
  • 1
    $\begingroup$ @Connor I believe we're all in the same mode of trying to figure out what it is you want to ask. Until that happens, we can't even be in a position to say whether anything is wrong or what it is: we're all guessing. $\endgroup$
    – whuber Mod
    Commented Mar 30, 2023 at 21:22
  • 1
    $\begingroup$ @whuber I think you're right. Thank you for engaging though. What you said made it instantly clear to me what the probable misunderstanding is! $\endgroup$
    – Connor
    Commented Mar 30, 2023 at 21:23
19
$\begingroup$

This ends up being a non-answer, but I figured I would share some analysis I did. For a while I've been concerned that the site's continual progress rests on the shoulders of a relatively few individuals. It is expected that a relatively few contribute a lot, but I felt the balance here was even more perilous than in some of the other sites I marginally participate on. Or put another way, if 5 of the top users depart the site in one fell swoop I worried we would not have a functioning site anymore.

My perceptions seem to be inaccurate though when examining the data - or at least several of the other sites mentioned face the same issues.

Comparing the long tails of several of the sites mentioned here in terms of the proportion of users that answer questions vs. the proportion of all answers we appear to be pretty similar to Physics, GIS, and MathOverflow in terms of the empirical CDF. The Math and Mathematica site have an even greater amount of inequality than we do here, with 20% of the people answering near 95% of the questions. TCS (theoretical computer science) is the most equitable of the sites listed here, and fits Pareto's 80-20 mark. We are closer to 85-20 - although the interesting part of the bend is closer to around 10% of the users answering questions.

enter image description here

Here is a slightly larger version of the figure with tool tips (although they don't help very much given the overlap of the lines).

So although I definitely agree attracting more users should be a long-term goal of the site, from this it doesn't appear to me that we have a problematic imbalance of expert users answering questions, at least relative to some of the other sites listed.

Data for this analysis can be found here. I used this query at the Data Explorer to gather the csv files for the different sites.

$\endgroup$
4
  • 1
    $\begingroup$ Useful analysis, but it is conditional on answered questions. What does it tell us for unanswered questions? To me it strengthens the impression that we do need more expert and diverse users, not because the existing ones are comparatively..."lazy", but because there are areas of expertise for which we get questions but we cannot provide answers. $\endgroup$
    – Alecos Papadopoulos
    Commented Jun 23, 2014 at 16:53
  • $\begingroup$ Yep I agree @AlecosPapadopoulos - my original hypothesis was that our curve would be above the others - which maybe has implications that we should nudge existing users to answer more questions. Some sites have group activity (like cleaning up particular tags) that would be types of behavior prompting action like that. But I was wrong, so it appears people are just asking too many questions per the group of experts to keep up. Case in point - physics has about 3,000 fewer questions, but has over 2,000 more people who have answered a question compared to stats. $\endgroup$
    – Andy W
    Commented Jun 23, 2014 at 17:04
  • 5
    $\begingroup$ +1 I feel sufficiently strongly about the need for breadth of participation that for the last year or so I have purposely cut back on my answering rate on both sites I moderate. I feel embarrassed when a very small number of individuals dominate the reputation boards: it's a sign of weakness, not strength. (I don't think this inaction will have affected the overall answering rates--my answers have simply been replaced by answers offered by others and this has given me a little more freedom to go after questions that seem not to be getting good answers.) $\endgroup$
    – whuber Mod
    Commented Jun 24, 2014 at 15:47
  • 2
    $\begingroup$ @AndyW thanks for bringing the data (and, especially for showing how to get it). If the concern were about the loss of a few people, though, we can't really see the potential impact in the plot, because that 'action' is in the extreme left of the plot, down at the bottom 20% or so. I think there are a very few users answering say 10-15% of questions (enough that a loss of even two would probably be very noticeable). While a few users on say math.SE might answer a lot of questions, there are so many other users answering that often others will be along in seconds, with multiple answers. $\endgroup$
    – Glen_b
    Commented Jun 24, 2014 at 22:53
9
$\begingroup$

Here's my take as a relatively new question-answerer, and as someone with far, far less knowledge than our superstar members.

I think you guys are spot-on so far. For instance, my question Computing inverse probability weights -- conditional (multivariate) density estimation? has an open bounty and, frankly, shouldn't be that complicated. And yet there are currently 17 (?) questions with open bounties, some three times as big, and mine is at the bottom of the list. I'm master's student -- I may be overzealous but I doubt I'm on the fringes of multivariate nonparametric statistics methodology. Someone out there knows the answer, and that someone would do me an enormous service in helping move my research along. Either there are many fewer competent statisticians out there than I think there are, or there's a talent mismatch. I'm guessing the latter.

But there's another issue. The superstar question answer-ers are too good at answering questions. It's not that they answer too many questions. It's that their answers are just too good. Every time I think about something I could contribute, I think "I'm just gonna sound like a dope if I try to post something, and I sure as hell don't want to steer anyone wrong." I've tried to pick out a niche of questions I can competently answer, and I'm grateful that the more experienced members are around to clean up my posts. But writing your first answer is intimidating, even moreso if it's an area at the limits of your own expertise, and when there are others out there who know way, way more than you do.

I also have to say: I've learned an enormous amount already from answering questions here, in large part because of the high standards set by others before me. That was the goal all along, but it's working out way better than I could have hoped. I agree that probably the best way to help get more, better answers is to recruit. I'm going to start recommending CV to classmates, but most of them really aren't the type to spend hours helping strangers on the Internet.

Also I suspect that a number of questions get answered, but don't get marked as answered, either because the answers don't actually answer the question, or the asker is a first-time poster and doesn't know how it works. If I wasn't already procrastinating as badly as I am by posting here, I'd try and figure out how to pull down that info and run some numbers on it.

Update: there are also lots of questions like Naive Bayes non-Dictionary Term in Test Document that get answered, get discussed in the comments, and then are never resolved enough to be marked as answered even though the asker almost surely got something like an answer out of it.

$\endgroup$
4
  • 6
    $\begingroup$ Thanks for this point of view (+1). About the "too good answers" part, I (imperfectly) copy from memory something I read on a meta-post on another site: "sites I stick around are those that I did not had to fight for every post". Look at my network profile, see how I started, see what happened eventually: Our super-star answerers do provide "too good" answers - but they are also too good to those with lesser knowledge. To maintain high standards, they are obliged to point out mistakes and ask for corrections -but they never, I mean never, do it in an intimidating or insulting way. $\endgroup$
    – Alecos Papadopoulos
    Commented Jun 25, 2014 at 3:40
  • 1
    $\begingroup$ And that's why I hope to stay. You guys work so hard to make this site as good as it is. $\endgroup$
    – shadowtalker
    Commented Jun 25, 2014 at 3:42
  • 1
    $\begingroup$ "experienced members are around to clean up my posts" -- that's just what those of us who have been around a while should be doing. Your answers seem to be a good deal better than a number of my early efforts (the worst of which are mostly now either improved, or sometimes deleted if other people gave reasonable answers; a few of them still exist and I still make plenty of mistakes now). I wouldn't worry that there might be others who know a bit more than you. I doubt there's more than a couple of answers I've given here where of the people who might answer I was the best person to answer. $\endgroup$
    – Glen_b
    Commented Jun 26, 2014 at 2:41
  • 5
    $\begingroup$ I'm even lower on the totem than you ssdecontrol. I just have an undergraduate education and a curiosity that gets me into trouble. I'm well aware of my limited education and experience. I look at a lot of questions before venturing to answer one but when I find one, I swallow my pride and answer knowing that I am sticking my neck out. My assumptions are that better answers will get better votes, correction will be communicated to me, and I can always delete it if it's fundamentally flawed. Honestly, I answer to relearn what I know and push a little further in. I haven't gotten a negative yet. $\endgroup$
    – Meadowlark Bradsher
    Commented Jun 26, 2014 at 17:44
5
$\begingroup$

Unsophisticated students ask many questions here. Our comments provide valuable assistance, helping to guide their thinking and studying. The students' questions themselves are often unanswerable (e.g., not clearly stated) or are so basic that they are well addressed by a comment such as "Have you considered _".

$\endgroup$
2
  • 1
    $\begingroup$ Thanks for participating in this discussion. Is it your impression that the phenomenon you describe happens relatively more often in CV than in the other sites that could be comparable to CV in other aspects, like e.g. the ones that I have sampled? $\endgroup$
    – Alecos Papadopoulos
    Commented Jul 4, 2014 at 16:56
  • 3
    $\begingroup$ I have no actual data, but here is my thinking. Statistics is a tool used by people in many fields; more so than, say, physics (one of your comparison groups). So a physics student might come here for help with statistics but a statistics student would almost never seek help with physics. $\endgroup$
    – Joel W.
    Commented Jul 4, 2014 at 21:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .