# How should performance answering questions best be evaluated

There are many statistics computed on the StackExchange websites that could be used to evaluate how well a person is performing at answering questions. We have (1) total reputation points (2) reputation points per question posted (3) number of answers accepted (4) percentage of answers accepted, (5) number of answers downvoted (as a negative) (6) percentage of answers downvoted and more. What do you think are the best measures and why do you think they are the best. High reputation point totals can be indicative of good answers but they may also be affected by the activity level or the amount of time the individual has been posting on StackExchange. Points per question may be better in that regard but number of upvoted on a question can be low just because the question was not interesting and only a few people looked at it. If the OP accepts the answer maybe that should carry some weight. Let me know what you think.

• You may be interested in this related thread on the main site, How can I improve my analysis of the effects of reputation on voting?. – chl Jun 18 '12 at 17:32
• I think no matter what criteria you choose, there will be examples of when that criteria, in isolation, fails as a measure of quality. That being said, I say we should choose a criteria that ranks me highly. How about number of silver badges? – Macro Jun 18 '12 at 22:43
• @Macro Well done! – whuber Jun 19 '12 at 1:14
• @whuber What is it with you guys? I raised this as a serious question and you treat it as a joke. – Michael R. Chernick Jun 19 '12 at 3:14
• I really would be interested to know what people think constitutes good answers. To add fuel to the fire here is an example as a point of discussion. A question was asked about characterizing accuracy of 4 km trips where the distance was measured receiving a GPS signl using a cell phone. The OP said the he knew that the minimum distance estimate was soemthing like 3.9 km and the maximum 4.05 km but he had no way of computing a mean value. – Michael R. Chernick Jun 19 '12 at 3:27
• The voting system here has too many flaws and sources of bias to provide a single metric or even set of metrics. Often the first answer will have many times more up votes than the best answer. I think any metric would be more valuable if weighted by voter rep - eg sum of the log (or sqrt) of voter rep over all votes for each question, perhaps also weighted by other metrics like number of votes for the OP, but all of these would also have biases. Just like impact factors and citation counts. How will the metric be used? – David LeBauer Jun 19 '12 at 3:34
• My suggestion was simply to use nonparametric tolerance intervals and I mentioned how many samples would be needed for the range to represent 90% confidence of 90% coverage for example. The OP liked the answer and accepted it and gave it an upvote. His comment was that he thought my answer gave him a sense of how little can be said based on this somewhat shaky data. Now Bill Huber suggested that the GPS could be highly biased and went through a very careful, detailed description of a simulation for GPS traced path of a circular trip along with a histogram of the error distribution. – Michael R. Chernick Jun 19 '12 at 3:35
• While my answer only got 1 upvote Bill's got 6. Was one asnwer better than the other? Mine directly addressed what the OP was interested in, a statistical method to assess the variability of the estimates when an estimate of the mean is not available. Bill's on the other hand was a beautifully displayed and detailed example of the intricacies of GPS errors and what its bias might be. Maybe we shouldn't be comparing answers this way. Both answers served a purpose related to the question but in vastly different ways. – Michael R. Chernick Jun 19 '12 at 3:41
• @MånsT: Please consider copying your comment into an answer so that we may evaluate your answering performance. ;-) (Seriously, your comment contains a handful of important nuggets.) Cheers. – cardinal Jun 19 '12 at 10:36
• At the risk of injecting more humor into the comments, look here: dilbert.com/strips/comic/1994-11-21 . The problem is that this is to some degree an adversarial game. – Patrick Caldon Jun 20 '12 at 2:05
• Dear god, no, @Patrick, not more humor! (+1) – cardinal Jun 20 '12 at 12:19
• Michael, Congratulations on the Sportsmanship badge! (I see @Gung recently earned it too: well done!) I believe you earned that one in record time. By requiring a combination of many answers and many votes, it reflects someone with a balanced and unbiased mode of participation, someone who is demonstrably an asset to our community. – whuber Jun 27 '12 at 4:07

## 3 Answers

I will take this question seriously, which may or may not be helpful.

Perhaps you are familiar with the discipline of psychometrics (which is the study of how to accurately measure psychological constructs). You will need to work out a theory of what it is you want to measure and how it is related to other theoretical constructs (or at least, available measures of those constructs). Often it is not possible to directly measure the target of your theorizing, but a variety of indirect measures can be found. Factor analysis is typically used to assess if the indirect measures really do 'hang together', and if they are unitary. From there, you can work on building a measurement system, and checking its reliability, and convergent and discriminant validity.

With some basics mentioned, let me throw out some specific thoughts. Raw values (e.g., reputation scores and badge counts) are rarely considered very useful. For instance, consider the total reputation score, an analogy can be made between it and p-values: Just as p-values measure effect size and $N$ simultaneously, and thus are often not a good thing to focus on (depending on what you want to know), so reputation measures both level of quality and amount of activity. Even mean upvotes minus downvotes per question is confounded with the amount of attention each question has received as you recognized. So a decent first pass at the typical quality of someone's (answering) efforts might be a weighted mean vote total, with the weights determined by each question's views. We can, of course, always push further. For example, some people may be only inclined to answer a question if they are confident they can give a really good answer, whereas others may answer anything where they suspect their marginal contribution is above zero (think of how some people are more talkative than others). Is that something that you want to capture in the measurement, or is it a contaminant you want to filter out? Note that we can run through analogous considerations with badges. For what it's worth, I agree with MansT (+1).

• Thanks. I like it that you took the time to write a serious and thoughtful answer and I agreed with MansT as well. Both very good answers. I thought yours deserves the check mark though. – Michael R. Chernick Jun 19 '12 at 20:46
• (+1) Interesting thoughts. That's funny because I had in mind an answer which started by considering a common formalisation for the expectation of a given outcome ($y$ = No. upvotes), $\mathbb{E}(y|x)=\text{fixed part} + \text{random part}$, where the 'fixed part' would be a mix of exactness, authoritativeness, creativity, openness, experience, among others, while the random component would include voting mimicry, accidental upvote, subjective bias and all measurement errors we can imagine. – chl Jun 19 '12 at 21:00
• (Con't) I haven't decide where to put 'reputation effect' yet, but that would probably go under the fixed part of the model :-) I have some fairly knowledge of basic psychometrics, but I think defining a measurement model will be a hard task. – chl Jun 19 '12 at 21:02
• @chl, I think you're right that figuring out the model would be extremely difficult. It's a big part of what I was trying to highlight. I'm sure you'd be able to do a much better job of it than I would, though. I'm interested in the topic, but not an expert (much like a number of other statistical topics, I suppose). – gung - Reinstate Monica Jun 19 '12 at 21:13

Why is there a need to evaluate answering performance? A person's reputation on this site - in the everyday sense of the word - is not determined by metrics.

There is certainly a purpose to the reputation system (write good answers => get more privileges) and to the badges (making people want to achieve goals that are good for the community), but what I (at least) think of people here is determined by their behaviour and their answers and not by those numbers. Using a different metric will not change that.

• +1 For not considering Freudian metrics ;) – user10525 Jun 19 '12 at 10:58
• Great answer. Couldn't agree more. You and Procrastinator are good examples of posters with relatively low numerical reputation but are (at least to me) trusted users. This is why I think quoting your (numerical) reputation (which is possibly what started this whole discussion) in order to further some other point doesn't make any sense. It's easy to optimize a metric if all you want to do is optimize a metric. – Macro Jun 19 '12 at 11:23
• +1 for MansT. So far I really like David's comments the most though. – Michael R. Chernick Jun 19 '12 at 11:50
• Maybe one take home message I am getting from this is that reputation points are overemphasized. I don't think the badges are really an incentive. Many of them are achieved based on doing unusal things. For me I get my badges mostly by chance through activity on the site. There are very few that I have looked at and made it a goal to achieve. – Michael R. Chernick Jun 19 '12 at 11:55
• I particularly like to answer questions on the bootstrap and on textbooks and maybe a little on time series. Those are areas where I think I have special expertise. – Michael R. Chernick Jun 19 '12 at 11:57
• @michael don't hesitate to reference and/or excerpt your own publications - I suspect that adding something like 'more details can be found on page x of Chernick 2020' with a link and even a quote or screenshot (as I did here) would give your answers more value and credibility (and as a result increase votes, sales, citations, students of your work). – David LeBauer Jun 19 '12 at 15:19
• @David I think there are a number of questions regarding bootstrap and variable selection where I referred to relevant work of mine, papers with LaBudde and Gunter and my forthcoming monograph with Lacey Gunter on variable selection and recently published book on bootstrap with LaBudde. I hope that people apprciate that the references are ther because I think they are relevant and not that I just want to toot my horn. – Michael R. Chernick Jun 19 '12 at 20:43
• Also we have well known experts on regression and variable selection in regression such as Frank Harrell and for time series modeling IrishStat and his Box-Jenkins time series/transfer function software autobox. So i toot other people's horns too. – Michael R. Chernick Jun 19 '12 at 20:44

Let's look at the question a little differently. What strategy would you like to use for answering questions? Lots of quick answers or a few well-crafted answers?

A look at this quarter's activity shows that some users have had to post (on the average) five or more answers in order to accrue the same number of upvotes and acceptances that a few other users acquire with a single answer. Often that single answer is clear, authoritative, correct, definitive, well-illustrated, linked to appropriate references, at most lightly edited, and even..."pretty." It is almost never downvoted.

(Although I could name many community members who have posted such answers, I would offer many of the posts by chl as examples: they have established a standard for us to aspire to.)

By comparison, those five quick answers--although they frequently convey useful replies--tend to be murky or technical (due to their brevity), lack references, occasionally err (at least originally), need extensive comments for clarification, have no illustrations or links, and have to be heavily edited for readability and $\TeX$ markup. They tend to collect anonymous downvotes and flags (which the community does not see but the moderators have to deal with, alas). A large minority of them collect only one vote, or no votes, or end up deleted altogether.

I imagine it takes about the same amount of time and effort to create one good answer as to toss off five quick ones, especially once one becomes familiar with the tools provided by this site for linking, importing graphics, marking up text, and so on, which expedite the production of quality posts.

Although it is difficult to pin down any statistic that consistently distinguishes the two strategies--and both produce contributions that are valued and appreciated--SE encourages the first kind of answer: it stands the test of time, it attracts searches from outside, and in the best cases it informs, delights, and inspires readers. I urge all in our community to aspire to that kind of contribution, even though I know it cannot be done all the time (and is not even appropriate for certain kinds of routine questions). Yet I also know, from having read almost all the posts on our site, that a great many of you have demonstrated this capability. So when deciding whether to spend your valuable time rushing to answer a lot of questions or answering a single one really well, I urge you to consider expressing your expertise and insight in a high-quality, well-crafted answer whenever you can. The reputation, badges, and other statistics will then take care of themselves.

• +1, For the record, I do aspire to that kind of contribution, I just usually fall short... – gung - Reinstate Monica Jun 22 '12 at 5:27
• I disagree. I frequently write short answers that get zero or one upvotes, sometimes require clarification in comments, and to my knowledge are not flagged. Sometimes I'll write a longer and more thoughtful answer. The time differential for me is not 1:5, but rather 1:20, or even more. I do not want to leave 20 questions alone where I could help someone with a quick answer, even if that does mean writing one long answer that may (or may not) help 10 people down the road. ... – Stephan Kolassa Aug 30 '18 at 7:02
• ... If I understand you correctly (and if you still hold this opinion), then your proposed course of action will result in more unanswered questions, which the community seems to believe is a problem. – Stephan Kolassa Aug 30 '18 at 7:07
• @Stephan That's right. But I feel strongly that your proposed solution--namely, degrade the quality of answers--is the wrong solution to the problem. – whuber Aug 30 '18 at 12:26
• Here is a query I wrote today for finding potential new users based on reputation from answers. If you set 'n' to let's say 10k you get 10 top all time users in reputation from answers in CV. Sorting by the 'Answer rep rate' column will rank 'chl' 2nd ('amoeba' is 1st, you are 3rd). – Andre Silva Feb 9 '19 at 1:31