How should performance answering questions best be evaluated

Question

There are many statistics computed on the StackExchange websites that could be used to evaluate how well a person is performing at answering questions. We have (1) total reputation points (2) reputation points per question posted (3) number of answers accepted (4) percentage of answers accepted, (5) number of answers downvoted (as a negative) (6) percentage of answers downvoted and more. What do you think are the best measures and why do you think they are the best. High reputation point totals can be indicative of good answers but they may also be affected by the activity level or the amount of time the individual has been posting on StackExchange. Points per question may be better in that regard but number of upvoted on a question can be low just because the question was not interesting and only a few people looked at it. If the OP accepts the answer maybe that should carry some weight. Let me know what you think.

Answer 1

I will take this question seriously, which may or may not be helpful.

Perhaps you are familiar with the discipline of psychometrics (which is the study of how to accurately measure psychological constructs). You will need to work out a theory of what it is you want to measure and how it is related to other theoretical constructs (or at least, available measures of those constructs). Often it is not possible to directly measure the target of your theorizing, but a variety of indirect measures can be found. Factor analysis is typically used to assess if the indirect measures really do 'hang together', and if they are unitary. From there, you can work on building a measurement system, and checking its reliability, and convergent and discriminant validity.

With some basics mentioned, let me throw out some specific thoughts. Raw values (e.g., reputation scores and badge counts) are rarely considered very useful. For instance, consider the total reputation score, an analogy can be made between it and p-values: Just as p-values measure effect size and $N$ simultaneously, and thus are often not a good thing to focus on (depending on what you want to know), so reputation measures both level of quality and amount of activity. Even mean upvotes minus downvotes per question is confounded with the amount of attention each question has received as you recognized. So a decent first pass at the typical quality of someone's (answering) efforts might be a weighted mean vote total, with the weights determined by each question's views. We can, of course, always push further. For example, some people may be only inclined to answer a question if they are confident they can give a really good answer, whereas others may answer anything where they suspect their marginal contribution is above zero (think of how some people are more talkative than others). Is that something that you want to capture in the measurement, or is it a contaminant you want to filter out? Note that we can run through analogous considerations with badges. For what it's worth, I agree with MansT (+1).

Answer 2

Why is there a need to evaluate answering performance? A person's reputation on this site - in the everyday sense of the word - is not determined by metrics.

There is certainly a purpose to the reputation system (write good answers => get more privileges) and to the badges (making people want to achieve goals that are good for the community), but what I (at least) think of people here is determined by their behaviour and their answers and not by those numbers. Using a different metric will not change that.

Answer 3

Let's look at the question a little differently. What strategy would you like to use for answering questions? Lots of quick answers or a few well-crafted answers?

A look at this quarter's activity shows that some users have had to post (on the average) five or more answers in order to accrue the same number of upvotes and acceptances that a few other users acquire with a single answer. Often that single answer is clear, authoritative, correct, definitive, well-illustrated, linked to appropriate references, at most lightly edited, and even..."pretty." It is almost never downvoted.

(Although I could name many community members who have posted such answers, I would offer many of the posts by chl as examples: they have established a standard for us to aspire to.)

By comparison, those five quick answers--although they frequently convey useful replies--tend to be murky or technical (due to their brevity), lack references, occasionally err (at least originally), need extensive comments for clarification, have no illustrations or links, and have to be heavily edited for readability and $\TeX$ markup. They tend to collect anonymous downvotes and flags (which the community does not see but the moderators have to deal with, alas). A large minority of them collect only one vote, or no votes, or end up deleted altogether.

I imagine it takes about the same amount of time and effort to create one good answer as to toss off five quick ones, especially once one becomes familiar with the tools provided by this site for linking, importing graphics, marking up text, and so on, which expedite the production of quality posts.

Although it is difficult to pin down any statistic that consistently distinguishes the two strategies--and both produce contributions that are valued and appreciated--SE encourages the first kind of answer: it stands the test of time, it attracts searches from outside, and in the best cases it informs, delights, and inspires readers. I urge all in our community to aspire to that kind of contribution, even though I know it cannot be done all the time (and is not even appropriate for certain kinds of routine questions). Yet I also know, from having read almost all the posts on our site, that a great many of you have demonstrated this capability. So when deciding whether to spend your valuable time rushing to answer a lot of questions or answering a single one really well, I urge you to consider expressing your expertise and insight in a high-quality, well-crafted answer whenever you can. The reputation, badges, and other statistics will then take care of themselves.