13
$\begingroup$

Background:
I was on Stack Overflow and they seem to have a lot more traffic and users there. It seems to make sense to me because knew many more CS majors than stats majors in college. This made me wonder about reputation-equality between sites. I saw some decent reputation folks (~7k+, ~35k+) making posts that were downvoted to negative net values for basic errors. That suggests that a ~7k on SO might mean much less than the same on CV in terms of representing expertise, or expectation of answer points given score. If someone with only moderate expertise can get into the "aristocracy" of 10k+ then I am much less likely to give credence to an answer based on answerer reputation.

Question/s:

  • What is the GINI score (or other appropriate measure) for user reputation across all of CV? SO?
  • How is each evolving over time?
  • How does each evolution relate to new and engaged participant "acquisition" rate or to user disengagement rate?
$\endgroup$
7
  • 6
    $\begingroup$ Would a higher or lower Gini score tell you whether 7K users were more reliable? My own thought would be to consider not just reputation but also net reputation per post; you can compute the second by clicking through to the user's page. [That's somewhat confounded by things like reputation cap and bounties given -- I've lost (/spent, respectively) in total about 5 reputation per answer on those -- but it shouldn't have much impact in general.] ... data.stackexchange.com/stats provides the ability to query the data base and some existing queries may partly answer your question. $\endgroup$
    – Glen_b
    Feb 22, 2016 at 20:33
  • 3
    $\begingroup$ However, reputation per post is difficult to compare cross-site because some sites upvote more on average; you'd probably want to scale for that too. You also may want to take a account of the fact that some queries include deleted or migrated answers in the answer total but not in the reputation total (so someone that answers a lot of R questions may look worse on those queries than they should). A carefully considered query will probably give good information and it can be done by user. $\endgroup$
    – Glen_b
    Feb 22, 2016 at 20:41
  • $\begingroup$ @Glen_b - that link is one of the sexiest things I have seen today. :) $\endgroup$
    – EngrStudent
    Feb 22, 2016 at 22:10
  • 1
    $\begingroup$ This is the answer per person ECDF comparison for a few sites. I imagine it will be close to the same for reputation. $\endgroup$
    – Andy W
    Feb 23, 2016 at 13:29
  • 3
    $\begingroup$ Well I wasted a few minutes getting the SQL as close as I could for the data explorer to calculate the GINI index. See here. $\endgroup$
    – Andy W
    Feb 23, 2016 at 14:32
  • 3
    $\begingroup$ You would also need to consider that different tags get differing numbers of views & upvotes. Ie for some tags, people contribute very high quality information, but get few upvotes. This will just be a very difficult question to answer. (Cf: How should performance answering questions best be evaluated.) $\endgroup$
    – gung - Reinstate Monica
    Feb 23, 2016 at 15:18
  • $\begingroup$ A very-easy to compute measure (that sure does have its limitations) would be just to look at the reputation it takes (at least for the sites that are about equally old) to get into, say, position 100, 200, etc. in the all-times rankings. $\endgroup$
    – Christoph Hanck
    Feb 26, 2016 at 15:46

1 Answer 1

Reset to default
12
$\begingroup$

Well thanks for nerdsniping me. My prior query in the comments is limited due to the idiosyncratic conventions of the data explorer to return only 50,000 rows. In that query I got around this problem by only selecting users with a reputation over 1 (people who sign up for the site and have their account associated de facto get a rep. of 1). Apparently the Gini coefficient is pretty sensitive to how you treat these people though. If you plot the ECDF, just imagine keeping the curve at the same place, but stretching the X axis to the right by a large amount.

Here is a new query that hops around that problem, in which you can input where the cut-off in reputation scores is considered. Here is a table of our site compared with several others, varying the cut-off at -100 (which effectively captures everyone - ?is Gini OK for negative values?), the cut-off above 1 (so eliminates those browsers who just had the 1 point) and 100 (who can also be browsers if they have the 100 boost from a neighboring site).

                     Rep CutOffs    
    Site       >-100     >1     >100
--------------------------------------
Cross Validated 0.81    0.68    0.54
Stack Overflow  0.95    0.91    0.77
Mathematics     0.90    0.85    0.76
Theor. CompSci  0.64    0.45    0.41
GIS             0.82    0.74    0.60

You can see the Gini metric is pretty sensitive to these cut-offs. The more people you cut-off the more equal the sites become. These appear to agree with my prior post based on answers off-hand. Theoretical computer science is the most egalitarian, CrossValidated is in the middle with a few others, and Mathematics and Stack Overflow have the most inequality. (Others feel free to add sites to my table.)

Now, does this have anything to do with your questions? I'm skeptical it does.

Off-topic rant - it is common in ecological models in social science for people to put inequality scores (Gini or others) on the right hand of regression models and interpret them as "more inequality results in some particular outcome". I have a difficult time relating such inequality measures though (which are based on economic ideas such as egalitarian transfers of funds) to micro-level individual behaviors. Take for instance a really simple 4 person society, and try to write out how exactly changing the cash one person has to members of the group. Measures like Gini imply a whole complicated lot of functions that on their face seem at best over-complicated.

This is all academic though. I don't know how to encourage more users for the site - which is the whole point of the question. Talking about metrics like Gini as if they matter as an outcome is a bit red herring.

For those interested, the GINI measure I calculate is based off the formula on this website. For a more academic reference though, Paul Allison gives two alternative formula in the reference below.

Allison, Paul D. "Measures of inequality." American sociological review (1978): 865-880 | PDF Link

$\endgroup$
9
  • 1
    $\begingroup$ Muahahaa. Yay, new term: nerdsniping. Btw, the probplot of reputation indicates the oligarchy has 4 members with Id's of 686,805,919,and 7290. They have cumulative reputation greater than the lowest 15,452 users, or 370,859 points. It is also ~2870x more than the average user. There should be a threshold of "expert" that is a function of GINI metric. For CV the minimum reputation to indicate superior quality might be, consistent with observation, lower than for SO. $\endgroup$
    – EngrStudent
    Feb 24, 2016 at 17:13
  • $\begingroup$ Note that the name Corrado Gini is honoured by at least three different coefficients in current use. That associated with the Lorenz curve is typically associated with positive values only as fractions of a total are key. $\endgroup$
    – Nick Cox
    Feb 24, 2016 at 21:06
  • 5
    $\begingroup$ An associated user who doesn't do anything on a site will actually have a rep of 101. In fact, CV has 13k users with rep = 101, and 9k with more than that, so your numbers based on >100 rep are 59% users who've associated and nothing else (well, plus some who happen to have earned exactly 101 rep on CV alone). Your query for CV changes from .54 to .67 with a cutoff of 101; presumably, other sites change too, though I didn't check. $\endgroup$
    – Danica
    Feb 24, 2016 at 22:57
  • $\begingroup$ @Dougal - yes, those who have rep on other sites get a 100 boost. Nice point about eliminating 101 and the GINI measure goes up, so it is not monotonically decreasing due to selection. I don't know whether it is reasonable to include them or not - presumably some of those browsers could legitimately answer questions. (And the one and done people it seems you are reasonable to keep as well in the metric.) $\endgroup$
    – Andy W
    Feb 25, 2016 at 18:59
  • $\begingroup$ @NickCox - I can't say I am familiar with any other coefficient than the one presented (citations added). There are quite a few different formulas floating around the produce that same statistic - but yes it ranges between 0 and 1. $\endgroup$
    – Andy W
    Feb 25, 2016 at 19:03
  • 1
    $\begingroup$ It's much more than there being several recipes for the measure you use, which can be related to the Lorenz curve. There are other quite different measures with the same label of Gini, $\endgroup$
    – Nick Cox
    Feb 25, 2016 at 19:05
  • 3
    $\begingroup$ When a 30k user on CV gives an answer - their answers tend to be "the mutant offspring of Clint Eastwood and Yoda" - they take the problem down with severe prejudice. I was surprised to see other fora where they can answer pretty bad. The number means different things in different contexts. If you can get a "high" reputation from a cracker-jack box, then your typical answers are likely to deserve less credibility. I am curious about the human physics behind that. $\endgroup$
    – EngrStudent
    Feb 26, 2016 at 19:08
  • $\begingroup$ In regards to the Gini coefficient, it is occasionally calculated with negative values (in regards to wealth, debt is a negative value). For the sake of describing CV, SO, etc. scores, it's not clear that you want use the raw scores: as you note, such a high percent of users have scores >= 101. They get their scores for "free", so we might be more interested in the distribution of "earned scores", ie score - 101. This would greatly affect the estimated Gini coefficient. $\endgroup$
    – Cliff AB
    Feb 27, 2016 at 2:49
  • 2
    $\begingroup$ The SO user rankings start tracking at 200 points, I suspect that is the value they came up with for an "engaged user". That is the cutoff I would sugest. $\endgroup$
    – Mike Wise
    Feb 28, 2016 at 9:53

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .