4
$\begingroup$

Background:

On CrossValidated some questions never get answered. Some questions get amazing answers. Some questions get 20 unique answers, and some only get one. Is it the fault of the question? I don't know how to answer that, but I wish that I knew where to start.

I know that there is a human answer to "what makes a great question" but how does that bear out in the scoring? Does the score reflect a good technical question, or a well written but less technical question? Does it reflect time of day, audience in the world? How well does the value applied to the questions and answers reflect their worth? In the same theme of Kevin Slavin, I would like to see the physics of the culture here in CrossValidated in enough clarity to evaluate how well the numerics line up with the human goals.

I have seen "rock-star" answers to average questions that boost the points and participation substantially. This was about the answerer, not just the question. Does "who answers the question" mean more, in terms of score, than the nature of the question or the other good answers - in defining the current point-related social phenomenology? Sometimes it can be good and sometimes bad, but the frequency of these events and their outcomes are not quantified.

Some folks will repeatedly post the same valid question. They may do it for more answers, or an answer of sufficient quality. They do it because it returns value. It returns value in this synthetic economy. Why is the two-year-old's repeated question working plausibly well for adults with technical questions in a technical forum?

The theme of the question behind these ideas feels like the Timbuk3 line "how well do we use our freedom to choose the illusions we create" but I would prefer to ask how we can inform improvement - increased return of value to the users. Malcolm Gladwell might tell you that the best Spaghetti sauces were not invented, they were measured into being. (link)

Question:

If I were to look at the data, what would the "moneyball" of CrossValidated look like?

An acceptable answer will umbrella other questions like the following:

  • How would the empirical answer inform improved participation/engagement/learning and merit driven point scoring? (show how the data speaks to current human metrics)
  • Is is possible to measure a better CV into being like extra chunky spaghetti sauce or a pleasant pepsi? (does the data show "hidden metrics" being served, and if so what, and should they be considered?)
  • What would the process of determining the "moneyball" look like? (show the approach)

The problem with this is that it is far too broad to be accessible. (Thank you @gung .)

Lets start simply, and if reasonable we can expand.

Lets assume:

  • that the score given to a question is the perfect measure of merit of a question (output)

Lets use as data (inputs):

  • the number of comments, weighted by their authors reputation (or a transform thereof), with +1's
  • the number of questions, weighted by their authors reputation (or a transform thereof)
  • the number of +1's given to the answers, and correct scores

oh and

  • we should have an askers name as part of the input, as well as date/time of asking

Strategic Question:

  • how well does the input correctly predict the output?

Tactic 1: Can I use a random forest to predict the output given the input?

  • If I use appropriate approaches to maximize the quality of the RF, so that the error is on the part of the model and not the modeler, what is the difference in variance between the data and the mean (whatever that means) versus the variance between the data and the model?
  • If I look at importance of the variables, does the "rock star" effect show in the names?

Tactic 2: If I were to make several variability charts, Bivariate fits, or logistic fits using appropriate columns columns what answers could I find to the following

  • how does day of week, and time of day of asking impact number of answers? ... mean or max rating of answerers? mean or max rating of commenters?
  • how does score of asker relate to quality of question?
  • how does score of answerer relate to score of answer?
  • how does the score of an answerer relate to the rating on the comment?

Now, do these together indicate or refute that CV has:

  • rock star effect in question, answer, or comment
  • time of day or day of week effect in response to answers

Thanks for helping.

$\endgroup$

migrated from stats.stackexchange.com Jul 3 '14 at 17:55

This question came from our site for people interested in statistics, machine learning, data analysis, data mining, and data visualization.

  • $\begingroup$ I don't think this question is appropriate for this forum, and if it belongs anywhere, it belongs in meta. The tagging is also misleading as none of the tags deal with the content of the question. $\endgroup$ – rocinante Jul 3 '14 at 17:04
  • 5
    $\begingroup$ @rocinante This is a difficult question to classify. Most of the questions on their own seem to belong on meta. In that it's a very chatty and unfocused post (it contains about 10 questions, some quite disparate), it's possibly too broad both here and on meta, and likely belongs on chat. I was quite unsure but plumped for voting to move it to meta in the hope that it might be possible to clean it up enough to be on topic there. $\endgroup$ – Glen_b -Reinstate Monica Jul 3 '14 at 17:36
  • 1
    $\begingroup$ I think this Q has potential merit. In light of @Glen_b's good points, can you focus it more? (If this leads to subsequent, follow-up questions on meta.CV, that will be fine, but this question should be sufficiently focused to be able to be given a single coherent answer.) $\endgroup$ – gung - Reinstate Monica Jul 3 '14 at 20:04
  • 1
    $\begingroup$ Part of this question should be possible to answer with access to meta-data (or indeed downloading whole posts and using text analysis. That would be an interesting question (could even be a topic for a statistics master thesis?). Without such analysis we are left to guessing! I have a quite sure impression that it is easier to get high votes with elementary questions, the pool of possible answerers/voters are bigger! Even very good questions, if they are too advanced, get little attention. The best way forward for a better CV is (many) more active members. $\endgroup$ – kjetil b halvorsen Jul 3 '14 at 21:49
  • $\begingroup$ This still seems too broad & unfocused to me to get a viable answer. Eg, what would the "moneyball" of CV be? Do you want the output of an algorithm to predict question quality? (I doubt you'll get one.) The part about people repeatedly asking the same question is a good point worth discussing, but seems tangential to the rest. Etc. $\endgroup$ – gung - Reinstate Monica Jul 4 '14 at 13:29
  • $\begingroup$ "assume ... that the score given to a question is the perfect measure of merit of a question" -- why would we assume something we know to be false? Scores given to questions reflect all sorts of things - including such trivialities as when they were posted (I have taken to posting questions at particular times because posting them at convenient times for me is counterproductive if I want the better answers that popularity/visibility tends to bring). $\endgroup$ – Glen_b -Reinstate Monica Jul 5 '14 at 5:41
  • $\begingroup$ (ctd) ... It's certainly possible I missed the point with that assumption, but that may be because I found the question unfocused and over-long and missed something because my concentration was flagging. $\endgroup$ – Glen_b -Reinstate Monica Jul 5 '14 at 5:51
  • $\begingroup$ It is ludicrously false assumption. Of course it is wrong. It might be useful. Its falsehood makes it more useful. It is like assuming that the IRS does optimal harvesting of the US personal income then use logistic harvesting to estimate the US GDP, and compare that "pseudo-optimal" GDP to the actual GDP and ask why they aren't the same. Sorry about "War and Peace". $\endgroup$ – EngrStudent - Reinstate Monica Jul 5 '14 at 23:37
  • $\begingroup$ @Glen_b - I think that you should be able to get great answers with asking any particular time. Is there a policy that would enable that as far as reasonable? Data has to answer that. $\endgroup$ – EngrStudent - Reinstate Monica Jul 6 '14 at 23:54
  • 1
    $\begingroup$ You should, especially given that answers tend to be spread over several days. But my own anecdotal sense is that time of initial posting makes enough difference to matter. I notice similar effects on several other forums (reddit, for example, where the effect is stronger) - posting when North America is in bed is less likely to be noticed that posting when North American readers are on the site. $\endgroup$ – Glen_b -Reinstate Monica Jul 7 '14 at 0:17

You must log in to answer this question.

Browse other questions tagged .