On CrossValidated some questions never get answered. Some questions get amazing answers. Some questions get 20 unique answers, and some only get one. Is it the fault of the question? I don't know how to answer that, but I wish that I knew where to start.
I know that there is a human answer to "what makes a great question" but how does that bear out in the scoring? Does the score reflect a good technical question, or a well written but less technical question? Does it reflect time of day, audience in the world? How well does the value applied to the questions and answers reflect their worth? In the same theme of Kevin Slavin, I would like to see the physics of the culture here in CrossValidated in enough clarity to evaluate how well the numerics line up with the human goals.
I have seen "rock-star" answers to average questions that boost the points and participation substantially. This was about the answerer, not just the question. Does "who answers the question" mean more, in terms of score, than the nature of the question or the other good answers - in defining the current point-related social phenomenology? Sometimes it can be good and sometimes bad, but the frequency of these events and their outcomes are not quantified.
Some folks will repeatedly post the same valid question. They may do it for more answers, or an answer of sufficient quality. They do it because it returns value. It returns value in this synthetic economy. Why is the two-year-old's repeated question working plausibly well for adults with technical questions in a technical forum?
The theme of the question behind these ideas feels like the Timbuk3 line "how well do we use our freedom to choose the illusions we create" but I would prefer to ask how we can inform improvement - increased return of value to the users. Malcolm Gladwell might tell you that the best Spaghetti sauces were not invented, they were measured into being. (link)
If I were to look at the data, what would the "moneyball" of CrossValidated look like?
An acceptable answer will umbrella other questions like the following:
- How would the empirical answer inform improved participation/engagement/learning and merit driven point scoring? (show how the data speaks to current human metrics)
- Is is possible to measure a better CV into being like extra chunky spaghetti sauce or a pleasant pepsi? (does the data show "hidden metrics" being served, and if so what, and should they be considered?)
- What would the process of determining the "moneyball" look like? (show the approach)
The problem with this is that it is far too broad to be accessible. (Thank you @gung .)
Lets start simply, and if reasonable we can expand.
- that the score given to a question is the perfect measure of merit of a question (output)
Lets use as data (inputs):
- the number of comments, weighted by their authors reputation (or a transform thereof), with +1's
- the number of questions, weighted by their authors reputation (or a transform thereof)
- the number of +1's given to the answers, and correct scores
- we should have an askers name as part of the input, as well as date/time of asking
- how well does the input correctly predict the output?
Tactic 1: Can I use a random forest to predict the output given the input?
- If I use appropriate approaches to maximize the quality of the RF, so that the error is on the part of the model and not the modeler, what is the difference in variance between the data and the mean (whatever that means) versus the variance between the data and the model?
- If I look at importance of the variables, does the "rock star" effect show in the names?
- how does day of week, and time of day of asking impact number of answers? ... mean or max rating of answerers? mean or max rating of commenters?
- how does score of asker relate to quality of question?
- how does score of answerer relate to score of answer?
- how does the score of an answerer relate to the rating on the comment?
Now, do these together indicate or refute that CV has:
- rock star effect in question, answer, or comment
- time of day or day of week effect in response to answers
Thanks for helping.