9
$\begingroup$

Update

This project has moved to the next phase. See What should be the data collection strategy for our first Polystats Project?. I have asked the moderators to lock this question.

End of Update

Some of you may have heard about the Polymath projects in research mathematics. Polymath is an approach that was conceptualized by Timothy Gowers to enable "massively collaborative research" in mathematics. Since the initial conception there are several polymath projects that the math community has started (some of which can be found at this wiki for polymath projects.)

Inspired by the success of polymath projects, I propose that we start a polystats project using the Q&A format provided by our site (polystats = "massively collaborative data analysis"). I envision the typical polystats project to follow the sequence of events outlined below:

  1. First question appears on the site to select a project.

    Once the community reaches a consensus on what project we will focus on (via voting) we lock the question and move to the next question.

  2. Second question (with a link to the first question) appears on the site requesting potential exploratory data visualizations that may shed light on potential model specifications and tentative answers the questions.

    Again once we reach consensus this question is locked and we move to an analysis phase.

  3. Third question (with a link to the second question) appears on the site requesting potential model specifications given what we know from questions 1 and 2.

    Once we reach consensus we lock this question and move to analysis phase. And so on.

With the above ideas in mind I want to set the ball rolling and ask you the first question for our first polystats project:

What should be our first polystats project?

Given below are some thoughts to what kinds of ideas may work well for our first polystats project.

  1. A potential answer should include the questions the project will answer along with a data source that is readily accessible.

  2. For our first project, it may make a lot of sense to choose an idea that is readily accessible to as many participants as possible. (to ensure wide participation)

    So, in particular avoid projects (at least for now) that need specialized domain knowledge or alternatively the project must be such that the required domain knowledge can be acquired by quickly skimming an article or two on the wiki.

  3. In order to maximize our chances of success perhaps we should avoid controversial ideas (e.g., global warming) or politically sensitive issues.

    As a first attempt let us choose a project that is fairly non-controversial but complex enough that we can put our collective minds to find reasonable answers.

I have marked this question CW so please feel free to upvote/downvote ideas so that the 'best' candidate floats up.

I suggest that you open a thread on https://stats.meta.stackexchange.com/ if you wish to open a discussion on the relative merits of some of these ideas or about anything else about the polystats project. You may also want to see the discussion on meta titled "How about starting a Polystats Project?"

$\endgroup$
8
  • $\begingroup$ Relevant questions on the site for free data sources: Datasets for running statistical analysis, Locating freely available data samples $\endgroup$
    – svadali
    Commented Sep 9, 2010 at 16:39
  • $\begingroup$ Another suggestion: It may be useful to suggest different project ideas based on the same dataset. For example, building on Shane's idea to use SE data, someone can suggest exploring a different aspect of Q&A sites. The votes would then decide what project we will look at. $\endgroup$
    – svadali
    Commented Sep 9, 2010 at 17:07
  • $\begingroup$ This thread should probably just decide on a subject for [polystats-project-1]. We can then have many questions on the same project tagged with the same polystats tag. So, for instance, this first project could be generally "Describe the SE communities", and a specific question could be "What makes a community succeed/fail?" and another could be "Which communities are more/less friendly?". $\endgroup$
    – Shane Mod
    Commented Sep 9, 2010 at 17:22
  • $\begingroup$ Will that not fragment our effort across multiple questions? From the point of maximizing contributors to the project and to achieve some coherence on polystats projects, we may want to stick to a somewhat linear format as outlined in the qn: Decide questions --> explore data --> model setup etc. Can't we use this thread to decide which questions to look at by posting the broad questions as separate answers? I like the idea of having version tags [polystats-project-1] and will add one along those lines. $\endgroup$
    – svadali
    Commented Sep 9, 2010 at 17:35
  • $\begingroup$ Then I would advise that you add a step: choose subject/data --> decide questions --> explore data --> model setup etc. Trying to use this question to manage the subject matter and the questions would seem too hard to follow (IMO). $\endgroup$
    – Shane Mod
    Commented Sep 9, 2010 at 17:41
  • $\begingroup$ I see what you are saying but I am guessing that most folks would decide to participate if the questions we are trying to solve are 'interesting' at some level. The data itself must anyway exist somewhere for a polystats-project to be viable. Therefore, it seems to me that the choice of subject/data is secondary to the questions we want to answer. Let us wait for others to chime in with their thoughts before we change the current setup. I have no issues with changing how we do this as long as the community arrives at some consensus. $\endgroup$
    – svadali
    Commented Sep 9, 2010 at 17:52
  • $\begingroup$ @Srikant, I also asked a more focused question about data set stats.stackexchange.com/questions/973/… $\endgroup$
    – robin girard
    Commented Sep 13, 2010 at 10:17
  • $\begingroup$ @robin Thanks for bringing that thread to my attention. I will post a meta thread which will serve as a repository of all these types of questions for future reference. $\endgroup$
    – svadali
    Commented Sep 13, 2010 at 12:57

5 Answers 5

Reset to default
3
$\begingroup$

I think that it could be interesting as a first project to analyze the data from the stackexchange sites. The goal would be to characterize the communities and the question/answer process itself.

We will need to further narrow the scope, but some possible questions:

  1. What makes a good question?
  2. What makes a good answer?
  3. How do the various communities differ?

The data is all cc-wiki and available both through the data dump and via the API. For those people doing their analysis in R, we can make the data available through the open-source overflowr package, but several other languages/tools are also available.

One nice thing about this project is that it opens the door for very different kinds of analysis that draws on our entire community, ranging from statistics to social network analysis and NLP/machine learning.

$\endgroup$
3
  • $\begingroup$ How about broadening the question: What factors determine success of a SE community? Is it no of questions, no of answers per question, no of users, voting patterns etc. $\endgroup$
    – svadali
    Commented Sep 9, 2010 at 16:41
  • $\begingroup$ That's a nice way of phrasing it, although it might also be interesting to classify which communities are more "friendly", "scholarly", "use your imagination here", etc. $\endgroup$
    – Shane Mod
    Commented Sep 9, 2010 at 16:56
  • $\begingroup$ @Shane On thinking about your three questions I feel that question 1 and 2 are qualitative whereas question 3 while quantitative seems to be a descriptive question. Would you agree with my assessment? Could you also highlight a bit more explicitly the specific project questions that you think we should be addressing? $\endgroup$
    – svadali
    Commented Sep 9, 2010 at 22:08
3
$\begingroup$

Are there reputation effects in SE communities and to what extent are reputation effects contingent on various variables such as size of the community, number of views per day, no of questions per day etc?

Reputation effect: Everything else being equal questions/answers of high rep users are voted up much more relative to questions/answers of low rep users.

Data source: See Shane's answer.

$\endgroup$
3
  • 1
    $\begingroup$ I was going to suggest the same question (this is the 'snowball effect' that Taleb talks about). However, the causality is difficult to tease out. One would have to look at how a user is voted up over the arc of their tenure on the site to control for the effect of 'merit'. $\endgroup$
    – shabbychef
    Commented Sep 9, 2010 at 18:44
  • $\begingroup$ Doesn't the difficulty make it more fun? :-) $\endgroup$
    – svadali
    Commented Sep 9, 2010 at 18:51
  • $\begingroup$ I didn't want people to mistakenly believe the problem was trivial, that's all ;) $\endgroup$
    – shabbychef
    Commented Sep 10, 2010 at 4:54
2
$\begingroup$

Update

Here is one of the two projects identified by @whuber. Please vote up/down this particular idea depending on whether you want to this to be our first polystats project. I am going to post the other project as a separate answer.

"Predictive Segmentation of Populations." Help improve clinical trials by distinguishing subpopulations of people into potential responders (to a treatment) and non-responders.

To spice it up a bit, what about engaging in a real problem with a prize offered for the best solution? Instances are easy to come by at places like http://www2.innocentive.com/ . Advantages include having a well-defined problem and ready access to appropriate data. Disadvantages include a time limit (usually a few months) and figuring out what to do with a prize if it's won ;-).

$\endgroup$
10
  • 1
    $\begingroup$ This is interesting, but probably too difficult to manage the "distribution" question. Not to mention that it might go against the openness of what we're doing. $\endgroup$
    – Shane Mod
    Commented Sep 9, 2010 at 20:06
  • $\begingroup$ Crowdsourcing at its finest. Those challenges you linked to seem to have pretty hefty prizes, I would be worried our hard work would be taken without recognition. $\endgroup$
    – Andy W
    Commented Sep 9, 2010 at 20:06
  • $\begingroup$ @Andy: On the contrary, I think recognition would go directly to this site, which anyone could review to see what happened. Or maybe you're saying the same thing Shane is? @Shane: When you work on a prize problem, you don't have to keep it secret. But maybe that's the point of your comment: what is to prevent a third party from coming in, lurking to follow a group effort, and then jumping the gun with a solution once one is imminent? I don't have an answer to that. But even so, knowing that a problem is of material interest to someone can get the juices flowing... $\endgroup$
    – whuber Mod
    Commented Sep 9, 2010 at 20:10
  • $\begingroup$ @whuber: Actually I mean that we all selflessly volunteer our time here for the greater good. Having a monetary reward, unless it was donated back to something, would feel weird to me. I would rather do something that is free, open and good for all. That said, I see no issue with people from this site getting together and doing this apart from the polystats project. $\endgroup$
    – Shane Mod
    Commented Sep 9, 2010 at 20:26
  • $\begingroup$ @Shane: I'm entirely in agreement. If anything done on this site ever resulted in a reward, it would have to go into the site itself or something closely related. The fame is worth more than the fortune, right? ;-) $\endgroup$
    – whuber Mod
    Commented Sep 9, 2010 at 21:20
  • $\begingroup$ @whuber Can you identify a specific project from that site which you can think is interesting and edit your answer to include that as your proposal? Right now, it is not clear what project we are voting on if we vote for your answer. $\endgroup$
    – svadali
    Commented Sep 9, 2010 at 21:58
  • $\begingroup$ @Srikant: That's a reasonable request, but it sort of misses the idea. These sites are dynamic: the problems on them are current and change regularly. Thus, you wouldn't be voting on any specific project, but on a method to identify worthy projects (in the spirit of an ongoing PolyStat rather than a one-shot effort). What I can add is that I have been monitoring Innocentive for the last year and have found some worthy projects--typically in the biomed/agriculture area--that are entirely statistical, kind of sophisticated versions of the better-known NetFlix challenges. $\endgroup$
    – whuber Mod
    Commented Sep 9, 2010 at 22:33
  • $\begingroup$ @whuber Fair enough. For what it is worth polystats-projects are supposed to be an ongoing effort and the first one is meant to test the waters (so to speak). Could you then identify a current promising candidate? I understand that the site is dynamic but we need some specific idea so that we can understand what we are voting for the first project. $\endgroup$
    – svadali
    Commented Sep 9, 2010 at 22:40
  • $\begingroup$ @Srikant: Here are some current ones due 12/1/10. (1) "Predictive Segmentation of Populations." Help improve clinical trials by distinguishing subpopulations of people into potential responders (to a treatment) and non-responders. (2) "Statistical Analysis of Genomic Variants." "...determine the optimal method of processing a large number of Single Nucleotide Polymorphism (SNP) data" (see the Wikipedia article on SNPs). There's a large synthetic dataset of 500,000 SNPs for 1000 patients available. We missed one this summer to help Innocentive itself develop statistical graphics. $\endgroup$
    – whuber Mod
    Commented Sep 10, 2010 at 22:05
  • $\begingroup$ @whuber Great. I added project one in this answer and project two as a separate answer. $\endgroup$
    – svadali
    Commented Sep 11, 2010 at 0:54
2
$\begingroup$

This is the other project suggested by @whuber. Please vote up/down this particular idea depending on whether you want to this to be our first polystats project.

"Statistical Analysis of Genomic Variants." "...determine the optimal method of processing a large number of Single Nucleotide Polymorphism (SNP) data" (see the Wikipedia article on SNPs). There's a large synthetic dataset of 500,000 SNPs for 1000 patients available

Original Context of @whuber's answer below:

To spice it up a bit, what about engaging in a real problem with a prize offered for the best solution? Instances are easy to come by at places like http://www2.innocentive.com/ . Advantages include having a well-defined problem and ready access to appropriate data. Disadvantages include a time limit (usually a few months) and figuring out what to do with a prize if it's won ;-).

$\endgroup$
1
$\begingroup$

Answer inspired by Srikant Vadali's comments on this question: Does herd behavior influence the SO sites? - please read question before reading further.

Can this be measured? I suppose the hypothesis is something like The rate at which up votes are given is directly(?) proportional to the number of up votes it already has.

Similar corollary for down votes.

I'm not a statistician so this probably needs some refinement - please feel free to make the appropriate suggestions and changes.

Update: I very much doubt it is a direct relationship because otherwise the number of votes would increase forever, but maybe some kind of predictable formula can emerge. (If you take number of edits (and hence visibility on active list) into consideration, it could become a complex, but very useful, formula.)

$\endgroup$

You must log in to answer this question.