I have asked the mods to lock this thread. I will post a question which will be a poll on what specific project question we want to investigate as part of the first project.
End of update
[Important Meta Note
Please do not vote-up this question as this question is a housekeeping type of question designed to move the polystats project to the next phase of the project. However, do vote up answers depending on their quality. I will add dummy edits at some regular frequency to push this question to the top from time to time. You may also wish to see this meta thread on the issue of rep and polystats project.]
The consensus (based on the votes/answers) to the first question What should be our first Polystats Project seems to be indicate that our first project would use Stackexchange data to answer several type of questions. The following summarizes the project ideas of that thread:
Are there reputation effects in SE communities and to what extent are reputation effects contingent on various variables such as size of the community, number of views per day, no of questions per day etc?
Quantify the quality of answers and questions (what makes a question/answer a good one?)
Classify how the communities differ.
Is there any evidence for herd behavior? (i.e., Is the probability of a question/answer being voted up/down dependent on the existing up/down totals?)
The above is my interpretation of the suggestions made in that thread. While we can debate the merits of the above questions on various dimensions (e.g., which ones are interesting, doable etc), I would like the focus of this question to be on our data collection strategy.
Currently, there exist three main SE sites (Stackoverflow, Superuser and ServerFault) and a whole bunch of beta sites including our site (I am not counting meta.SE and SE 1.0 sites). It seems to me that the beta sites are not well established enough for us to answer any of the questions we have in a reasonable way. Thus, I propose that we restrict attention to Stackoverflow, Superuser and Server Fault.
However, the amount of data that is available on these sites is staggering. The following is a snapshot of the current statistics for these three sites:
Clearly, we cannot download all the data to work with and need some form of sampling strategy.
Given the above background, my questions are:
Do you agree with the restriction to Stackoverflow, Superuser and Serverfault?
What data sampling strategy would you recommend that we follow?
Do you think that all of us should work with the same dataset (to ensure that we can reproduce each other's results/bugs)? If so, where/how should we store it?
What variables do you like to see in the data dump? The following shows the data that we can potentially download using the API/data-dump provided by SE. Table names are in capital letters with corresponding fields below the table name.
POSTS Id PostTypeId AcceptedAnswerId CreationDate Score ViewCount Body OwnerUserId LastEditorUserId LastEditDate LastActivityDate Title Tags AnswerCount CommentCount FavoriteCount ClosedDate ParentId CommunityOwnedDate USERS Id Reputation EmailHash CreationDate DisplayName LastAccessDate WebsiteUrl Location Age AboutMe Views UpVotes DownVotes COMMENTS Id PostId Score Text CreationDate UserId BADGES Id UserId Name Date POSTTAGS PostId TagId TAGS Id TagName VOTES Id PostId VoteTypeId CreationDate BountyAmount UserId VoteTypes Id Name