What should be the data collection strategy for our first Polystats Project?

Question

Update

I have asked the mods to lock this thread. I will post a question which will be a poll on what specific project question we want to investigate as part of the first project.

End of update

[Important Meta Note

Please do not vote-up this question as this question is a housekeeping type of question designed to move the polystats project to the next phase of the project. However, do vote up answers depending on their quality. I will add dummy edits at some regular frequency to push this question to the top from time to time. You may also wish to see this meta thread on the issue of rep and polystats project.]

The consensus (based on the votes/answers) to the first question What should be our first Polystats Project seems to be indicate that our first project would use Stackexchange data to answer several type of questions. The following summarizes the project ideas of that thread:

• Are there reputation effects in SE communities and to what extent are reputation effects contingent on various variables such as size of the community, number of views per day, no of questions per day etc?

• Quantify the quality of answers and questions (what makes a question/answer a good one?)

• Classify how the communities differ.

• Is there any evidence for herd behavior? (i.e., Is the probability of a question/answer being voted up/down dependent on the existing up/down totals?)

The above is my interpretation of the suggestions made in that thread. While we can debate the merits of the above questions on various dimensions (e.g., which ones are interesting, doable etc), I would like the focus of this question to be on our data collection strategy.

Currently, there exist three main SE sites (Stackoverflow, Superuser and ServerFault) and a whole bunch of beta sites including our site (I am not counting meta.SE and SE 1.0 sites). It seems to me that the beta sites are not well established enough for us to answer any of the questions we have in a reasonable way. Thus, I propose that we restrict attention to Stackoverflow, Superuser and Server Fault.

However, the amount of data that is available on these sites is staggering. The following is a snapshot of the current statistics for these three sites:

Clearly, we cannot download all the data to work with and need some form of sampling strategy.

Given the above background, my questions are:

1. Do you agree with the restriction to Stackoverflow, Superuser and Serverfault?

2. What data sampling strategy would you recommend that we follow?

3. Do you think that all of us should work with the same dataset (to ensure that we can reproduce each other's results/bugs)? If so, where/how should we store it?

4. What variables do you like to see in the data dump? The following shows the data that we can potentially download using the API/data-dump provided by SE. Table names are in capital letters with corresponding fields below the table name.

POSTS Id PostTypeId AcceptedAnswerId CreationDate Score ViewCount Body OwnerUserId LastEditorUserId LastEditDate LastActivityDate Title Tags AnswerCount CommentCount FavoriteCount ClosedDate ParentId CommunityOwnedDate USERS Id Reputation EmailHash CreationDate DisplayName LastAccessDate WebsiteUrl Location Age AboutMe Views UpVotes DownVotes COMMENTS Id PostId Score Text CreationDate UserId BADGES Id UserId Name Date POSTTAGS PostId TagId TAGS Id TagName VOTES Id PostId VoteTypeId CreationDate BountyAmount UserId VoteTypes Id Name

Answer 1

My thoughts:

1. I agree that we should either (a) restrict to those sites or else (b) do everything but them. The latter option would give more variety in the communities so I think that it could be more interesting, and it would give a picture of communities at much different stages of development and in completely different fields. So we can get a nice picture of community development and decay.

2. I don't think that we should necessarily need to sample initially based on doing step 1. People can then engage in dimension reduction or sampling exercises as part of the actual analysis if there is too much data, although I'm optimistic that we can make available all of the data given the restriction from 1.

3. I'm happy to act as the "data tsar" to get that started (or someone else can), and post a complete dataset in a readable format (or at least provide a script to get it). That being said, I think the data processing part of this task could be part of what someone's value add in this whole project. So long as everyone has access to the same data (which they do already), then whether everyone uses the same data isn't especially important in my view.

4. Clearly what data one uses will depend on what question they're trying to address.
   I myself am interested in describing how the communities differ. For that, I would probably start with basic descriptive statistics about questions, users, votes (I'm somewhat less interested in the comments and badges tables), look at some of the distributions (I'm sure that most voting patterns follow a power law, for instance). Maybe we can use a gini coefficient to classify the inequality of the various communities. And I would also want to look at social network relationships (e.g. connectedness)? Is there a stagnation effect as communities grow older (we seem to casually observed this)? Do new users behave differently than old users? Needless to say, there are a very large number of interesting questions that can be asked of this data.

Alternatively, to answer the "what makes a good question/answer", we can do something like this.

Answer 2

I agree with Shane for everything but I will put in my responses as well. I think it will be useful though because I have seperate interests in the project than Shane. I am solely interested in the question "Are there repuation effects in SE communities" (not because the other things aren't interesting, simply because that is how I personally like to pursue topics in a simple, counter-factual framework).

1- Restricting to Stackoverflow, Superuser and Serverfault is fine to answer my question, but if people want to consider a wider array of topics it is benificial to have a broader range of communities (its necessary to examine between community variation). It also doesn't hinder my quest to only consider every other community than these three as Shane suggests (I don't have any theoretical reasons to expect differences between reputation effects in different communities apriori). So I concur with Shane's suggestion of every other community besides Stackoverflow, Superuser and Serverfault.

2 - I agree with Shane as well, and I think more data is better at this point. If people need to reduce the data to either make it more manageable or address particular questions they can do it on their own. If we had more refined goals I wouldn't disagree with a sampling strategy, but I don't want to limit potential projects before we get started.

3 - Again I agree with Shane, and I think it will be simpler to manage if we all work with the same data dump. This will allow easier replication and I think will make trouble shooting easier to handle (although if people use different sampling schemes this will become more difficult). I could volunteer some space either in my Dropbox account or google docs and publish the link (as long as people are fine with delimited text I don't see any space problems on my end). I suppose in the future we may want to think about how to store results/code in a manageable and retrievable fashion.

4 - I'll come back to this when I take a closer look at the variables and what information is contained. Although we may want to include everything just so it does not limit anyones ability to answer different questions at this point.

Sounds like a really cool project, and I hope it gets some really good user input from the community. I am really interested to see how different people approach the subject. I think I may use this as personal motivation to learn some more R so I can disseminate results.

Answer 3

First, a meta point- We probably should not have moved to this phase without finalizing the project question. So, I will wait a few days for any more opinions to come in and then get this thread locked. The next question will focus on which one of the four ideas we should work on our first project. Ideally, all of us should commit to working on the most voted idea (even if it is not our personal favorite) to maximize participation and to divide-and-conquer the project. After all, the point of Polystats Projects = 'massively collaborative data analysis'.

Here are my answers to my four questions from the top:

1. Depending on the project selected we should use data from the SO, SU and SF OR the data from the beta sites OR perhaps all sites. I think the answer to this is dependent on the question we select.

2. IMO, if we select SO, SU and SF we definitely need a sampling strategy. If we want to maximize contribution then it would make sense to work with 'small' datasets as the community will have different computing limitations (e.g., some folks may not have the RAM to load all the data in memory etc). We want to be as inclusive as possible and hence large datasets are a negative as far as this issue is concerned.

   Perhaps, we can do some sort of stratified sampling with the strata being: time period, No of votes for qns during a time period (high, medium, low etc), rep of users during a time period (again high, medium, low etc).

3. Yes, we should work with the same dataset. Not only does this help with reproducibility but it would also ensure that we are on the same page. One advantage here is that if we use different methods (machine learning, stats, data mining etc) to investigate the same question then we can attribute any differences in conclusions to the methods used rather than any idiosyncrasies of the data we used.

   Some form of delimited, text file is ideal (perhaps, csv). Storage is much less of an issue as long as it is a stable link.

4. What data we need to extract is dependent on the questions we seek to investigate. So, I will defer answering this question.