7
$\begingroup$

Our first polystats project research question is:

Are there reputation effects in SE communities and to what extent are reputation effects contingent on various variables such as size of the community, number of views per day, no of questions per day etc?

See: Which one of the following questions should we address for our first Polystats Project? for votes on various proposed questions.

I am posting below some issues pertaining to data collection. As soon as reach consensus on these issues we can proceed to the exploratory data analysis phase of the project. Feel free to post any proposals that may occur to you and on which you think we need consensus.

Similar to the previous question, this poll will be open for 3 days till Sep 25th 00 hrs UTC.

Please vote up or down vote as appropriate as this is a CW question.

Previous Questions reg Polystats-Project-1:

Question 1: What should be our first Polystats Project?
Question 2: What should be the data collection strategy for our first Polystats Project?
Question 3: Which one of the following questions should we address for our first Polystats Project?

Relevant Meta Thread:

How about starting a Polystats Project?

$\endgroup$

9 Answers 9

Reset to default
5
$\begingroup$

As the many different responses in this thread already attest, it would be helpful to address some preliminary issues before asking this question. The important ones include:

  • The study question itself ought to be well defined. Exactly what is a "reputation effect"? (I have followed all the referenced threads, and have quickly reviewed them again, but can find no description or definition of this term. Am I the only one who doesn't know precisely what it means?)

  • Guidance from previous work can be helpful. E.g., does anyone know of any specific examples of "reputation effects"? Is there some background literature describing a theory of such effects? If so, what does it suggest about effect sizes, important covariates, explanatory factors, appropriate ways to measure them, and so on?

  • What do we hope the scope of the study will be? Do we aim do draw conclusions about all "SE Communities" or only about the ones whose data will be studied? Is there any intent or hope of drawing more general conclusions about "reputation" itself?

  • Because this appears to be research ab initio--that is, not building on or relying on previous studies--we will need a preliminary EDA phase. It may be difficult to recommend a good sampling plan until some EDA is accomplished. Indeed, a minimum of three phases of sampling seems to be required: some sampling--not necessarily even random--to conduct EDA, followed by more formal randomized sampling to obtain a small representative subset of data whose aim is to estimate effect sizes and variances well enough to determine how much more data will be needed to test the study questions to given degrees of confidence.

  • ...which implies we need to establish some targets for confidence in the study results.

  • Those targets may have to be adjusted to account for multiple comparisons.

  • We cannot do any such adjusting until a set of specific hypotheses has been established, which is one of the aims of the EDA.

  • Can we be more specific about the actual cost (or its equivalent in volunteer effort) of sampling the data? This would help us make tradeoffs between the desire for high confidence (achieved by larger sample sizes) and the pain of gathering the data needed to achieve such confidence. If necessary, addressing this question can be delayed until after the EDA or preliminary sampling phase.

Once these questions have been answered it should be straightforward do specify a sampling protocol. I don't imagine these questions will be difficult to answer, but the answers ought to be given and documented somewhere. Such formalities seem especially important in light of the collaborative, distributed nature of the proposed research, so that a common understanding is established (and not just assumed).

Apropos this last point, it's starting to look like developing a study protocol and all its documentation solely as a set of threads on this site will quickly become unwieldy. Who wants to wade through a long set of links to questions/comments/answers in order to look up basic information? It would be very useful to have a central location in which a small number of key documents are dynamically maintained. These would eventually include a statement of the study goals (including background information), the study protocol, a data dictionary, and the various datasets themselves.

$\endgroup$
4
  • 1
    $\begingroup$ +1, Really excellent points. At some point I would have suggested a wiki as a way to document progress. Perhaps, we need one right now before the threads get out of hand. Can someone volunteer to set one up for this project? $\endgroup$
    – user28
    Sep 23, 2010 at 15:07
  • 1
    $\begingroup$ I can set one up, yes. $\endgroup$
    – Shane
    Sep 23, 2010 at 15:20
  • $\begingroup$ @Shane Great. Please let us know the link once it is up and running. $\endgroup$
    – user28
    Sep 23, 2010 at 15:23
  • $\begingroup$ @Srikant: It's here: statalgo.com/polystatsproject/wiki. I'll try to make it a little more attractive, etc. as time goes on. I may also install the R mediawiki plugin. But this should do the trick for now. $\endgroup$
    – Shane
    Sep 23, 2010 at 15:39
6
$\begingroup$

Data should be in CSV format.

If you disagree suggest an alternative format.

$\endgroup$
1
  • $\begingroup$ Having CSV format is the best overall, although it would be good for R users to also have Rdata format available. This compressed and can store complex data structures. $\endgroup$
    – Shane
    Sep 23, 2010 at 16:34
6
$\begingroup$

We need a few volunteers to get this project moving into high-gear. Building on @whuber's excellent answer to this question I have identified a few tasks that we need to do. I have divided up the tasks in such a way that they should not take up too much of any one person's time. Please edit this answer and write your name against a task if you wish to do that task. Please update this answer also with relevant pointers/answers to each task under its respective bullet point.

  1. Setting up a wiki - Shane has volunteered to set this up.

    The wiki link is at: http://www.statalgo.com/polystatsproject/wiki/

  2. We need a definition for reputation as used by various fields and what it means in the context of Q&A communities. Some preliminary definitions from the top of my head:

    • Reputation = "Expertise in a specific domain"
    • Reputation = "Trustworthiness"
    • Wiki has a section on 'Online Reputation'
    • As per this sites FAQ "Reputation is a rough measurement of how much the community trusts you."

  3. Can someone find out if there is a formal definition of reputation effect? (Srikant)

    (a) Reputation Effect as Analogy to Network effects

    The wiki states: "In economics and business, a network effect (also called network externality) is the effect that one user of a good or service has on the value of that product to other people. When network effect is present, the value of a product or service increases as more people use it.[Emphasis added]"

    Thus, in our context, reputation effect would be: "When reputation effects are present the reputation of a user increases faster the higher his/her reputation is."

    (b) Another definition would be what?

    (Changing the unit of analysis to answers/questions as opposed to people) A reputation effect would be: "When an individual has a higher reputation, their questions/answers receive more up-votes, ceteris paribus to answers by individuals with lower reputation". Added by Andy W

  4. Can a few people dig up at a few references relevant to the project?

  5. What theoretical framework(s) would be appropriate for this project?

  6. Anything else we need to do please add as additional items.

$\endgroup$
5
  • $\begingroup$ I can make a citeulike group library with which to store suggested references if people would like me to. You don't need an account to see the library and follow the DOI, but you do need an account to post articles to the group. I have found plenty of work examine all sorts of aspects of online communities in just a short time span. I think this article looks pretty interesting, portal.acm.org/citation.cfm?id=1341549 PDF here cs.cornell.edu/~lars/wsdm08.pdf $\endgroup$
    – Andy W
    Sep 23, 2010 at 20:32
  • $\begingroup$ Also does anybody know of a Sociological/Psychological/Anthropological literature that is applicable to this topic (keywords or specific scholars)? $\endgroup$
    – Andy W
    Sep 23, 2010 at 20:35
  • $\begingroup$ @Andy I would go ahead with whatever you think makes sense. If you could collect a few relevant references then perhaps we can divide up the task of reading the relevant articles and providing a summary on the wiki. $\endgroup$
    – user28
    Sep 24, 2010 at 13:20
  • $\begingroup$ @all: Once we have some reading materials we need some volunteers to take up the task of reading some of that material. Could you please add your name as a comment or in the answer so that we can decide how to divide the readings? $\endgroup$
    – user28
    Sep 24, 2010 at 13:21
  • $\begingroup$ @all, I've gone ahead and made a group library on Citeulike, which can be found here, citeulike.org/groupfunc/14031/home , you can make suggestions for articles to post either here or at the actual site. And if you have a citeulike account you can post articles yourself. So make some recommendations, as it is pretty sparse at the moment. $\endgroup$
    – Andy W
    Sep 24, 2010 at 16:40
4
$\begingroup$

All of us should analyze the same dataset.

$\endgroup$
2
$\begingroup$

We should restrict data collection to SE 2.0 beta communities only.

$\endgroup$
1
$\begingroup$

We should restrict data collection to Stackoverflow, Server Fault and Super User only.

$\endgroup$
1
$\begingroup$

Data should be collected for all SE 2.0 beta communities, Stackoverflow, Server Fault and Super User.

$\endgroup$
1
$\begingroup$

We should collect a representative sample of the data as downloading the entire data (especially for Stackoverflow, SuperUser and Server Fault) is impractical.

If you vote for this please suggest a sampling strategy as well.

$\endgroup$
0
$\begingroup$

The following variables are available from the data dump:

POSTS 
Id PostTypeId AcceptedAnswerId CreationDate Score ViewCount Body 
OwnerUserId LastEditorUserId LastEditDate LastActivityDate Title Tags 
AnswerCount CommentCount FavoriteCount ClosedDate ParentId CommunityOwnedDate 

USERS 
Id Reputation EmailHash CreationDate DisplayName LastAccessDate WebsiteUrl 
Location Age AboutMe Views UpVotes DownVotes 

COMMENTS 
Id PostId Score Text CreationDate UserId BADGES Id UserId Name Date 

POSTTAGS 
PostId TagId TAGS Id TagName VOTES Id PostId VoteTypeId CreationDate 
BountyAmount UserId VoteTypes Id Name

We do not need to download comments and tags.

From the users table, we need:

ID Reputation CreationDate Upvotes Downvotes Views

From the posts table we need:

ID PostTypeID OwnerUserID CreationDate Score ViewCount AnswerCount

For each one of the sites, we need:

SiteName NoOfUsers CreationDate VariousSummaryStats (No of views per day etc)

Edit this 'answer' to add data that you think we will require for our analysis.

$\endgroup$

You must log in to answer this question.