I am trying to figure out where the line is for acceptable questions about sources of data and what is the best way to deal with such questions. In my mind, this is a special case of data collection queries, and constitutes "practical, answerable questions based on actual problems". For example, the GIS stackexchange folks seem perfectly happy to answer such questions, but that is a niche area and not all questions could be redirected there. There are also many data source questions where issues of sampling and estimation seem relevant (where can I find block level data on consumer expenditures or ACS vs long form census). There are also questions about comparing potential data sources where folks on this forum have substantial amount of domain expertise that would be valuable and cannot be found elsewhere. Finally, I have not found links to zanran and infochimps to be particularly helpful in my own searches.

  • 2
    $\begingroup$ Have you discovered the meta thread where this topic was originally discussed? That looks like a duplicate to me--maybe we should continue the discussion there. $\endgroup$ – whuber Jul 23 '12 at 18:27
  • 1
    $\begingroup$ I saw that when I was searching, but I thought the question suffered from picking an example where the line was very clear. Gini coefficient for taxation seems to be pretty specific and the answer made sense (at least when the econ site was around). I want to understand where that line is. Are there any such questions that are appropriate? $\endgroup$ – Dimitriy V. Masterov Jul 23 '12 at 19:07
  • 4
    $\begingroup$ In my experience, separating the data folks from the analysis folks is problematic. It's a two-sided market, and cleaving the two is like opening a singles bar with gender-specific sections. Questions about data should be handled by the people who analyze it. They have the best understanding of its limitations and potential. I've spent a lot of time dealing with data librarians and database admins who did not really have a clue. But I guess the consensus is that these questions do not belong. $\endgroup$ – Dimitriy V. Masterov Jul 25 '12 at 22:58

I still oppose this. I really sympathize with the difficulties in getting quality information about data sources, but I don't believe that this site is the appropriate venue.

There's a reason why the Data StackExchange site is failing: data sourcing questions are boring. People don't want to open up a screen full of questions like

  • "Where can I get data on X?"
  • "Where can I get data on X, but from 1970-1978?"
  • "Where can I get data on X in HDF5?"

This site works because there's a good probability that any user can click on any question and either be able to answer it or be able to learn something from it. Domain-specific data sourcing questions are, by definition, useless to users outside of that domain. The same could be said of language-specific programming questions, which is why we have a similar policy for migrating those elsewhere.

Following on your bar analogy, opening Stats.SE to data sourcing questions would be like bringing busloads of grandparents to the bar. I love my grandma, but there's no way I want her and forty of her dearest friends in my favorite pub. They'll take up too much room, distract me from what I'm there for, and eventually change the very nature of the place into something I don't enjoy. Sure, they'll love it for a while - until all their grandkids stop showing up!

I think the reason why GIS and other domain Q&A sites are more open to and (I believe) the correct venues for these types of questions is that they only have to deal with their own grannies. That keeps the volume to a manageable level and lets the site carry on its main purposes. We have no basis for limiting the volume or topic of those requests here, so they might just inundate us (especially around final exam time, I suspect).

For me, the key struggle of this site is keeping the experts interested. And the key to that is, I think, following this fundamental guideline in our FAQ: "If the question requires statistical expertise to be answered, ask it here". Data sourcing questions usually require a wholly different, domain-specific sorts of expertise. It happens that many of our excellent members have expertise in many domains (programming, too), but that's not why they're here - so even though it might be a useful and convenient place to ask data sourcing or programming questions, I think we have to honor the purpose of this venue to preserve its quality.

  • 6
    $\begingroup$ (Sorry, Grandma and other grandparents) $\endgroup$ – Matt Parker Jul 26 '12 at 20:14

I guess it depends on what you mean by "X". If the question is, 'where can I get data on US state-by-state unemployment rates?' (answer: FRED), then I don't think it belongs here just as was decided previously in the question @whuber links. I further agree with @Procrastinator that the proposed data Q&A site is a good idea for stuff like that where someone is looking for a particular public data set, but is having trouble finding it.

However, there could be a different version of this question that I think could be worthwhile for CV. Specifically, I have often thought of asking a CW question to have a place where links for statistical data sets of different kinds for demonstrating statistical properties or practicing different kinds of analyses could be listed (an example link would be to DASL). The idea is that it would serve as a recourse similar to A Handbook of Small Data Sets. We have such a repository of links to resources for learning R, for example, and one for R packages that can read data from feeds. My idea would be analogous to those, but I've never asked because I've never been sure it it's really appropriate.

If this is what @Dimitriy means, then I do think there could be real value in such a thing, although I recognize that the community is ambivalent on this type of stuff. For what it's worth, we do have this, which is also similar but more focused and doesn't have the open-ended 'big-list' quality to it.

Update: I don't know how I had missed it, but it turns out we do have something like this: Locating freely available data samples.

  • $\begingroup$ That is not a use I had in mind, but this does seem useful. $\endgroup$ – Dimitriy V. Masterov Jul 26 '12 at 3:47
  • 2
    $\begingroup$ I confess to a prejudice here: I react negatively to requests for generic datasets intended solely to demonstrate some statistical technique or another. It's hard to resist the feeling, perhaps unwarranted, that anybody making such a request needs to get out and do some real statistics. They'll collect loads of interesting rich datasets in short order. The world is literally at our fingertips on the Web and the data are pouring out at us: people shouldn't have to come here to ask where to find it! $\endgroup$ – whuber Jul 26 '12 at 21:52
  • 2
    $\begingroup$ @whuber, in a way, this is the argument concerning the homework tag. The world is full of good statistics books... heck, you have Wikipedia right on the same computer on which you are submitting the question to CV. Go on and read what the lack of memory of the exponential distribution means. Or how to add two independent Poisson random variables. Does it mean that we have to boycott all the homework smelling questions? (I tagged two or three today, and thought of giving Michael Chernick a yellow card for solving all the problems for the slacking students ;) ) $\endgroup$ – StasK Aug 5 '12 at 15:40
  • 2
    $\begingroup$ @whuber, I think we may disagree here. As I answered here, playing w/ data is an important part of getting yourself up to speed. You can use the datasets in your textbook, but might want more. It could be valuable for teachers too. I note that other (less useful, IMHO) 'big-list' answers (eg, jokes) are among the most popular, upvoted, & heavily commented / answered Q's on CV. But I respect the community's ambivalence, & good citizenship is an important part of keeping the ship running smoothly. $\endgroup$ – gung - Reinstate Monica Aug 5 '12 at 15:51

There's now an SE site in beta called Open Data for such questions.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .