Public sphere statistical analyses

Question

Is there much precedent or openness to Q&A on the public sphere issues involving statistical methods? This question has three parts: a motivating example, a statistical perspective, and a perspective as it relates to utility of such questions on CV.SE.

The motivating example comes from reading this article in the New York Times. The gist is that someone, somewhere has produced a scoring of NYC teachers, based on a "value added model", and the results of those evaluations will be made public. I'd like to put aside issues regarding evaluation of teachers (e.g. Bill Gates has addressed this) and any more general matters regarding education, and focus on the statistical issues. This topic is in the public sphere: public resources and attention are shaped by the results of a statistical project.

The statistical perspective is that there are several things that can be addressed: what is the method that is employed? What are its assumptions? Where can one access the data? What are alternative methods? How can/did the authors check the assumptions? What happens if the assumptions are wrong? How sensitive is the model to errors in the data, etc.

The goal isn't to make a judgement about either the utility of the practice nor to attack or defend the research. Instead, the statistical questions, like all research and consulting projects, are based on utility and reproduceability. Are the statistical questions well posed, do the data allow us to answer the questions, were the questions answered, and what may be done in a subsequent iteration (either of analyses, data collection, or application of the methodology in another context)?

As for the CV perspective, there are some questions that seem quite at home here, such as: What is the statistical model that's used? What are its assumptions? There are other questions that are not so good, and are more opinion based: What's wrong with this paper? What's good about this paper?

I believe that outright opinion questions not appropriate. Could they be reformulated? Perhaps. An example: what is the most suspect assumption of this application? Time and again, I take a project, identify the key assumptions, and then determine which ones would not hold up to scrutiny in either the same context, using data from a different sample, or in a related context. In other words, which assumptions only work this one time, for this one data set? Those are particularly suspect assumptions. Many other assumptions are tolerably false.

Now, my opinion:

I believe that such questions could make CV quite attractive to a larger set of users and serve a serious educational and public benefit: people could learn about statistical methodology and engage in understanding a use of statistics in the public sphere. I am not interested in potential questions that relate merely to reports on surveys and census results: these can be tar pits of debate over sampling methodologies. Instead, I'm concerned with more methodological and model-based types of questions, where assumptions about data and models play a role.

I have used such analyses of published works in the past to teach students statistical methods, and it has worked very well. At some point, perhaps it would be of interest to expand to other published works, but I think that public sphere questions are a good start.

Answer 1

For the folks that are old enough, and who happen to be present in the US, the 2000 presidential elections with contoversial measurement errors on the split ballot created a strong splash of analyses and re-analyses. My adviser, Richard Smith, took a little longer than others, probably three or four months (I've seen at least two dozen analyses, and that was yet in the era when you had to write plain HTML rather than hit a "Post to blog" button on your iPhone), and he produced a very nicely rounded paper with good regression influence diagnostics, various ways of modeling the outlier, Poisson regressions with demographics of the counties, etc. -- see http://www.stat.unc.edu/faculty/rs/palmbeach.html.

Somewhat similar uproar of grassroots statistical analyses was produced by the recent parliamentary election in Russia -- how many votes were actually fabricated? In normal elections, this would have been a plot with five Gaussian densities representing the votes cast for each of the participating parties: http://oude-rus.livejournal.com/542295.html (the horizontal axis is the percent voted for a party, the vertical axis is the number of polling stations showing this percent, with a bin of 0.5%). I saw photos of protesters carrying this graph on rallies in Moscow and St. Petersburg.

So I think the opportunities present themselves where moderately complicated statistical analysis can create some visual aids that would aid the public in understanding a phenomenon. Great examples are provided in the joint journal of ASA and RSS, Significance. For instance, they had an article about how difficult it is to visualize the radiation exposures that may differ by 9 orders of magnitude in a meaningful way, without expecting people to understand the log="xy" option. This was in a special issue on Fukushima in the fall of 2011, about half a year after the disaster. I don't think you can excite the general public by saying, "Look, I've found a p-value of 0.0000001!" (or, for that matter, "I've found a Bayes factor of 1000000"). It has to be easily digestible, without the need to explaining why Bayes factors are better than p-values because they respect the likelihood principle, and yada-yada-yada. It has to be a picture telling a story that's worth a thousand words. Journalists have their Picture of the Year international contests; may be we could launch "Plot of the Year" contest aiming at explaining a social phenomenon with numbers and figures.

Answer 2

First off, I think this idea could be woven effectively into the site in various ways, and I agree it could be a great and valuable resource to both this community and to the general public. So, the question becomes appropriate implementation of such a task.

While it could be woven directly into the Q/A format of the site, I don't believe per se this is the best venue for such a discussion for a few reasons;

• The question needs to be formulated in such a way that it fits within the purview of the site. It's possible, but it may entail asking several questions to limit the scope within reason for one question.
• I believe a satisfying response would be very large in scope compared to normal questions.

For these reasons, I think our blog is a potentially more appropriate venue for such an endeavor. But, the problem so far with the blog is getting more community involvement in the creation/editing of posts. That venue is available though if you personally want to take the initiative.

When writing about any polarizing topic one needs to choose their words very wisely, so it may be difficult to recruit authors for such a task (in any form). It is on your shoulder's so to speak to start the task. Given your standing in the community and quality of your questions/answers I have no doubt you could accomplish such a task, but IMO you shouldn't assume anyone else in the community will write a satisfying response given the nature and scope of what you are asking.

---

In reply to @Iterator's comments. Just to reiterate, the main goal of my answer was to suggest that to get such a project underway it will likely need to be undertaken in large part by yourself.

On a more opinionated manner, I believe it could be woven into several questions on the site, although offhand I don't particularly like the prospect. It may be a useful hueristic for a reviewer to dichotomize any particular project into several smaller components, but I'm not sure if it is a good idea to evaluate any particular small slice of some work in absentia of the other components of the research methodology. This is in addition to the fact that you have to constrain the original questions in such a way to fit within the scope of the site to begin with. Again I think it could be done, but I think it is better/easier just to write one all-inclusive opinion piece on the blog.

That being said if your the one undertaking the project you have the academic freedom to pursue it in anyway you want. I have faith given your contributions here and on SO that you have enough understanding of the nature of the sites to attempt such a project.

As far as how the blog works, so far basically we have published pretty much whatever we feel like! We have attempted a few thematic type series of posts, but participation has been too scarce to keep such contributions sustainable. Invitations are open to any member of the community to post a contribution to the blog if they want to.

IMO the blog is an excellent platform for things that don't quite fit into the Q/A of the site. My recent post on tables started as an answer to the site, then I realized it was just a pain to say what I wanted to say that directly answered the question. So instead of not saying what I wanted to say to fit within the scope of the site, I wrote a blog post. I think it is a bad idea to change the question you want an answer to just to fit within the status quo of the Q/A site.