How to stem the tide of duplicate Qs

Question

Please note that I searched on the keywords "duplicate" and "exact-duplicate" before posting this :-)

It seems to be a point of pride for some that this site has hosted so many thousand Qs. But this counter produces rather empty numbers since a great many of these Qs are duplicates. Almost daily we receive new questions that are variants of "How can I analyze data from survey rating items?" or "How should I choose predictors for my model?" To deal with this, should we...

...find a way to make the FAQ more prominent? ...prominently display links to keyword searches for the most popular terms, such as Likert and Variable-selection? ...ask people to affirm that they have read the FAQ and/or conducted a keyword search before allowing them to pose a Q?

I realize that some people will object to anything that acts a disincentive to participation by new people. But think about what this site is becoming when we allow these duplicates or near-duplicates to populate so much of it. It makes it harder for the answers to stand out.

Answer 1

Some of our "sister sites" in the SE network currently receive much more traffic and there may be lessons learned that we can transfer here.

Here are some that come to mind:

1. Judicious use of the faq tag. We don't appear to have a faq tag. But, the math website has taken to marking particularly common questions with this tag in order to facilitate quick identification of the most common duplicates. All such tagged questions also show up on a special tab to the right of the "featured" tab on their site. We could potentially use such an approach here.
2. Meta list to track major topics and the most common questions within each topic. Again, this comes from the math site and I find it to be a pretty useful feature, as long as you know it exists. The thread List of Generalizations of Common Questions contains the generalized versions of the most common questions by topic. (They even have a "statistics" topic, though the questions there are closer to elementary probability.) I think if we wanted to do something similar on this site, we'd probably want to adapt it a bit, but we get lots of repeat questions on linear regression, AN(C)OVA, Likert scales, basic time-series, etc. It would take some work on the front end to put together, but once started would be fairly easily updated. Note that each such question is also supposed to be marked faq.
3. Searching on tags and keywords. The tags can at least help narrow down the topic when searching for duplicates, but it's not easy, especially, e.g., if you're looking for something under r or a similarly popular tag. The search facilities are pretty poor, in my opinion, but whuber and others have some nice tricks for squeezing as much out of them as possible. (Helpful hint: Doing an empty search via the search box reveals lots of handy options on how to narrow your search.)
4. Google. The built-in search facilities being what they are, often a good Google search can uncover a duplicate pretty quickly. One particularly useful feature is to enter site:stats.stackexchange.com at the end of your search to limit the search to our domain. I've used this pretty often to find a duplicate for which I vaguely recall some feature (like an interesting phrase), but not the authors or title.

I do not see an easy solution to more rapidly acclimating new users. They all have different incentives and different experiences with technology. Some come seeking immediate help and wouldn't be bothered to look at the FAQ anyway. I fear that is a problem that will persist. Asking politely whether they've searched for duplicates can help, but ultimately I suspect the burden will be on the active members to identify duplicates and vote to close.

Thus, I believe our most productive angle is to try to have some simple tools and procedures that accelerate our work.

Answer 2

The answer is simple, however it is also an appeal for everyone -- rather giving the same answer again, use close as an exact duplicate option or (if you can't) flag.

Answer 3

Comments on questions are so regular I think it is reasonable to suggest the OP to search some of the prior questions on the site (of course in addition to closing the question if a duplicate is known).

This might include re-tagging with more appropriate tags, which should change the list of related questions on the right hand side I believe, or being even more explicit (e.g. you should peruse the tag likert for similar questions).

Of course such comments can be taken in a negative light depending on inferred tone (whether malice was intended or not). Perhaps we can add to our library of helpful responses a response for such a situation. My first stab at a response is;

Your question on ?X? is fairly broad, and has likely been covered in previous questions on the site. Have you perused the related questions on the site? See for instance [tag:your-tag-here] and/or [link to some search term](your link here).

I agree it can be annoying, but it just seems to be another part of the community moderation in an attempt to up-keep the site. This isn't terribly helpful in deciding when such prodding should be undertaken, but I guess I will leave that to the better judgement of regular contributors to particular topics.

Answer 4

I do see your point, but I guess it doesn't bother me as much as it does you. I also suspect that it's pretty easy to overestimate the number of questioners who are 'lazy' and not scanning through enough of the suggested 'related questions', although there's no doubt that happens. Moreover, I'm disinclined to vote to close questions that aren't egregious (ironically, I have done so several times over the last few days, though). Among other things, I suspect you're right that it would become a disincentive.

I think that part of the issue here is that there is an unavoidable tension between the simple question-and-answer nature of the site, and the intention to have the site build up a repository of information over time. Some of this is indicated by the discussion of how CV sees itself; there's a lot of overlap between this site and other venues on the internet. We want to build a lasting knowledge base, but a lot of new and one-time users will necessarily see this as a place to ask a simple question without thinking of those questions as something that gets stored for posterity.

In my opinion, two strategies are appropriate. I agree with @AndyW and others who suggest using comments that suggest other questions. Setting aside the effect this has on the questioner, it's been tremendously helpful for me. I can't count how many times I've gone to a question, and seen the comment '[this] question is pretty similar and has some really good answers', and then navigated over to the other question, and learned a lot more about the topic than whatever I would have said. Again, maybe that questioner never goes there, but that doesn't bother me; maybe a subsequent person ends up stumbling across the link and does end up reading the good question with the comprehensive answers. If nothing else, having lots of links pointing towards the best of CV helps Google to find the most helpful discussions.

It is definitely true that certain types of questions / issues come up over and over. Since I only know 3 or 4 things about statistics, I often end up over time with lists of questions on just a few topics where I've given similar answers. Eventually, I put together what I think of as my clearest and fullest statement on the issue. Afterwards, when those questions still arise, I can give a perfunctory answer and link to where I've discussed the topic more thoroughly. (I have also linked to other contributors' answers, of course, but I'm most familiar with what I've written.) For me at least, having tried to answer a question several times helps the ideas 'gel' in my head--I may well not have been able to give my best answer the first time. Furthermore, I think this process does tend to elevate better questions and answers over time, and is part of the resolution of the tension I mentioned. I guess what I'm saying is that I think the process is actually fairly healthy and organic, even though I can appreciate the frustration in the interim.

Answer 5

I noticed that when I start writing a question there is an automatic list of "similar questions" based on the words I use in my question. This can be helpful in not duplicating questions when the association of my word to words in other posts is good but it doesn't always work.