14
$\begingroup$

Stack Exchange recently started a collaboration project with the University of Melbourne, in an attempt to improve the automatic detection of duplicate questions.

More information on the project and a call for help from the community can be found here: Project Reduplication of Deduplication Has Begun!

In it, Tim Post mentions that Doris Hoogeveen (me) will post a message on the different meta sites to let you know that we're ready for help. This is it!

As a brief summary, the project is about finding ways to automatically determine if two questions are duplicates, and also to detect the opposite: falsely tagged duplicates. To validate our methods, we need manual judgements from people who really know the data. This is where we need your help. It is impossible to manually annotate all the data from Stack Exchange, and so we have applied several filters to make sure we only show you questions pairs with a high likelihood of being duplicates. This does not mean that most of the pairs will be duplicates. Instead, it means that the percentage of duplicates in there will be much higher than in a random sample, and it means that the pairs will be informative for our model, even if they are labelled as non-duplicates.

Participation is completely optional, and there's no minimum time requirement. There's a reputation requirement however: only people who have duplicate question tagging rights on the site can participate. That is people with at least 3000 reputation points, or a gold badge. This is to ensure the annotations are compatible with StackExchange's quality standard.

If you are interested in participating in the project, please head over to the annotation interface, which can be found here: http://hum.csse.unimelb.edu.au/se-annotate/

You can log in with your Stack Exchange account, so no sign up is required. The site also contains some more information on the project, which might answer some questions you may have. And I'm also happy to answer them here.

$\endgroup$
31
  • 2
    $\begingroup$ "That is people with at least 3000 reputation points, or a gold badge" ... do you mean something like "a gold badge in a relevant tag" or "any gold badges at all" there? [Edit: from the linked post, it's gold tag badge; aside from Anony-Mousse (19.4K) every other gold tag badge holder here (me, gung whuber, Peter Flom completes the entire list I think) has at least 50K reputation. So 3K reputation includes everyone who can do it here.] $\endgroup$
    – Glen_b Mod
    Commented Nov 11, 2016 at 3:54
  • 1
    $\begingroup$ Also, are these assessed by multiple reviewers (we don't always agree! Far from it) or is it one assessment per pair of potential dupes? $\endgroup$
    – Glen_b Mod
    Commented Nov 11, 2016 at 3:56
  • $\begingroup$ Wow, it's glacially slow. Hmm. Will there be rendering of Mathjax? I haven't hit one yet $\endgroup$
    – Glen_b Mod
    Commented Nov 11, 2016 at 3:59
  • 1
    $\begingroup$ The first three I saw weren't even remotely close to duplicates. $\endgroup$
    – Glen_b Mod
    Commented Nov 11, 2016 at 3:59
  • 1
    $\begingroup$ Also, date of posting may sometimes be a relevant consideration if one is trying to work out which one is to be treated as canonical -- other things being equal, choosing the older one as canonical (essentially to break the tie) seems to be common. $\endgroup$
    – Glen_b Mod
    Commented Nov 11, 2016 at 4:09
  • $\begingroup$ I'm looking into the slowness. That's far from ideal for annotating of course, so hopefully I can fix it. Yes, some questions will be very far from duplicates. Unfortunately I don't have a method to filter those out yet. This is exactly why we are doing this project! I left out the date to avoid a bias towards choosing the older one as the canonical one, but let me discuss this again with the other researchers. I'll get back to you on that. $\endgroup$
    – Monozygotic
    Commented Nov 11, 2016 at 4:18
  • 5
    $\begingroup$ Ah, well, if you want to avoid people relying on the date, perhaps that could be mentioned on the first screen. I'm not sure why that should be a problem since it's one of the criteria used in practice. $\endgroup$
    – Glen_b Mod
    Commented Nov 11, 2016 at 4:27
  • 1
    $\begingroup$ Is it intentional to offer closed questions for consideration? $\endgroup$
    – Glen_b Mod
    Commented Nov 11, 2016 at 6:26
  • 6
    $\begingroup$ Regarding gold tag badges, it's worth remembering that it takes 1k upvotes to earn them. Even if a user had never done anything else, that's 10k rep. In theory, someone could have given away >70% of their rep through bounties, but I'll bet it's never happened. If the idea is just people w/ vote to close privileges that's fine, or if it's that there is a 2-tiered system w/ additional credence given to the ratings of gold tag badge holders, that should be clarified. $\endgroup$
    – gung - Reinstate Monica
    Commented Nov 11, 2016 at 13:44
  • 2
    $\begingroup$ There appears to be almost no relationship between the any pair of questions that I've reviewed. For each pair, I've been trying to sort out what the common thread might be -- in case the relationship is non-obvious -- but the best I can tell is that the narrowest commonality between the questions is that both merely are on-topic. $\endgroup$
    – Sycorax Mod
    Commented Nov 13, 2016 at 1:01
  • 1
    $\begingroup$ @Sycorax, that's completely fine. I'm getting the impression that many people think they are presented with the output of a system, and therefore expect to see many duplicate pairs. This is not the case. The annotated pairs will be used to build such a system. You will be presented with many more non-duplicate pairs than duplicate pairs, but both types are useful for us. $\endgroup$
    – Monozygotic
    Commented Nov 13, 2016 at 23:06
  • 7
    $\begingroup$ Perhaps "we have applied several filters to make sure we only show you questions pairs with a high likelihood of being duplicates" is creating that expectation - it might help to edit your question to clarify the point. $\endgroup$
    – Scortchi Mod
    Commented Nov 14, 2016 at 9:30
  • 2
    $\begingroup$ @Firebug, we do use historical data from Stack Exchange, but after doing a manual analysis we found out that there are quite a few duplicate question labels missing. Improving this is essential for building a better system. So what we're doing now is showing people about 0.0003% of all question pairs to annotate manually, which will result in an estimated 45% increase in the number of duplicate question labels. Looking at the review queues is a good suggestion. I'm not sure if I have access to those, but it's worth having a look. $\endgroup$
    – Monozygotic
    Commented Nov 15, 2016 at 0:29
  • 1
    $\begingroup$ @Monozygotic I find myself unsure of what the desired boundary between "related" and plain "not duplicates" should be. A number of pairs on broadly the same topic have come up, but I have been marking as not duplicate on the assumption that "related" would be for cases where I'd want to link the two (i.e. where answers might be somewhat helpful even if they're not directly answering the question), but it occurred to me that this might be stricter than you want. Can you clarify? $\endgroup$
    – Glen_b Mod
    Commented Nov 16, 2016 at 5:03
  • 1
    $\begingroup$ @ssdecontrol, haha I'm sure they would be! A combination of two simple methods is used to do the selection. You can find the details in this paper: people.eng.unimelb.edu.au/tbaldwin/pubs/sigir2016-webqa.pdf The motivation behind it is threefold: 1. we wanted to use fast methods which would result in a relatively high percentage of duplicate questions in the set; 2. we wanted the set to include pairs that standard IR methods fail on; and 3. we wanted to catch the low hanging fruit. Let me know if you have questions about the paper! $\endgroup$
    – Monozygotic
    Commented Dec 7, 2016 at 22:05

6 Answers 6

Reset to default
8
$\begingroup$

Possible problem:

When presenting this question:

How to find set of directions in Stahel-Donoho outlyingness measure?

it screws up over the "link" to the paper listed at the bottom, though this is probably the fault of the poster. Anyway, I thought I'd give it a mention:

Original:

enter image description here

As presented:

enter image description here

(Another issue:) It seems to have a fixation on "What is the meaning of p values and t values in statistical tests?", which it has paired with ... I don't know, maybe 5-7 different posts so far.

At least one of your algorithms seems to be infatuated with this post. They should get a room.

If that's what's supposed to happen then fine. But if not, I figured you would probably want to know. Two other posts have come up several times each.

$\endgroup$
4
  • $\begingroup$ :) I'm aware of some posts showing up much more often than others. In the selection process I have filtered out some that really showed up a ridiculous number of times. They will have to be investigated separately. The others are actually interesting. I'm hoping to find out why they show up so often. Such cases will appear in the system we'll build too, so more annotations for these are good to have, to see if we can avoid identifying them as duplicates all the time. $\endgroup$
    – Monozygotic
    Commented Nov 11, 2016 at 7:03
  • $\begingroup$ Not sure what happened to the link to the paper, but I think I may leave that for now. I don't think it's a problem that will show up often, and I don't think it will affect people's annotations. It just shows I'm not actually a web developer. ;) $\endgroup$
    – Monozygotic
    Commented Nov 11, 2016 at 7:05
  • $\begingroup$ Yeah, it's not a critical problem, but I thought I'd point it out. $\endgroup$
    – Glen_b Mod
    Commented Nov 11, 2016 at 7:06
  • $\begingroup$ I do appreciate that! $\endgroup$
    – Monozygotic
    Commented Nov 11, 2016 at 7:11
7
$\begingroup$

Unfortunately in my 164th review I misclicked as duplicate two posts that were very clearly not duplicates. There doesn't seem to be any way to go back and fix it. [The site sort of loads the page and then seems to "shift" the pages as it finishes putting everything on it/redraws it - my browser was being slow to load after review 163 and I quickly saw the next was not a duplicate, but the screen for review 164 "re-drew" just as I was about to click, changing which button was under the cursor.]

This (not being able to fix the resulting error from a misplaced click, however generated) is a major problem -- as far as I recall I have only marked one previous pair as duplicate so (if I remembered that right), there's now a 50% error rate in my marking as duplicate. With duplicates so rare in your current data set, that's likely to screw up what data you're likely to end up with pretty badly (certainly the part of it that's from me, anyway)

This is likely to be a more widespread problem than just me, since the chance of such a misclick goes up the more other people review too -- and not everyone will notice. If only one in several hundred posts is a duplicate the chance that a substantial fraction of the marked duplicates is nothing of the kind will become quite high (e.g. imagine people make a mistake about one time in 50, marking a duplicate as not or vice-versa ... and also that about one pair of posts in 200 is really a duplicate... then the majority of posts marked as duplicates won't be duplicates!)

Not providing a way to way to review and fix that is likely to leave the resulting data set with greatly decreased value.

$\endgroup$
5
  • 1
    $\begingroup$ sorry for the late reply. I was abroad with very limited internet access. I understand your concern. I'll try to find your particular mistake and fix it, but of course that doesn't fix the general problem you're presenting here. I'm actually not expecting big problems with this, for the following three reasons: $\endgroup$
    – Monozygotic
    Commented Jan 9, 2017 at 22:13
  • $\begingroup$ 1. when multiple people annotate the same question pairs, we'll be able to find the odd annotations and review them (or discard them); 2. the data presented in this annotation project is not the only data that is used in experiments. These are the pairs that seem to confuse the standard methods most, but we'll also use the existing duplicate questions, so the error rate will be lower than the 50% you indicated; 3. 100% perfect data is not necessarily guaranteed to give you the best performing system. Having a few errors in there may result in more robust models. $\endgroup$
    – Monozygotic
    Commented Jan 9, 2017 at 22:13
  • $\begingroup$ I was concerned that you may not have enough reviews that you'd have multiple responses on most of the pairs, and while I agree that a few errors among the set of what is labelled as duplicates is not a bad thing, having a high proportion would be (and it seemed to me that a proportion even remotely near that high would be a problem). If you're satisfied, then that's fine, but it would make sense to me to have at least a "previous" button or link so you could fix the one you just did if you knew you made a mistake (of course it won't catch the mistakes you don't notice). $\endgroup$
    – Glen_b Mod
    Commented Jan 10, 2017 at 1:55
  • $\begingroup$ However, I guess you're done collecting data now. $\endgroup$
    – Glen_b Mod
    Commented Jan 10, 2017 at 2:10
  • $\begingroup$ yes, you're right, and I should've mentioned it in my reply: having a back button to allow people to fix things is a good idea! I wish I had thought of it at the start of the project. I'll stop collecting data this weekend, and will let people know either later today or tomorrow morning. Thank you so much for all your help! $\endgroup$
    – Monozygotic
    Commented Jan 11, 2017 at 4:28
5
$\begingroup$

Problem:

This question has MathJax $\beta$'s in it. They're rendering as $?$'s for me. ($\hat{?}$ amuses me in particular -- it tickles my funny bone for some reason):

enter image description here

Note that the rest of the MathJax is fine there.

$\endgroup$
3
  • $\begingroup$ I fixed this problem, but now I have created a different issue: overlapping text in inline MathJax. I'll keep working on it. $\endgroup$
    – Monozygotic
    Commented Nov 11, 2016 at 5:07
  • $\begingroup$ Yep I experienced some of that now. Not enough to stop one from figuring out if it was a duplicate but it would be a lot easier without it. $\endgroup$
    – Glen_b Mod
    Commented Nov 11, 2016 at 6:06
  • $\begingroup$ I managed to fix some of it, but strangely enough not all it seems. I'm gonna need more time for this.. $\endgroup$
    – Monozygotic
    Commented Nov 11, 2016 at 7:18
4
$\begingroup$

This reminds me a bit of the issues with the review queues. The interface is not as nice as the regular site, and it doesn't present the information the way I'm used to looking at it. It would be good to have links to the threads so I can view them on the normal site. Right now I'm spending some time searching for them.

$\endgroup$
7
  • $\begingroup$ Yeah, I found myself missing some of the usual cues, but the information page indicates they're deliberately not linking to the originals, presumably so we avoid using cues the AI won't have access to. $\endgroup$
    – Glen_b Mod
    Commented Nov 11, 2016 at 23:40
  • 3
    $\begingroup$ @Glen_b, I think of this as trying to get the true labels that can be used to train a model subsequently. I don't think it should matter if we have access to more or different information than the AI will. $\endgroup$
    – gung - Reinstate Monica
    Commented Nov 12, 2016 at 0:54
  • 1
    $\begingroup$ Glen_b, that's exactly why we left them out, but gung, you do have a point. I'll discuss it again with the other researchers. $\endgroup$
    – Monozygotic
    Commented Nov 12, 2016 at 23:47
  • 1
    $\begingroup$ @Monozygotic, from a machine learning standpoint, it doesn't matter where the labels come from; the algorithm will simply take them as a given. There is no reason for us to be limited in some way. Moreover, if I'm going to rate these, I'm probably going to look at the original pages one way or another to figure it out, so there isn't much point in trying to obscure it. (Also, for Glen_b to be notified, you need to ping him by putting "@" before his username as I did for you at the beginning of this comment.) $\endgroup$
    – gung - Reinstate Monica
    Commented Nov 13, 2016 at 0:01
  • $\begingroup$ @gung, interesting, I didn't know the @ triggered a notification. Thanks for telling me! We had some initial ideas about obscuring the URLs, along the lines of giving people minimal information to mimic somewhat what a computer has to deal with. After getting a lot of feedback from people we came back from that and added tags, comments and answers. We may have to take it further even and add URLs too. I'll discuss and will get back to you. $\endgroup$
    – Monozygotic
    Commented Nov 13, 2016 at 0:24
  • $\begingroup$ You don't need the "@" for the author of a post (question / answer) when the comment is below the post at issue. (Thus, eg, you don't need it for me here.) But you do need it to notify other commenters. (So you would need it for Glen_b to be notified here, or for me to be notified regarding my comment to the question post at the top.) $\endgroup$
    – gung - Reinstate Monica
    Commented Nov 13, 2016 at 0:31
  • 4
    $\begingroup$ Regarding the AI, there is really no reason (at all) that we should try to mimic the conditions the computer has to deal w/. If you had true labels already, you could do this as an experiment to compare how well your algorithm performs relative to how well a person can do w/ the same information. However, you need the true labels in advance (that's what we're doing here), & that is an empirical study in cognitive science to investigate an entirely different issue. It really makes no sense in this context. If you wanted, you could ask about this on the main site or on the Psychology & Neuroscience SE site. $\endgroup$
    – gung - Reinstate Monica
    Commented Nov 13, 2016 at 0:35
3
$\begingroup$

The project is drawing to a close, and I am planning to take down the annotation interface this weekend.

Thank you very much to everyone who helped out, and thank you for all the valuable feedback and suggestions. Hopefully the collected annotations will lead to the development of better duplicate question detection systems, especially for StackExchange.

$\endgroup$
-1
$\begingroup$

I am really concerned that this will prevent people from asking questions or getting help on stack exchange. Here are my reasons:

  1. Sythesis Questions: On rare occassions, I have re-asked a question to get a sythesis or concluding answer. I do this because the question has no definitive (well-defined) answer either because of the following reasons below:
  1. The answer changed over the years over the multiple times it has been asked
  2. It has been asked with so many different variations and these variations have completely different answers
  3. It has been answered in a non-direct means often with extra-enous information
  1. Potential for Overfitting It will take a long time for accuracy and precision to be good enough to avoid overfitting and be able to release it into the wild. How do I know there will be overfitting? Well, I personally have noticed that suggested answers for coding on stack overflow do not match up well usually with the questions I am asking while I am typing it. Considering how this question matching algorithm performs on stack exchange, I think this shows that the unstructured data from stack exchange posts is prone to overfitting and we have to be careful of that.
  2. Skewed toward/against the questions of new users: Depending on how you code it the algorithm is going to be skewed toward or against news users in a signficant way. Right now my guess is toward new users; however, I want to show both because we have to be aware of it. I think you need to closely monitor growth new users on stack exchange to make sure there is not a significant drop in activity as you more you automate this process.
    1. [Reason for the skew against] Basically new users are prone toward producing missing values this text mined dataset then older users because reputation system promotes such behavior. Consider for a minute on what @Glen_b pointed out with the [0] and the misspelled name. Do you think that these syntax errors are random human error? No, they are intentional and have non-random reasons much like the missing values or you might find in a database. The major reason the person probably misspelled it the way he did was because of stack exchange limits both the content and type of questions new users can ask especially well documented questions. Personally, I cannot tell you the number of times I have had misspell or format a question/answers in an odd way just so I can keep my references while getting around the reference limit from the subjective filter, link limit, spam filter (Not the actual question text but the actual references themselves).
    2. [Reason for the Skew Toward] On the other hand, new users are less familiar with the site are more prone not to be familar with how to navigate stack exchange and thus will ask duplicated questions.
  3. StackOverflow will be an outlier in performance: In terms of behavior of the algorithm I think we can predicted that stack overflow will be an outlier for two major reasons: 1) There is a lack of good syntax thesaurus between the top 5 most popular languages. So to ask how to code something in one language means there are at least 5 valid duplicates of how to code that in a different language. 2) The dependency on tags as classifier, cluster, or filter of question @IanRingrose mentioned. So it either going to perform really well for stackoverflow or really poorly for stackoverflow.

The duplicate questions tag does not automatically delete questions, so Synthesis Questions, Overfitting, and Outliers are currently dealt with because people can object to having a closed thread or duplicate tag. I am worried that this safe guard from duplicate questions tag will be removed when one releases Project Reduplication of Deduplication into the full wild of stack overflow.

P.S. Also I just posted an example of reason 1.2 (Synthesis Question) in Cross Validated

$\endgroup$
4
  • 2
    $\begingroup$ I understand your concern, and I think you've made some valid points if the current system would be replaced by a fully automatic one. I don't think that is StackExchange's plan though. As far as I know, right now they use a very simple retrieval mechanism based on content word matching, to potential duplicate questions. My goal is to improve this, which means users will get better suggestions. $\endgroup$
    – Monozygotic
    Commented Dec 10, 2016 at 23:18
  • 1
    $\begingroup$ StackExchange might decide to automatically label duplicates with a very high confidence, but since duplicate question detection is a difficult task, I don't think this will include many cases, and so for the vast majority, human confirmation will still be necessary. The suggestions will just be better. $\endgroup$
    – Monozygotic
    Commented Dec 10, 2016 at 23:21
  • $\begingroup$ Overfitting, skews in the data, differences between different sites... of course these are all things I will have to watch out for. These problems appear in many classification tasks, and so of course I will look out for them and try to minimise their impact. Your first point is a really tricky one, which I'll definitely need to think about. And your point about misspellings that are done on purpose is new to me. That's a really interesting thing, which I hadn't thought about yet. Thanks for pointing that out! $\endgroup$
    – Monozygotic
    Commented Dec 10, 2016 at 23:24
  • $\begingroup$ Does this take away some of your concerns? $\endgroup$
    – Monozygotic
    Commented Dec 10, 2016 at 23:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .