-7
$\begingroup$

This topic is motivated from this question: Why is leave-one-out cross-validation (LOOCV) variance about the mean estimate for error high?

The top answer starts with

Oct 2015. My original answer here was irrelevant, as pointed out in the comments below and in a couple of downvotes. I replace it with a new answer now (but save my original answer below).

This is a complicated topic, and I must admit that I don't have a full understanding of it.

This question has 11 up-votes, and it is arguably not a good answer. The user himself is saying that the original answer was irrelevant and that he "does not have a full understanding of it".

I think the problem has to do with the question being very popular, or the problem being a difficult one. In fact, it has been asked several of times, see e.g. this, this and this.

I am myself a newcomer, yet I still think answers should be up-voted by veterans, to avoid misleading answers. Or a mechanism for too popular questions' answers to be up-vote protected.


This post was to warn about a problem I see: misleading up-voted answers, answers that might be up-voted by beginners because they like it or empathize with the answerer saying "This is a complicated topic, and I must admit that I don't have a full understanding of it", or because they see an easy to understand answer, or whatever reason. And I think the problem comes from the original question being a hot topic.

Disclaimer: I do not think that the answer provided should be deleted, I just find misleading that it has that amount of up-votes when, in my opinion, does not answer the question, which is exactly

"Is there a mathematical formula, visual, or intuitive way to understand why that average has a higher variance compared with the k-fold cross validation?"

| |
$\endgroup$
  • 6
    $\begingroup$ This seems confused on whether you are referring to questions or answers. A post can't be both. A poster might post a question and an answer. Either way, the reputation needed for voting is a Stack Exchange-wide issue. Often the number of votes is not what you (I, we) think a post deserves: in abstraction we all agree, I imagine. Otherwise what decides what is "too popular"? The measure of popularity is voting; there is no independent measure of merit, just many personal opinions. In short, I don't think you've identified a problem we can do anything about, let alone a solution. $\endgroup$ – Nick Cox Jul 17 '18 at 9:44
  • $\begingroup$ I am referring to answers in popular questions. Up-voting a question is okay, it can just mean you have the same question or same doubt. Now I'm talking about up-voting an answer. "what decides what is "too popular"" Number of visits. Look, e.g. at this question: stats.stackexchange.com/questions/89251/… . It has far less views than the one linked, and they are equally old. $\endgroup$ – D1X Jul 17 '18 at 9:47
  • 1
    $\begingroup$ No; I don't think that will walk, let alone run. Too popular is a pejorative but popular means that many people have the same question, which is not a problem. Voting rules in any case are Stack Exchange-wide rules. In principle any proposal from down below can be discussed on its merits; in practice village hall discussions don't lead to changes in national or international policy. $\endgroup$ – Nick Cox Jul 17 '18 at 9:56
  • $\begingroup$ Hmm. Regarding this specific answer, it is mine. Do you think the thread would improve if I delete it? I was hesitant to delete it because I think there is useful discussion in the comments and the updated version of the answer clearly points to the relevant literature and to the other relevant threads on CV. Alternatively, I can delete the disclaimer about the original answer and the original answer itself, leaving only the "new answer". Would this improve the answer in your opinion? $\endgroup$ – amoeba Jul 17 '18 at 10:22
  • 1
    $\begingroup$ By the way, I did see your new answer in the linked thread that is now marked as accepted. Unfortunately you don't provide any specific evidence showing that LOO does indeed have high[er] variance, only some heuristic arguments (the same ones that I give in my answer). So I don't really see how your answer resolved the issues that I raised in my answer ("However, note that Hastie et al. do not give any citations here, and while this reasoning sounds plausible, I would like to see some more convincing evidence that this is the case.") $\endgroup$ – amoeba Jul 17 '18 at 10:26
  • $\begingroup$ @amoeba To the first question, yeah, you are right, there is interesting discussion. While I do not think it should be deleted, as it indeed contains valuable information, I also don't think the answer should have that amount of up-votes. Because pointing to the relevant literature does not answer the question. $\endgroup$ – D1X Jul 17 '18 at 10:32
  • 3
    $\begingroup$ In practice readers will not all read all the links. I think you should declare an interest as an participant in these threads. You're fine asking a question arising partly out of your experience, but declaring an interest in the examples is still advisable. $\endgroup$ – Nick Cox Jul 17 '18 at 10:43
  • 1
    $\begingroup$ @amoeba Then, no offense intended, read again. I think I do answer the user's question and clarify his misconception. Please do note that it is the user himself that has decided to accept my answer. What you want or request might not be what that question is asking, it may be as simple as that. By the way, I do think this is an interesting discussion. $\endgroup$ – D1X Jul 17 '18 at 11:09
  • 4
    $\begingroup$ A "declaration of interest" means here merely that you should make explicit that you're a party to the threads you cite. It's one thing to make assertions about the merits of other people's threads and not quite the same thing to make assertions about threads in which you participate. $\endgroup$ – Nick Cox Jul 17 '18 at 11:12
  • 1
    $\begingroup$ This is getting too messy. Your general suggestions (which I don't see merit in, but that's just one view) are too tangled up with examples in which you have a personal stake. Your main example is compromised by strong personal opinion such as "definitely not a good answer". I would rewrite the question and go for neutral examples. It is not out of order to cite threads in which you participate. but this long discussion with @amoeba suggests that in this case the examples raise too many side-issues. $\endgroup$ – Nick Cox Jul 17 '18 at 11:18
  • 1
    $\begingroup$ For the record, I took absolutely no offense and I don't think you need to delete this thread. $\endgroup$ – amoeba Jul 17 '18 at 11:30
  • 1
    $\begingroup$ I am not suggesting deletion, just rewriting. That should be clear. There are upvoted answers, so I don't think you can delete it any way. $\endgroup$ – Nick Cox Jul 17 '18 at 11:31
  • 2
    $\begingroup$ You shouldn't just say "a bad answer" as if it were objective fact. Regardless, the reaction to what you think is a bad answer is comments or a better answer explaining your view of the question. $\endgroup$ – Nick Cox Jul 17 '18 at 12:35
  • 1
    $\begingroup$ Bad answer, wrong answer, or, as you say "answer that does not make a lot of sense". It was just me trying to get to the point. Call it as you wish. $\endgroup$ – D1X Jul 17 '18 at 12:38
  • 3
    $\begingroup$ No; I don't think you can dismiss the point as merely one about wording. It's got to the point where adding another answer is a better reaction. $\endgroup$ – Nick Cox Jul 17 '18 at 13:02
7
$\begingroup$

This site allows any user* to vote +/- on any question and any answer (unless some very specific circumstances), you can like it or not. The natural consequence is that the older Q&A's and the more popular ones, get more votes because they are more visible, they had more time to accumulate the votes. If you don't agree with some vote, you can give an opposite one. This is not something that we can change, as this is a general feature of the StackExchange network and we have no influence on it whatsoever. If you wish, you can post a feature request on http://meta.stackexchange.com/, but I doubt that it would have any chances of succeeding, because this is how this network works from the beginning.

* - More precisely: any user with reputation at least 15 to vote up or with reputation 125 to vote down, as noted by Nick Cox.

| |
$\endgroup$
  • $\begingroup$ This answers the question, yet I'm afraid I'm going to delete the question, as suggested by comments. $\endgroup$ – D1X Jul 17 '18 at 11:26
7
$\begingroup$

I agree with the answers by @Tim and @amoeba but a dimension in the discussion deserves more elaboration than comments make easy.

The question -- a little perversely, as at the moment of writing you have to read all the links or all the comments to discern that the OP is a party to the threads mentioned -- conflates several issues: various specific reactions to some related threads, on which I won't comment further, and some general questions.

I'll re-phrase the more general issue as two distinct questions. Clearly I am not writing on behalf of the OP.

  1. If I see a question or answer that I regard as bad, or not good, what should I do if no-one has pointed that out?

What you can and should do is react with comments explaining what's wrong and what's right or a new answer that explains more fully. (Asking a new and different question and even answering it too might also be a way to clarify a tangled discussion.) It's fine to cross-refer to other answers or questions. Downvoting is within the scope of dissent, although it's generally better to accompany a downvote with an explanation, unless someone has done that already (in which case you can add +1 to their explanatory comment(s)).

  1. What is the community to do about questions or answers upvoted more than they deserve? Should there be different rules for voting to cope with this problem?

The question fails at the first hurdle. There is no community (e.g. CV or SE-wide) view on, or meaning to, what are bad questions or answers other than what the community has collectively conveyed through voting, answering and commenting. A large number of upvotes can mean an excellent post or an over-rated post, but such judgments are personal opinion. The situation is no different from the last election in which you voted for a losing candidate: the stupid people won, but that's democracy, and unsurprisingly their view is different.

Even if there were an identifiable community view, a change in voting rules remains a Stack Exchange-wide matter.

| |
$\endgroup$
4
$\begingroup$

Regarding your general feature request I agree with @Tim's answer: you can raise it on Meta.SE if you want but I think it has zero chances of being implemented.


Regarding your specific example of my answer https://stats.stackexchange.com/a/90903 you are right to notice an important issue: the 1st version of my answer was entirely missing the point (it did not contain any false claims but it did not answer the question either). Nevertheless it got 12 upvotes. As I wrote in the comment from 2015:

@Jake, you are right, my answer (from over a year ago) does not make a lot of sense; I have already noticed it myself but forgot to deal with it. Funny that it got 12 upvotes :-/

Too bad! Lesson to learn here: answers that are missing the point can get many upvotes.

My advice: if you spot such a case, leave a comment and maybe even mention it in chat or on Meta, and perhaps other people will agree with you and help downvoting the answer. The original answerer, if they are still around, can then either delete the answer or fix it.

This is exactly what happened in this case: back in 2015, motivated by comments and downvotes, I have fixed my answer, and now it gives exactly the same reasoning (right in the 1st paragraph under the disclaimer) as your answer here https://stats.stackexchange.com/a/357404. So I'm not quite sure what's your problem with my answer now in 2018.

| |
$\endgroup$
  • $\begingroup$ The questions are not the same. The exact question the user makes in what you answered is exactly "Is there a mathematical formula, visual, or intuitive way to understand why that average has a higher variance compared with the k-fold cross validation?" Again, I still think your answer, while instructive, does not answer this. While it might resemble mine, (I don't think so, but let's just assume it) we are NOT answering the same question. What the question I answer asks, is, precisely If the above argument is right, why would models learned with leave-one-out CV have higher variance? $\endgroup$ – D1X Jul 17 '18 at 11:41
  • $\begingroup$ The point I am trying to make here is that two answers may be equal yet one might be correct and the other one not, depending on the question. In one of the questions the user specifically asks something and in the other one the other user asks another thing. $\endgroup$ – D1X Jul 17 '18 at 11:45
  • $\begingroup$ You are making a distinction between "Why X?" and "Is there an intuitive way to understand X?". I don't quite understand this distinction. $\endgroup$ – amoeba Jul 17 '18 at 12:02
  • $\begingroup$ No, that is not true. I myself don't understand why you don't understand what I mean. It is clear that the one who asks your ( let's call it like that for simplification ) question has already understood the meaning of variance in the context of CV. Mine has not. Mine is asking about variance of the model and is confused about which variance we are talking about, which is something I do my best to clarify in the answer. $\endgroup$ – D1X Jul 17 '18 at 12:07
  • $\begingroup$ The original askers might indeed have had different level of understanding, but nevertheless they were asking essentially the same question. Anyway, I decided to edit my answer, see updated version at stats.stackexchange.com/a/90903. I inserted specific links to simulations and papers to support my claim that this topic is controversial. $\endgroup$ – amoeba Jul 17 '18 at 21:48

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .