-8
$\begingroup$

There are a few topics on this site (unbalanced data, up-weighting, accuracy, F1 score, precision/recall etc) that whenever a user asks a question regarding one of these methods/metrics, a few high reputation users immediately ignore the question at hand and post copy pasted responses about how that method is bad. Usually the question has nothing to do with the merits of that method/metric and rather the implementation of it and so this response has nothing to do with the question actually being asked.

For example see Class imbalance: training set is balanced but test set is imbalanced, how to train?

The user asks about the implementation of training a model with under-sampled data and a user responds with 3 comments back to back about how imbalanced data isn't a problem (this is entirely unrelated as the asker was only under-sampling for computational reasons) and how accuracy/precision/recall etc shouldn't be used.

Again the user asked no questions about whether someone should use any of these methods; they simply asked if they were to use them how would they do it.

The responses by these users are spam. They detract from the actual question being asked and make new users feel frustrated when an honest question is answered with "don't use that; it is bad".

I will add that all of the methods that these users dislike are widely used, in peer-reviewed papers, in statistical textbooks, across various internet sources. I am not saying that this means they are in fact valid, but I am also saying that a few guys here on cross validated agreeing that they are bad does not either make them invalid. Further the only proof offered against the validity of these methods are cross-validated posts written by the commenter (that have since been closed). If the user wants to discuss the merits of those methods then there are already questions for that.

Should comments/answers that answer something other than the question being asked be allowed?

See other examples here:

Model accuracy versus F1

Is it appropriate to use NPV and PPV when performing undersampling?

Reporting F1 Scores

These are just a few examples, I am sure there are others

$\endgroup$
3
  • 3
    $\begingroup$ Spam is a defined, technical word, within this SE network of sites. For the definition, see meta.stackexchange.com/questions/58032/… $\endgroup$ Dec 10, 2021 at 2:17
  • $\begingroup$ Okay, so when a user critiques these types of posts by linking to their website, Twitter, and/or textbook without disclosing the affiliation then that meets this network’s definition of spam $\endgroup$
    – astel
    Dec 10, 2021 at 2:57
  • 10
    $\begingroup$ You're allowed to disagree with a method, practice, or advice, but the fact that you disagree doesn't make the method, practice, or advice an instance of spam. The definition of spam is very specific. $\endgroup$
    – Sycorax Mod
    Dec 10, 2021 at 3:12

3 Answers 3

10
$\begingroup$

We should all agree in abstraction that good answers (and good comments) should be correct and relevant, but who is the judge of either? Even in a community defined by overlapping technical interests, there can be marked disagreements about principles or good practice. (Supposedly, R.A. Fisher defined variance informally as the attitude of one statistician to another.)

I see no policy issue here.

  • If existing answers or comments are inaccurate or irrelevant, that means there is always scope for posting better answers or comments.

  • Further, it can be in order to answer or comment that a question is misguided and that there is a better way to proceed or a better way to pose the analysis. As in teaching, sometimes a poster needs an explanation that they would be better off backing up and seeing what they are doing differently. CV is not a help line with an absolute commitment to try to adopt the poster's point of view; it is a forum with freedom to express other points of view.

  • In addition to answers, comments can be used to broaden the discussion or even to make wry or witty asides. If a comment strays too far from the main topic, it can always be flagged for deletion.

In the question the examples given are some distance from my usual territory so I won't attempt any judgment on what in particular threads is relevant or irrelevant or fair comment or not.

However, calling such answers spam is unhelpful (if not offensive, as my impression is that almost all contributions to CV are made in good faith). StackExchange has a specific definition of spam: "Exists only to promote a product or service, does not disclose the author's affiliation." Other way round, we have a clear policy that whatever is spam should just be flagged for deletion by moderators. So, again, if anything really is spam there is no policy issue to be discussed here.

There are some strong generalisations here and [in the original post] just one example by way of a particular thread. I guess that you don't want to single out particular people but the difficulty remains that strong claims require strong evidence.

$\endgroup$
6
  • $\begingroup$ To address your comment that I have too much faith in statistical journals etc. if you read the question, I state that I don't believe that their existence in journals makes it valid, but I also don't believe that a few posters on cross validated agreeing makes it invalid. By continuously posting on every topic that these methods are bad, the user is attempting to shift the narrative rather than inform. It is not settled that these methods are bad. See for example the original example I linked. Another very respected user argues for these methods in the comments $\endgroup$
    – astel
    Dec 9, 2021 at 15:49
  • 7
    $\begingroup$ Thanks for your comments. If posting similar comments to similar questions counts as spam, then many of us are spammers too. You're misusing a useful word there. $\endgroup$
    – Nick Cox
    Dec 9, 2021 at 16:30
  • 5
    $\begingroup$ We have a concise community definition of spam, as allowed grounds for flagging an answer: Exists only to promote a product or service, does not disclose the author's affiliation. $\endgroup$
    – Nick Cox
    Dec 9, 2021 at 16:51
  • 3
    $\begingroup$ I basically agree with all of this, but regarding the last paragraph, I don't see such a need for extensive evidence. The pattern is pretty familiar to me from my experience here. I'm happy to believe that when someone asks a question about, say, stepwise regression, that some users will immediately comment that it's a bad thing to do. $\endgroup$ Dec 9, 2021 at 20:07
  • $\begingroup$ Again, I think the thing missing in all of this is whether or not these methods and metrics have been proven bad. You can read through the comments in the first example I provided that show enough argument about them that maybe you’d agree they haven’t $\endgroup$
    – astel
    Dec 9, 2021 at 21:42
  • 6
    $\begingroup$ @gung makes a very good point. Many of us do have hobby horses -- mine include thinking that normality tests, quantile binning and several popular plots are oversold. Hence posters with prejudices -- which should be based on reasoned stances -- can often be observed making the same points in different threads. As Richard Feynman almost said, the same questions have the same answers (or in CV provoke the same comments). The issue is whether this is a problem, and I don't think it is. $\endgroup$
    – Nick Cox
    Dec 12, 2021 at 12:37
9
$\begingroup$

As the user whose comments you take offense at, I would like to offer a word of explanation. (Incidentally, I would have appreciated a pointer to this Meta question, e.g., in a reply to one of my comments. Complaining about me behind my back is not good style.)

I agree that I will often post comments under questions about imbalanced data and over-/undersampling, even if the questions themselves are not about whether this is a good idea, but about how to go about over-/undersampling (or similar). True. My reasoning is that over-/undersampling is, in my opinion, akin to shooting yourself in the foot. When someone asks for the best way to shoot yourself in the foot (use a shotgun? a crossbow?), I'll post a comment pointing out that the foot-shooting itself may be misguided.

I personally think that these comments add value. And note that I am posting them as comments, not as answers - which they are not, as you point out.

Now, you could make a point that if I did post these as answers, they could at least be downvoted if enough users felt the way you do. True again. I think reasonable people can disagree about whether my comments should really be answers. But judging from the upvotes my comments sometimes get, I suspect that any such answers would also garner their share of upvotes - and getting rep out of these non-answers-only-posted-as-answers-so-people-can-downvote-them feels at least as spammy to me as posting comments.

Funnily enough, I posted a Meta question about this precise issue here: What to do about "wrong" questions? The top voted answer there was indeed to post an answer.

You also complain that I use boilerplate comments. Again, I plead guilty as charged. Why am I doing it? Because we get one or more of these questions every single day. Every one a slightly different question about shooting yourself in the foot. One day it's shotguns vs. crossbows. The next day it's about right foot or left foot first. Then whether it might not be best to start by shooting yourself in the calf first. I still believe that a boilerplate answer "don't shoot yourself in the foot at all" is useful, even if the question is one about shooting yourself in the hand or in other body parts. After all, the alternative would be to either spend a lot of time crafting tailored answers to each separate question, which I honestly don't have the time or inclination to do, or not to do anything. Call me obsessive, but I believe that if my comments make someone think twice about unbalanced data and over-/undersampling, then they are useful, and not posting anything because I didn't have the time to do a thorough job but did not want to post boilerplate would have been a loss.

Also, note that we have a very useful Meta question here: How best to use the review queue? It contains a lot of boilerplate you can use for comments, and which I also use often, especially for what I think are self-study questions. Yes, boilerplate has its place.

Incidentally, it is not the case that I post my boilerplate "without even considering the question at hand", as you allege. Per above, my comments may not address the specific foot-shooting question, but also per above, I believe it's better to have a short imprecise comment than nothing at all.

Finally, you note that my position is a minority one, and that lots of people are thinking deeply about the imbalance "problem" (you will excuse the scare quotes), SMOTE etc, even in peer reviewed articles. Yes, that is true. With Nick Cox, I have little faith in the statistics published in peer reviewed journals and conferences, especially in papers from non-statisticians. Among whom I count ML scientists, who are typically far more competent in computer science, algorithm design, database operations, software development and straight-up coding than in statistics. I started my statistical "career" in psychology, and the statistical half-knowledge that is published in psychology journals is breathtaking - because authors, reviewers and editors are psychologists, not statisticians, and they frankly are misremembering half of what they learned in their one undergrad stats course. Thus, the fact that there are peer-reviewed articles on foot-shooting does not carry much weight with me. Sorry to be blunt.

Conversely, yes, I know that I am one of the more vocal people here when it comes to the topic under question, but I am certainly not the only one to weigh in here. It is also not the case that the only arguments I give are posts I wrote myself. Yes, I do link to my own posts in my comments - because (a) I know them best, and (b) I wrote one of them for the express purpose of having a canonical question to link to (and invested quite some time and effort in it), after noticing this deluge of "class imbalance problem" questions. I assume you have seen the comments below this question of mine, which to me indicate my mystification is shared by some.

Believe me, I would be the first to rejoice if this particular stream of "wrong questions" were to stop, and we could go back to answering interesting questions. Unfortunately, it doesn't.

If the moderators believe that my commenting is not welcoming to new users, and that I should cut back on it, then I will certainly do so. If so, I ask them to contact me. Until then, I will probably continue commenting, because I actually think it's the right thing to do.

Let me close by thanking you for bringing the fact that my commenting is off-putting to some to my attention. The feedback is appreciated, and I will try to moderate my behavior. Conversely, I would like to ask you to consider my points in good faith.

$\endgroup$
3
  • 4
    $\begingroup$ Particularly if the question is along the lines of "If I want to run faster, what should I use to shoot my foot?" $\endgroup$
    – Dave
    Dec 9, 2021 at 19:05
  • $\begingroup$ Just because you think I am going to shoot myself in the foot doesn’t mean I am. And all your warnings about me shooting myself in the foot turn my question about how to use my gun into debates about why my gun is bad. As I said the fact that these methods are bad is not solidified. Just as you believe that statistical papers/textbooks using these methods doesn’t mean they are correct, I don’t believe that a few persons on cross validated thinking they are bad makes them bad. I am not taking a position on whether they are valid or not. I am just saying it’s not as final as you think it is. $\endgroup$
    – astel
    Dec 9, 2021 at 21:38
  • 4
    $\begingroup$ @Stephan Kolassa: I propose that you continue to warn against foot-shooting! $\endgroup$ Dec 10, 2021 at 2:25
7
$\begingroup$

Answers should answer the question. Comments should comment on the question. Stephan and I are using Cross Validated as it is intended to be used.

I have learned in statistical consulting that, yes, I should answer questions that the customer has. However, it is important to answer the questions that the customer should have had.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .