7
$\begingroup$

We have an ambiguous tag that has 23 questions but no usage guidance and seems to be being used for several quite different things.

Some questions are asking about benchmarks for statistical models to be compared against eg here. There was a failed edit request suggesting that this should become the recommended usage for the tag:

Refers to a forecasting model which is used as a standard by which a more complected model (i.e VARs and VECM) must out preform. Usually the historical mean, median or some form of an ARIMA class of model is used as a the model which the more complected one must outperform to display the model's effectiveness in comparison.

But there seem to be rather more questions about benchmarking code runtimes etc eg here.

I have a feeling that the tag should be split into two cases and "benchmark" itself dropped for being potentially ambiguous. But I'm not an expert on either of these two areas.

There are a few threads that do not seem to fit within either of these categories, eg here (which might be better regarded as a request for a canonical exemplar data set) and here (where "benchmark" seems to be referring to educational standards), but these usages seem too rare to require disambiguating and perhaps the tag should simply be removed.

$\endgroup$
14
  • $\begingroup$ I am not convinced that these are really so different meanings. Benchmarking means comparing performance of several things with each other. One can benchmark statistical models and one can benchmark algorithms. Is "benchmarking in forecasting" really something so distinct? Otherwise the suggested excerpt looks way too specific for me. $\endgroup$
    – amoeba
    Nov 9, 2017 at 18:22
  • $\begingroup$ If you think the tag should be split in two, then what two names would you suggest? $\endgroup$
    – amoeba
    Nov 9, 2017 at 18:22
  • $\begingroup$ @amoeba Thanks for the feedback. I don't know enough to make a sensible suggestion, though it was clear the suggested excerpt is more specific than the current usage. "How to analyse code runtime benchmarks I have run" vs "what benchmark I should compare a set of forecasts to" seem different to me, but whether the difference is so fundamental that it deserves splitting I am not 100% sure. At any rate, it does seem a fairly heterogeneous collection of questions in its current form. $\endgroup$
    – Silverfish
    Nov 9, 2017 at 19:01
  • $\begingroup$ Well, I certainly do agree that the tag is not very useful (if at all). It it vanished, I would not regret. On the other hand, it does not look meaningless enough to actually spend the effort to kill it. On the yet another hand, 23 threads are so few that it's not much of an effort. $\endgroup$
    – amoeba
    Nov 9, 2017 at 21:18
  • $\begingroup$ Perhaps we could ping one of our active [time-series] and/or [forecasting] contributors (StephanKolassa?) to ask what they think. $\endgroup$
    – amoeba
    Nov 9, 2017 at 21:18
  • 1
    $\begingroup$ Really my first thought here is to kill off this tag. $\endgroup$
    – gung - Reinstate Monica
    Nov 9, 2017 at 21:46
  • $\begingroup$ Wow my edit made it as a meta post on CV. looks like I made it. $\endgroup$
    – EconJohn
    Nov 10, 2017 at 18:52
  • $\begingroup$ How about just creating a benchmark-forecasting tag? $\endgroup$
    – EconJohn
    Nov 10, 2017 at 18:53
  • 1
    $\begingroup$ @EconJohn If we were to go down that line, I think [forecasting-benchmark] sounds more natural to me, but it would be good to get some feedback from a forecaster! $\endgroup$
    – Silverfish
    Nov 12, 2017 at 0:17
  • $\begingroup$ @Silverfish well I am one, just a relatively new one with little academic qualification. $\endgroup$
    – EconJohn
    Nov 12, 2017 at 0:20
  • $\begingroup$ I am with @gung: kill this tag. A special tag for benchmarks in forecasting looks too specialized to me. But I am not interested/knowledgeable in forecasting. Would be good to ping StephanKolassa and/or RobHyndman. $\endgroup$
    – amoeba
    Nov 12, 2017 at 0:48
  • 1
    $\begingroup$ Welcome to this rarefied club, @EconJohn. The air's thin up here, though. $\endgroup$
    – gung - Reinstate Monica
    Nov 12, 2017 at 1:38
  • $\begingroup$ @gung on EconSE I'm a Mod. here im a peasant. yelp $\endgroup$
    – EconJohn
    Nov 12, 2017 at 1:45
  • $\begingroup$ My apologies, @EconJohn, I hadn't realized. I spend little time elsewhere in the SE system, but when I do, I'm a peon, so I understand. $\endgroup$
    – gung - Reinstate Monica
    Nov 12, 2017 at 2:01

1 Answer 1

Reset to default
4
$\begingroup$

In forecasting, comparing a proposed new method against a benchmark is fundamental, for the reasons I describe here. (My impression is that the analogue is not equally common in Machine Learning, but I may well be misinformed here.) So it's an important point.

However, I don't see it as a big topic. There is really not all that more to it than what I wrote in the link above, plus this thread (which ironically enough doesn't have the tag) and maybe a few threads on relative error measures like the .

I don't think a question on how to report performance on software benchmarks should carry a "benchmark" tag - any more than a question on how to analyze apple yields should carry an "apples" tag.

Bottom line: I'd suggest we remove the tag and possibly burninate it. It doesn't seem to be worth the housekeeping. If this answer gets upvoted enough and no mods weigh in, I'll start removing it in small batches.

$\endgroup$
9
  • 2
    $\begingroup$ Thanks for the feedback. I think the difference between an "important" topic and a "big" one is worthwhile for tags. $\endgroup$
    – Silverfish
    Nov 12, 2017 at 21:04
  • $\begingroup$ +1 Kill the tag! $\endgroup$
    – amoeba
    Nov 16, 2017 at 8:22
  • 1
    $\begingroup$ I have started removing the tags a few questions at a time. Will take a few days. $\endgroup$
    – Stephan Kolassa
    Nov 20, 2017 at 13:38
  • $\begingroup$ Just checked: three more to go :-) $\endgroup$
    – amoeba
    Dec 14, 2017 at 8:50
  • $\begingroup$ @amoeba: gah, completely forgot about that! Thanks for pinging me! I'll finish this off now. $\endgroup$
    – Stephan Kolassa
    Dec 14, 2017 at 8:53
  • $\begingroup$ Done. Flagged the question to be tagged resolved. $\endgroup$
    – Stephan Kolassa
    Dec 14, 2017 at 8:57
  • $\begingroup$ Hmm, we don't usually do it (marking with resolved) with tag related Meta posts but actually it might be not a bad idea. What we usually do, is to edit the answer with "Update: done" and ping the OP to accept it :-) $\endgroup$
    – amoeba
    Dec 14, 2017 at 8:58
  • $\begingroup$ @Silverfish See above. $\endgroup$
    – amoeba
    Dec 14, 2017 at 8:59
  • $\begingroup$ Thank you Stephan! $\endgroup$
    – Silverfish
    Dec 14, 2017 at 18:50

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .