7
$\begingroup$

We have an ambiguous tag that has 23 questions but no usage guidance and seems to be being used for several quite different things.

Some questions are asking about benchmarks for statistical models to be compared against eg here. There was a failed edit request suggesting that this should become the recommended usage for the tag:

Refers to a forecasting model which is used as a standard by which a more complected model (i.e VARs and VECM) must out preform. Usually the historical mean, median or some form of an ARIMA class of model is used as a the model which the more complected one must outperform to display the model's effectiveness in comparison.

But there seem to be rather more questions about benchmarking code runtimes etc eg here.

I have a feeling that the tag should be split into two cases and "benchmark" itself dropped for being potentially ambiguous. But I'm not an expert on either of these two areas.

There are a few threads that do not seem to fit within either of these categories, eg here (which might be better regarded as a request for a canonical exemplar data set) and here (where "benchmark" seems to be referring to educational standards), but these usages seem too rare to require disambiguating and perhaps the tag should simply be removed.

$\endgroup$
  • $\begingroup$ I am not convinced that these are really so different meanings. Benchmarking means comparing performance of several things with each other. One can benchmark statistical models and one can benchmark algorithms. Is "benchmarking in forecasting" really something so distinct? Otherwise the suggested excerpt looks way too specific for me. $\endgroup$ – amoeba says Reinstate Monica Nov 9 '17 at 18:22
  • $\begingroup$ If you think the tag should be split in two, then what two names would you suggest? $\endgroup$ – amoeba says Reinstate Monica Nov 9 '17 at 18:22
  • $\begingroup$ @amoeba Thanks for the feedback. I don't know enough to make a sensible suggestion, though it was clear the suggested excerpt is more specific than the current usage. "How to analyse code runtime benchmarks I have run" vs "what benchmark I should compare a set of forecasts to" seem different to me, but whether the difference is so fundamental that it deserves splitting I am not 100% sure. At any rate, it does seem a fairly heterogeneous collection of questions in its current form. $\endgroup$ – Silverfish Nov 9 '17 at 19:01
  • $\begingroup$ Well, I certainly do agree that the tag is not very useful (if at all). It it vanished, I would not regret. On the other hand, it does not look meaningless enough to actually spend the effort to kill it. On the yet another hand, 23 threads are so few that it's not much of an effort. $\endgroup$ – amoeba says Reinstate Monica Nov 9 '17 at 21:18
  • $\begingroup$ Perhaps we could ping one of our active [time-series] and/or [forecasting] contributors (StephanKolassa?) to ask what they think. $\endgroup$ – amoeba says Reinstate Monica Nov 9 '17 at 21:18
  • 1
    $\begingroup$ Really my first thought here is to kill off this tag. $\endgroup$ – gung - Reinstate Monica Nov 9 '17 at 21:46
  • $\begingroup$ Wow my edit made it as a meta post on CV. looks like I made it. $\endgroup$ – EconJohn Nov 10 '17 at 18:52
  • $\begingroup$ How about just creating a benchmark-forecasting tag? $\endgroup$ – EconJohn Nov 10 '17 at 18:53
  • 1
    $\begingroup$ @EconJohn If we were to go down that line, I think [forecasting-benchmark] sounds more natural to me, but it would be good to get some feedback from a forecaster! $\endgroup$ – Silverfish Nov 12 '17 at 0:17
  • $\begingroup$ @Silverfish well I am one, just a relatively new one with little academic qualification. $\endgroup$ – EconJohn Nov 12 '17 at 0:20
  • $\begingroup$ I am with @gung: kill this tag. A special tag for benchmarks in forecasting looks too specialized to me. But I am not interested/knowledgeable in forecasting. Would be good to ping StephanKolassa and/or RobHyndman. $\endgroup$ – amoeba says Reinstate Monica Nov 12 '17 at 0:48
  • 1
    $\begingroup$ Welcome to this rarefied club, @EconJohn. The air's thin up here, though. $\endgroup$ – gung - Reinstate Monica Nov 12 '17 at 1:38
  • $\begingroup$ @gung on EconSE I'm a Mod. here im a peasant. yelp $\endgroup$ – EconJohn Nov 12 '17 at 1:45
  • $\begingroup$ My apologies, @EconJohn, I hadn't realized. I spend little time elsewhere in the SE system, but when I do, I'm a peon, so I understand. $\endgroup$ – gung - Reinstate Monica Nov 12 '17 at 2:01
4
$\begingroup$

In forecasting, comparing a proposed new method against a benchmark is fundamental, for the reasons I describe here. (My impression is that the analogue is not equally common in Machine Learning, but I may well be misinformed here.) So it's an important point.

However, I don't see it as a big topic. There is really not all that more to it than what I wrote in the link above, plus this thread (which ironically enough doesn't have the tag) and maybe a few threads on relative error measures like the .

I don't think a question on how to report performance on software benchmarks should carry a "benchmark" tag - any more than a question on how to analyze apple yields should carry an "apples" tag.

Bottom line: I'd suggest we remove the tag and possibly burninate it. It doesn't seem to be worth the housekeeping. If this answer gets upvoted enough and no mods weigh in, I'll start removing it in small batches.

$\endgroup$
  • 2
    $\begingroup$ Thanks for the feedback. I think the difference between an "important" topic and a "big" one is worthwhile for tags. $\endgroup$ – Silverfish Nov 12 '17 at 21:04
  • $\begingroup$ +1 Kill the tag! $\endgroup$ – amoeba says Reinstate Monica Nov 16 '17 at 8:22
  • 1
    $\begingroup$ I have started removing the tags a few questions at a time. Will take a few days. $\endgroup$ – S. Kolassa - Reinstate Monica Nov 20 '17 at 13:38
  • $\begingroup$ Just checked: three more to go :-) $\endgroup$ – amoeba says Reinstate Monica Dec 14 '17 at 8:50
  • $\begingroup$ @amoeba: gah, completely forgot about that! Thanks for pinging me! I'll finish this off now. $\endgroup$ – S. Kolassa - Reinstate Monica Dec 14 '17 at 8:53
  • $\begingroup$ Done. Flagged the question to be tagged resolved. $\endgroup$ – S. Kolassa - Reinstate Monica Dec 14 '17 at 8:57
  • $\begingroup$ Hmm, we don't usually do it (marking with resolved) with tag related Meta posts but actually it might be not a bad idea. What we usually do, is to edit the answer with "Update: done" and ping the OP to accept it :-) $\endgroup$ – amoeba says Reinstate Monica Dec 14 '17 at 8:58
  • $\begingroup$ @Silverfish See above. $\endgroup$ – amoeba says Reinstate Monica Dec 14 '17 at 8:59
  • $\begingroup$ Thank you Stephan! $\endgroup$ – Silverfish Dec 14 '17 at 18:50

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .