The current tag except (and description) of reads

Hypothesis testing assesses whether data are inconsistent with a given hypothesis rather than being an effect of random fluctuations.

It seems to me this is a false dichotomy, as data can often be considered outcomes of a random data generating process regardless of whether $H_0$ holds or not. I wonder if we could get a better excerpt (and description).

I do not feel competent enough to write a really good excerpt myself, but we do have highly competent users around; maybe they will find my concern valid and will chip in.

  • 3
    $\begingroup$ I think this is a result of our preference to have short excerpts. W/ a little elaboration, it'll be fine. $\endgroup$ Sep 1 '21 at 11:24
  • 2
    $\begingroup$ Maybe I'm overthinking it, but the right definition depends on whether you want the tag to encompass all forms of hypothesis testing, including Bayesian alternatives (e.g. BEST), or whether it should strictly be about null-hypothesis significance testing. $\endgroup$ Sep 2 '21 at 6:16
  • 3
    $\begingroup$ @FransRodenburg, my first concern is to remove whatever is misleading or incorrect. After that I think hypothesis-testing should cover hypothesis testing in general (including Bayesian), not only specific instances (frequentist). $\endgroup$ Sep 2 '21 at 6:28
  • $\begingroup$ Do you disagree with the spirit behind the description or only with the particular way that it has been phrased? $\endgroup$ Nov 1 '21 at 23:28
  • $\begingroup$ @SextusEmpiricus, hmm, at the moment I find it hard to add anything to what I have already written. $\endgroup$ Nov 2 '21 at 5:06
  • 2
    $\begingroup$ @Frans Rodenburg: It should be about all forms of hypothesis testing, including Bayesian ones (and others). At least, that is how it is used until now ... . Then other tags can modify that! $\endgroup$ Nov 3 '21 at 15:46

In the spirit of keeping the description short, here are some suggestions for removing the imprecise language of "random fluctuations":

  • Hypothesis testing provides an empirical answer to a yes or no question. (See comments.)
  • Hypothesis testing assesses whether data provides sufficient evidence to favor one hypothesis over another.

Or a slightly longer one for NHST specifically:

  • A frequentist statistical tests assesses how likely it is that the data were generated under the null-hypothesis, and then decides whether or not to reject it in favor of the alternative.
  • $\begingroup$ I prefer your second definition to your first, which seems to encompass all binary decisions in decision theory (and much else besides) $\endgroup$
    – Glen_b
    Sep 7 '21 at 5:33
  • $\begingroup$ @Glen_b I tried to write something as short as possible, but I agree it is too broad like that, so I've crossed it out. $\endgroup$ Sep 7 '21 at 15:49
  • 1
    $\begingroup$ +1 Frans I like these, but I would reframe the conclusion of the NHST sentence as decides whether evidence to support the alternative hypothesis was found or not, since substantive conclusions from NHST are always about the alternative, not the null. $\endgroup$
    – Alexis
    Oct 28 '21 at 16:58
  • $\begingroup$ @Alexis I find it difficult to do so while still conveying that the procedure relies on evidence against the null, and not for the alternative. What do you think of the edit? $\endgroup$ Oct 29 '21 at 17:23
  • $\begingroup$ Well I disagree that evidence against the null is not evidence for the alternative. But again: I think conclusions are always about the alternative. Also: not all null hypotheses take the form $\text{H}_0\text{: }\theta=0$, for example $\text{H}_0\text{: }|\theta|=\Delta$ is the null of an equivalence test, and does not fit with the edited conclusion (i.e. rejecting that null one concludes that there is evidence for equivalence (within $\pm \Delta$), not difference). $\endgroup$
    – Alexis
    Oct 29 '21 at 18:30
  • $\begingroup$ @Alexis Of course there are exceptions to the rule, but wouldn't you agree that the name is derived from some statement of no difference, or no effect? Like kjetil mentioned in the comments above, for an equivalence test you can just write in the description how it differs from the usual. On a different note, if we are going with the most general description rather than one that pertains only to NHST, what do you think of the other suggestion? $\endgroup$ Nov 5 '21 at 14:35
  • $\begingroup$ No I would not. In general, (and leaving aside the omnibus test hypotheses corresponding to all four of these forms), I see: $\text{H}_{0}\text{: }\theta \le 0$, $\text{H}_{0}\text{: }\theta \ge 0$, $\text{H}_{0}\text{: }\theta = 0$, and $\text{H}_{0}\text{: }| \theta | \ge \Delta$ as viable and important forms of null hypothesis in every day application. $\endgroup$
    – Alexis
    Nov 5 '21 at 16:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .