12
$\begingroup$

We have two tags that seem closely related:

  • (69 threads) Excerpt:

    Also called "perfect separation". In logistic regression, if all 1 & 0 outcomes lie on different sides of some value for a variable, the coefficient estimate for that variable will tend towards infinity.

  • (37 threads) Excerpt:

    separability is when two classes in the data can be completely separated, i.e. by some (hyper)plane, such that for instance a logistic regression, svm or discriminant analysis can give perfect predictions on the training data. See the wikipedia article on Separation (statistics)

There are 12 threads that have both tags.

I'm wondering if the distinction between them, and when one or the other should be used, can be made clearer. Alternatively, should one be made a synonym of the other? (Which?)

$\endgroup$
  • 3
    $\begingroup$ Clarification on terminology: does HDE refer to what happens in logistic regression, or does it describe the property of the data? $\endgroup$ – amoeba says Reinstate Monica Nov 27 '17 at 8:33
  • 3
    $\begingroup$ @amoeba If we argue from historical principles then H&D's paper is entitled "Wald's Test as Applied to Hypotheses in Logit Analysis" so I think they saw it as a property of the analysis method. It seems to me that separability may be slightly more general and so it would be better if it were the only term but I would not want to man the barricades over the choice. $\endgroup$ – mdewey Nov 27 '17 at 13:54
  • 2
    $\begingroup$ We seem to have quite a few tagged [logistic] mentioning "separation" but tagged with neither [hauck-donner-effect] nor [separability] so if we do re-tag one of these existing threads then we might re-tag some which lack either. $\endgroup$ – mdewey Nov 27 '17 at 14:10
  • 3
    $\begingroup$ That's a perceptive comment, @amoeba. I suppose this situation is analogous to the one w/ zero-inflation. Prima facie, it seems more appropriate to me to have a single tag here than there; but I'm not sure I can fully articulate why. $\endgroup$ – gung - Reinstate Monica♦ Nov 27 '17 at 14:37
  • $\begingroup$ Some evidence: recently this question stats.stackexchange.com/questions/314842/… about what look like hauck-donner effect, but in a logistic model estimated by gee, was closed as an exact duplicate of a question about hacck-donner in usual maximum likelihood logistic regression. I'm not totally sure that was absolutely correct ... what do you think? My guess is that the Hauck-donner effect will manifest itself even if estimation is not by maximum likelihood, but I know of no references ... $\endgroup$ – kjetil b halvorsen Nov 27 '17 at 17:30
6
$\begingroup$

Let me make a concrete proposal so that people can comment and vote.

I propose that we re-tag all the [hauck-donner-effect] ones as [separability] and for at least a selected few of those not tagged with either but which are clearly about separation we tag them [separability] too.

======= edit 29 November 2017 ==========

Following discussion I now propose the new tag [separation] for both of these. For an excerpt I suggest the following, taken and adapted from Wikipedia

Separation occurs when some value(s) of a predictor (or linear combination of predictors) is associated with only one value of a categorical outcome.

I have tried to make this as non-technical as possible but am happy for anyone to edit it (as they can later anyway).

$\endgroup$
  • 3
    $\begingroup$ I can always make a synonym & merge them, so I don't see the need for you to do all that work. The question is more just, should they stay separate or come together (somehow)? If they should stay apart, how should we define their scope more clearly such that they will be used correctly? $\endgroup$ – gung - Reinstate Monica♦ Nov 27 '17 at 18:33
  • 1
    $\begingroup$ All in all,there are so few questions in total so it makes meaning to make hauck-donner-effect a synonym of separability even it they are not synonyms in strict sense, but clearly [separability] is the more general term. $\endgroup$ – kjetil b halvorsen Nov 28 '17 at 11:45
  • $\begingroup$ To me [separability] sounds not very clear as a tag name. Maybe [complete-separation] could be clearer? Or is it just because I am not very familiar with this terminology? @gung $\endgroup$ – amoeba says Reinstate Monica Nov 28 '17 at 22:25
  • $\begingroup$ I agree that separability is not very clear, complete-separation might be better. $\endgroup$ – kjetil b halvorsen Nov 28 '17 at 22:28
  • 1
    $\begingroup$ @amoeba, technically, there is also "quasi-separation" at the boundary between complete separation & overlapping categories. We could try [category-separation], which isn't an official term, but is very descriptive & intuitive. H-D is rather esoteric, IMHO. $\endgroup$ – gung - Reinstate Monica♦ Nov 29 '17 at 1:08
  • $\begingroup$ I read up on what "quasi-separation" is and I see it called "quasi-complete separation" too. I think I would be fine mapping it under "complete-separation" which does have a clearer name for a tag than "category-separation" IMHO. @gung $\endgroup$ – amoeba says Reinstate Monica Nov 29 '17 at 8:08
  • 1
    $\begingroup$ My feeling is that most of the people who post questions about separation have no idea what is happening to them and so many of the tags are done later by us or people like us. Ditto HDE. I would prefer to leave it as [separability] even if usually the problem is indeed complete-separation but I am happy to go with the majority. $\endgroup$ – mdewey Nov 29 '17 at 12:05
  • $\begingroup$ @amoeba, would just plain, old [separation] be OK? $\endgroup$ – gung - Reinstate Monica♦ Nov 29 '17 at 12:56
  • $\begingroup$ @gung I don't know, to me personally [complete-separation] is rather clear while [separation] (or [separability]) sounds mysterious. But then there is this Wiki page en.wikipedia.org/wiki/Separation_(statistics) so maybe "separation" is a clear enough term. It's not my area, so I don't know the terminology. Feel free to ignore my opinions here. $\endgroup$ – amoeba says Reinstate Monica Nov 29 '17 at 15:01
  • $\begingroup$ cc above to @mdewey $\endgroup$ – amoeba says Reinstate Monica Nov 29 '17 at 15:01
  • 1
    $\begingroup$ @amoeba there is a highly voted question with an extremely good answer by scortchi stats.stackexchange.com/questions/11109/… (which has both tags incidentally) if you want to extend your horizons. Its accepted answer is not so good in my view. $\endgroup$ – mdewey Nov 29 '17 at 15:09
  • 3
    $\begingroup$ Regarding separation, when I talk to people about this, I never say 'complete separation' (or 'partial separation', or 'quasi separation', etc, or for that matter 'separability' or 'h-d-e'). I always just say 'separation'. I may follow up w/ a brief explanatory description. To me, plain, old 'separation' would be the clearest, phrase. $\endgroup$ – gung - Reinstate Monica♦ Nov 29 '17 at 15:09
  • 1
    $\begingroup$ @mdewey, if you like the idea of using just [separation], why not add it to your answer? If everyone is in agreement, I can accept it & implement this after a few days w/o objection. Also, you could suggest your preferred excerpt text, if you'd like (or I could do it from the existing 2 excerpts). $\endgroup$ – gung - Reinstate Monica♦ Nov 29 '17 at 17:15
  • 1
    $\begingroup$ Maybe something along the lines "Separation occurs when the values of the categorical outcome (classes) can be completely separated by a linear combination of predictors. Can cause problems in logistic regression." ? CC @gung $\endgroup$ – amoeba says Reinstate Monica Dec 6 '17 at 10:21
  • 2
    $\begingroup$ I adapted @amoeba's text. Both synonyms are made & merged. $\endgroup$ – gung - Reinstate Monica♦ Dec 6 '17 at 14:17
3
$\begingroup$

The tag's definition of the Hauck-Donner effect appears to be different than the classical one. To my knowledge, Hauck-Donner effect refers to an erratic behaviour (non-monotonicity) of the power of the Wald test, which is unrelated to the separability issue. As the following four citations show:

  1. A common interpretation of Hauck Donner effect is about power of Wald test

  2. None of these papers (seems to) mention separability

  3. Three next references show the presence of the Hauck Donner effect in other settings than the logit (such as GLM or ANOVA).

    • Original paper: Hauck, Donner (1977) Wald's Test as Applied to Hypotheses in Logit Analysis, Journal of the American Statistical Association, Vol. 72, No. 360

      For tests of a single parameter in the binomial logit model, Wald's test is shown to behave in an aberrant manner.

    • Vaeth (1985) On the Use of Wald's Test in Exponential Families:

      Hauck & Donner (1977) showed that Wald's test (the maximum likelihood test statistic) behaves in an aberrant manner when applied to hypotheses about a single parameter in a binomial logit model. [...] Finally, the use of Wald's test in the analysis of generalized linear models is discussed.

    • Mantel (1987) Understanding Wald's Test for Exponential Families:

      [...] demonstration by Hauck and Donner (1977) that in certain multiparameter situations in which interest focuses on one of several parameters, Wald's test statistic (the maximum likelihood test statistic) can decrease to 0.

    • Fears et al (1996) A Reminder of the Fallibility of the Wald Statistic

      Hauck and Donner and Vaeth have identified situations in which the Wald statistic has poor power. We consider another example that is not in the classes discussed by those authors. We present data from a balanced one-way random effects analysis of variance (ANOVA) that illustrate the poor power of the Wald statistic.

    • Venables and Ripley (2002, p. 198) Modern Applied Stats with S, give a similar definition of Hauck and Donner. Interestingly, they mention Hauck Donner and separation together:

      There is one fairly common circumstance in which both convergence problems and the Hauck–Donner phenomenon can occur. This is when the fitted probabilities are extremely close to zero or one.

Now it needs to be shown that the tag's definition of Hauck Donner effect is also an accepted one? What are the references for that one? In absence of sources, a deeper cleaning of the tags might be required?

$\endgroup$
  • 1
    $\begingroup$ This is interesting, but I have never seen this generalized use in practice. $\endgroup$ – kjetil b halvorsen Dec 1 '17 at 13:46
  • 2
    $\begingroup$ Looking again at the questions tagged HDE even the titles usually contain "separation" so would you like to spell out the implications of your answer for what we should do next? $\endgroup$ – mdewey Dec 1 '17 at 15:53
  • 1
    $\begingroup$ But are there any references for the common use of Hauck Donner as separability @kjetilbhalvorsen ? A google search links to... cross-validated itself, blogs or stata lists. Any papers or books in the next 30-50 google links refer to the Wald test issue it seems? $\endgroup$ – Matifou Dec 1 '17 at 17:18
  • $\begingroup$ @mdewey fair point, as I am not sure about where the "common use" of HDE as separation issue comes from, I am not sure at this point what the implications should be. $\endgroup$ – Matifou Dec 1 '17 at 17:20
  • 1
    $\begingroup$ MASS 4th edition, page 198 says: "There is one fairly common circumstance in which both convergence problems and the Hauck-Donner phenomenon occur. This is when the fitted probabilities are exytremely close to zero or one". So, it does not equate, just says that it very commonly occurs together. That may explain the "common use" issue. $\endgroup$ – kjetil b halvorsen Dec 1 '17 at 18:18
  • $\begingroup$ Indeed @kjetilbhalvorsen, I had just added that quote into the list! I am also wondering whether this quote led to confusion among practitioners. But is this "common use" broader than cross-validated itself? $\endgroup$ – Matifou Dec 1 '17 at 18:48
  • 2
    $\begingroup$ I'm not sure about this. These quotes suggest H-D is about problems with the Wald test, but the examples mostly refer to logistic regression, it seems. When there is separation, the Wald test has problems. That is clear. If there are similar problems with the Wald test outside of separation in logistic models, I'm not familiar w/ it. Moreover, all the threads grouped by the tag seem to be about separation. If we just edited them, we would eliminate the tag. Is that what you're suggesting? I doubt there is sufficient demand to support a tag for the Wald test blowing up outside of separation. $\endgroup$ – gung - Reinstate Monica♦ Dec 2 '17 at 1:12
  • $\begingroup$ It seems h-d-e tag was created by Sycorax as mentioned here stats.meta.stackexchange.com/a/3112/28666 so maybe you guys should ping him to join this conversation? CC @gung $\endgroup$ – amoeba says Reinstate Monica Dec 3 '17 at 9:40
  • 2
    $\begingroup$ I was under the impression that HDE exclusively referred to the "perfect separation" issue, but it appears that was a misuse or conflation of terminology. But from a tagging perspective, I think the problem largely disappears if we're agreed to retag all posts referring to separation as "HDE" to "separation." I'm not aware of any posts asking about the HDE and referring to Wald tests. $\endgroup$ – Reinstate Monica Dec 3 '17 at 17:49
  • $\begingroup$ @gung I forgot to tag you in my comment which you solicited. $\endgroup$ – Reinstate Monica Dec 3 '17 at 22:54
  • 1
    $\begingroup$ @Matifou, I appreciate your contribution. Would you mind going through the h-d-e threads & seeing if you find any that are about problems w/ the Wald test unrelated to separation & letting me know which you find? If you don't find any, would you be willing to go along w/ us making the synonyms as suggested above for the purposes of keeping the site's content organized? (Nb, the way tags are used is that they don't really have to be perfect synonyms in the everyday language sense.) $\endgroup$ – gung - Reinstate Monica♦ Dec 4 '17 at 1:06
  • $\begingroup$ @gung. Sure. I did go through and indeed, it looks like all of them relate to the (quasi-)separation issue. That's impressive indeed :-) So I think indeed going ahead by making sure they have a separation tag, and removing the HDE tag. The only one I found discussing HDE didn't have the tag! stats.stackexchange.com/questions/5304/… $\endgroup$ – Matifou Dec 4 '17 at 5:18
  • $\begingroup$ This is not necessarily very surprising because it seems that in the vast majority of cases [h-d-e] tag was not added by the OPs but by one of the few people who are participating in this Meta discussion. So I wouldn't take the evidence about how [h-d-e] tag is being used on CV as an evidence of how this term is used in a broader community. That said, I agree with the current proposal as stated above by @gung. $\endgroup$ – amoeba says Reinstate Monica Dec 4 '17 at 8:38
  • 1
    $\begingroup$ OK, I'm going to go ahead & make the synonyms at this point. $\endgroup$ – gung - Reinstate Monica♦ Dec 4 '17 at 14:21

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .