3
$\begingroup$

The current excerpt of is wrong; it's not necessarily about violations, just about variations in inputs, possibly unseen:

Sensitivity analysis refers to methods to see if violations of assumptions of a model make large differences to results.

What's the right definition? For example, the definition on the Wikipedia page for sensitivity analysis says:

Sensitivity analysis is the study of how the uncertainty in the output of a mathematical model or system (numerical or otherwise) can be apportioned to different sources of uncertainty in its inputs.

(I was about to create the same tag at DataScience.SE and copy whatever definition we decide on.)

| |
$\endgroup$
  • $\begingroup$ I think the WP definition is wrong, in the context of statistics. $\endgroup$ – gung - Reinstate Monica May 7 '18 at 11:25
  • $\begingroup$ @gung: Can you cite a better definition? $\endgroup$ – smci May 7 '18 at 11:27
  • 1
    $\begingroup$ I don't know of a specific place where there's an official definition, but it is very common in statistics to use the term 'sensitivity analysis' loosely to refer to things like the top statement. (Nb, "assumptions" doesn't strictly mean, eg, homoscedasticity.) The discussion on WP sounds like error propagation (mostly in reverse). Indeed, the page repeatedly references the Wikipedia page for Uncertainty analysis. It reads like it is written by / for engineers trying to simulate complex systems (or has that in mind). $\endgroup$ – gung - Reinstate Monica May 7 '18 at 13:06
  • 1
    $\begingroup$ @gung I find the term used quite frequntly in papers I review for the situation where authors check what would happen if they deleted certain cases. For instance in meta-analysis what would happen if I deleted the low-quality studies? The Wikipedia definition seems to cover that in a rather abstract and indirect way. $\endgroup$ – mdewey May 7 '18 at 13:17
  • 1
    $\begingroup$ @mdewey, my interpretation of the top definition covers that. IMHO, the WP definition, read in the context of the whole page, seems to largely have something else in mind. $\endgroup$ – gung - Reinstate Monica May 7 '18 at 13:42
  • $\begingroup$ Then what do you call "measuring how much varying an individual variable affects the model output"? (and mainly measuring the influence, not the uncertainty or error). This is the meaning I've heard other data scientists use. Seems there are multiple meanings out there? Do we need a numbered list on the tag info? $\endgroup$ – smci May 7 '18 at 22:41
  • $\begingroup$ It's certainly possible that there are people who use the term that way, @smci. My point is that in applications of statistics to scientific research, it isn't what I'm familiar with. What you are referring to sounds like importance. $\endgroup$ – gung - Reinstate Monica May 8 '18 at 1:07
  • $\begingroup$ @gung: No, (variable or feature) importance is only a ranking of how much influence (+ve or -ve) a variable has on the idv (and then I've only ever seen it with tree-based models, not LR or NN). Importance is not the variable's actual numerical influence, e.g. "when Age changes from 18 to 25, (average) income changes from 22K to 40K". Compare to "variable importances: Age 1, Zipcode 2, Gender 3". $\endgroup$ – smci May 8 '18 at 1:12
  • $\begingroup$ @smci, variable importance is a topic in statistics beyond just random forests. I don't know of a measure for ANN's per se, but I wouldn't be surprised if there are some. For an overview of some suggested measures for linear regression, see here. I don't know this material well, but it seems to me that the idea of seeing how much the predicted value changes as a function of changing an input variable could fit within this topic, broadly construed. $\endgroup$ – gung - Reinstate Monica May 8 '18 at 20:50
  • $\begingroup$ I think we could modify the top definition to explicitly mention that it could include assumptions that are not specifically about the assumptions of the method used - e.g normality of residuals or whatnot - but also about the various inputs to the model. $\endgroup$ – Peter Flom May 9 '18 at 11:00
  • $\begingroup$ What do we call "measuring how much varying an individual variable affects the model output"?" As in quantifying the ratio of change in input[i] to change in output (not just ranking whether input[i] has more influence than input[j]. It isn't importance. I believe "sensitivity analysis" is used in that sense (among the others you also mention). If you say it isn't, then what's the right term? $\endgroup$ – smci May 11 '18 at 8:34
  • $\begingroup$ @smci, that would be a decent question for the main site. 'What should the SA tag excerpt say?', is the question on this thread (& is appropriate for meta.CV). $\endgroup$ – gung - Reinstate Monica May 11 '18 at 12:57
  • 1
    $\begingroup$ @smci FYI, I have updated the wiki excerpt based on the suggestion in my answer. I realize that we did not reach consensus and you will probably be unhappy about this new excerpt, but if better (more consensus) suggestions appear in this thread, we can easily update the excerpt again. $\endgroup$ – amoeba May 15 '18 at 9:02
3
$\begingroup$

I tried to modify the current excerpt in the general direction outlined by @AdamO, but keeping it easy to grasp. My understanding of what "sensitivity analysis" is, follows @gung's answer.

Auxiliary methods intended to check if the outcome of some statistical analysis strongly depends on the model assumptions, preprocessing steps, presence of outliers, etc.

| |
$\endgroup$
  • $\begingroup$ +1, I might make this slightly shorter by substituting "an" for "some statistical". $\endgroup$ – gung - Reinstate Monica May 11 '18 at 13:40
  • 1
    $\begingroup$ @gung I have updated the wiki excerpt (with your edit). If better suggestions appear here, we can update it again. $\endgroup$ – amoeba May 14 '18 at 19:50
  • $\begingroup$ This is great, except I'd change "the outcome of some statistical analysis" to "... of some statistical model" $\endgroup$ – smci May 17 '18 at 2:41
  • $\begingroup$ @smci It currently says "outcome of an analysis", as gung suggested. "Outcome of a model" sounds slightly weird to me, and there is the world "model" in this sentence already. $\endgroup$ – amoeba May 17 '18 at 13:30
2
$\begingroup$

I would say sensitivity analysis is...

a set of non-primary statistical analyses intended to verify the accuracy and generalizability of estimates and inference derived from a primary analysis and to make corrections where appropriate.

| |
$\endgroup$
  • $\begingroup$ +1, this seems right to me. However, it is rather general; I wonder if someone who isn't very familiar w/ statistics & modeling would recognize what this really means in practice. $\endgroup$ – gung - Reinstate Monica May 7 '18 at 17:55
  • 1
    $\begingroup$ @gung Agreed. Excerpts are general summaries. This is what the user sees while hovering their cursor over the tag. If the user in't familiar with stats/modeling, my hope would be they'd click through to the Wiki and read on. $\endgroup$ – AdamO May 7 '18 at 21:44
  • 1
    $\begingroup$ True, @AdamO. It was more a caveat than a criticism. There is a legitimate issue here, though. If people wouldn't be able to recognize what is meant, it wouldn't be as effective as an excerpt. That's my concern. Note that my answer (ie, the set of examples) is consistent w/ your excerpt. So again, I don't disagree. $\endgroup$ – gung - Reinstate Monica May 8 '18 at 1:10
  • 1
    $\begingroup$ How about starting "An analysis conducted after the primary analysis ...". The word non-primary looks ugly to me but that may just be a matter of taste. $\endgroup$ – mdewey May 9 '18 at 14:13
  • 1
    $\begingroup$ @mdewey fine by me, all I want to underscore is that the findings of sensitivity analyses are always considered incidental and never interesting in their own right. Otherwise they are secondary hypotheses. It all boils down to what you report in a summary statement. Sensitivity analyses are behind the scenes and not all that relevant to the science. $\endgroup$ – AdamO May 9 '18 at 15:21
  • 1
    $\begingroup$ To be honest, I don't like this particular wording too much. It sounds too convoluted for an excerpt. I'd rather go with the one we have currently, modifying it in the general direction suggested by AdamO: "Auxiliary methods intended to check if the outcome of some statistical analysis strongly depends on the model assumptions, preprocessing steps, presence of outliers, etc." (CC @gung) $\endgroup$ – amoeba May 11 '18 at 8:50
  • $\begingroup$ Also cc @mdewey. $\endgroup$ – amoeba May 11 '18 at 8:50
  • $\begingroup$ @amoeba: that's nice and concise, but as I mention doesn't resolve the conflict in usage(s) I've heard. $\endgroup$ – smci May 11 '18 at 9:15
  • 1
    $\begingroup$ @smci Yes, it seems there might be different meanings of "sensitivity analysis". Gung and AdamO are talking about the sensitivity of analysis results on non-normality/outiers/etc. You seem to be talking about the sensitivity of prediction on a particular independent variable. Can you give an example of some source where "sensitivity analysis" is used in this latter meaning? But not Wikipedia; a paper or book that uses "sensitivity analysis" in this sense. $\endgroup$ – amoeba May 11 '18 at 9:34
  • $\begingroup$ @amoeba, why not make your excerpt suggestion into an official answer? $\endgroup$ – gung - Reinstate Monica May 11 '18 at 13:04
  • 2
    $\begingroup$ BTW, I think 'sensitivity analysis' is used in the latter way, it's just common in other fields & not so much in statistics. W/ a regression model, it's really obvious what will happen to y-hat if you vary Xj, but in simulation models of complex phenomena, there might be no way to just look at the model & tell what is going to happen. So people do 'experiments' on the model by plugging in different values & running it to get the output. This elucidates the model's sensitivity to Xj in a manner analogous to how d-prime is a measure of a detection system's sensitivity to a signal. $\endgroup$ – gung - Reinstate Monica May 11 '18 at 13:05
  • 3
    $\begingroup$ @amoeba On thinking about this further, I am inclined to agree with you here. There are two other kinds of "non-primary" analyses that are worth mentioning: secondary analyses and post-hoc analyses: these often augment the primary findings but aren't meant to validate them. For instance, I might be interested in a causal effect of smoking on lung cancer. I might also want to know if sex is an effect modifier. In that case, testing for interaction is a secondary analysis. It is meant to augment rather than advise the primary analysis. $\endgroup$ – AdamO May 11 '18 at 13:20
  • $\begingroup$ @gung OK, I posted my suggestion as an answer. $\endgroup$ – amoeba May 11 '18 at 13:31
  • $\begingroup$ @AdamO, you could upvote amoeba's answer below if you like his phrasing. $\endgroup$ – gung - Reinstate Monica May 12 '18 at 0:37
1
$\begingroup$

Let me start by grounding this. In a scientific situation, people typically have theories about important relationships between variables. These theories are, however, incomplete in some way, or are not universally accepted. Thus, it is worthwhile to test specific hypotheses derived from those theories. The point of designing a study is to create a context in which there will be sufficient data that are relevant to that question. The point of building a regression model is to create a statistical context in which the hypothesis can be tested. This is the kind of situation that I typically work with, and what I generally have in mind when answering questions on CV.

At this point, let's say we've tested some treatment (intervention, exposure, etc.) effect and gotten a result. The model constitutes making a whole host of assumptions (not just independence, homoscedasticity, and normality). One might be concerned that the result we have is contingent on one or another assumption that makes our conclusion fragile. In that case, one way to proceed is to rerun the analysis / fit a new model that is robust to that kind of violation. It is common to call this a 'sensitivity analysis', although I don't know of this being an official definition anywhere. Here are some example write ups that might come out of this process:

... A concern is that our result could be driven by two high-leverage datapoints. As a sensitivity analysis, we refit our model without those points. The treatment remains significant and the mean difference is similar (-.51 vs. -.72).

or

... A concern is that our result could be driven by two high-leverage datapoints. As a sensitivity analysis, we fit a robust regression model using Tukey's bisquare as the loss function. The treatment remains significant and the mean difference is similar (-.51 vs. -.72).

... A concern is that the residuals may not be sufficiently normal to rely on normal theory to determine the p-values. As a sensitivity analysis, we bootstrapped the residuals to compute the p-value without assuming normality. The treatment remains significant (p=0.0031).

... We treated the summed score from the questionnaire as sufficiently equal-interval to use standard linear regression methods, because the scores are not against the bounds of the scale and this facilitates easier interpretation. However, one could argue that this is too cavalier for the statistical test of our primary hypothesis. As a sensitivity analysis, we replicated this result with an ordinal logistic regression (proportional odds) model. The intervention is significant in this model as well (p<0.02).

... Complete case analysis is valid under the assumption that the missingness is MAR. We have argued that this assumption is reasonable in our case. However, as a sensitivity analysis, we used the largest change observed in our dataset and assigned it, in the 'incorrect' direction, to those follow up visits that were missed. The question then is, how many do we have to add before the treatment effect becomes nonsignificant? ...

Etc. The above are the kind of thing that I think of when I see the phrase 'sensitivity analysis'. Note that @mdewey's meta-analysis example falls within this continuum.


On the other hand, no terms are all that well standardized. The same terms are used different ways by different fields. The Wikipedia definition sounds like error propagation in reverse. You start with a certain amount of uncertainty in the model's output, and "apportion" that to uncertainty in different inputs. (That is, '26% of the uncertainty is due to measurement error in X1', etc.) That is perfectly reasonable when developing a simulation model of a complex system, but it really doesn't fall within the category of analyses illustrated by the examples above.

| |
$\endgroup$
  • 2
    $\begingroup$ +1. This could (should?) be an answer on the main site to a question "What is sensitivity analysis?" $\endgroup$ – amoeba May 7 '18 at 21:18
  • $\begingroup$ But what's your suggested tag excerpt? I was only asking about the tag excerpt (a paragraph), not the full definition. $\endgroup$ – smci May 11 '18 at 8:31
  • $\begingroup$ @smci An excerpt suggestion based on gung's answer: stats.meta.stackexchange.com/questions/5245/…. $\endgroup$ – amoeba May 11 '18 at 8:52
  • $\begingroup$ @smci, I think amoeba's suggested excerpt might be the best. $\endgroup$ – gung - Reinstate Monica May 11 '18 at 12:59

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .