6
$\begingroup$

Since this question asks for an intuition behind the idea:

What is the intuition behind the idea that for linear regression, the number of observations should exceed the number of parameters?

I thought I might also add something that is not about a linear regression but might lead to the same intuition. Therefore, I wrote about the problem of a logistic regression instead, and my answer got deleted. I see the point that this was too far-fetched, but my idea when I heard that observations should exceed the number of parameters was that Why is logistic regression particularly prone to overfitting in high dimensions? also showed this intuition to me.

I doubt that it was even right to see both questions as dealing with the same intuition, so that this question here is not about reopening the deleted answer (unless someone wants to). Yet, the answer was not deleted for being wrong, as far as I can see, but just since it was not about linear regression, at least if you check the only upvoted remark for this (which was again upvoted right before the answer was deleted).

If I show that in other models, you also have to avoid that the number of parameters (or dimensions) are too many, is this clearly off-topic for a question that asks about "the intuition of the idea that for linear regression, ..."? Mind that the question also becomes clearer in the body:

If a population model has k independent variables and 1 intercept, why are k+1 observations required to perform OLS estimates?

What is the intuition behind this?

But even then, the other link about logistic regression might lead to this intuition, or take anything else that is not about linear regression and still helps understanding this.

Is the wording of the question right? English is not my mother tongue, and I understood it as a much broader question.

$\endgroup$
9
  • 5
    $\begingroup$ I suspect the post had been deleted to spare you any more downvotes. I have undeleted it so that all readers of this thread will have access to it. As far as this actual (meta) question goes, I do favor general answers for their potential to expose deeper concepts. In this case, though, I have been unable to see why the additional complications associated with logistic regression might shed any light on the original question. $\endgroup$
    – whuber Mod
    Dec 13, 2023 at 15:46
  • $\begingroup$ I had difficulties understanding your generalisation and in particular in the case of the link with "Why is logistic regression particularly prone to overfitting in high dimensions?". Logistic regression is already able to overfit in the case of a single dimension, and the problem with logistic regression there is a different issue. Linear regression can also do this, but then the observations need to be in a lower dimensional subspace of the model, which has a low probability. $\endgroup$ Dec 15, 2023 at 15:02
  • $\begingroup$ @SextusEmpiricus That question is about an overfit in high dimensions, not just the one in the first that you can get since it is asymptotic. I remember having read the underlying paper of that question - at least until I found this chapter on rare feature crosses. I think that the accepted answer there has not understood the main problem of rare crossings of rare features with rare events. Or I have not understood it. $\endgroup$ Dec 15, 2023 at 22:09
  • $\begingroup$ Check What are the consequences of rare events in logistic regression? with "The standard rule of thumb for linear (OLS) regression is that you need at least 10 data per variable or you will be 'approaching' saturation. However, for logistic regression, the corresponding rule of thumb is that you want 15 data of the less commonly occurring category for every variable." Both deal with too little sample size, but in two ways, the latter will overfit rare features at rare events. $\endgroup$ Dec 15, 2023 at 22:10
  • $\begingroup$ @questionto42 while the dimensionality is a problem in practical cases, the accepted answer does correctly explain the principle behind the problem with logistic regression, and that is that it is easily prone to overfitting because of it's function... even with a single parameters it can happen. That is a different situation/problem requirement n >= p in OLS which relates to the rank of the design matrix. $\endgroup$ Dec 15, 2023 at 22:38
  • $\begingroup$ @SextusEmpiricus The question links to a Google guide that tells about the problem in high dimensions = with (too) many features at hand. Of course the asymptotic function itself builds the ground for this to happen, but the question is about why it is such a problem in higher dimensions. And that is where the rare feature crosses come in. You add so many paramters that you can tell the story of a few rare events to a large degree with a few rare features. And since unconstrained MLEs can go up to infinity, the overfitting needs to be fought with regularization or feature reduction. $\endgroup$ Dec 15, 2023 at 22:55
  • $\begingroup$ See the Google guide Regularization in Logistic Regression "Imagine that you assign a unique id to each example, and map each id to its own feature. If you don't specify a regularization function, the model will become completely overfit." ... and more, I quoted this also in my answer at Why is logistic regression particularly prone to overfitting in high dimensions?. $\endgroup$ Dec 15, 2023 at 22:59
  • 4
    $\begingroup$ @questionto42 both deal with problems of high dimensionality but the underlying idea is different. That logistic regression is being particularly prone to overfitting has a different motivation than the $n \geq p$ requirement for OLS to work. Peter Flom mentions the two in his answer “First, you need k+1 observations for the regression to work at all, but you need a lot more than that to avoid overfitting. ”. The issues 'to work at all' and 'to avoid overfitting' or two different things. The question is about the former, the $k+1$ requirement. $\endgroup$ Dec 15, 2023 at 23:04
  • $\begingroup$ @SextusEmpiricus Good stuff. I am glad you point this out so clearly, this wording is what I wanted to find out all the time and did not know how to say it. I was seeking for something, this could be it :). I still dare to disagree "a bit" since the two things seem to be in the end the same: to work at all or the path to become good work afterwards are on the same path, even though "to work at all" is just the first step. Fine-tuning fights the same problem, just in a lower intensity, it is still linked to this "work at all" as the first step. $\endgroup$ Dec 16, 2023 at 13:57

1 Answer 1

10
$\begingroup$

It's perfectly valid (even good) to explain that a question is a special case of a more general phenomenon and then address the general case. It's also fine to explain a concept in a different domain and then explain the connection with the topic of the original question. But I don't think your answer is a good example of either of these, and the intuition you claim it offers is not evident. The question is about linear regression and OLS. You bring up logistic regression, perfect separation, MLE, and overfitting, but don't make much of an attempt to explain why invoking these concepts is useful to address the original question.

Note I have not seen your answer before this and so was not involved in the decision, but would probably also have voted to delete.

$\endgroup$
3
  • $\begingroup$ I spent some time on it. I am out on a limb here, but I somehow see that this could help getting from linear to something else, then back, to see that this is a broad challenge in many guises. I also see that it was undeleted in the meantime. As I am not a professional, I see this also from a third viewpoint, not just from my own. I am not sure whether I am right with the answer, but for my own understanding of the right balance between parameters against observations, intuition grew while I wrote :))). $\endgroup$ Dec 13, 2023 at 16:01
  • $\begingroup$ Thanks for undeleting it, I was already wondering why it was undeleted again when I had finished changing the answer. I see your other remark above only now. In the meantime, I had already changed the answer a lot. With some luck, it can survive. The downvotes are OK (I even understand them, I still do not know whether I am right with the rare feature crosses - which I only took from the paper underlying the other question about the overfitting - and their link to unconstrained MLE that is my own guess), I seek for something here, that weighs more. $\endgroup$ Dec 13, 2023 at 16:14
  • 1
    $\begingroup$ (+1) Skimming through the upvoted answers to the original question I feel that other people obviously do not mean the same by intuition as I do so I am not surprised that the OP gave an answer rather at a tangent. $\endgroup$
    – mdewey
    Dec 14, 2023 at 16:47

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .