15
$\begingroup$

I see a fair number of these questions from new members wanting to know how to interpret their model result. This is usually followed by a series of comments explaining that there is no general way to judge whether R2 (or any other such value) is high or not, that this cannot be answered without more expertise in the field, and so on. This type of question not infrequently ends up being closed as unclear.

Do you think it would be useful to create a question/answer that addresses this general point thoroughly, as a duplicate target for these questions? This question is inspired in part by the useful and popular thread What should I do when my neural network doesn't generalize well?

$\endgroup$
4
  • 1
    $\begingroup$ Would be useful, go ahead! $\endgroup$
    – kjetil b halvorsen Mod
    Jun 18, 2019 at 19:38
  • $\begingroup$ @kjetilbhalvorsen Will do so soon, thanks! $\endgroup$
    – mkt
    Jun 18, 2019 at 20:33
  • 1
    $\begingroup$ And when you do it please post here to point to it so it gets maximum publicity $\endgroup$
    – mdewey
    Jun 22, 2019 at 12:36
  • $\begingroup$ @mdewey Done, thanks! Link in the answer below. $\endgroup$
    – mkt
    Jun 23, 2019 at 20:09

2 Answers 2

Reset to default
7
$\begingroup$

Thanks for all your feedback. I have now posted a question and answer here:

Is my model any good, based on the diagnostic metric ($R^2$ / AUC / accuracy / etc) value?

Edits, suggestions, feedback, and additional answers welcome. I hope this is useful.

$\endgroup$
-5
$\begingroup$

Interpretability actually fundamentally necessary to answer questions about overfitting. There are a few recent books on the interpretability of models that disagree with the notion that there is no universal way of judging models. Instead, there is a universal way, the problem is it cannot be automated with the current level of computer vision that we have. For example see this book: https://christophm.github.io/interpretable-ml-book/ Fundamentally, the ability to generalize necessary to answer questions about AIC. There are a few recent books on the interpretability of models that disagree with the notion that there is no universal way of judging models. Instead, there is a universal way, the problem is it cannot be automated with the current level of computer vision that we have. For example see this book: https://christophm.github.io/interpretable-ml-book/

The main problem is that calculating AUC, Adjusted R^2, and is not universal for all algorithms in Python or R even if the concepts themselves are agnostic and universal. The moderators would have to allow questions that they have previously struck down as "how to code questions"? (*)

Because of the issue with a canonical question

(*)P.S. I can be more code specific if we are allowed too.

The main problem is that calculating AUC, Adjusted R^2, and is that methods are not universal for all algorithms in Python or R even if the concepts themselves are agnostic. To really talk about this we would need to talk about computer code, so moderators would have to allow questions that they have previously struck down as "how to code questions"? (*)

I am still for it, I am just thinking the moderators might need to adjust the a threshold for acceptable questions/answers just this once if this is the direction you guys want to go.

(*)P.S. To prove my point, I can be more code specific about packages and libraries if we are allowed too related to universal methods to interpret models and prevent overfitting.

$\endgroup$
5
  • $\begingroup$ If you really read about interpretability and about AUC and R^2, you would realize that interpretability and explainability contains that part of statistics that fundamentally inseparable from code or pseudo-code. Dr. Moran is not the first statistician to make this point. $\endgroup$
    – mlane
    Jun 22, 2019 at 23:42
  • $\begingroup$ Thus, I would hate to see mkt efforts struck down because we "don't allow code questions" on this StackExchange. Because such a move fundamentally goes against the ethics of data science and the knowledge of machine learning when it comes to interpreting models and preventing overfitting. Machine learning should be transparent $\endgroup$
    – mlane
    Jun 22, 2019 at 23:48
  • 3
    $\begingroup$ I appreciate your effort in writing this, but I am afraid I don't fully understand it. You are of course welcome to post your own answer to the question (now been posted, link in my answer). $\endgroup$
    – mkt
    Jun 23, 2019 at 20:29
  • $\begingroup$ @mkt I realized after reading various post you linked to and other did stack overflow. So I worked the last 2 months on a new thread: stats.meta.stackexchange.com/questions/5725/… That the connection is poorly understood. I am still working on the feature one to cite specific code, but the model on with links to code is mostly done. $\endgroup$
    – mlane
    Jul 30, 2019 at 6:33
  • $\begingroup$ I'm afraid I don't follow the new post either. I hope someone else will be able to offer more useful feedback. $\endgroup$
    – mkt
    Jul 30, 2019 at 11:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .