12
$\begingroup$

I know that accuracy is an improper scoring rule, thus it shouldn't be used to choose among classifiers. In particular, it should not be used to choose the best hyperparameters for a NN. However, since the popularization of Deep Learning, lots of people have been using validation accuracy for NN model selection. All of them, at some point, rediscover the fact that even if the validation loss is increasing, the validation accuracy may keep increasing or oscillate:

Can it be over fitting when validation loss and validation accuracy is both increasing?

I think at least some of them would be interested in an authoritative answer to the question: "How can validation accuracy increase or oscillate, if the validation loss keeps increasing?". Do we have a canonical Q&A on this? It would not only be interesting in itself, but also in order to close duplicates.

$\endgroup$
5
$\begingroup$

I think that

Why is accuracy not the best measure for assessing classification models?

is the closest answer about this issue; I don't believe it exactly addresses what you're asking about, the change in accuracy or loss across epochs during training, but only the shortcomings of comparing models on the basis of accuracy reported after training has completed.

$\endgroup$
  • 2
    $\begingroup$ Given that this thread is pretty highly voted (partly through ending up on the HNQ list), I think it would be a good idea if DeltaIV or someone else added an answer addressing the specific aspect in question. After all, it is yet another reason why accuracy can be misleading. (Yes, I may be biased.) $\endgroup$ – Stephan Kolassa Dec 31 '18 at 19:46
  • $\begingroup$ Could this be a case where OP (in this case, @Tim) edits the question to be slightly more broad than it is now? Or would creating a new question be in order? $\endgroup$ – Sycorax♦ Jan 1 '19 at 0:43
  • $\begingroup$ I agree that this is the closest answer, but I don't like with @StephanKolassa proposal of adding my answer to that question. I asked the question "why can accuracy increase when loss decreases", I wouldn't accept an answer "this is another manifestation of the inadequacy of accuracy as a classification metric". I would like to understand how accuracy could increase, when loss is also increasing. $\endgroup$ – DeltaIV Jan 1 '19 at 23:29
  • 1
    $\begingroup$ @DeltaIV This question just got bumped to the front page by the Community user. It seems to be related. stats.stackexchange.com/questions/317699/… $\endgroup$ – Sycorax♦ Jan 2 '19 at 0:45
  • 1
    $\begingroup$ I think I found exactly the question I was looking for: stats.stackexchange.com/questions/282160/… I think it should be used to close duplicates in the future. Looks like I won't be able to score some points by asking & answering such a question, in the end ☺️ $\endgroup$ – DeltaIV Jan 2 '19 at 7:14
  • 2
    $\begingroup$ @DeltaIV Happy new year! I only noticed this Meta thread now. Replying to your last comment above: I think the Q that you found is great but the answers (including the accepted one) are really mediocre. They don't provide any direct evidence that these are the correct explanations, and don't cite any relevant literature... I'm pretty sure a better answer is possible. I'd be very interested to see an authoritative answer in that thread. If you can answer it better then please go ahead & let me know in case I miss it. $\endgroup$ – amoeba Jan 7 '19 at 22:48
  • 1
    $\begingroup$ @amoeba thanks for the trust :-) I also think that the current answers are...improvable. However: 1) I think the comment that refers to stats.stackexchange.com/questions/258166/… is spot on. It's still not the answer, but it's closer. 2) Holidays are over for me, and I just changed job function, so I don't think I'll be able to write an authoritative answer anytime soon. $\endgroup$ – DeltaIV Jan 8 '19 at 13:30
  • 1
    $\begingroup$ @DeltaIV Hmm, congratulations, I guess ;-) $\endgroup$ – amoeba Jan 8 '19 at 14:35
  • $\begingroup$ @amoeba thanks :-) $\endgroup$ – DeltaIV Jan 8 '19 at 16:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .