14
$\begingroup$

Thanks to Sycorax, we have an excellent question which can be used to close as duplicate (most of) those questions about neural networks which can't overfit the training set:

What should I do when my neural network doesn't learn?

This helped to close a lot of "my network doesn't work, please fix" questions:

What's the best way to answer "my neural network doesn't work, please fix" questions?

I think we would also need a similar question for cases when the validation loss doesn't decrease (or does so only slightly). I tried to look for one, but I couldn't find it. Do we have one?

If the answer is "No", I'll write one with the same aim as Sycorax's. Now, to abuse the usual paraphrase from Tolstoy, "training issues are all alike; every generalization issue is particular in its own way", so it might be harder to give good answers to such a question, but I think that, if framed the right way, it could work. For example:

"My neural network doesn't reach the generalization error it should be able to reach, based on reputable sources or experiments with very similar NNs: which things should I check"?

For example, the first thing you need to check in this case is if you're overfitting the training set, and when (at which epoch) the training and validation losses start to diverge. There are a few questions where the OP only reports the accuracy on the validation set at a fixed number of epochs, which is pretty useless.

Update: this question has been created: What should I do when my neural network doesn't generalize well?

$\endgroup$
10
  • 1
    $\begingroup$ Looking forward to your Q&A. $\endgroup$
    – amoeba
    Commented Sep 7, 2018 at 7:21
  • $\begingroup$ @amoeba I take it this means there's no such question, currently. Ok! Then I'll ask one later today. Hopefully it doesn't get closed as a dupe after 5 minutes ;-) $\endgroup$
    – DeltaIV
    Commented Sep 7, 2018 at 8:13
  • $\begingroup$ Having your question closed as a duplicate can be frustrating, but at least you would have found a way to ask your question that presumably nobody thought of before... when you have written your Q&A, can you please link there in an answer here? $\endgroup$
    – Stephan Kolassa
    Commented Sep 7, 2018 at 12:52
  • 2
    $\begingroup$ @StephanKolassa I've written the Q. I plan to write the A by the end of the day (and I will share the link here) but if anyone wants to answer it before, be my guest. $\endgroup$
    – DeltaIV
    Commented Sep 7, 2018 at 13:14
  • $\begingroup$ @DeltaIV "by the end of the day" :-/ $\endgroup$
    – amoeba
    Commented Sep 10, 2018 at 8:21
  • 1
    $\begingroup$ @amoeba ah, but then Sycorax wrote the currently accepted answer to this question, where he correctly notes that several hours of careful crafting should go in the answer....also, you know: weekend ;-) $\endgroup$
    – DeltaIV
    Commented Sep 10, 2018 at 10:25
  • $\begingroup$ Sure I understand. But if possible, don't wait too long: you are rapidly losing chances to get on the Hot Network list which would increase the visibility of the thread (and the number of upvotes) by an order of magnitude. @Sycorax'es thread got there right after posting (I think he posted Q and A together or maybe in quick succession) and that brought 100s of upvotes. $\endgroup$
    – amoeba
    Commented Sep 10, 2018 at 21:43
  • $\begingroup$ :-($\phantom{}\phantom{}$ $\endgroup$
    – amoeba
    Commented Sep 25, 2018 at 20:23
  • $\begingroup$ @amoeba sorry, you're right! The answer ended up becoming so huge that I could never find the time complete it...I'll try to do better in the next days. Concerning the Hot Network list, I don't think the question ever made it to that list. $\endgroup$
    – DeltaIV
    Commented Sep 25, 2018 at 22:25
  • 1
    $\begingroup$ @amoeba at long last, I posted an answer! Looking forward to constructive criticism. $\endgroup$
    – DeltaIV
    Commented Sep 30, 2018 at 19:04

2 Answers 2

Reset to default
8
$\begingroup$

Just so this question has an answer...

I'm not aware of a canonical thread on this topic. I think that making one is a good idea.

I have some general advice for anyone who is considering writing an answer. From my experience working on What should I do when my neural network doesn't learn?, I discovered that the trick is to be comprehensive and organized. Take the time to plan your answer. Use the formatting options to separate the post into distinct sections of related content. It will take time, but it's often the case that specific topics have specialized threads, so it's well-worth the time to dig through the site content to find relevant discussions.

It took me several hours to write my answer, and more time to provide incremental edits and revisions. It's well worth the time, since these threads will be viewed thousands of times or more.

$\endgroup$
0
4
$\begingroup$

I finally posted an answer here. It's less comprehensive that what I had in mind initially, but I could never find the time to complete the "unabridged edition", thus I went for a shorter and more focused answer.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .