Thanks to Sycorax, we have an excellent question which can be used to close as duplicate (most of) those questions about neural networks which can't overfit the training set:
This helped to close a lot of "my network doesn't work, please fix" questions:
I think we would also need a similar question for cases when the validation loss doesn't decrease (or does so only slightly). I tried to look for one, but I couldn't find it. Do we have one?
If the answer is "No", I'll write one with the same aim as Sycorax's. Now, to abuse the usual paraphrase from Tolstoy, "training issues are all alike; every generalization issue is particular in its own way", so it might be harder to give good answers to such a question, but I think that, if framed the right way, it could work. For example:
"My neural network doesn't reach the generalization error it should be able to reach, based on reputable sources or experiments with very similar NNs: which things should I check"?
For example, the first thing you need to check in this case is if you're overfitting the training set, and when (at which epoch) the training and validation losses start to diverge. There are a few questions where the OP only reports the accuracy on the validation set at a fixed number of epochs, which is pretty useless.
Update: this question has been created: What should I do when my neural network doesn't generalize well?