A few months ago, I read an interesting comment by user shimao to a question, which linked to a YouTube video where some lecturer (a mathematician, I think) showed how ReLU activations don't solve the vanishing gradient problem (contrary to "popular" belief), but ResNets do, and went on to prove the second result for linear residual connection networks. I tried to find the question, but searching for things like "ReLU solve vanishing gradient" or even just "vanishing gradient" doesn't help. I think the question was not too old. Can you help me find it? As the link was in a comment, finding it is particularly hard.
The reason you probably did not find the comment is because it was not
shimao who posted, but
Jan Kukacka (see here).
I recommend watching this talk by Moritz Hardt (youtube.com/watch?v=l1YxQ1Od1Y0) where he explains that ReLU in fact does not solve the vanishing gradient problem. – Jan Kukacka Dec 25 '17 at 20:20
I found the question putting the following in CV's search box:
vanishing gradient problem ReLU (it was the first/top Q in the 'relevance' tab).