5
$\begingroup$

A few months ago, I read an interesting comment by user shimao to a question, which linked to a YouTube video where some lecturer (a mathematician, I think) showed how ReLU activations don't solve the vanishing gradient problem (contrary to "popular" belief), but ResNets do, and went on to prove the second result for linear residual connection networks. I tried to find the question, but searching for things like "ReLU solve vanishing gradient" or even just "vanishing gradient" doesn't help. I think the question was not too old. Can you help me find it? As the link was in a comment, finding it is particularly hard.

$\endgroup$
3
  • 3
    $\begingroup$ The site search engine doesn't look at comments, but sometimes Google does. Try keywords like "shimao relu resnet site:stats.stackexchange.com". $\endgroup$
    – whuber Mod
    Commented Jul 14, 2018 at 19:27
  • 1
    $\begingroup$ @AndreSilva good suggestion, I tried to do that without success. $\endgroup$
    – DeltaIV
    Commented Jul 15, 2018 at 14:49
  • $\begingroup$ @whuber good point, tried to do that , but still couldn't find anything. Such a pity, it was a great video....Actually Google did return a few hits that sounded promising, but they were pointing to deleted questions. That would also explain why it doesn't show up in my favorites list - years of burns with the SE search engine have taught me to immediately add to Favorites any question/answer that I find interesting. Thus, I was expecting it to be there, but if the question has been deleted in meantime, that would explain why it's no more among my favorites. $\endgroup$
    – DeltaIV
    Commented Jul 15, 2018 at 14:52

1 Answer 1

Reset to default
8
$\begingroup$

I believe I found it: Vanishing gradient vs. dying ReLU? (youtube video here).

The reason you probably did not find the comment is because it was not shimao who posted, but Jan Kukacka (see here).

I recommend watching this talk by Moritz Hardt (youtube.com/watch?v=l1YxQ1Od1Y0) where he explains that ReLU in fact does not solve the vanishing gradient problem. – Jan Kukacka Dec 25 '17 at 20:20

I found the question putting the following in CV's search box: vanishing gradient problem ReLU (it was the first/top Q in the 'relevance' tab).

$\endgroup$
1
  • 1
    $\begingroup$ ah! That explains the mystery. Thanks a lot :-) $\endgroup$
    – DeltaIV
    Commented Jul 15, 2018 at 16:49

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .