11
$\begingroup$

There are quite a few questions asking how to run backpropagation when dealing with skip connections, residual networks, "mixed" RNN-CNNs, attention mechanisms, etc.

I suspect the reason for the confusion is that backpropagation is usually taught as "manual automatic differentiation": the teacher or student manually writes out the symbolic form of the derivative at each layer of a simple feed-forward network using the chain rule.

However backprop / automatic differentiation in general works on arbitrary computation graphs (DAGs), so that any computation composed of differentiable primitives can be backpropagated, including all those different types of networks mentioned above.

(There are also questions about how to backprop through batch norm, which is itself composed of some simple mathematical primitives and is effortlessly handled by autodiff. Authors of highly optimized libraries might want to write out the "fused" gradient of batch norm to maximize performance, but I suspect that's not what people asking the question are trying to do.)

So my question is, how should I answer these types of questions? I could just post an outline of how backprop works on arbitrary computation graphs, and then add some good references for backprop/autodiff. But I'd have to repeat this on every question, and whenever someone wants to know how to perform backprop on a newly invented foo-network, they'll still start a new question because it's not obvious to them that BP in foo-networks is the same as BP in every other network.

$\endgroup$
  • 3
    $\begingroup$ I've felt that we needed a complete and comprehensive answer about backprop for some time, so I absolutely support your asking and answering of such a question. I look forward to reading and upvoting it when it's ready. $\endgroup$ – Sycorax May 2 at 21:42
  • $\begingroup$ Note that there is already a thread describing backprop at length: stats.stackexchange.com/questions/5363/… $\endgroup$ – Jan Kukacka May 9 at 11:55
  • $\begingroup$ @JanKukacka although the answer there isn't wrong, I feel that it falls into the pitfall of teaching "manual automatic differentiation", and only shows how it is applied to feedforward networks (leaving unanswered how it is applied to arbitrary neural network architectures, which are the questions i see a lot). Also Neilsen's book, upon which that answer is based, seems to have some misunderstandings on what backprop requires (stats.stackexchange.com/questions/365590/…). $\endgroup$ – shimao May 9 at 18:57
7
$\begingroup$

Consider writing your own question that will get at the underlying issue in the clearest and most concise manner, then answering it yourself. (Be open to others contributing an answer as well, though.) This could become a canonical thread for this topic, and could serve as a duplicate target in the future.

For some examples, on the main site @Sycorax has done the sort of thing I'm advocating here: What should I do when my neural network doesn't learn?. On meta.CV, he has discussed doing this sort of thing here: Do we need a canonical thread about stock trading? What would it address?

In the future, you could write a smaller, perfunctory answer to the specific question (e.g., backpropagation in CNN), link to the canonical thread, and vote to close as a duplicate. That way the OP would get some information specific to their situation, and a pointer to the more general and detailed solution. If there were subsequent questions that were essentially the same (e.g., backpropagation in CNN again), you could just immediately vote to close as a duplicate of the former. Note that you can flag closed questions to have moderators add additional duplicate targets, so that thread could end up with both the prior more specific and the more general threads listed.

$\endgroup$
  • 3
    $\begingroup$ I don't really agree with writing a perfunctory answer + voting to close as a duplicate. If it needs spelling out how the canonical answer applies; well, this question's not a duplicate & an answer doing just that with a cross-reference is the way to go: if it doesn't need spelling out, then this question can be closed as a duplicate, with no need for an answer. Agree with everything else, of course. $\endgroup$ – Scortchi♦ May 2 at 18:17
  • $\begingroup$ That's a reasonable position, @Scortchi. I think it's a bit of a gray area. Providing something in addition to VTC seems nicer to the OP (but also possibly unnecessary). I'm pretty sure I've done that before, but I can't figure out how to search for my answers on closed Qs. $\endgroup$ – gung♦ May 2 at 18:31
  • 5
    $\begingroup$ +1 One piece of advice I'd like to submit: if shimao (or anyone else) does go the route of writing & answering a question that you hope will be canonical, do put some time into planning out your answer ahead of time, writing a draft, and then rewriting it. Well-articulated canonical threads can be viewed tens of thousands of times, so it's well worth the effort to write clearly and with full coverage of the topic and include citations and links to other questions as necessary. You can write a question and answer it at the same time, so there's no "rush" to hit submit. $\endgroup$ – Sycorax May 2 at 18:35
  • 1
    $\begingroup$ @gung: If the something in addition is simply an assurance that once the OP's read the linked post they'll find that it does in fact apply to their problem, despite what a glance at it might suggest; then I'd put that in a comment. The important thing's that the two are linked, anyway. It's just that it could seem a bit off to close a question you've just answered - if you're providing information pertaining to the OP's specific situation, perhaps someone else thinks they could provide more, or do a better job. $\endgroup$ – Scortchi♦ May 2 at 19:11
  • $\begingroup$ I'd +1 but I have the same reservation as @Scortchi. $\endgroup$ – amoeba May 2 at 20:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .