# Should we make [deep-learning] tag a synonym for [neural-networks]?

We already have a thread on tag synonyms and merging the *-learning tags, but since deep learning is currently a hot topic, it probably would be better to discuss it separately.

Should we make (2,136 questions) a synonym of (4,304 questions)?

Currently people seem to be using "deep learning" to mean neural networks, as "shallow" networks are rarely used. I personally don't find any good reason why keeping two separate tags would have any merits, but I may be wrong.

• Anyone knows it that would help us get some more neural-networks gold badge user privileges out there? – Firebug Apr 29 '19 at 14:20
• @Firebug Sycorax is just a few votes away from a gold badge in [neural-networks]. They will probably get it in a few days anyway. It does not look like anybody else would be close even after the suggested tag merge. – amoeba Apr 30 '19 at 8:39
• Deep learning is mostly used as a marketing term. +1 for the merge. – Franck Dernoncourt May 3 '19 at 2:27
• the merge is ok if the neural-network tag is kept, not the other way around – user May 6 '19 at 17:36

I'm in favor of the merger.

Any questions which are specifically about deep neural networks, in contrast to neural networks in general, can be described as such in the text.

One motivation for the distinction in the literature is that training a network with more layers can be harder than training a network with fewer, among other similar types of problems (cf vanishing gradient etc). Recently, this challenge has been ameliorated (better initialization, activation, optimization and network wiring). Hence, deep neural networks are accessible to novice and expert practitioners alike, and given the success of deep networks in solving problems, it seems plausible to expect that scientists will tend to prefer using deep networks to solve problems going forward. That is, deep networks will probably come to envelope neural networks.

More to the point, as a matter of tagging, this distinction does not seem necessary. The purposes of tags is to organize posts; even though experts understand that deep and shallow networks are distinct, that fact doesn't imply that we need to preserve that distinction in tagging. If a person is in favor of maintaining the distinction, the justification must hinge on making the tagging system better.

On the other hand, combining the tags makes the usage clear and makes the organization less haphazard. Today, a user will search twice using [deep-learning] keyword and [neural-networks] keyword, or subscribe to both [deep-learning] and [neural-networks] tags. I don't understand why this is a good system when something like 90% of the [neural-networks] tags could plausibly also be [deep-learning] tags and vice versa.

Consider duplicate questions. If we commit to honoring the distinction between deep and shallow networks, can a general question about deriving back-propagation that bears the [deep-learning] but not [neural-networks] tag be closed as a duplicate of the same question bearing the [neural-networks] tag but not the [deep-learning] tag? On the one hand, the question content could be identical but for the tag, so decision Vote to Close as Duplicate may be obvious; on the other hand, if we reason that the distinction is vitally important, then we're signing up to double the number of questions under each tag. In the cases where the question contents are agnostic to network depth, this duplication is pointless from every perspective except that of tagging.

• I +1ed this but I suspect that it will make the life for users who actually have a deep learning question instead of a relative generic NN issue, harder. – usεr11852 Apr 27 '19 at 22:36
• @usεr11852 Thanks! Can you elaborate on how using the DNN tag instead of the NN tag would help? It seems like concerns peculiar to DNNs would be enumerated in a clear Question. But I might have missed something subtle. – Sycorax Apr 27 '19 at 23:27
• I think that most users employ tags to bring things to their attention and this will create more noise (to an already noise tag). Especially long and involved question on DNN will probably be harder to stand out. As mentioned, I am in favour of the merger because deep-learning tag has come to be (colloquially) used as a synonym with NN. I suspect that other tags like cnn, lstm, etc. will have to be used more prominently. – usεr11852 Apr 27 '19 at 23:35
• @usεr11852 I think I understand: the "tag subscription" feature will work differently. I suppose someone who subscribes to deep-learning but has no interest in neural-networks would lose a little bit of functionality. On the other hand, I think the merger improves things because something like 90% of questions about neural-networks would appropriately have the deep-learning tag and vice versa. I agree that the LSTM/RNN/CNN tags should be used more often. – Sycorax Apr 27 '19 at 23:39
• Sure, fair enough. As mentioned, I am in favour of the merger. Probably the disruption to these DNN user will be small. – usεr11852 Apr 28 '19 at 9:15
• @Sycorax unrelated to this post, but: Congrats on the first golden one [neural-networks] badge of all times! – Jan Kukacka May 3 '19 at 10:01
• Sycorax & @JanKukacka would you be willing to help to improve the neural-networks tag wiki for the fact that now it covers both deep and shallow networks? – Tim May 6 '19 at 6:53
• @Tim I've taken a stab at improving the tag description. One concern: on the one hand, I think we need to explain why some models have this exotic name ("neural networks") which might not be obvious to a newcomer. On the other hand, I really dislike the name because I think it gives newcomers some false expectations about how NNs should work ("My brain is a neural network, therefore my model should be as good as my brain."). What do you make of this? – Sycorax May 6 '19 at 14:25
• @Sycorax agree, but I don't think that this is something to be explained on the tag page. Saying that it is "loosely inspired" by biological NNs is probably enough. – Tim May 6 '19 at 21:01
• I made some further edits to the tag wiki. I felt the wiki excerpt was trying to give some lengthy explanations that are fine for the tag wiki but are not needed in a wiki excerpt. (CC @Tim) – amoeba May 8 '19 at 20:47
• I made some further edits in wiki (cc @amoeba). – Tim May 8 '19 at 20:56

In favor.

Looking at the recent questions with these tags, there seems to be no pattern which get tagged , which get , and which get both. On the other hand, there are questions tagged that have absolutely nothing to do with it. Making deep learning an explicit synonym of neural networks might (hopefully) even prevent some users from using the tag for any learning (that perhaps feels deep to them, or whatever was their reason for using that tag for questions like this one).

The question received +20/-0, Sycorax's answer in favor +17/-0, Jan Kukacka's answer in favor +9/-0, and Rob's answer against +2/-3 votes. We should consider the fact that at least some of the votes are by the same users, so the counts cannot be considered as independent. On another hand, there were voices in favor from some of the highest ranking users in [deep-learning] and [neural-network] tags, including Sycorax, Franck Dernoncourt (comment), and Jan Kukacka. All this means for me that the voting was in favor for merge, so the tags will be merged.

Update: The tags are synonimized.

• If anyone is wondering how to vote for a synonym, you can do so from the tag's page if you have sufficient reputation. Go to neural-networks and then click Synonyms. – Sycorax May 4 '19 at 22:12
• By the way, how did you -- as a mod -- manage to suggest a synonym without automatically approving it? – amoeba May 6 '19 at 7:37
• @amoeba I'm not sure. I found only this meta.mathoverflow.net/questions/3731/… – Tim May 6 '19 at 7:45

Should we make [deep-learning] tag a synonym for [neural-networks]?

No. The neural-networks tag is for both SNN and DNN, whether NN tagged questions should be retagged deep or shallow is a different question. So keep NN when the question applies to both and use deep or shallow when the other isn't applicable, same for the resolution (width).

Depth and width, or lack thereof, are differentiating factors.

What are the effects of depth and width in deep neural networks?

See this question from 3 1/2 years ago: What is the difference between a neural network and a deep neural network, and why do the deep ones work better? (100 up / 0 down) and the useful image included in amoeba's (Jan 18 2018) answer with 136 upvotes, 0 down, (and the other 9 answers barely scoring 2 dozen votes total):

... for real-world tasks deep architecture are often beneficial and shallow architecture would be inefficient and require a lot more neurons for the same performance.

But it's far from proven. Consider e.g. Zagoruyko and Komodakis, 2016, Wide Residual Networks. Residual networks with 150+ layers appeared in 2015 and won various image recognition contests. This was a big success and looked like a compelling argument in favour of deepness; here is one figure from a presentation ...

Article by Bernard Marr & Co.: "Deep Learning Vs Neural Networks - What’s The Difference?". While I disagree that the line should be 3 I agree that there's a difference.

Nicolas Neubauer, PhD in network mining & analysis
"Neural networks" can be used to refer to the whole class of machine learning architectures where individual units are connected via weights and those weights are adjusted as the network is trained. In that sense, deep learning is just a particular branch of network architecture and training.

In a more narrow sense, neural networks might refer to the "old-school" way of constructing and training networks, where you have few layers (typically input, output, and 1 or 2 layers in-between), and then deep learning is the "new" way of doing this.

Wikipedia: "Deep Learning":

Overview

Most modern deep learning models are based on an artificial neural networks, specifically, Convolutional Neural Networks (CNN)s, although they can also include propositional formulas or latent variables organized layer-wise in deep generative models such as the nodes in deep belief networks and deep Boltzmann machines.

In deep learning, each level learns to transform its input data into a slightly more abstract and composite representation. In an image recognition application, the raw input may be a matrix of pixels; the first representational layer may abstract the pixels and encode edges; the second layer may compose and encode arrangements of edges; the third layer may encode a nose and eyes; and the fourth layer may recognize that the image contains a face. Importantly, a deep learning process can learn which features to optimally place in which level on its own. (Of course, this does not completely obviate the need for hand-tuning; for example, varying numbers of layers and layer sizes can provide different degrees of abstraction.)

The "deep" in "deep learning" refers to the number of layers through which the data is transformed. More precisely, deep learning systems have a substantial credit assignment path (CAP) depth. The CAP is the chain of transformations from input to output. CAPs describe potentially causal connections between input and output. For a feedforward neural network, the depth of the CAPs is that of the network and is the number of hidden layers plus one (as the output layer is also parameterized). For recurrent neural networks, in which a signal may propagate through a layer more than once, the CAP depth is potentially unlimited. No universally agreed upon threshold of depth divides shallow learning from deep learning, but most researchers agree that deep learning involves CAP depth > 2. CAP of depth 2 has been shown to be a universal approximator in the sense that it can emulate any function.[citation needed] Beyond that more layers do not add to the function approximator ability of the network. Deep models (CAP > 2) are able to extract better features than shallow models and hence, extra layers help in learning features.

Etc.

• The question is: there's 4300 questions tagged as [neural-network] and assuming that you need one minute to read question and re-tag it, are you willing to spend approximately 72 hours non-stop on re-taggig the questions? Also notice that nobody here considers neural networks and deep learning as completely the same. You don't seem to give arguments why it would be useful to keep separate tags, while there seem to be arguments against it that got some +1s. – Tim Apr 29 '19 at 15:55
• @Tim The "usefulness" of seperate tags (deep and unspecified) is only when the question is intended to be about neural networks in general and not specifically about only deep ones. Perhaps the solution was too subtle, that being to find the few questions that are shallow and remove them from the tag, retagging them shallow, then merge; and from here on hopefully people will put the applicable tag on their questions. What you (and some others) propose is just lump them all together, a few minutes discussion, and a few seconds work to synonymize - that's what I object to. – Rob Apr 29 '19 at 16:59
• Could you give a few examples of questions that specifically ask about shallow neural networks? At the moment, this seems to me like a rather hypothetical scenario. – Jan Kukacka Apr 30 '19 at 7:17
• @Rob: most of the results in this search are not asking specifically about shallow nets... Lots of them ask for "differences between shallow and deep nets", which in my opinion is covered by tag [neural-networks]. Using this query brings us down to 10 questions... – Jan Kukacka Apr 30 '19 at 11:55
• @JanKukacka - Yes, that's the point made. It's easy to find a few, and miss a few. Even easier to just say everything is deep even when it's not. Take the first example returned by the link you offered, it's tagged "deep-learning". --- We only need to do a few simple searches (we can see the last one is a false-positive, so that's 6 results). If it takes a couple of hours to do a half decent job that's the price, to do a better than minimum effort. – Rob Apr 30 '19 at 13:04
• Rob, we made the tags synonyms, but if you feel that there are questions that are specific for single-layer neural networks and not for multilayer ones, & tagging them would be meaningful, then maybe you should start another thread and we will discuss creating a separate tag for such questions? – Tim May 6 '19 at 6:56
• @Tim Thanks for keeping me updated on the status. With a synonymization it can be undone, merge has a permanency. The link above shows there's ~ a dozen for someone to look at. – Rob May 6 '19 at 9:33
• Rob, the tags are only synonymized at the moment, they are not merged (@Tim). Our usual policy is to merge tags some time (~1 month) after synonymizing if no problems are uncovered. – amoeba May 6 '19 at 10:06