We already have a thread on tag synonyms and merging the *-learning tags, but since deep learning is currently a hot topic, it probably would be better to discuss it separately.
Should we make deep-learning (2,136 questions) a synonym of neural-networks (4,304 questions)?
Currently people seem to be using "deep learning" to mean neural networks, as "shallow" networks are rarely used. I personally don't find any good reason why keeping two separate tags would have any merits, but I may be wrong.
I'm in favor of the merger.
Any questions which are specifically about deep neural networks, in contrast to neural networks in general, can be described as such in the text.
One motivation for the distinction in the literature is that training a network with more layers can be harder than training a network with fewer, among other similar types of problems (cf vanishing gradient etc). Recently, this challenge has been ameliorated (better initialization, activation, optimization and network wiring). Hence, deep neural networks are accessible to novice and expert practitioners alike, and given the success of deep networks in solving problems, it seems plausible to expect that scientists will tend to prefer using deep networks to solve problems going forward. That is, deep networks will probably come to envelope neural networks.
More to the point, as a matter of tagging, this distinction does not seem necessary. The purposes of tags is to organize posts; even though experts understand that deep and shallow networks are distinct, that fact doesn't imply that we need to preserve that distinction in tagging. If a person is in favor of maintaining the distinction, the justification must hinge on making the tagging system better.
On the other hand, combining the tags makes the usage clear and makes the organization less haphazard. Today, a user will search twice using [deep-learning] keyword and [neural-networks] keyword, or subscribe to both [deep-learning] and [neural-networks] tags. I don't understand why this is a good system when something like 90% of the [neural-networks] tags could plausibly also be [deep-learning] tags and vice versa.
Consider duplicate questions. If we commit to honoring the distinction between deep and shallow networks, can a general question about deriving back-propagation that bears the [deep-learning] but not [neural-networks] tag be closed as a duplicate of the same question bearing the [neural-networks] tag but not the [deep-learning] tag? On the one hand, the question content could be identical but for the tag, so decision Vote to Close as Duplicate may be obvious; on the other hand, if we reason that the distinction is vitally important, then we're signing up to double the number of questions under each tag. In the cases where the question contents are agnostic to network depth, this duplication is pointless from every perspective except that of tagging.
cnn, lstm, etc. will have to be used more prominently.
$\endgroup$
Commented
Apr 27, 2019 at 23:35
In favor.
Looking at the recent questions with these tags, there seems to be no pattern which get tagged neural-networks, which get deep-learning, and which get both. On the other hand, there are questions tagged deep-learning that have absolutely nothing to do with it. Making deep learning an explicit synonym of neural networks might (hopefully) even prevent some users from using the tag for any learning (that perhaps feels deep to them, or whatever was their reason for using that tag for questions like this one).
The question received +20/-0, Sycorax's answer in favor +17/-0, Jan Kukacka's answer in favor +9/-0, and Rob's answer against +2/-3 votes. We should consider the fact that at least some of the votes are by the same users, so the counts cannot be considered as independent. On another hand, there were voices in favor from some of the highest ranking users in [deep-learning] and [neural-network] tags, including Sycorax, Franck Dernoncourt (comment), and Jan Kukacka. All this means for me that the voting was in favor for merge, so the tags will be merged.
Update: The tags are synonimized.
nn wiill not return the dl-tagged threads.
$\endgroup$
Commented
Nov 1, 2020 at 21:22
Should we make [deep-learning] tag a synonym for [neural-networks]?
No. The neural-networks tag is for both SNN and DNN, whether NN tagged questions should be retagged deep or shallow is a different question. So keep NN when the question applies to both and use deep or shallow when the other isn't applicable, same for the resolution (width).
See: Why are neural networks becoming deeper, but not wider?
Depth and width, or lack thereof, are differentiating factors.
What are the effects of depth and width in deep neural networks?
See this question from 3 1/2 years ago: What is the difference between a neural network and a deep neural network, and why do the deep ones work better? (100 up / 0 down) and the useful image included in amoeba's (Jan 18 2018) answer with 136 upvotes, 0 down, (and the other 9 answers barely scoring 2 dozen votes total):
... for real-world tasks deep architecture are often beneficial and shallow architecture would be inefficient and require a lot more neurons for the same performance.
But it's far from proven. Consider e.g. Zagoruyko and Komodakis, 2016, Wide Residual Networks. Residual networks with 150+ layers appeared in 2015 and won various image recognition contests. This was a big success and looked like a compelling argument in favour of deepness; here is one figure from a presentation ...
Article by Bernard Marr & Co.: "Deep Learning Vs Neural Networks - What’s The Difference?". While I disagree that the line should be 3 I agree that there's a difference.
Quora Q&A: "What is the difference between Neural Networks and Deep Learning?":
Nicolas Neubauer, PhD in network mining & analysis
Answered May 27, 2015
"Neural networks" can be used to refer to the whole class of machine learning architectures where individual units are connected via weights and those weights are adjusted as the network is trained. In that sense, deep learning is just a particular branch of network architecture and training.In a more narrow sense, neural networks might refer to the "old-school" way of constructing and training networks, where you have few layers (typically input, output, and 1 or 2 layers in-between), and then deep learning is the "new" way of doing this.
Wikipedia: "Deep Learning":
Overview
Most modern deep learning models are based on an artificial neural networks, specifically, Convolutional Neural Networks (CNN)s, although they can also include propositional formulas or latent variables organized layer-wise in deep generative models such as the nodes in deep belief networks and deep Boltzmann machines.
In deep learning, each level learns to transform its input data into a slightly more abstract and composite representation. In an image recognition application, the raw input may be a matrix of pixels; the first representational layer may abstract the pixels and encode edges; the second layer may compose and encode arrangements of edges; the third layer may encode a nose and eyes; and the fourth layer may recognize that the image contains a face. Importantly, a deep learning process can learn which features to optimally place in which level on its own. (Of course, this does not completely obviate the need for hand-tuning; for example, varying numbers of layers and layer sizes can provide different degrees of abstraction.)
The "deep" in "deep learning" refers to the number of layers through which the data is transformed. More precisely, deep learning systems have a substantial credit assignment path (CAP) depth. The CAP is the chain of transformations from input to output. CAPs describe potentially causal connections between input and output. For a feedforward neural network, the depth of the CAPs is that of the network and is the number of hidden layers plus one (as the output layer is also parameterized). For recurrent neural networks, in which a signal may propagate through a layer more than once, the CAP depth is potentially unlimited. No universally agreed upon threshold of depth divides shallow learning from deep learning, but most researchers agree that deep learning involves CAP depth > 2. CAP of depth 2 has been shown to be a universal approximator in the sense that it can emulate any function.[citation needed] Beyond that more layers do not add to the function approximator ability of the network. Deep models (CAP > 2) are able to extract better features than shallow models and hence, extra layers help in learning features.
Etc.
