34
$\begingroup$

I've observed that we tend to get a number of questions in the vein of "I'm training a neural network and the model is not effective (for some definition of effective). Please help me fix it." (I can provide examples of these types of questions, but I don't think it would be helpful to highlight the authors at this time.)

These questions are not exactly programming questions, since OP has syntactically-correct code; instead, OP's problem is that the model isn't very effective at whatever task they're working on. (But there could be semantic errors, in which case the question is, in my view, a programming question -- but whether or not this is the case is not obvious without a detailed study of the code and what task OP is trying to accomplish with the model. Both are often unclear.)

In some sense, they are on-topic because they are about how to carry out modeling which is, essentially, a core component of modern machine learning.

On the other hand, since all neural networks are bespoke and parameterized with a truly massive number of knobs (hyperparameters) to twiddle, these questions don't seem to generalize very well. And these threads tend to take on the form of a frustrating IT support dialogue:

Answerer: "Try adjusting the learning rate."

OP: "That didn't work."

Answerer: "Try using a smaller hidden layer."

OP: "That didn't work."

Answerer: "Try using a different activation function."

OP: "That didn't work."

We could undertake to write a comprehensive answer enumerating the various knobs to attempt improving the model, perhaps in the form of "symptom/diagnosis/treatment." I don't know how effective this would be, since OP will often insist "but I tried that!" Moreover, it's usually true that many knobs will need to be turned in combination.

By way of example, in working on a hobby neural network project, it took me more than 150 different model configurations to get a model that worked to my satisfaction, and this involved tuning a number of different model attributes in combination (and many hours of computing time).

Is there a way that we can give helpful answers to the rising tide of these rather under-specified questions? Tuning neural networks is important, and we have a large volume of questions about how to do it, but I have yet to see a good answer to any one question.

$\endgroup$
19
  • 6
    $\begingroup$ My feeling is that -- without very careful focus - most such questions are too broad (and often unclear), but I've been hoping the many machine learning experts on site would curate these posts - initially, placing most of them on hold. While I know some topics that would now be considered machine learning well enough, I'm not in a position to curate other topics - e.g. many of the technical NN posts. The standards applied by the high rep users in these topics seem considerably more lenient than the rest of the site; we'd expect to see some difference, but it seems like a lot. .... ctd $\endgroup$
    – Glen_b
    Commented May 30, 2018 at 2:00
  • 14
    $\begingroup$ ctd... I appreciate the motivation to help & the increasing importance of the topic but we cannot solve every kind of problem. In particular, we're not a clinic. Vague questions in the "it doesn't work, please fix it" style aren't even questions. They should be on hold until they're a clear question answerable in a few paragraphs, or they belong on some kind of site nearer to a forum. StackExchange sites are supposed to be a repository for questions whose answers will help more than one user. There's some greyness in the placement of the line, but I think almost all of these go beyond it. $\endgroup$
    – Glen_b
    Commented May 30, 2018 at 2:02
  • 3
    $\begingroup$ I couldn't answer even the simplest questions on neural networks, but these seem on all fours with any question of the form Why doesn't my method work well (OR at all) with my data? The latter can often be answered with a small or moderate dataset and a simple procedure, but once the dataset is too large to post, the model or method or estimation is complicated, and there may even be doubt about whether in principle the procedure is a good choice for the data, or vice versa, all bets are usually off. $\endgroup$
    – Nick Cox
    Commented May 30, 2018 at 8:59
  • 1
    $\begingroup$ Thanks for the input, @Glen_b and Nick Cox. I think that at least some of these problems could be resolved with a more vigorous close-as-unclear or close-as-dupe, but since expertise is required to make those determinations, and that expertise is uncommon among our regular reviewers, the review queue is not working quite how we want it to. So right now I'm undertaking to get a gold badge in neural-network, so that I have "dupe-hammer" power. But this will take some time to accumulate that much rep in neural-network. $\endgroup$
    – Sycorax Mod
    Commented May 30, 2018 at 15:34
  • $\begingroup$ That's a noble aim. One or two users with gold badges would be a big help. $\endgroup$
    – Glen_b
    Commented May 30, 2018 at 16:34
  • $\begingroup$ Can't such questions be marked as duplicate of some other question (in a general way, 'how do I make a NN of type ... more effective?' 'which knows can I turn with this type ... of NN?')? Otherwise (if no duplicate) then I guess they may be considered usefull, although they might be worded in a very specific way such that general application and searchability for future users might be difficult. $\endgroup$
    – Sextus Empiricus
    Commented May 30, 2018 at 17:30
  • 2
    $\begingroup$ @MartijnWeterings In general, this is how the queue works. Over the past few months, I've noticed that a number of my attempts to close-as-dupe fail. My speculation is that other reviewers are, in instances where they do not feel competent to judge, are voting to leave open/skipping voting entirely. This is all well and good (people shouldn't close-as-dupe unless they understand what's going on in the questions), but it has the side-effect that some dupes remain open because the 5-vote threshold is not reached. $\endgroup$
    – Sycorax Mod
    Commented May 30, 2018 at 17:32
  • 1
    $\begingroup$ You could create a sort of support question like is done with general question about software. Then all (or many) of those questions can be closed as being duplicates (or at least receive little upvotes) if they do not provide sufficient novelty. $\endgroup$
    – Sextus Empiricus
    Commented May 30, 2018 at 17:33
  • 1
    $\begingroup$ @MartijnWeterings We could. And I think that I will, over the next few weeks, try to write & answer such a question. The challenging point is that askers might feel shortchanged by that result: "But I tried all the suggestions in that thread and it didn't work!" And maybe they have! Usually the problem is getting all of the different dials configured "just right" in combination. $\endgroup$
    – Sycorax Mod
    Commented May 30, 2018 at 17:35
  • $\begingroup$ @Sycorax "Usually the problem is getting all of the different dials configured "just right" in combination."... this can be part of the general solution/answer. If people want some ready made solution like an answer to homework or get somebody else to finish their project, then I guess it is off-topic. It is just debugging or doing regular work which makes the question useless for the general public unless there is some (new) 'principle' explained by the answer. $\endgroup$
    – Sextus Empiricus
    Commented May 30, 2018 at 18:31
  • $\begingroup$ I am among those who wouldn't read questions on NNs or presume to judge what is and what is not a good question on NNs and so am part of the problem. $\endgroup$
    – Nick Cox
    Commented May 31, 2018 at 23:37
  • $\begingroup$ If the code is "syntactically correct" shouldn't OPs be directed to: codereview.stackexchange.com? (I've been gently asked to go there myself when posting a question when asking for help improving "working" code). $\endgroup$
    – D. Woods
    Commented Jun 3, 2018 at 19:32
  • 1
    $\begingroup$ hey, I'm glad to see I'm not the only one who has issues with this kind of DNN questions! I've been asking similar questions before here on Meta several times. Last year, I even predicted that these kind of questions would become more and more common 1/ $\endgroup$
    – DeltaIV
    Commented Jun 4, 2018 at 9:09
  • 2
    $\begingroup$ Recently I considered posting the same meta question. These answers are generally impossible to answer other than using some generic recipes from Deep learning 101, and have very low value for future users (whose network also doesn't work but they need to tune other hyperparameters to make it work). In this sense I think they are off-topic. $\endgroup$
    – Jan Kukacka
    Commented Jun 4, 2018 at 14:04
  • 2
    $\begingroup$ @JanKukacka But I think that repeating the generic advice could be a valid answer. The trick is to correctly scope the question so that we write that answer exactly once. $\endgroup$
    – Sycorax Mod
    Commented Jun 4, 2018 at 14:48

3 Answers 3

Reset to default
20
$\begingroup$

Yes, it's common, and also on DataScience, AI, and Stackover itself. We are the nicest community, questions like that would easily downvoted to -6 on Stackover.

I think we should:

  • Try to spot any obvious mistake/inconsistency from the question. For example, not using softmax properly
  • Ask for visualization clues

But we shouldn't:

  • Download their dataset, run the network ourself
  • Provide IT-support like answers such as "try a bigger network!"

I don't see how we will be able to offer any useful advice on debugging. Anything could happen; very likely the code is just bugged. We don't have time for debugging a network implementation. Unfortunately, neural network programming is a skill set that require extensive training and understanding. OP would need to seek for paid consultant services.

We should vote to close the questions until OP comes back with something more concrete to answer.

$\endgroup$
3
10
$\begingroup$

Basically helping out with these neural networks is (often) like debugging and/or solving a complex unclear problem. It is like doing a (big) task rather than answering a clear (and abstract) question. So it should be deemed off-topic when a question does not involve the solution or clarification of specific principles and is not useful for the site other than helping out the single person who asked the question.

The same issue exists when people ask for support with problems related to software. For these cases there is a reference Internet Support for Statistics Software . That place provides resources for general help to solve software problems. This link is then used in place of many questions which can be closed with a reference to this general help and support FAQ about software.

If many questions exist surrounding similar support issues related to neural networks, then a similar type of standard community support question could be created. That would provide resources for general support on neural networks and allow us to close most of the questions. Questions that still involve a novel and abstract component of general interest can still be answered, as they are useful for the database of questions and answers.

$\endgroup$
3
  • 1
    $\begingroup$ Lightly edited for style, etc. Please check that I have been faithful to your intent. $\endgroup$
    – Nick Cox
    Commented Jun 6, 2018 at 19:22
  • $\begingroup$ This could be a canonical thread on the main site, & the new questions could be closed as duplicates. $\endgroup$
    – gung - Reinstate Monica
    Commented Jun 7, 2018 at 15:38
  • 1
    $\begingroup$ I have posted what I hope will become the canonical answer for "how to train a neural network" questions here: stats.stackexchange.com/questions/352036/… $\endgroup$
    – Sycorax Mod
    Commented Jun 19, 2018 at 0:37
10
$\begingroup$

I have decided to make a single-destination thread for "debugging" neural network performance, with the goal that other questions which ask similar questions can be marked as a duplicate of this one.

My neural network doesn't learn

$\endgroup$
1
  • 3
    $\begingroup$ I've favourited it so I can hopefully find it again $\endgroup$
    – Glen_b
    Commented Jun 19, 2018 at 2:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .