14
$\begingroup$

I've seen a few questions like this lately:

Backpropagation in multi-layer perceptron (MLP) doesn't converge

They don't seem programming questions to me: the code is syntactically correct, because it runs, but the result is not the desired one (semantic error). In this case the loss function doesn't decay with epochs or something like that...the OP wasn't completely clear.

Now, do we consider this to be a programming question or not? In my experience, a Neural Network may not converge either because of a bug in the code (off-topic!) or because of issues in the data set, in the weight initialization, hyperparameters setup, wrong regularization approach, etc. (I would say all this is on-topic). However, it's hard in advance to say what is what. In this case, if I had to guess I would say "programming error" because it seems the OP is trying to reproduce a simple case from a book. Anyway, do we consider the question on-topic or not?

$\endgroup$
3
  • 13
    $\begingroup$ Often it is a conceptual error that caused code to fail or be incorrect. But if the question essentially amounts to "what's wrong with this code?", then it ought to be closed: we're not a code review site. We should at least apply the same standards as Stack Overflow: the code needs to be stripped to a simple, reproducible, executable example using the least amount of data needed to exhibit the problem; and we should expect--no, demand--that the question show what steps have been taken to establish the correctness of the code and debug the problem. $\endgroup$
    – whuber Mod
    Commented Oct 20, 2017 at 21:21
  • 2
    $\begingroup$ @whuber sounds extremely reasonable. Maybe we should add it in the Help section? I'm not sure the part about MREs (Minimal Reproducible Examples) is there. On the other hand, adding suggestions on how to include code might be seen as a recommendation to include code, which is not what we want really....I don't know what's the best course of action. $\endgroup$
    – DeltaIV
    Commented Oct 23, 2017 at 12:07
  • 1
    $\begingroup$ @whuber Perhaps one way to trick users into adhering to these best practices would be to require the users to show unit tests as well as their actual code; the lack of a unit test is demonstrable evidence that the poster has made no effort to check correctness, while the presence of the test gives the context of what the poster's assumptions are about how it should work (i.e. the user's conceptual understanding of the problem). $\endgroup$
    – Sycorax Mod
    Commented Nov 2, 2017 at 21:11

2 Answers 2

Reset to default
15
$\begingroup$

This is a subtle issue, and I think you have it right, but it will always be tricky in practice. My own guideline for what is on-topic is (conceptually) straightforward: What does the OP need explained? If what they don't know / misunderstand is something about how the code works (e.g., in that language), then this is ultimately a programming question and should be considered off-topic here (but it is potentially on topic on Stack Overflow). On the other hand, if what they don't understand is a topic in machine learning (statistics, etc.), then the question is on-topic here.

There are a few difficulties in practice, though. One is that it can be hard for someone to read a thread and recognize that the real issue is a conceptual misunderstanding. This will be true if your background is in, say, statistics and the question is about a machine learning topic with which you aren't very familiar (e.g., neural networks). It will be even more true if the question is only / primarily stated in code you don't read. I sometimes worry that we are liable to close threads that we should answer because the users in the close vote queue are unfamiliar with the topic / language. For example, we seem to have better coverage of statistics than machine learning among our active users on CV. Something you can do, if you come across such a case, is to leave a comment that this appears to be an on-topic machine learning issue, not actually about the code. You might also prompt the OP to explain their question more clearly and in a software-neutral way. Indeed, when I am faced with such a thread in the close vote queue, I look for a comment from a user who I know has expertise on the topic / language, and/or try to see if any have voted to close or leave open.

Another issue is what to do if a question is based on several misunderstandings that constitute a mixture of conceptual and programming issues. In this case, the thread should be considered on-topic due to the existence of issues we should address. It is perfectly fine to not address the programming issues (or to do so) as answerers choose. It would presumably be most helpful to the OP to point out that those topics should be asked on SO, though.

$\endgroup$
12
$\begingroup$

This is not just an issue with ML code, it's an issue with any code that doesn't do what the OP wanted that's posted to the site at all.

It could either be a problem of understanding what the algorithm is (not really knowing the correct thing to code to achieve the aim) or it could be a problem of converting a correct algorithm into code (implementation).

The first kind of problem would be on topic, but the second would usually be off topic. As whuber says, we're not a code review site.

Here's the kicker: It could also be both -- but even if it is only one or the other, disentangling the two kinds of issues in block of code is often very difficult.

We should as far as feasible focus on questions that are about the first thing -- not just because we're not a code review site but

  • that's where such questions have to begin; it's too hard to make progress on such problems if you're not clear what you're trying to do

  • because that's where the statistical issues mostly will be

* We can't completely eliminate questions with code, however, for a variety of reasons, nor do I think we should. This is more about "how should questions where there's an ambiguity of problem-category be structured?" -- a question that doesn't do a good job of separating the two kinds of issues is generally a bad question -- it's a problem of premature coding.

So to work on a questions "what should I be implementing?" we should not in general be looking at a chunk of code. We should usually be talking about a problem expressed in words and algebra (or pictures or whatever other way of transmitting concepts works for this problem). That focuses them on the part of the problem they they should focus on first and allows us to focus on the part of the problem we can talk about here. It helps to disentangle the mire of "what am I trying to do" from "did I do it right" -- such disentangling should generally be the province of the question, not of an answer.

Once that "what specific approach should I use to solve this" kind of question is clear a suitable thing to code up may result -- which might be expressed in various ways -- often in words and algebra but perhaps in pseudocode or possibly even in code in some instances. So answers might occasionally give code but code should usually be avoided in those kinds of questions, for their own sake. (At the very least, code should be right at the end, after a proper description of the problem, not the problem description itself! I think "is this code right? ", where you must infer the issue from the code, should usually just be closed.)

Once the OP is sure they have the algorithm/approach they need, the question "does this chunk of code correctly implement this clearly stated algorithm" would nearly always be off topic here, but set up properly could be a great question for another site.

$\endgroup$
4
  • 3
    $\begingroup$ I get your general point, but in my opinion ML is different: Neural Networks are much less "standard" than other methods, because "*we actually don't know why they work* (it's like we're in the steam engine era, before the discovery of thermodynamics). Even a "classic" architecture such as a CNN can have many variations, and each one is an opportunity for errors at the coding and/or at the conceptual level. Contrast it with Gaussian Processes, where 1) usually you just call a function and 2) at most you only change the covariance function. There's much less opportunity for... $\endgroup$
    – DeltaIV
    Commented Oct 21, 2017 at 7:24
  • 2
    $\begingroup$ ...errors at a coding or a conceptual error. As a matter of fact, on average I see far more lines of code in ML questions than in questions on GLM, or OLS, etc. It may also be just a bad habit: maybe ML (and especially DL) practitioners mostly come from the computer science world, and err on the side of too much coding. $\endgroup$
    – DeltaIV
    Commented Oct 21, 2017 at 7:31
  • 5
    $\begingroup$ To some extent you see similar issues (though perhaps at a different scale) with some of the more tricky parts of MCMC implemented on big models, where even if you screw up it might look plausible and even if you implement it correctly it might not work well. I don't think any of that changes a reasonable policy for our site though -- if you can't easily tell whether it's an implementation error or not, it's likely not in a suitable state for a question here. $\endgroup$
    – Glen_b Mod
    Commented Oct 21, 2017 at 8:24
  • 2
    $\begingroup$ 100% agree - the policy is great, I just wanted some clarifications, because I had the impression that more ML questions with a lot of code have recently been asked and have not been closed. $\endgroup$
    – DeltaIV
    Commented Oct 21, 2017 at 11:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .