I have been reading the paper Efficient Global Optimization of Expensive Black-Box Functions by Jones, et al. and they describe maximizing the expectation of a metric they created, called the improvement metric, for global optimization and seems like a really neat idea. I thought to myself it would be a fun idea to try and follow their analysis but to formulate it under my own new metric. However, I have been unable to come up, conceptually, with any metric that seems like an intuitive thing to try to maximize (or minimize) to help guide an optimization algorithm.

So, with that said, I don't have any idea what sort of metric I would pitch as being a good one to explore, and so I wanted to know if there are any places on stack exchange that would be a good (or acceptable) place to ask a question like this? I basically want to be able to ask the community as a whole to help me brainstorm ideas but I wasn't sure if this is allowable given that I don't have an idea for the metric formulated myself.

• Chat is intended for that. BTW, this is the first time I have seen our community referred to as a "hole" :-).
– whuber Mod
Feb 4, 2016 at 20:01
• @whuber is it better to post this question in ten-fold, create my own room for it, or...? Feb 4, 2016 at 20:27
• Nobody would visit a private room! Try starting a conversation in tenfold. Be prepared to wait a few days for responses. You might also get some friendly advice about how to generate focused, answerable questions while you are reading a paper like this.
– whuber Mod
Feb 4, 2016 at 20:29
• If anyone gives you substantive help with a research problem that leads to publication, they should normally be a co-author (e.g. made actual contribution to the way you did things for example, in a way that things would have been clearly different had they not helped). If they give you more minor help (perhaps suggesting a more appropriate analysis) they should usually at least get an acknowledgement. [At best it's very bad manners when effusive thanks and praise for lots of free help don't even correspond to a half dozen words of acknowledgement in a footnote.] Feb 5, 2016 at 1:08
• @Glen_b I just realized you have an open bounty that is highly related to my question here! Want to be my co-author? ;) Feb 5, 2016 at 20:47
• Having an open bounty wouldn't be a reason to make me a coauthor; but in any case the rules for using material that's on site here are already spelled out Feb 6, 2016 at 1:34
• @Glen_b I was merely kidding around, as indicated by the good only winky ;) Feb 6, 2016 at 1:43
• I considered that you might be, but figured it was better to assume you weren't, just in case. I'll probably delete the last three comments (counting this one) soon Feb 6, 2016 at 1:45
• why is the title of the question really general but the discussion not? I think what matters is not this particular example that might have an easy answer, but to know who questions that are boundary research should be handled. Feb 13, 2016 at 20:14
• I have a quite cool and novel metric that I use for outlier detection. We would have to talk be email. Feb 13, 2016 at 22:12
• @EngrStudent sure that would be fine by me. However, I am curious to know why you would think outlier detection would solve this problem. Feb 14, 2016 at 5:13
• @RustyStatistician - It is a metric, and while it was derived using log-likelihood ratio's and symbolic genetic algorithms built around outlier detection, it is really a data transformation that optimizations are able to be built around. It is a continuous transform so it gives "distance" from in, and not just a binary in vs. out. Feb 14, 2016 at 12:52
• @EngrStudent sure let's talk. Feb 14, 2016 at 14:49

That paper is good but pretty old, many things happened since then.

In the literature of Bayesian Optimization what you are trying to do is to come up with a new acquisition function $\alpha(x)$, where $x$ is a point in the domain to be optimized (Jones et al. use Expected Improvement as their acquisition function) and $\alpha$ is some metric that takes into account exploration vs exploitation. People have been working on this problem for a long time.

You might want to check the literature review papers that I listed in my answer to this other question. In particular, you'll see that there are several common acquisition functions that people use in Bayesian Optimization:

1. Probability of Improvement (PI).
2. Expected improvement (EI), as per Jones et al.
3. Lower confidence bound (LCB), or UCB for maximization (Srinivas et al., 2010)
4. Expected information gain (ES, or Entropy Search); see Hennig and Schuler (2012) and Hernández-Lobato et al. (2014).

Note that 1 and 2 belong to a more general class of generalized expected-improvement measures (see Schonlau et al., 1998).

Metrics based on entropy are somewhat more principled but much harder to compute. EI is the "standard" choice nowadays, although it is not always the best choice. Sometimes it is good to use a combination of acquisition functions (see Hoffman et al., 2011).

References:

[1] Srinivas, N., Krause, A., Kakade, S. M., & Seeger, M. (2009). Gaussian process optimization in the bandit setting: No regret and experimental design. arXiv preprint arXiv:0912.3995.

[2] Hennig, P., & Schuler, C. J. (2012). Entropy search for information-efficient global optimization. The Journal of Machine Learning Research, 13(1), 1809-1837.

[3] Hernández-Lobato, J. M., Hoffman, M. W., & Ghahramani, Z. (2014). Predictive entropy search for efficient global optimization of black-box functions. In Advances in Neural Information Processing Systems (pp. 918-926).

[4] Schonlau, M., Welch, W. J., & Jones, D. R. (1998). Global versus local search in constrained optimization of computer models. Lecture Notes-Monograph Series, 11-25.

[5] Hoffman, M. D., Brochu, E., & de Freitas, N. (2011). Portfolio Allocation for Bayesian Optimization. In UAI (pp. 327-336).

• Thanks for all the references. I have read and aware of all of those and indeed I am trying to come up with a new acquisition function. But really, I guess that is the hardest part... Feb 15, 2016 at 5:51
• This no doubt is helpful on the original specific question, but it doesn't address the Meta question here. Feb 15, 2016 at 9:47
• (+1) Good point @Nick, thanks -- I am a new user to Meta. Perhaps -- and please take this as a Meta comment to improve Meta -- you should apply the same standard to all users. I wrote and spent time on my answer because I saw another answer from a very experienced user which addresses the original question; this seemed to be fine and nobody pointed out its off-topic-ness. This is confusing for new users. (It might be formally incorrect, but looking at what more experienced members do, e.g. what look like acceptable answers, is a common way human beings learn the rules of a community.) Feb 15, 2016 at 16:00
• Sorry, but I don't follow what you are suggesting here. In practice, despite being a frequent and keen user of this forum, I don't have time to look at more than a very few threads. So, that is unfortunately a more than sufficient explanation if you are suggesting that I am being inconsistent. It's not too late to add the suggestion that your reply might belong to the other thread, except that I have not studied it carefully. Feb 15, 2016 at 16:16
• My comment was addressed to you but more in general to other users of Meta I suppose (that's why I meant it as "a Meta comment to improve Meta"). I appreciate that you personally cannot read all answers -- from the upvotes to your comment I reckoned that other people were following this. However, the other answer is in this very thread, so I (erroneously) assumed you (or other users) had read it, but this was a wrong assumption on my part. Thanks again for your clarification. Feb 15, 2016 at 16:33
• You're right. Indeed I am now also puzzled by the answer from the redoubtable @gung, which addresses the substantive question, not the Meta question, as far as I can see. Comment added above for symmetry. Feb 15, 2016 at 18:14
• @NickCox (lacerbi), basically it was a comment that was too long to fit as a comment, so I posted it as an answer. This is an admittedly boundary-pushing practice that people sometimes engage in, but probably shouldn't. I will delete my answer. I agree that this post isn't really a meta answer either, but I am indifferent to whether or not it stays. Ie, my deletion is not meant to suggest that this post should be deleted as well. Feb 15, 2016 at 18:22
• I would suggest this answer stay as I think, although it does not answer my original question of where can we post things, it is a valuable answer in its own right to the secondary topic at hand of optimization of acquisition functions. Feb 16, 2016 at 3:55
• Thanks all for your answers. I agree with you all that my answer is off-topic here (although it might be of interest for the non-Meta question). I'd be happy if it was moved somewhere else more appropriate. (Perhaps if at some point @RustyStatistician opens a related thread in CrossValidated it could be moved there.) Feb 17, 2016 at 22:32