Where can we ask open ended questions about research?

Question

I have been reading the paper Efficient Global Optimization of Expensive Black-Box Functions by Jones, et al. and they describe maximizing the expectation of a metric they created, called the improvement metric, for global optimization and seems like a really neat idea. I thought to myself it would be a fun idea to try and follow their analysis but to formulate it under my own new metric. However, I have been unable to come up, conceptually, with any metric that seems like an intuitive thing to try to maximize (or minimize) to help guide an optimization algorithm.

So, with that said, I don't have any idea what sort of metric I would pitch as being a good one to explore, and so I wanted to know if there are any places on stack exchange that would be a good (or acceptable) place to ask a question like this? I basically want to be able to ask the community as a whole to help me brainstorm ideas but I wasn't sure if this is allowable given that I don't have an idea for the metric formulated myself.

Answer 1

That paper is good but pretty old, many things happened since then.

In the literature of Bayesian Optimization what you are trying to do is to come up with a new acquisition function $\alpha(x)$, where $x$ is a point in the domain to be optimized (Jones et al. use Expected Improvement as their acquisition function) and $\alpha$ is some metric that takes into account exploration vs exploitation. People have been working on this problem for a long time.

You might want to check the literature review papers that I listed in my answer to this other question. In particular, you'll see that there are several common acquisition functions that people use in Bayesian Optimization:

1. Probability of Improvement (PI).
2. Expected improvement (EI), as per Jones et al.
3. Lower confidence bound (LCB), or UCB for maximization (Srinivas et al., 2010)
4. Expected information gain (ES, or Entropy Search); see Hennig and Schuler (2012) and Hernández-Lobato et al. (2014).

Note that 1 and 2 belong to a more general class of generalized expected-improvement measures (see Schonlau et al., 1998).

Metrics based on entropy are somewhat more principled but much harder to compute. EI is the "standard" choice nowadays, although it is not always the best choice. Sometimes it is good to use a combination of acquisition functions (see Hoffman et al., 2011).

---

References:

[1] Srinivas, N., Krause, A., Kakade, S. M., & Seeger, M. (2009). Gaussian process optimization in the bandit setting: No regret and experimental design. arXiv preprint arXiv:0912.3995.

[2] Hennig, P., & Schuler, C. J. (2012). Entropy search for information-efficient global optimization. The Journal of Machine Learning Research, 13(1), 1809-1837.

[3] Hernández-Lobato, J. M., Hoffman, M. W., & Ghahramani, Z. (2014). Predictive entropy search for efficient global optimization of black-box functions. In Advances in Neural Information Processing Systems (pp. 918-926).

[4] Schonlau, M., Welch, W. J., & Jones, D. R. (1998). Global versus local search in constrained optimization of computer models. Lecture Notes-Monograph Series, 11-25.

[5] Hoffman, M. D., Brochu, E., & de Freitas, N. (2011). Portfolio Allocation for Bayesian Optimization. In UAI (pp. 327-336).