question posed by @BRSutherland that I thought warranted surfacing here
Is there some level of dimensionality where BO starts to fail? If data regime stays small and dimensionality continues to increase, at some point there are no optimization algorithms that will work efficiently and you should redesign your campaign.
My response:
As you pointed out, curse of dimensionality is a general problem, not something specific to BO. My perhaps dissatisfying answer is that I recommend choosing a model that tends to not overfit in high dimensions. As a recent example, there is a project optimizing a chemical reaction. There are four ingredients, one processing parameter (time), and one output. I jumped in recently to support, and they’ve collected ~40 datapoints so far. For this specific dataset, if I choose a model that tends not to overfit, the next batch of suggestions does a better job at exploring the boundaries of the space which were largely unexplored in the original 40 datapoints. An alternative would be to simply run quasi-random search to get the same behavior, but the former is likely to be more sample-efficient.
The rule of thumb is up to a couple dozen parameters, but there are cases where for problems with certain behavior (especially when only a subset of parameters truly matter), it’s been shown to do OK with a hundred or hundreds of dimensions. For example https://arxiv.org/pdf/2103.00349.
Pulling a quote from someone at fb 4 years ago:
128 hyperparameters is a lot – way more than our standard GPEI algorithm can handle comfortably
Within aluminum alloys, there are ~23 unique periodic elements represented in common mixtures. Would it make sense to expand the search space to all non-radioactive materials? Probably not. Could a high-dimensional algorithm quickly determine that those other elements aren’t relevant? Maybe. Would that be a moot point because many of those configurations wouldn’t be synthesizable or measurable? (i.e., unable to get to point of running a tensile test). Probably.
However, I would lean away from restricting it to be lower than those 23 periodic elements, and instead focusing on finding suitable lower and upper bounds for them and honing in on things like processing conditions and purity. Likewise, I would spend the extra computation on “fully Bayesian” models and more computationally expensive models like SAASBO (I would rather wait a couple hours for an algorithm to suggest a higher quality experiment prior to running something in the lab – experiments carry a lot of hidden costs compared with running an algorithm).