Benchmarks and performance metrics for lab autonomy

[npaulson]

What are the best ways to quantify the performance of an autonomous laboratory? How do we formulate benchmark problems to help compare different autonomous laboratories? How can we leverage these benchmarks and metrics to justify the benefits of laboratory autonomy?

[sgbaird]

Xref:

Benchmarking acceleration in materials discovery with AI

I’ve been discussing with a lot of people recently about the best approaches to benchmarking “acceleration” in materials discovery with AI. There are few examples of this done in literature. Baselines are often either easy to establish but not highly relevant, such as a human with random sampling—or more relevant but requires intensive effort, such as comparing a human with design-of-experiments vs. human with AI. We have some ideas internally at the Acceleration Consortium about how assess th…