AutODEx: Automated Optimal Design of Experiments Platform with Data- and Time-Efficient Multi-Objective Optimization

1MIT CSAIL 2Independent Researcher
Neural Information Processing Systems (NeurIPS) 2023 RealML Workshop

Abstract

We introduce AutODEx, an automated machine learning platform for optimal design of experiments to expedite solution discovery with optimal objective trade-offs. We implement state-of-the-art multi-objective Bayesian optimization (MOBO) algorithms in a unified and flexible framework for adaptive experimental design, along with efficient asynchronous batch strategies extended to MOBO to harness experiment parallelization. For users with little or no experience with coding or machine learning, we provide an intuitive graphical user interface (GUI) to help quickly visualize and guide the experiment design. For experienced researchers, our modular code structure serves as a testbed to quickly customize, develop, and evaluate their own MOBO algorithms. Extensive benchmark experiments against other MOBO packages demonstrate AutODEx's competitive and stable performance. Furthermore, we showcase AutODEx's real-world utility by autonomously guiding hardware experiments with minimal human involvement.


Video Demonstration

Platform Features

Automated Workflow

The GUI serves as the user's main interaction point. Users initially configure optimization settings through the GUI, initiate optimization runs, and set stopping criteria. Automation is employed with an evaluation program (Python/C/C++/Matlab) to repeatedly optimize and evaluate designs. Manual evaluations are also supported; users can view proposed designs in the GUI, input results, and receive updates on progress.

Modular Algorithm Framework

MOBO consists of four core modules in our framework:

  1. an inexpensive surrogate model for the black-box objective functions;
  2. an acquisition function that trades-off exploration and exploitation of the design space;
  3. a cheap multi-objective solver to approximate the Pareto set and front;
  4. a selection strategy that proposes experiment candidates to evaluate next.
These modules are implemented as decoupled building blocks, making it highly modularized and easy to develop new algorithms.

Based on this framework, we implement several popular and state-of-the-art MOBO methods. With necessary modules of the MOBO framework implemented, the algorithms can be easily composed by specifying the choice of each module and inheriting the base class MOBO.

Intuitive Graphical User Interface

The key strength of AutODEx is the user-friendly GUI that significantly lowers the barrier of applying MOBO to real-world problems. Users can easily configure problem settings (e.g., design and performance space properties), run optimization and inspect progress.

The iterface for configuring problem settings.

The partial interface for running optimization and inspecting progress.

The GUI guides the user through a set of simple steps to configure the problem, such as the number of design and performance parameters, the parameter bounds and constraints, parallelization settings, and selection of the optimization algorithm without the need of coding. The GUI also includes a real-time display of the design and performance space which allows users to easily understand the current status of optimization. We also support displaying and exporting the whole optimization history (including database and statistics).

Efficient Asynchronous Optimization

While standard MOBO optimizes for the Pareto front in a data-efficient manner, often, when multiple experiment setups are available, evaluations can be executed in batch by parallel workers to further speed up the whole optimization process. To this end, we extend several asynchronous strategies to the multi-objective scenario including our novel and effective method. All the algorithms in AutODEx are implemented to support batch synchronous/asynchronous evaluation.

Experiments

Comparison Across Platforms

We compare AutODEx against other BO libraries with multi-objective optimization capabilities across 12 standard multi-objective benchmark problems. We measure as the performance the log of the difference between the hypervolume of the ground-truth Pareto front and hypervolume of the best Pareto front approximation found by the algorithms (lower is better).

The performance of AutODEx is generally stable and ends up either the best or comparable on most benchmark problems. Especially, AutODEx takes a major lead in several challenging problems such as ZDT1, ZDT3, and VLMOP3, which shows that our platform handles high-dimensional MOBO problems well.

BoTorch, OpenBox and AutODEx outperform other libraries by a great margin overall. While BoTorch and OpenBox share a better median rank, AutODEx appears to be the stablest that consistently ranks between 1 and 3 on all problems and has a higher lower-bound.

AutODEx has a competitive performance to BoTorch and OpenBox, and outperforms other baselines on a wider range of benchmarks.

Comparison on Asynchronous Strategies

We implement the extension of Kriging Believer (KB) and Local Penalization (LP) strategies to MOBO algorithms in AutODEx for asynchronous batch optimization. To address the failure cases of both approaches, we propose a novel strategy Believer-Penalizer (BP) that naturally combines KB and LP by applying KB to designs with certain predictions and LP to designs with uncertain predictions.

To test whether Believer-Penalizer is effective, we compare four asynchronous MOBO algorithms on all benchmark problems. Our Async BP strategy consistently outperforms other methods and follows the best of Async KB and Async LP.

Real-World Optimal Design of Experiments

We further demonstrate the real-world applicability of AutODEx through applying to optimize a PID heater controller to have optimal response time and minimal overshoot in a fully automated way. To automate the experiment, we simply link the Python evaluation program of this experiment setup to AutODEx through GUI then start the iterative optimization. Finally, a set of solutions are discovered with optimal trade-offs. Using MOBO algorithms provided by AutODEx is able to discover better designs compared to random exploring at the presence of evaluation noise. This example, with all the components that people can easily buy off-the-shelf, serves as a simple proof of concept that AutODEx is applicable to optimize real physical systems.