We introduce AutODEx, an automated machine learning platform for optimal design of experiments to expedite solution discovery with optimal objective trade-offs. We implement state-of-the-art multi-objective Bayesian optimization (MOBO) algorithms in a unified and flexible framework for adaptive experimental design, along with efficient asynchronous batch strategies extended to MOBO to harness experiment parallelization. For users with little or no experience with coding or machine learning, we provide an intuitive graphical user interface (GUI) to help quickly visualize and guide the experiment design. For experienced researchers, our modular code structure serves as a testbed to quickly customize, develop, and evaluate their own MOBO algorithms. Extensive benchmark experiments against other MOBO packages demonstrate AutODEx's competitive and stable performance. Furthermore, we showcase AutODEx's real-world utility by autonomously guiding hardware experiments with minimal human involvement.
The GUI serves as the user's main interaction point. Users initially configure optimization settings through the GUI, initiate optimization runs, and set stopping criteria. Automation is employed with an evaluation program (Python/C/C++/Matlab) to repeatedly optimize and evaluate designs. Manual evaluations are also supported; users can view proposed designs in the GUI, input results, and receive updates on progress.
MOBO consists of four core modules in our framework:
Based on this framework, we implement several popular and state-of-the-art MOBO methods. With necessary modules of the MOBO framework implemented, the algorithms can be easily composed by specifying the choice of each module and inheriting the base class MOBO.
The key strength of AutODEx is the user-friendly GUI that significantly lowers the barrier of applying MOBO to real-world problems. Users can easily configure problem settings (e.g., design and performance space properties), run optimization and inspect progress.
The GUI guides the user through a set of simple steps to configure the problem, such as the number of design and performance parameters, the parameter bounds and constraints, parallelization settings, and selection of the optimization algorithm without the need of coding. The GUI also includes a real-time display of the design and performance space which allows users to easily understand the current status of optimization. We also support displaying and exporting the whole optimization history (including database and statistics).
While standard MOBO optimizes for the Pareto front in a data-efficient manner, often, when multiple experiment setups are available, evaluations can be executed in batch by parallel workers to further speed up the whole optimization process. To this end, we extend several asynchronous strategies to the multi-objective scenario including our novel and effective method. All the algorithms in AutODEx are implemented to support batch synchronous/asynchronous evaluation.
We compare AutODEx against other BO libraries with multi-objective optimization capabilities across 12 standard multi-objective benchmark problems. We measure as the performance the log of the difference between the hypervolume of the ground-truth Pareto front and hypervolume of the best Pareto front approximation found by the algorithms (lower is better).
The performance of AutODEx is generally stable and ends up either the best or comparable on most benchmark problems. Especially, AutODEx takes a major lead in several challenging problems such as ZDT1, ZDT3, and VLMOP3, which shows that our platform handles high-dimensional MOBO problems well.
We implement the extension of Kriging Believer (KB) and Local Penalization (LP) strategies to MOBO algorithms in AutODEx for asynchronous batch optimization. To address the failure cases of both approaches, we propose a novel strategy Believer-Penalizer (BP) that naturally combines KB and LP by applying KB to designs with certain predictions and LP to designs with uncertain predictions.
To test whether Believer-Penalizer is effective, we compare four asynchronous MOBO algorithms on all benchmark problems. Our Async BP strategy consistently outperforms other methods and follows the best of Async KB and Async LP.
We further demonstrate the real-world applicability of AutODEx through applying to optimize a PID heater controller to have optimal response time and minimal overshoot in a fully automated way. To automate the experiment, we simply link the Python evaluation program of this experiment setup to AutODEx through GUI then start the iterative optimization. Finally, a set of solutions are discovered with optimal trade-offs. Using MOBO algorithms provided by AutODEx is able to discover better designs compared to random exploring at the presence of evaluation noise. This example, with all the components that people can easily buy off-the-shelf, serves as a simple proof of concept that AutODEx is applicable to optimize real physical systems.