SED:SED toolbox

Introduction

The SED Toolbox (Sequential Experimental Design) is a powerful Matlab toolbox for the sequential Design of Experiments (DoE). In traditional experimental design, all the design points are selected up front, before performing any (computer or real-life) experiment, and afterwards, no additional design points are selected. This traditional approach is prone to oversampling and/or undersampling, because it is often very difficult to estimate up front the required number of design points.

The SED Toolbox solves this problem by providing the user with state-of-the-art algorithms that generate an experimental design in a sequential way, i.e. one design point at a time, without having to provide the total number of design points in advance. This is called sequential experimental design (SED). The SED Toolbox was designed to be extremely fast and easy to use, yet very powerful.

Central to the experimental design problem is the trade-off between the intersite (maximin) and projected (non-collapsing) requirements. The intersite distance is the smallest distance between two design points in the design space; this value should be as high as possible, in order to have the points spread out as evenly as possible. In addition to the intersite distance, the projected distance is also important. The projected distance is the smallest distance between all the points after they have been projected on one axes of the design space. This measure is especially important if the relative importance of the design parameters is unknown. E.g., if one of the design parameters does not influence the output behavior, two design points which only differ in this (irrelevant) parameter have the same behavior, and can be seen as the "same" design point. Thus, the projected distance must also be maximized.

All the algorithms in the SED Toolbox were optimized to produce designs that score well on both the intersite and projected distance.

Download

See: download page

Quick start guide

IMPORTANT: Before the toolbox can be used, you have to set it up for use, by browsing to the directory in which the toolbox was unpacked and running the startup command:

startup

Now the toolbox is ready to be used. The SED Toolbox can be used in several ways, based on how much freedom you want in configuring and fine-tuning the parameters of the algorithms. We will now describe the three ways the toolbox can be used, in order of complexity, based on your requirements. If you prefer to learn by example, you can check out the examples directory in the distribution, which contains several applications and example problems for the toolbox.

You want an ND design of X points

In order to quickly generate a good ND design in X points, you can use the following code:

startup % configure the toolbox
config.inputs.nInputs = N; % set the number of inputs in the config struct
generator = SequentialDesign(config); % set up the sequential design
generator = generator.generateTotalPoints(X); % generate a total of X points
points = generator.getAllPoints(); % return the entire design

% optional:
generator.plot(); % plot the design
generator.getMetrics(); % get some metrics about the quality of the design

You want to use the more advanced features of the SED Toolbox

If you want to use some of the more advanced features of the SED Toolbox, such as input ranges and weights and constraints, you have two options. The first one is to use Matlab structs as in the previous example. The second one is to use simple XML files to configure the toolbox. Note that constraints will only work with XML configuration. You can open the 'problem.xml' file in the SED directory to get an idea of how a problem configuration looks like. You can edit this file to suit your needs and use it to configure the toolbox using the following command:

% generate a sequential design for the problem defined in problem.xml:
generator = SequentialDesign('problem.xml');

% generate a sequential design using the specified method for the problem defined in problem.xml:
generator = SequentialDesign('problem.xml', 'methods/mc-intersite-projected-threshold.xml');

If you instead prefer to use Matlab structs, you can use the following code to configure the toolbox:

config.inputs.nInputs = 2; % this is a 2D example
config.inputs.minima = [-1 -1]; % define the minimum of each input
config.inputs.maxima = [3 1]; % define the maximum of each input
config.inputs.weights = [2 0]; % the first input is twice as important as the second one
generator = SequentialDesign(config); % set up the sequential design

You want full control over all the method parameters

If you want full control over all the parameters of both the problem specification and the sequential design method, XML files are the only option. By editing the method XML files, you can tweak each method to your own preferences. Even though the options are documented, it might be difficult to understand their effect on the sampling process. Note that the default settings have been chosen based on extensive studies and comparisons, and are in most cases the best choice. If you have any questions or suggestions, please contact the authors at Karel dot Crombecq at ua.ac.be.

In addition to the methods provided by the XML files packaged with the SED Toolbox, SED also contains a huge library of components (such as candidate generators, optimizers, metrics) from which the user can compose his own sequential design methods. This feature is undocumented and unsupported, but users are free to experiment with them.

SED toolbox interface

A reference of all the functions available in the SED Toolbox can be found on this page.

Rules of thumb for selecting the right sequential design method

The default sequential design method for the SED Toolbox is mc-intersite-projected-threshold.xml. This is an intelligent Monte Carlo method which generates Monte Carlo points only in parts of the design space where the projected distance is above a certain threshold. From the remaining points, the best point in terms of intersite distance is picked as the next design point.

This method is very fast and can be applied to highly dimensional problems and for large designs. It also works well with constraints and input weights. However, there are some cases in which one of the other methods might be a better choice. Below you can find a table with rules of thumb for picking the right method for the right job.

Constraints

the default method mc-intersite-projected-threshold can run into problems when you are using very strict constraints. Because the Monte Carlo points are filtered by the projected distance threshold, it might be possible that no candidates remain that satisfy the constraints. In that case, mc-intersite-projected can be a good alternative. It produces slightly worse designs but is much more robust in terms of constraints. Additionally, mc-intersite-projected-threshold and all other methods besides mc-intersite-projected need the corner points [-1,...,-1] and [1,...,1] to start, and if they violate the constraints they will still be selected. You can later request the design without these corner points using the getAllPointsWithoutInitialDesign() function, so this might not be an issue, but keep it in mind.

Quality vs speed

The slowest method available in SED is optimizer-intersite, but this method also generates the best designs (slightly better than mc-intersite-projected-threshold). If you have the time, consider using this method instead. It also supports constraints, but might also run into problems with very tight constraints.

If time is of no concern, you can also consider increasing some of the method parameters to further improve the design. For mc-intersite-projected-threshold, the candidatesPerSample option can be increased to improve the quality at the cost of speed. For optimizer-intersite, both the nPop and maxIterations options can be increased.

Dimensionality

The Monte Carlo methods scale very well with the number of dimensions and points and should work for high-dimensional problems. However, the optimizer methods suffer more from the curse of dimensionality. optimizer-intersite should work up to 10D, but will run into memory problems for higher dimensions.

Contribute

Suggestions on how to improve the SED toolbox are always welcome. For more information please see the feedback page.