Toolbox configuration

The toolbox can be configured by means of an XML file. Examples can be found in the 'config' directory. The default configuration file is 'config/default.xml'.

Structure

Plans and Runs

The general structure of the toolbox is as follows: The top-level <Plan> type defines a surrogate modeling experiment, and an experiment may consist of multiple <Run> tags. Each <Run> tag can be configured separately. For example, say you want to model some problem from electronics and you have at your disposal 3 algorithms for selecting data points. Now lets assume you want to compare the different algorithms on your problem and see which one gives you the best model with the least number of data samples. In this case your <Plan> tag would contain 3 <Run> tags and each <Run> tag would contain a different <SampleSelector> tag. For example:

<Plan>
   ...
   <Run name="gradient-run" repeat="1">
     <SampleSelector>gradient</SampleSelector>
   </Run>
  <Run name="grid-run" repeat="1">
     <SampleSelector>grid</SampleSelector>
   </Run>
   <Run name="random-run" repeat="1">
     <SampleSelector>random</SampleSelector>
   </Run>
   ...
</Plan>
...

Thus, this concept of a plan and multiple runs allows you to setup different configurations beforehand and try them one in one go. As you can see it is also possible to specify a repeat attribute. Setting it to 5, for example, will ensure that that particular run is repeated 5 times. This is usually a good idea if there is a lot of randomness in the algorithms (as is usually the case). Remember though to set the seedRandomState option in the M3 tag to random, for otherwise you will still get deterministic results:

  <Option key="seedRandomState" value="random"/>	

Declarations and Definitions

Each component in the toolbox has its own configuration section. Inside the <Run> tag you declare what components you would like to use. This declaration refers to the definition of each component, further down the file. So when you see line like:

<SampleSelector>gradient</SampleSelector>

This means we want to use the gradient sample selection algorithm, the word gradient is a unique identifier that refers to the <SampleSelector> tag that has gradient as its "id" attribute. In this case your configuration file would have the following structure:

<Plan>
   <Run>
     <!-- Here we declare what components to use, for example, the gradient sample selection algorithm -->
     <SampleSelector>gradient</SampleSelector>
     ...
   </Run>
   ...
<Plan>
...

<!-- The definition of the sample selector -->
<SampleSelector id="gradient">
   ...
   <!-- all the options for the gradient sample selection algorithm -->
</SampleSelector>
...

If you would like to use a different algorithm (e.g., grid for the Grid sample selector), you simply fill in a different id in the the <SampleSelector> tag in the <Run> tag:

<SampleSelector>grid</SampleSelector>

You just have to make sure there is a matching definition lower down the file for the id you have filled in. All the other components (AdaptiveModelBuilder, SampleEvaluator, ...) work in exactly the same way.

Configuration File

Also see the Running page for more information about running the toolbox.

Default configuration

The default toolbox configuration file can be found in the config directory under the filename default.xml. It contains all the possible options you can set.

We advise you to use this file as a basis for your own experiments.

Custom configuration

If you know what you are doing, you can merge your own custom configuration with the 'merge' option. Options or tags that are missing in this custom file will then be filled up with the values from the default configuration. Merging works as follows:

go('customConfig.xml',[],[],'-merge');

Components

We are well aware that the list below is incomplete and possibly even out of date in some cases. We try to document everything as best we can but much is limited by available time and manpower. The most up to date documentation can always be found (if not here) in the default.xml configuration file and, of course, in the source files.

The following components can be configured separately:

• Plan
• Outputs
  • Measures
• Inputs
• Simulator tag
• ContextConfig
• SUMO
  • Initial Designs
• Sample Evaluators
• Sample Selectors
• Adaptive Model Builders

General guidelines

Some general guidelines on how to configure the toolbox for different situations can be found on this page.