Whats new
This page gives a high level overview of the major changes in each toolbox version. For the detailed list of changes please refer to the Changelog page. For a list of features in the current version see the about page.
6.0 - To be released late summer 2008
Originally this was supposed to be 5.1 but after many fixes and added features we decided to to promote it to 6.0. Some of the things that can be expected for 6.0 are:
- A nice graphical user interface (GUI) for loading models, browsing through dimensions, plotting errors, generating movies, ... (A sneak peek is available here)
- Introduction of project directories. All files belonging to a particular problem (simulation code, datasets, XML files, documentation, ...) are now grouped together in a project directory instead of being spread out over 3 different places.
- Support for autosampling, one or more dimensions can be ignored during adaptive sampling. This is useful if the simulation code can generate samples for that dimension itself (e.g., frequency samples in the case of a frequency domain simulator in Electro-Magnetism)
- Models now remember axis lables, measure scores, and output names
- An export function has been added to export models to a standalone Matlab script (.m file). Not supported for all model types yet.
- Proper support for Matlab R2008
And of course countless bugfixes, performance, and feature enhancements
5.0 - Released April 2008
SUMO Toolbox
In April 2008, the first public release of the SUrrogate MOdeling (SUMO) Toolbox occurred.
The sample selection and evaluation backends have seen some major improvements.
The number of samples selected each iteration need no longer be chosen a priori but is determined on the fly based on the time needed for modeling, the average length of the past 'n' simulations and the number of compute nodes (or CPU cores) available. Of course, a user specified upper bound can still be specified. It is now also possible to evaluate data points in batches instead of always one-by-one. This is useful if, for example, there is a considerable overhead for submitting one point.
In addition, data points can be assigned priorities by the sample selection algorithm. These priorities are then reflected in the scheduling decisions made by the sample evaluator. It now also becomes possible to add different priority management policies. For example, one could require that 'interest' in sample points be renewed, else their priorities will degrade with time.
A new sample selection algorithm has been added that can use any function as a criterion of where to select new samples. This function is able to use all the information the surrogate provides to calculate how interesting a certain sample is. Internally, a numeric global optimizer is applied on the criterion to determine the next sample point(s). There are several criterions implemented, mostly for global optimization. For instance the 'expected improvement criterion' is very efficient for global optimization as it balances between optimization itself and refining the surrogate.
Finally the handling of failed or 'lost' data points has become much more robust. Pending points are automatically removed if their evaluation time exceeds a multiple of the average evaluation time. Failed points can also be re-submitted a number of times before being regarded as permanently failed.
The modeling code has seen some much needed cleanups. Adding new model types and improving the existing ones is now much more straightforward.
Since the default Matlab neural network model implementation is quite slow, two additional implementations were added based on FANN and NNSYSID which are much faster. In addition the NNSYSID implementation also supports pruning. However, though these two implementations are faster, the Matlab implementation still outperforms them accuracy wise.
An intelligent seeding strategy has been enabled. The starting point/population of each new model parameter optimization run is now chosen intelligently in order to achieve a more optimal search of the model parameter space. This leads to better models faster.
- The Optimization framework was removed due to several reasons.
- Added an optimizer class hierarchy for solving subproblems transparently.
- Added several criterions for optimization, available through the InfillSamplingCriterion.
Various changes
The default 'error function' is now the root relative square error (= a global relative error) instead of the absolute root mean square error.
The memory usage has been drastically reduced when performing many runs with multiple datasets (datasets are loaded only once).
The default settings have been harmonized and much improved. For example the SVM parameter space is now searched in log10 instead of loge. The MinMax measure is now also enabled by default if you do not specify any other measure. This means that if you specify minimum and maximum bounds in the simulator xml file, models which do not respect these bounds are penalized.
Finally this release has seen countless cleanups, bug fixes and feature enhancements.
