Multi-Objective Modeling
THIS PAGE IS UNDER CONSTRUCTION
Motivation
Please first read the technical report available here and the page about Measures.
Often it makes sense to use multiple measures. For example you may be interested in minimizing the average relative error AND the maximum absolute error. Alternatively you may have a problem with multiple outputs (see Running#Models_with_multiple_outputs) and it may make sense to model them in a model them together in a multi-objective way. This section is about those topics.
Note that we are talking about optimization of the model parameters (hyperparameter optimization), NOT optimization of the simulator!!
Using Multiple Measures
To enable multiple measures simply specify multiple <Measure> tags in your configuration file and make sure the use attribute is set to on. For example:
<Measure type="ValidationSet" errorFcn="rootMeanSquareError" target=".001" use="on"/> <Measure type="LRMMeasure" target="0" use="on"/>
What the toolbox then does with this depends on some other settings.
Selecting the best model
The toolbox keeps track of the k best models found so far. Each time a model is found that is better than the previous best model it will be processed (plotted, profiled, ...) and saved to disk. If only one measure is active the best model is simply the model with lowest measure score. However, when multiple measures are used, an intelligent pareto-based method is used to decide which model is the best choice. Models that score high on a particular measure but low on another are not discarded immediately, but are given a chance to set things right in future iterations of the toolbox. This encourages variety in the models, while still ensuring convergence to the optimal accuracy for each measure. An often used combination is CrossValidation with the MinMax measure, to ensure that no poles are present in the model domain when using rational models.
Weighted Single Objective
If you specify nothing else the toolbox will simply minimize the sum of both (= scalarization) and everything continues as normal. However, if the scale of the measures differs greatly this might not be very fair. Or, it might be that you consider one more important than the other. In that case you can add weights as follows:
<Measure weight="0.6" type="ValidationSet" errorFcn="rootMeanSquareError" target=".001" use="on"/> <Measure weight="0.4" type="LRMMeasure" target="0" use="on"/>
Now the toolbox will generate models (by optimizing the model parameters) that minimize:
0.6*(validation score) + 0.4*(LRM score)
So this gives you more fine grained control of the importance of each measure. Note that it is up to you to ensure the weights are normalized. If no weight is specified it defaults to 1.
Multi-Objective
Sometimes a weighting scheme is not enough and you want to do true multi-objective optimization (e.g., to see the trade-off between the measures). In this case there is no longer a single best model but a set of models. This set of models will be saved every k iterations of the multi-objective optimization routine.
To enable multi-objective hyperparameter optimization there are two adaptive model builders you can use:
- GeneticModelBuilder (e.g., anngenetic) : this uses the multi-objective version of the GA implemented in the Matlab GADS Toolbox. In this case you must the option paretoMode="true"
- ParetoModelBuilder (e.g., krigingnsga) : this uses the standard NSGA-II algorithm (but other algorithms could be easily added).
Note that when using the GeneticModelBuilder you should increase the population size and number of generations in order to get a good pareto front.
If you want to plot the search trace of a multi-objective run you can use plotModelParetoFront function in the tools directory. To extract the k-th Pareto front you can use the nonDominatedSort function.
