Difference between revisions of "Multi-Objective Modeling"

From SUMOwiki
Jump to navigationJump to search
Line 7: Line 7:
  
 
Often it makes sense to use multiple measures.  For example you may be interested in minimizing the average relative error AND the maximum absolute error.  Alternatively you may have a problem with multiple outputs (see [[Running#Models_with_multiple_outputs]]) and it may make sense to model them in a model them together in a multi-objective way.  This section is about those topics.
 
Often it makes sense to use multiple measures.  For example you may be interested in minimizing the average relative error AND the maximum absolute error.  Alternatively you may have a problem with multiple outputs (see [[Running#Models_with_multiple_outputs]]) and it may make sense to model them in a model them together in a multi-objective way.  This section is about those topics.
 +
 +
'''Note that we are talking about optimization of the model parameters (hyperparameter optimization), NOT optimization of the simulator!!'''
  
 
== Using Multiple Measures ==
 
== Using Multiple Measures ==
Line 18: Line 20:
  
 
What the toolbox then does with this depends on some other settings.
 
What the toolbox then does with this depends on some other settings.
 +
 +
=== Selecting the best model ===
 +
 +
The toolbox keeps track of the k best models found so far.  Each time a model is found that is better than the previous best model it will be processed (plotted, profiled, ...) and saved to disk.  If only one measure is active the best model is simply the model with lowest measure score.
 +
However, when multiple measures are used, an intelligent pareto-based method is used to decide which model is the best choice. Models that score high on a particular measure but low on another are not discarded immediately, but are given a chance to set things right in future iterations of the toolbox. This encourages variety in the models, while still ensuring convergence to the optimal accuracy for each measure. An often used combination is CrossValidation with the MinMax measure, to ensure that no poles are present in the model domain when using rational models.
  
 
=== Weighted Single Objective ===
 
=== Weighted Single Objective ===
Line 28: Line 35:
 
</nowiki></pre>
 
</nowiki></pre>
  
Now the toolbox will minimize:
+
Now the toolbox will generate models (by optimizing the model parameters) that minimize:
  
 
<pre><nowiki>
 
<pre><nowiki>
Line 38: Line 45:
 
=== Multi-Objective ===
 
=== Multi-Objective ===
  
== Multi-output modeling ==
+
Sometimes a weighting scheme is not enough and you want to do true multi-objective optimization (e.g., to see the trade-off between the measures).  In this case there is no longer a single best model but a set of models.  This set of models will be saved every k iterations of the multi-objective optimization routine.
 +
 
 +
To enable multi-objective hyperparameter optimization there are two adaptive model builders you can use:
 +
 
 +
# GeneticModelBuilder (e.g., anngenetic) : this uses the multi-objective version of the GA implemented in the [http://www.mathworks.com/products/gads/ Matlab GADS Toolbox]. In this case you must the option ''paretoMode="true"''
 +
# ParetoModelBuilder (e.g., krigingnsga) : this uses the standard NSGA-II algorithm (but other algorithms could be easily added).
 +
 
 +
Note that when using the GeneticModelBuilder you should increase the population size and number of generations in order to get a good pareto front.
  
--
+
If you want to plot the search trace of a multi-objective run you can use ''plotModelParetoFront'' function in the tools directory. To extract the k-th Pareto front you can use the ''nonDominatedSort'' function.
the best alternative is ValidationSet, which by default behaves as cross validation in which only one fold is considered, reducing the cost of the measure by a factor 5. A second measure, called MinMax, is also activated by default, enabling the user to force the model to remain within certain bounds, to speed up the convergence. See below for more details in how to set these bounds.
 
  
However, in certain situations it might be very effective to use different measures, or use multiple measures together. When multiple measures are used, an intelligent pareto-based method is used to decide which model is the best choice. Models that score high on a particular measure but low on another are not discarded immediately, but are given a chance to set things right in further iterations of the toolbox. This encourages variety in the models, while still ensuring convergence to the optimal accuracy for each measure. An often used combination is CrossValidation with the MinMax measure, to ensure that no poles are present in the model domain.
+
== Multi-output modeling ==

Revision as of 00:35, 8 February 2009

THIS PAGE IS UNDER CONSTRUCTION


Motivation

Please first read the technical report available here and the page about Measures.

Often it makes sense to use multiple measures. For example you may be interested in minimizing the average relative error AND the maximum absolute error. Alternatively you may have a problem with multiple outputs (see Running#Models_with_multiple_outputs) and it may make sense to model them in a model them together in a multi-objective way. This section is about those topics.

Note that we are talking about optimization of the model parameters (hyperparameter optimization), NOT optimization of the simulator!!

Using Multiple Measures

To enable multiple measures simply specify multiple <Measure> tags in your configuration file and make sure the use attribute is set to on. For example:

<Measure type="ValidationSet" errorFcn="rootMeanSquareError" target=".001" use="on"/>
<Measure type="LRMMeasure" target="0" use="on"/>

What the toolbox then does with this depends on some other settings.

Selecting the best model

The toolbox keeps track of the k best models found so far. Each time a model is found that is better than the previous best model it will be processed (plotted, profiled, ...) and saved to disk. If only one measure is active the best model is simply the model with lowest measure score. However, when multiple measures are used, an intelligent pareto-based method is used to decide which model is the best choice. Models that score high on a particular measure but low on another are not discarded immediately, but are given a chance to set things right in future iterations of the toolbox. This encourages variety in the models, while still ensuring convergence to the optimal accuracy for each measure. An often used combination is CrossValidation with the MinMax measure, to ensure that no poles are present in the model domain when using rational models.

Weighted Single Objective

If you specify nothing else the toolbox will simply minimize the sum of both (= scalarization) and everything continues as normal. However, if the scale of the measures differs greatly this might not be very fair. Or, it might be that you consider one more important than the other. In that case you can add weights as follows:

<Measure weight="0.6" type="ValidationSet" errorFcn="rootMeanSquareError" target=".001" use="on"/>
<Measure weight="0.4" type="LRMMeasure" target="0" use="on"/>

Now the toolbox will generate models (by optimizing the model parameters) that minimize:

0.6*(validation score) + 0.4*(LRM score)

So this gives you more fine grained control of the importance of each measure. Note that it is up to you to ensure the weights are normalized. If no weight is specified it defaults to 1.

Multi-Objective

Sometimes a weighting scheme is not enough and you want to do true multi-objective optimization (e.g., to see the trade-off between the measures). In this case there is no longer a single best model but a set of models. This set of models will be saved every k iterations of the multi-objective optimization routine.

To enable multi-objective hyperparameter optimization there are two adaptive model builders you can use:

  1. GeneticModelBuilder (e.g., anngenetic) : this uses the multi-objective version of the GA implemented in the Matlab GADS Toolbox. In this case you must the option paretoMode="true"
  2. ParetoModelBuilder (e.g., krigingnsga) : this uses the standard NSGA-II algorithm (but other algorithms could be easily added).

Note that when using the GeneticModelBuilder you should increase the population size and number of generations in order to get a good pareto front.

If you want to plot the search trace of a multi-objective run you can use plotModelParetoFront function in the tools directory. To extract the k-th Pareto front you can use the nonDominatedSort function.

Multi-output modeling