Difference between revisions of "Measures"

From SUMOwiki
Jump to navigationJump to search
 
(41 intermediate revisions by 5 users not shown)
Line 1: Line 1:
A measure is used to measure the quality of a new model. The measure decides wether the modelling process can be halted (the target accuracy has been reached) or wether new samples should be selected / new models should be built. The choice of measure is therefore very important for the success of the toolbox.
+
== What is a Measure ==
 +
A crucial aspect of generating models is estimating their quality.  Some metric is needed to decide whether a model is poor, good, very good, etc. In the SUMO Toolbox this role is played by the ''Measure'' component. A measure is an object that, given a model, returns an estimation of its quality.  This can be something very simple as the error of the model fit in the training data, or it may involve a complex calculation to see if a model satisfies some physical constraint.
  
The rule of thumb is that the default measure, CrossValidation, is the best choice, but is also very expensive, as it requires that a number (5 by default) of models are built for each new model that has to be evaluated. Especially when using neural networks this can become an unacceptable overhead. If the modelling takes unacceptably long, the best alternative is TestSamples, which by default behaves as cross validation in which only one fold is considered, reducing the cost of the measure by a factor 5.
+
The Measure is used in the SUMO Toolbox to drive the [[Add_Model_Type#Models.2C_Model_builders.2C_and_Factories|model parameter optimization]] (see also the [[Running#Understanding_the_control_flow|toolbox control flow]]). This can be done in a single objective or [[Multi-Objective Modeling|multi-objective]] fashion.
  
However, in certain situations it might be very effective to use different measures, or use multiple measures together. When multiple measures are used, an intelligent pareto-based method is used to decide which model is the best choice. Models that score high on a particular measure but low on another are not discarded immediately, but are given a chance to set things right in further iterations of the toolbox. This encourages variety in the models, while still ensuring convergence to the optimal accuracy for each measure. An often used combination is CrossValidation with the MinMax measure, to ensure that no poles are present in the model domain.
+
There are two aspects to a Measure:
  
Below is a list of available measures and the configuration options available for each of them. Each measure also has a target accuracy attribute, which can be ommitted and which defaults to 0.001. In certain cases, such as the binary MinMax measure, the target accuracy is irrelevant.
+
# The quality estimation algorithm
 +
# The error function
  
 +
The first is the algorithm used to estimate the model quality.  This is for example the in-sample error, or the 5-fold crossvalidation score.  The error function determines what kind of error you want to use.  You can calculate the in-sample error using: the average absolute error, the root mean square error, a maximum relative error, etc.  Note that the error function may not be relevant for every type of measure (eg., AIC).
  
 +
It cannot be stressed enough that:
  
== CrossValidation ==
+
<center>'''A proper choice of Measure and Error function is CRUCIAL to the success of your modeling'''</center>
 +
 
 +
That choice will depend on your problem characteristics, the data distribution and the model type you will use to fit the data.  It is '''extremely important''' to think about this well ("what do you want?") before starting any modeling.
 +
 
 +
As a side remark, note that Measures and sequential designs are closely related.  The same criteria that is used to decide if a model is good or bad, can be used to identify interesting locations to select new data points.
 +
 
 +
Also note that some model builders (notably GeneticModelBuilder) support constraints directly.  This means you can also implement what you would otherwise implement as a Measure as constraint in the model parameter optimization algorithm itself.  This is a superior solution to using a Measure.
 +
 
 +
== Using measures ==
 +
 
 +
To use a particular measure in the SUMO Toolbox, you have to edit the <Measure> tag in Plan section of the configuration XML-file. If you are using the configuration XML-file default.xml, please note that the <Measure> tag does NOT refer to any configuration sections below the Plan section. Instead they are defined directly in the Plan section. You can of course always add Measure configuration sections and refer to them in the Plan section using their id. Here are three equivalent examples of defining measure in the configuration xml. The default.xml also contains examples of measures in the commented out sections.
 +
 
 +
The first way is the one used in the default.xml and can be found within the Plan section; the measure type used is the SampleError which scores models on how well the capture the training data. The error function used to aggregate the scores for each training sample is meanSquareError. The target determines the target accuracy that has to be satisfied by the models, in this case it means that mean square error of all training examples of a model should be 0.01 or less. The option <code>use="on"</code> indicates that this measure should be used to guide the model selection criterion.
 +
 
 +
<source xmlns:saxon="http://icl.com/saxon" lang="xml">
 +
<Plan>
 +
    ...
 +
    <Measure type="SampleError" target=".01" errorFcn="meanSquareError" use="on"/>
 +
    ...
 +
</Plan>
 +
</source>
 +
 
 +
The second way uses an id in the Plan section to refer to the same measure, but which is now defined separately in a configuration section:
 +
 
 +
<source xmlns:saxon="http://icl.com/saxon" lang="xml">
 +
<Plan>
 +
    ...
 +
    <Measure>ExampleMeasure</Measure>
 +
    ...
 +
</Plan>
 +
 
 +
...
 +
 
 +
<Measure id="ExampleMeasure" type ="SampleError">
 +
    <Option key="use" value="on"/>
 +
    <Option key="target" value="0.0"/>
 +
    <Option key="errorFcn" value="meanSquareError"/>
 +
</Measure>
 +
</source>
 +
 
 +
And finally the configuration section can also be placed directly into the Plan section.
 +
 
 +
<source xmlns:saxon="http://icl.com/saxon" lang="xml">
 +
<Plan>
 +
    ...
 +
   
 +
    <Measure id="ExampleMeasure" type ="SampleError">
 +
        <Option key="use" value="on"/>
 +
        <Option key="target" value="0.0"/>
 +
        <Option key="errorFcn" value="meanSquareError"/>
 +
    </Measure>
 +
    ...
 +
</Plan>
 +
</source>
 +
 
 +
== Using multiple measures ==
 +
 
 +
The SUMO-Toolbox also allows you to use several measures to guide modeling process. You may want to use more than one measure because for example you need a model which has a mean square error of 0.01 but its largest error cannot exceed a certain threshold. Or you want to use several measures to monitor the modeling process.
 +
 
 +
Go to the [[Multi-Objective_Modeling|Multi-Objective Modeling page]] to find out how to build models using multiple measures.
 +
If you want to use more than one measure but only for monitoring purposes (i.e. they will not be used for model selection), you can do this by adding the desired measure(s) you want to use for monitoring to the Plan but with the <code>use="off"</code> keyword.
 +
 
 +
<source xmlns:saxon="http://icl.com/saxon" lang="xml">
 +
<Plan>
 +
    ...
 +
    <Measure type="SampleError" target=".01" errorFcn="meanSquareError" use="on"/>
 +
    <Measure type="CrossValidation" target=".05" errorFcn="maxAbsoluteError" use="off"/>
 +
    ...
 +
</Plan>
 +
</source>
 +
 
 +
== Tips on using measures ==
 +
 
 +
A recommended read is [http://sumo.intec.ugent.be/sites/sumo/files/sumo/2009_08_EWC.pdf the paper available here].
 +
 
 +
In general, the default measure, 5 fold CrossValidation, is an acceptable choice.  However, it is also very expensive, as it requires that a model be re-trained for each fold.  This may slow things down if a model is expensive to train (e.g., neural nets).  Also CrossValidation can give biased results if data is clustered or scarce.  Increasing the number of folds may help here.  A cheaper alternative is ValidationSet (see below) or AIC.  For a full list of available measures see the <code>src/matlab/measures</code> subdirectory.
 +
 
 +
Note that multiple measures may also be combined.  For more information see [[Multi-Objective Modeling]].
 +
 
 +
For how to change the error function see [[FAQ#How_do_I_change_the_error_function_.28relative_error.2C_RMS.2C_....29.3F| this FAQ entry]].
 +
 
 +
Below is a list of some available measures and the configuration options available for each of them. Each measure also has a target accuracy attribute, which can be omitted and which defaults to 0.001. In certain cases, such as the binary MinMax measure, the target accuracy is irrelevant.
 +
 
 +
== Defining your own Measure ==
 +
 
 +
see [[Add_Measure]]
 +
 
 +
== Measure types ==
 +
 
 +
 
 +
''We are well aware that documentation is not always complete and possibly even out of date in some cases. We try to document everything as best we can but much is limited by available time and manpower.  We are are a university research group after all. The most up to date documentation can always be found (if not here) in the default.xml configuration file and, of course, in the source files.  If something is unclear please dont hesitate to [[Reporting problems|ask]].''
 +
 
 +
=== CrossValidation ===
  
 
The CrossValidation measure is the default choice and performs an n-fold cross validation on the model to create an efficient estimation of the accuracy of the model. Several options are available to customize this measure.
 
The CrossValidation measure is the default choice and performs an n-fold cross validation on the model to create an efficient estimation of the accuracy of the model. Several options are available to customize this measure.
Line 21: Line 117:
 
}}
 
}}
 
{{Option
 
{{Option
|name        = storeModels
+
|name        = randomThreshold
 +
|values      = positive integer
 +
|default    = 1000
 +
|description = If the number of samples is greater than this number a random partitioning is used
 +
}}
 +
{{Option
 +
|name        = partitionMethod
 +
|values      = [uniform,random]
 +
|default    = uniform
 +
|description = This option defines whether the test sets for the folds are chosen randomly, or are chosen in such a way as to maximize the domain coverage. Random is generally much faster, but might result in pessimistic scoring, as unlucky test set choice can result in an inaccurate error. This can partly be fixed by enabling the resetFolds option.
 +
}}
 +
{{Option
 +
|name        = resetFolds
 
|values      = boolean
 
|values      = boolean
 
|default    = no
 
|default    = no
|description = When this option is turned on, the models built for each fold are stored on disk for future reference.
+
|description = Folds are generated from scratch for each model that is evaluated using this measure. If the same model is evaluated twice (for example, after a rebuild), new folds are used. Enabling this feature can be very costly for large sample sizes. As a rule of thumb, enable this in combination with the random partition method, or disable this when using the uniform method.
}}
 
{{Option
 
|name        = modelDifference
 
|values      = [0,1]
 
|default    = 0
 
|description = When this option is turned on, the average is calculated of the models built in different folds, and the score is based partly on the cross validation error and partly on the error of the average model in the samples. This option can be used to detect poles that will otherwise go unnoticed using the CrossValidation measure. A sensible value is 0.2.
 
 
}}
 
}}
  
== TestSamples ==
+
Here is an example configuration for the CrossValidation measure.
 +
 
 +
<source xmlns:saxon="http://icl.com/saxon" lang="xml">
 +
<Measure type ="CrossValidation">
 +
    <Option key="use" value="on"/>
 +
    <Option key="target" value="0.0"/>
 +
    <Option key="errorFcn" value="meanSquareError"/>
 +
    <Option key="folds" value="5"/>
 +
    <Option key="randomThreshold" value="1000"/>
 +
    <Option key="partitionMethod" value="uniform"/>
 +
    <Option key="resetFolds" value="no"/>
 +
</Measure>
 +
</source>
 +
 
 +
=== ValidationSet ===
  
The TestSamples measure has two different methods of operation.
+
The ValidationSet measure has two different methods of operation.
  
In the first method, the list of samples that have been evaluated is split into a test set (validation set) and a training set. A model is then built using the training set, and evaluated using the test set (which is by default 20% of the total sample pool). However, an external data file containing a test set can also specified. In this case, all the evaluated samples are used for training, and the external test set is used for validation only. Which of these two operation methods is used, depends on the configuration options below. By default, no external test set is loaded.
+
# In the first method, the list of samples that have been evaluated is split into a validation set and a training set. A model is then built using the training set, and evaluated using the validation set (which is by default 20% of the total sample pool).
 +
# However, an external data file containing a validation set can also specified. In this case, all the evaluated samples are used for training, and the external set is used for validation only. Which of these two operation methods is used, depends on the configuration options below. By default, no external validation set is loaded.
  
If you want to use an external test set, you will have to provide a SampleEvaluator configuration so that the test set can be loaded from an external source. Here is an TestSamples configuration which loads the test set from the scattered data file provided in the simulator file:
+
If you want to use an external validation set, you will have to provide a DataSource configuration so that the validation set can be loaded from an external source. Here is a ValidationSet configuration example which loads the validation set from the scattered data file provided in the simulator file:
  
<pre><nowiki>
+
<source lang="xml">
<Measure type="TestSamples" target=".001">
+
<Measure type="ValidationSet" target=".001">
 
<Option key="type" value="file"/>
 
<Option key="type" value="file"/>
<SampleEvaluator type="be.ac.ua.coms.m3.SampleEvaluators.ScatteredDatasetSampleEvaluator"/>
+
<DataSource type="iminds.sumo.DataSource.datasets.ScatteredDatasetDataSource"/>
 
</Measure>
 
</Measure>
</nowiki></pre>
+
</source>
  
 
{{OptionsHeader}}
 
{{OptionsHeader}}
Line 69: Line 186:
 
}}
 
}}
  
== LeaveNOut ==
+
=== MinMax ===
 
 
  
== MinMax ==
+
The MinMax measure is used to eliminate models whose response falls below a given minimum or above a given maximum. This measure can be used to detect models that have poles in the model domain and to guide the modeling process in the right direction. If the output is known to lie within certain value bounds, these can be added to the simulator file as follows:
  
the MinMax measure is used to eliminate models that go below a given minimum or above a given maximum. This measure can be used to detect models that have poles in the model domain and to guide the modelling process in the right direction. If the output is known to lie within certain value bounds, these can be added to the simulator file as follows:
+
<source lang="xml">
 
 
<pre><nowiki>
 
 
<OutputParameters>
 
<OutputParameters>
 
<Parameter name="out" type="real" minimum="-1" maximum="1"/>
 
<Parameter name="out" type="real" minimum="-1" maximum="1"/>
 
</OutputParameters>
 
</OutputParameters>
</nowiki></pre>
+
</source>
 +
 
 +
When the MinMax measure is defined, these values will be used to ensure that all models stay within these bounds. If only the minimum or only the maximum is defined, naturally only these are enforced. There are no further configuration options for this measure. In case of complex outputs the modulus is used.
 +
 
 +
Remember though, that no guarantee can be given that the poles will really disappear. Using this measure only combats the symptoms and not the cause of the problem. Also, this measure can be reasonably slow, because it evaluates a dense grid to decide wether the model is crossing boundaries. If the model is slow to evaluate, this can take a considerable amount of time. Also note that the model is checked on a dense grid.  However, in higher dimensions the grid is sparser and thus no absolute certainty is given.
 +
Finally note that this is quite a strong constraint on the model building process.  It means that a model which is otherwise very good, but simply overshoots the data in one place, will be penalized quite heavily.  This, if you can, try not to to give too strict bounds.
 +
 
 +
'''Tips''':
 +
# even if you don't know the exact bounds on your output, you can still use this measure by specifying very broad bounds (e.g., [-10000 10000]).  This can still allow you to catch poles since, by definition, they reach until infinity.
 +
# if you are using the ANN models and you want your output to lie within [0 1] you can simply use a 'logsig' transfer function for the output layer.  Then BY DEFINITION your output will lie in [0 1] and no extra measure is needed. To get a different range you could even rename logsig.m and add your own scaling. This solution is far superior than using MinMax.
 +
# if you are using the GeneticModelBuilder with population type custom, the MinMax idea can also be implemented in the GA itself as a constraint.  Then  you do not need the measure, this would also be a better solution.
 +
 
 +
=== SampleError ===
 +
 
 +
This measure simply calculates the error in the training data.  Note that this measure is useless for interpolating methods like Kriging and RBF.
 +
 
 +
=== ModelDifference ===
 +
 
 +
The toolbox keeps track of the n best models found so far.  The ModelDifference measure uses the disagreement between those models as a heuristic for ranking them.  A model that differs considerably from the other models is assumed to be of poor quality.  Remember that this is just a heuristic! We recommend ModelDifference never be used alone, but always in combination with some other model.
  
When the MinMax measure is defined, these values will be used to ensure that all models stay within these bounds. If only the minimum or only the maximum is defined, naturally only these are enforced. There are no further configuration options for this measure.
+
=== LRMMeasure ===
  
'''Tip''': even if you don't know the exact bounds on your output, you can still use this measure by specifying very broad bounds (e.g., [-10000 10000]).  This can still allow you to catch poles since, by definition, they reach till infinity.
+
This is a very useful measure that can be used to complement other measures (see [[Multi-Objective Modeling]]) with some good resultsFor example, as a cheaper alternative for crossvaliation with neural networks.  It will penalize models where they show unwanted 'bumps' or 'ripples' in the response.
  
== ModelDifference ==
+
=== AIC ===
  
== SampleError ==
+
Implements [http://en.wikipedia.org/wiki/Akaike_information_criterion Akaikes Information Criterion].  Note that this requires a proper implementation of the freeParams of a Model.

Latest revision as of 11:49, 26 July 2014

What is a Measure

A crucial aspect of generating models is estimating their quality. Some metric is needed to decide whether a model is poor, good, very good, etc. In the SUMO Toolbox this role is played by the Measure component. A measure is an object that, given a model, returns an estimation of its quality. This can be something very simple as the error of the model fit in the training data, or it may involve a complex calculation to see if a model satisfies some physical constraint.

The Measure is used in the SUMO Toolbox to drive the model parameter optimization (see also the toolbox control flow). This can be done in a single objective or multi-objective fashion.

There are two aspects to a Measure:

  1. The quality estimation algorithm
  2. The error function

The first is the algorithm used to estimate the model quality. This is for example the in-sample error, or the 5-fold crossvalidation score. The error function determines what kind of error you want to use. You can calculate the in-sample error using: the average absolute error, the root mean square error, a maximum relative error, etc. Note that the error function may not be relevant for every type of measure (eg., AIC).

It cannot be stressed enough that:

A proper choice of Measure and Error function is CRUCIAL to the success of your modeling

That choice will depend on your problem characteristics, the data distribution and the model type you will use to fit the data. It is extremely important to think about this well ("what do you want?") before starting any modeling.

As a side remark, note that Measures and sequential designs are closely related. The same criteria that is used to decide if a model is good or bad, can be used to identify interesting locations to select new data points.

Also note that some model builders (notably GeneticModelBuilder) support constraints directly. This means you can also implement what you would otherwise implement as a Measure as constraint in the model parameter optimization algorithm itself. This is a superior solution to using a Measure.

Using measures

To use a particular measure in the SUMO Toolbox, you have to edit the <Measure> tag in Plan section of the configuration XML-file. If you are using the configuration XML-file default.xml, please note that the <Measure> tag does NOT refer to any configuration sections below the Plan section. Instead they are defined directly in the Plan section. You can of course always add Measure configuration sections and refer to them in the Plan section using their id. Here are three equivalent examples of defining measure in the configuration xml. The default.xml also contains examples of measures in the commented out sections.

The first way is the one used in the default.xml and can be found within the Plan section; the measure type used is the SampleError which scores models on how well the capture the training data. The error function used to aggregate the scores for each training sample is meanSquareError. The target determines the target accuracy that has to be satisfied by the models, in this case it means that mean square error of all training examples of a model should be 0.01 or less. The option use="on" indicates that this measure should be used to guide the model selection criterion.

<Plan>
    ...
    <Measure type="SampleError" target=".01" errorFcn="meanSquareError" use="on"/>
    ...
</Plan>

The second way uses an id in the Plan section to refer to the same measure, but which is now defined separately in a configuration section:

<Plan>
    ...
    <Measure>ExampleMeasure</Measure>
    ...
</Plan>

...

<Measure id="ExampleMeasure" type ="SampleError">
    <Option key="use" value="on"/>
    <Option key="target" value="0.0"/>
    <Option key="errorFcn" value="meanSquareError"/>
</Measure>

And finally the configuration section can also be placed directly into the Plan section.

<Plan>
    ...
    
    <Measure id="ExampleMeasure" type ="SampleError">
        <Option key="use" value="on"/>
        <Option key="target" value="0.0"/>
        <Option key="errorFcn" value="meanSquareError"/>
    </Measure>
    ...
</Plan>

Using multiple measures

The SUMO-Toolbox also allows you to use several measures to guide modeling process. You may want to use more than one measure because for example you need a model which has a mean square error of 0.01 but its largest error cannot exceed a certain threshold. Or you want to use several measures to monitor the modeling process.

Go to the Multi-Objective Modeling page to find out how to build models using multiple measures. If you want to use more than one measure but only for monitoring purposes (i.e. they will not be used for model selection), you can do this by adding the desired measure(s) you want to use for monitoring to the Plan but with the use="off" keyword.

<Plan>
    ...
    <Measure type="SampleError" target=".01" errorFcn="meanSquareError" use="on"/>
    <Measure type="CrossValidation" target=".05" errorFcn="maxAbsoluteError" use="off"/>
    ...
</Plan>

Tips on using measures

A recommended read is the paper available here.

In general, the default measure, 5 fold CrossValidation, is an acceptable choice. However, it is also very expensive, as it requires that a model be re-trained for each fold. This may slow things down if a model is expensive to train (e.g., neural nets). Also CrossValidation can give biased results if data is clustered or scarce. Increasing the number of folds may help here. A cheaper alternative is ValidationSet (see below) or AIC. For a full list of available measures see the src/matlab/measures subdirectory.

Note that multiple measures may also be combined. For more information see Multi-Objective Modeling.

For how to change the error function see this FAQ entry.

Below is a list of some available measures and the configuration options available for each of them. Each measure also has a target accuracy attribute, which can be omitted and which defaults to 0.001. In certain cases, such as the binary MinMax measure, the target accuracy is irrelevant.

Defining your own Measure

see Add_Measure

Measure types

We are well aware that documentation is not always complete and possibly even out of date in some cases. We try to document everything as best we can but much is limited by available time and manpower. We are are a university research group after all. The most up to date documentation can always be found (if not here) in the default.xml configuration file and, of course, in the source files. If something is unclear please dont hesitate to ask.

CrossValidation

The CrossValidation measure is the default choice and performs an n-fold cross validation on the model to create an efficient estimation of the accuracy of the model. Several options are available to customize this measure.

Template:OptionsHeader Template:Option Template:Option Template:Option Template:Option

Here is an example configuration for the CrossValidation measure.

<Measure type ="CrossValidation">
    <Option key="use" value="on"/>
    <Option key="target" value="0.0"/>
    <Option key="errorFcn" value="meanSquareError"/>
    <Option key="folds" value="5"/>
    <Option key="randomThreshold" value="1000"/>
    <Option key="partitionMethod" value="uniform"/>
    <Option key="resetFolds" value="no"/>
</Measure>

ValidationSet

The ValidationSet measure has two different methods of operation.

  1. In the first method, the list of samples that have been evaluated is split into a validation set and a training set. A model is then built using the training set, and evaluated using the validation set (which is by default 20% of the total sample pool).
  2. However, an external data file containing a validation set can also specified. In this case, all the evaluated samples are used for training, and the external set is used for validation only. Which of these two operation methods is used, depends on the configuration options below. By default, no external validation set is loaded.

If you want to use an external validation set, you will have to provide a DataSource configuration so that the validation set can be loaded from an external source. Here is a ValidationSet configuration example which loads the validation set from the scattered data file provided in the simulator file:

<Measure type="ValidationSet" target=".001">
	<Option key="type" value="file"/>
	<DataSource type="iminds.sumo.DataSource.datasets.ScatteredDatasetDataSource"/>
</Measure>

Template:OptionsHeader Template:Option Template:Option Template:Option

MinMax

The MinMax measure is used to eliminate models whose response falls below a given minimum or above a given maximum. This measure can be used to detect models that have poles in the model domain and to guide the modeling process in the right direction. If the output is known to lie within certain value bounds, these can be added to the simulator file as follows:

<OutputParameters>
	<Parameter name="out" type="real" minimum="-1" maximum="1"/>
</OutputParameters>

When the MinMax measure is defined, these values will be used to ensure that all models stay within these bounds. If only the minimum or only the maximum is defined, naturally only these are enforced. There are no further configuration options for this measure. In case of complex outputs the modulus is used.

Remember though, that no guarantee can be given that the poles will really disappear. Using this measure only combats the symptoms and not the cause of the problem. Also, this measure can be reasonably slow, because it evaluates a dense grid to decide wether the model is crossing boundaries. If the model is slow to evaluate, this can take a considerable amount of time. Also note that the model is checked on a dense grid. However, in higher dimensions the grid is sparser and thus no absolute certainty is given. Finally note that this is quite a strong constraint on the model building process. It means that a model which is otherwise very good, but simply overshoots the data in one place, will be penalized quite heavily. This, if you can, try not to to give too strict bounds.

Tips:

  1. even if you don't know the exact bounds on your output, you can still use this measure by specifying very broad bounds (e.g., [-10000 10000]). This can still allow you to catch poles since, by definition, they reach until infinity.
  2. if you are using the ANN models and you want your output to lie within [0 1] you can simply use a 'logsig' transfer function for the output layer. Then BY DEFINITION your output will lie in [0 1] and no extra measure is needed. To get a different range you could even rename logsig.m and add your own scaling. This solution is far superior than using MinMax.
  3. if you are using the GeneticModelBuilder with population type custom, the MinMax idea can also be implemented in the GA itself as a constraint. Then you do not need the measure, this would also be a better solution.

SampleError

This measure simply calculates the error in the training data. Note that this measure is useless for interpolating methods like Kriging and RBF.

ModelDifference

The toolbox keeps track of the n best models found so far. The ModelDifference measure uses the disagreement between those models as a heuristic for ranking them. A model that differs considerably from the other models is assumed to be of poor quality. Remember that this is just a heuristic! We recommend ModelDifference never be used alone, but always in combination with some other model.

LRMMeasure

This is a very useful measure that can be used to complement other measures (see Multi-Objective Modeling) with some good results. For example, as a cheaper alternative for crossvaliation with neural networks. It will penalize models where they show unwanted 'bumps' or 'ripples' in the response.

AIC

Implements Akaikes Information Criterion. Note that this requires a proper implementation of the freeParams of a Model.