Difference between revisions of "Config:Plan"

From SUMOwiki
Jump to navigationJump to search
m
(AutoConfig for SUMO 6.2)
 
(4 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 +
'''Generated for SUMO toolbox version 7.0'''.
 +
''We are well aware that documentation is not always complete and possibly even out of date in some cases. We try to document everything as best we can but much is limited by available time and manpower.  We are an university research group after all. The most up to date documentation can always be found (if not here) in the default.xml configuration file and, of course, in the source files.  If something is unclear please dont hesitate to [[Reporting problems|ask]].''
 
== Plan ==
 
== Plan ==
=== LevelPlot ===
 
Only change if you are using levelplots
 
<source xmlns:saxon="http://icl.com/saxon" lang="xml">
 
 
<!--Only change if you are using levelplots-->
 
<[[Config:LevelPlot|LevelPlot]]>default</[[Config:LevelPlot|LevelPlot]]></source>
 
 
=== ContextConfig ===
 
=== ContextConfig ===
ContextConfig should (normally) always be set to 'default'
+
Default components, these should normally not be changed unless you know what you are doing
 
<source xmlns:saxon="http://icl.com/saxon" lang="xml">
 
<source xmlns:saxon="http://icl.com/saxon" lang="xml">
  
<!--ContextConfig should (normally) always be set to 'default'-->
+
<!--Default components, these should normally not be changed unless you know what you are doing-->
<[[Config:ContextConfig|ContextConfig]]>default</[[Config:ContextConfig|ContextConfig]]></source>
+
  <[[Config:ContextConfig|ContextConfig]]>default</[[Config:ContextConfig|ContextConfig]]></source>
 
=== SUMO ===
 
=== SUMO ===
SUMO should (normally) always be set to 'default'
+
Default components, these should normally not be changed unless you know what you are doing
 
<source xmlns:saxon="http://icl.com/saxon" lang="xml">
 
<source xmlns:saxon="http://icl.com/saxon" lang="xml">
  
<!--SUMO should (normally) always be set to 'default'-->
+
<!--Default components, these should normally not be changed unless you know what you are doing-->
<[[Config:SUMO|SUMO]]>default</[[Config:SUMO|SUMO]]></source>
+
  <[[Config:SUMO|SUMO]]>default</[[Config:SUMO|SUMO]]></source>
=== AdaptiveModelBuilder ===
+
=== LevelPlot ===
The AdaptiveModelBuilder specifies the model type and the modeling algorithm to use The default value 'rational' refers to rational functions. 'rational' is an id that refers to an AdaptiveModelBuilder tag that is defined below
+
Default components, these should normally not be changed unless you know what you are doing
 
<source xmlns:saxon="http://icl.com/saxon" lang="xml">
 
<source xmlns:saxon="http://icl.com/saxon" lang="xml">
  
<!--The AdaptiveModelBuilder specifies the model type and the modeling algorithm to use The default value 'rational' refers to rational functions. 'rational' is an id that refers to an AdaptiveModelBuilder tag that is defined below-->
+
<!--Default components, these should normally not be changed unless you know what you are doing-->
<[[Config:AdaptiveModelBuilder|AdaptiveModelBuilder]]>rational</[[Config:AdaptiveModelBuilder|AdaptiveModelBuilder]]></source>
+
  <[[Config:LevelPlot|LevelPlot]]>default</[[Config:LevelPlot|LevelPlot]]></source>
=== SampleSelector ===
+
=== Simulator ===
The method to use for selecting new samples. Again 'gradient' is an id that refers to a SampleSelector tag defined below
+
This is the problem we are going to model, it refers to the name of a project directory in the examples/ folder. It is also possible to specify an absolute path or to specify a particular xml file within a project directory
 
<source xmlns:saxon="http://icl.com/saxon" lang="xml">
 
<source xmlns:saxon="http://icl.com/saxon" lang="xml">
  
<!--The method to use for selecting new samples. Again 'gradient' is an id that refers to a SampleSelector tag defined below-->
+
<!--This is the problem we are going to model, it refers to the name of a project directory in the examples/ folder. It is also possible to specify an absolute path or to specify a particular xml file within a project directory-->
<[[Config:SampleSelector|SampleSelector]]>gradient</[[Config:SampleSelector|SampleSelector]]></source>
+
  <[[Config:Simulator|Simulator]]>Math/Academic2DTwice</[[Config:Simulator|Simulator]]></source>
 
=== Run ===
 
=== Run ===
Runs can given a custom name by adding a name="the_name" attribute, a repeat attribute is also possible to repeat a run multiple times
+
Runs can given a custom name by using the name attribute, a repeat attribute is also possible to repeat a run multiple times. Placeholders available for run names include: #adaptivemodelbuilder# #simulator# #sampleselector# #output# #measure#
 
<source xmlns:saxon="http://icl.com/saxon" lang="xml">
 
<source xmlns:saxon="http://icl.com/saxon" lang="xml">
  
<!--Runs can given a custom name by adding a name="the_name" attribute, a repeat attribute is also possible to repeat a run multiple times-->
+
<!--Runs can given a custom name by using the name attribute, a repeat attribute is also possible to repeat a run multiple times. Placeholders available for run names include: #adaptivemodelbuilder# #simulator# #sampleselector# #output# #measure#-->
<[[Config:Run|Run]] name="" repeat="1">
+
  <[[Config:Run|Run]] name="" repeat="1">
  <!-- Configuration components, refer to those defined below
+
      <!-- Enties listed here override those defined on plan level -->
        Enties listed here override those defined on plan level -->
 
 
 
  <!-- This is the problem we are going to model, refers to an xml file in the examples/ directory -->
 
  <[[Config:Simulator|Simulator]]>Academic2DTwice.xml</[[Config:Simulator|Simulator]]>
 
 
 
  <!--
 
  How is the simulator implemented:
 
    - Matlab script (matlab)
 
    - scattered dataset (scattered),
 
    - local executable (local)
 
    - etc
 
  -->
 
  <[[Config:SampleEvaluator|SampleEvaluator]]>matlab</[[Config:SampleEvaluator|SampleEvaluator]]>
 
 
 
  <!--
 
  The default behavior is to model all outputs and score models using
 
  crossvalidation.  See below how to override this. Note that
 
  crossvalidation is very a expensive measure and can significantly
 
  slow things down when using computationally expensive model types
 
  (eg. neural networks)
 
  -->
 
 
 
  <!-- Define inputs that are to be modelled this run. This optional setting
 
        reduces the dimension of the problem by keeping inputs that were not
 
        selected at 0. When this section is not specified, all inputs are used.
 
        In this example, input x is filtered out (not mentioned) and input z is set to a constant and will have
 
      no role in the modelling process. -->
 
  <!--
 
  <[[Config:Inputs|Inputs]]>
 
      <[[Config:Input|Input]] name="y" />
 
      <[[Config:Input|Input]] name="z" value="1.5" />
 
  </[[Config:Inputs|Inputs]]>
 
  -->
 
 
 
 
 
 
 
  <!--  Complex example of a modeling run of the InductivePosts example with many different
 
      output configurations.
 
 
 
  <[[Config:Outputs|Outputs]]>
 
 
        
 
        
       Model the modulus of complex output S22 using cross-validation and the default model builder
+
       <!-- What experimental design to use for the very first set of samples -->
       and sample selector.
+
       <[[Config:InitialDesign|InitialDesign]]>lhdWithCornerPoints</[[Config:InitialDesign|InitialDesign]]>
 
        
 
        
       <[[Config:Output|Output]] name="S22" complexHandling="modulus">
+
       <!--
        <[[Config:Measure|Measure]] type="[[Measure#CrossValidation|CrossValidation]]" target=".05" />
+
          The method to use for selecting new samples. Again 'default' is an id that refers to a
      </[[Config:Output|Output]]>
+
          SampleSelector tag defined below.  To switch off sampling simply remove this tag. -->
 +
      <[[Config:SampleSelector|SampleSelector]]>default</[[Config:SampleSelector|SampleSelector]]>
 
        
 
        
 +
      <!--
 +
      How is the simulator implemented (ie, where does the data come from):
 +
        - Matlab script (matlab)
 +
        - scattered dataset (scatteredDataset),
 +
        - local executable or script (local)
 +
        - etc
 +
       
 +
        Make sure this entry matches what is declared in the simulator xml file
 +
        in the project directory.  For example, it makes no sense to put matlab here if you only
 +
        have a scattered dataset to work with.
 +
      -->
 +
      <[[Config:SampleEvaluator|SampleEvaluator]]>matlab</[[Config:SampleEvaluator|SampleEvaluator]]>
 +
 +
      <!--
 +
          The AdaptiveModelBuilder specifies the model type and the hyperparameter optimization
 +
          algorithm (= the algorithm to choose the model parameters, also referred to as the
 +
          modeling algorithm or model builder) to use. The default value 'kriging' refers to Kriging models.
 +
          'kriging' is an id that refers to an AdaptiveModelBuilder tag that is defined below.
 +
      -->
 +
      <[[Config:AdaptiveModelBuilder|AdaptiveModelBuilder]]>kriging</[[Config:AdaptiveModelBuilder|AdaptiveModelBuilder]]>
 
        
 
        
       Model the modulus of complex output S22, but introduce some normally-distributed noise
+
       <!-- How the quality of a model is assesed is determined by one or more Measures.  You can try different combinations
       (variance .01 by default).
+
          of measures by specifying different measure tags.  It is the measure score(s) that drive the model parameter optimization.
 +
          We recommend you do not use more than one measure unless you know what you are doing.
 +
         
 +
          If the use attribute is set to 'off' then the measure score is printed and logged, but is not used in the modeling itself.
 +
          More examples of measures are shown below.
 +
       -->
 
        
 
        
       <[[Config:Output|Output]] name="S22" complexHandling="modulus">
+
       <[[Config:Measure|Measure]] type="[[Measure#CrossValidation|CrossValidation]]" target="0.01" errorFcn="rootRelativeSquareError" use="on"/>
        <[[Config:Measure|Measure]] type="[[Measure#CrossValidation|CrossValidation]]" target=".05" />
 
        <[[Config:Modifier|Modifier]] type="[[Modifier#Noise|Noise]]" />
 
      </[[Config:Output|Output]]>
 
 
        
 
        
 +
      <!-- By default all inputs are modeled.  If you want to only model a couple of inputs you can specify an Inputs tag as follows:
 
        
 
        
       Model the modulus of complex output S22, but introduce normally-distributed noise
+
       <[[Config:Inputs|Inputs]]>
      with variance .1. However, when Nan or Inf values are returned from the simulator,
+
        <[[Config:Input|Input]] name="x" />
       we ignore these errors and let the toolbox process them normally. By default,
+
        <[[Config:Input|Input]] name="y" />
       samples with NaN or Inf values are ignored.
+
        // Setting a simulator input to a constant (default is 0):
 +
        <[[Config:Input|Input]] name="z"  value="14.6"/>
 +
       </[[Config:Inputs|Inputs]]>
 +
       -->
 
        
 
        
       <[[Config:Output|Output]] name="S22" ignoreNaN="no" ignoreInf="no">
+
       <!--        
        <[[Config:Measure|Measure]] type="[[Measure#CrossValidation|CrossValidation]]" target=".05" />
+
      By default the toolbox will model every single output using a separate model. If you want to change this
        <[[Config:Modifier|Modifier]] type="[[Modifier#Noise|Noise]]" distribution="normal" variance=".1" />
+
       e.g., you only want to model a specific output, or you want to use different settings for each output; then you
      </[[Config:Output|Output]]>
+
       can specify an Outputs tag.
  </[[Config:Outputs|Outputs]]>
 
  -->
 
 
 
 
 
  <!--
 
 
 
  An example configuration for the Academic2DTwice example used here.
 
     
 
  <[[Config:Outputs|Outputs]]>
 
       <[[Config:Output|Output]] name="out">
 
        <[[Config:SampleSelector|SampleSelector]]>gradient</[[Config:SampleSelector|SampleSelector]]>
 
        <[[Config:AdaptiveModelBuilder|AdaptiveModelBuilder]]>rational</[[Config:AdaptiveModelBuilder|AdaptiveModelBuilder]]>
 
        <[[Config:Measure|Measure]] type="[[Measure#CrossValidation|CrossValidation]]" target=".0001" use="on" />
 
       </[[Config:Output|Output]]>
 
 
        
 
        
       <[[Config:Output|Output]] name="outinverse">
+
       The following is an example for the Academic2DTwice problem used in this file.  Remember that if you change
        <[[Config:SampleSelector|SampleSelector]]>grid</[[Config:SampleSelector|SampleSelector]]>
+
      the problem you are modeling, you will have to change this section too.
        <[[Config:AdaptiveModelBuilder|AdaptiveModelBuilder]]>kriging</[[Config:AdaptiveModelBuilder|AdaptiveModelBuilder]]>
+
      -->
        <[[Config:Measure|Measure]] type="[[Measure#ValidationSet|ValidationSet]]" target=".05" use="on" />
+
      <[[Config:Outputs|Outputs]]>
      </[[Config:Output|Output]]>
+
        <[[Config:Output|Output]] name="out">
  </[[Config:Outputs|Outputs]]>
+
            <!--
  -->
+
                You can specify output specific configuration here
 +
               
 +
            <[[Config:SampleSelector|SampleSelector]]>lola</[[Config:SampleSelector|SampleSelector]]>
 +
            <[[Config:AdaptiveModelBuilder|AdaptiveModelBuilder]]>rational</[[Config:AdaptiveModelBuilder|AdaptiveModelBuilder]]>
 +
            <[[Config:Measure|Measure]] type="[[Measure#CrossValidation|CrossValidation]]" target=".01" errorFcn="meanSquareError" use="on" />
 +
            -->
 +
        </[[Config:Output|Output]]>
 +
       
 +
        <[[Config:Output|Output]] name="outinverse">
 +
            <!--
 +
            <[[Config:SampleSelector|SampleSelector]]>delaunay</[[Config:SampleSelector|SampleSelector]]>
 +
            <[[Config:AdaptiveModelBuilder|AdaptiveModelBuilder]]>rbf</[[Config:AdaptiveModelBuilder|AdaptiveModelBuilder]]>
 +
            <[[Config:Measure|Measure]] type="[[Measure#ValidationSet|ValidationSet]]" target=".05" use="on" />
 +
            -->
 +
        </[[Config:Output|Output]]>
 +
       
 +
      </[[Config:Outputs|Outputs]]>
  
  <!--
+
      <!--   
  Measure examples:
+
        This is a more complex example of how you can have different configurations per output.
 
+
      -->
  * 5-fold crossvalidation (warning expensive on some model types (eg: takes a long time on neural networks))
+
      <!--
  <[[Config:Measure|Measure]] type="[[Measure#CrossValidation|CrossValidation]]" target=".001" use="on">
+
      <[[Config:Outputs|Outputs]]>
                                <Option key="folds" value="5"/>
 
                        </[[Config:Measure|Measure]]>    
 
 
 
  * Using a validation set, the size taken as 20% of the available samples
 
    <[[Config:Measure|Measure]] type="[[Measure#ValidationSet|ValidationSet]]" target=".001">
 
                                <Option key="percentUsed" value="20"/>
 
                        </[[Config:Measure|Measure]]>
 
  
 +
        * Model the modulus of complex output S22 using cross-validation and the default model
 +
        builder and sample selector.
 +
       
 +
        <[[Config:Output|Output]] name="S22" complexHandling="modulus">
 +
            <[[Config:Measure|Measure]] type="[[Measure#CrossValidation|CrossValidation]]" target=".05" />
 +
        </[[Config:Output|Output]]>
 +
       
 +
        * Model the real part of complex output S22, but introduce some normally-distributed noise
 +
        (variance .01 by default).
 +
       
 +
        <[[Config:Output|Output]] name="S22" complexHandling="real">
 +
            <[[Config:Measure|Measure]] type="[[Measure#CrossValidation|CrossValidation]]" target=".05" />
 +
            * for other types of modifiers see the datamodifiers subdirectory
 +
            <[[Config:Modifier|Modifier]] type="[[Modifier#Noise|Noise]]" />
 +
        </[[Config:Output|Output]]>
 +
      -->
  
  * Using a validation set defined in an external file (scattered data)
+
       <!--
       <[[Config:Measure|Measure]] type="[[Measure#ValidationSet|ValidationSet]]" target=".001">
+
       More complex examples of how you can use measures:
      * the validation set come from a file
 
       <Option key="type" value="file"/>
 
      * the test data is scattered data so we need a scattered sample evaluator to load the data
 
      and evaluate the points. The filename is taken from the <[[Config:ScatteredDataFile|ScatteredDataFile]]> tag in the simulator
 
      xml file.  Optionally you can specify an option with key "id" to specify a specifc dataset if there
 
      is more than one choice.
 
      <[[Config:SampleEvaluator|SampleEvaluator]] type="ibbt.sumo.SampleEvaluators.datasets.ScatteredDatasetSampleEvaluator"/>
 
                        </[[Config:Measure|Measure]]>
 
  
  * Used for testing optimization problems
+
       * 5-fold crossvalidation (warning expensive on some model types!)
      * Calculates the (relative) error between the current minimum and a known minimum.
 
        Often one uses this just as a stopping criterion for benchmarking problems.
 
       * trueValue: a known global minimum
 
  <[[Config:Measure|Measure]] type="[[Measure#TestMinimum|TestMinimum]]" errorFcn="relativeError" trueValue="-5.0" target="0.1" use="on" />
 
 
 
 
 
 
 
  * Examples of combined measures:
 
  Measure the model based on a set of test samples, taken as a subset from the list of evaluated samples.
 
  This subset is selected to cover the design space as good as possible.
 
  <[[Config:Measure|Measure]] type="[[Measure#ValidationSet|ValidationSet]]" target=".001">
 
     
 
      <Option key="percentUsed" value="20"/>
 
      <Option key="type" value="gridded"/>
 
      <Option key="randomThreshold" value="1000"/>
 
     
 
      Submeasures can be defined to work on the model produced by the supermeasure.
 
      In this case, the ValidationSet measure will generate a new model using a subset
 
      of the entire list of evaluated samples, and will then do an additional
 
      cross-validation check on this new model.
 
 
       <[[Config:Measure|Measure]] type="[[Measure#CrossValidation|CrossValidation]]" target=".001" use="on">
 
       <[[Config:Measure|Measure]] type="[[Measure#CrossValidation|CrossValidation]]" target=".001" use="on">
 
         <Option key="folds" value="5"/>
 
         <Option key="folds" value="5"/>
 +
      </[[Config:Measure|Measure]]> 
 +
 +
      * Using a validation set, the size taken as 20% of the available samples
 +
      <[[Config:Measure|Measure]] type="[[Measure#ValidationSet|ValidationSet]]" target=".001" errorFcn="meanAbsoluteError">
 +
        <Option key="percentUsed" value="20"/>
 
       </[[Config:Measure|Measure]]>
 
       </[[Config:Measure|Measure]]>
        
+
 
  </[[Config:Measure|Measure]]>
+
       * Using a validation set defined in an external file (scattered data)
 
+
            <[[Config:Measure|Measure]] type="[[Measure#ValidationSet|ValidationSet]]" target=".001">
  <[[Config:Measure|Measure]] type="[[Measure#ModelDifference|ModelDifference]]" target=".001" use="off">
+
            * the validation set come from a file
      <Option key="LHS" value="1000"/>
+
            <Option key="type" value="file"/>
 
+
            * the test data is scattered data so we need a scattered sample evaluator
      <[[Config:Measure|Measure]] type="[[Measure#SampleError|SampleError]]" target=".001" use="off">
+
            to load the data and evaluate the points. The filename is taken from the
       
+
            <[[Config:ScatteredDataFile|ScatteredDataFile]]> tag in the simulator xml file.
        <[[Config:Measure|Measure]] type="[[Measure#LeaveNOut|LeaveNOut]]" target=".001" use="off">
+
            Optionally you can specify an option with key "id" to specify a specifc
            <Option key="count" value="5"/>
+
            dataset if there is more than one choice.
        </[[Config:Measure|Measure]]>
+
            <[[Config:SampleEvaluator|SampleEvaluator]]
          
+
            type="ibbt.sumo.sampleevaluators.datasets.ScatteredDatasetSampleEvaluator"/>
       </[[Config:Measure|Measure]]>
+
                    </[[Config:Measure|Measure]]>
     
+
 
  </[[Config:Measure|Measure]]>
+
      * Used for testing optimization problems
  -->
+
         * Calculates the (relative) error between the current minimum and a known minimum.
</[[Config:Run|Run]]></source>
+
          Often one uses this just as a stopping criterion for benchmarking problems.
 +
        * trueValue: a known global minimum
 +
       <[[Config:Measure|Measure]] type="[[Measure#TestMinimum|TestMinimum]]" errorFcn="relativeError" trueValue="-5.0" target="0.1" use="on" />  
 +
      -->
 +
  </[[Config:Run|Run]]></source>

Latest revision as of 11:31, 25 March 2010

Generated for SUMO toolbox version 7.0. We are well aware that documentation is not always complete and possibly even out of date in some cases. We try to document everything as best we can but much is limited by available time and manpower. We are an university research group after all. The most up to date documentation can always be found (if not here) in the default.xml configuration file and, of course, in the source files. If something is unclear please dont hesitate to ask.

Plan

ContextConfig

Default components, these should normally not be changed unless you know what you are doing

<!--Default components, these should normally not be changed unless you know what you are doing-->
   <[[Config:ContextConfig|ContextConfig]]>default</[[Config:ContextConfig|ContextConfig]]>

SUMO

Default components, these should normally not be changed unless you know what you are doing

<!--Default components, these should normally not be changed unless you know what you are doing-->
   <[[Config:SUMO|SUMO]]>default</[[Config:SUMO|SUMO]]>

LevelPlot

Default components, these should normally not be changed unless you know what you are doing

<!--Default components, these should normally not be changed unless you know what you are doing-->
   <[[Config:LevelPlot|LevelPlot]]>default</[[Config:LevelPlot|LevelPlot]]>

Simulator

This is the problem we are going to model, it refers to the name of a project directory in the examples/ folder. It is also possible to specify an absolute path or to specify a particular xml file within a project directory

<!--This is the problem we are going to model, it refers to the name of a project directory in the examples/ folder. It is also possible to specify an absolute path or to specify a particular xml file within a project directory-->
   <[[Config:Simulator|Simulator]]>Math/Academic2DTwice</[[Config:Simulator|Simulator]]>

Run

Runs can given a custom name by using the name attribute, a repeat attribute is also possible to repeat a run multiple times. Placeholders available for run names include: #adaptivemodelbuilder# #simulator# #sampleselector# #output# #measure#

<!--Runs can given a custom name by using the name attribute, a repeat attribute is also possible to repeat a run multiple times. Placeholders available for run names include: #adaptivemodelbuilder# #simulator# #sampleselector# #output# #measure#-->
   <[[Config:Run|Run]] name="" repeat="1">
      <!-- Enties listed here override those defined on plan level -->
      
      <!-- What experimental design to use for the very first set of samples -->
      <[[Config:InitialDesign|InitialDesign]]>lhdWithCornerPoints</[[Config:InitialDesign|InitialDesign]]>
      
      <!--
          The method to use for selecting new samples. Again 'default' is an id that refers to a
          SampleSelector tag defined below.  To switch off sampling simply remove this tag. -->
      <[[Config:SampleSelector|SampleSelector]]>default</[[Config:SampleSelector|SampleSelector]]>
      
      <!--
      How is the simulator implemented (ie, where does the data come from): 
        - Matlab script (matlab)
        - scattered dataset (scatteredDataset), 
        - local executable or script (local)
        - etc
        
        Make sure this entry matches what is declared in the simulator xml file
        in the project directory.  For example, it makes no sense to put matlab here if you only
        have a scattered dataset to work with.
      -->
      <[[Config:SampleEvaluator|SampleEvaluator]]>matlab</[[Config:SampleEvaluator|SampleEvaluator]]>

      <!--
          The AdaptiveModelBuilder specifies the model type and the hyperparameter optimization
          algorithm (= the algorithm to choose the model parameters, also referred to as the
          modeling algorithm or model builder) to use. The default value 'kriging' refers to Kriging models.
          'kriging' is an id that refers to an AdaptiveModelBuilder tag that is defined below.
      -->
      <[[Config:AdaptiveModelBuilder|AdaptiveModelBuilder]]>kriging</[[Config:AdaptiveModelBuilder|AdaptiveModelBuilder]]>
      
      <!-- How the quality of a model is assesed is determined by one or more Measures.  You can try different combinations
           of measures by specifying different measure tags.  It is the measure score(s) that drive the model parameter optimization.
           We recommend you do not use more than one measure unless you know what you are doing.
          
           If the use attribute is set to 'off' then the measure score is printed and logged, but is not used in the modeling itself.
          More examples of measures are shown below.
      -->
      
      <[[Config:Measure|Measure]] type="[[Measure#CrossValidation|CrossValidation]]" target="0.01" errorFcn="rootRelativeSquareError" use="on"/>
      
      <!-- By default all inputs are modeled.  If you want to only model a couple of inputs you can specify an Inputs tag as follows: 
      
      <[[Config:Inputs|Inputs]]>
         <[[Config:Input|Input]] name="x" />
         <[[Config:Input|Input]] name="y" />
         // Setting a simulator input to a constant (default is 0):
         <[[Config:Input|Input]] name="z"  value="14.6"/>
      </[[Config:Inputs|Inputs]]>
      -->
      
      <!--          
      By default the toolbox will model every single output using a separate model.  If you want to change this
      e.g., you only want to model a specific output, or you want to use different settings for each output; then you
      can specify an Outputs tag.
      
      The following is an example for the Academic2DTwice problem used in this file.  Remember that if you change
      the problem you are modeling, you will have to change this section too.
      -->
      <[[Config:Outputs|Outputs]]>
         <[[Config:Output|Output]] name="out">
            <!--
                You can specify output specific configuration here
                
            <[[Config:SampleSelector|SampleSelector]]>lola</[[Config:SampleSelector|SampleSelector]]>
            <[[Config:AdaptiveModelBuilder|AdaptiveModelBuilder]]>rational</[[Config:AdaptiveModelBuilder|AdaptiveModelBuilder]]>
            <[[Config:Measure|Measure]] type="[[Measure#CrossValidation|CrossValidation]]" target=".01" errorFcn="meanSquareError" use="on" />
            -->
         </[[Config:Output|Output]]>
         
         <[[Config:Output|Output]] name="outinverse">
            <!--
            <[[Config:SampleSelector|SampleSelector]]>delaunay</[[Config:SampleSelector|SampleSelector]]>
            <[[Config:AdaptiveModelBuilder|AdaptiveModelBuilder]]>rbf</[[Config:AdaptiveModelBuilder|AdaptiveModelBuilder]]>
            <[[Config:Measure|Measure]] type="[[Measure#ValidationSet|ValidationSet]]" target=".05" use="on" />
            -->
         </[[Config:Output|Output]]>
         
      </[[Config:Outputs|Outputs]]>

      <!--   
         This is a more complex example of how you can have different configurations per output.
      -->
      <!--
      <[[Config:Outputs|Outputs]]>

         * Model the modulus of complex output S22 using cross-validation and the default model
         builder and sample selector.
         
         <[[Config:Output|Output]] name="S22" complexHandling="modulus">
            <[[Config:Measure|Measure]] type="[[Measure#CrossValidation|CrossValidation]]" target=".05" />
         </[[Config:Output|Output]]>
         
         * Model the real part of complex output S22, but introduce some normally-distributed noise
         (variance .01 by default).
         
         <[[Config:Output|Output]] name="S22" complexHandling="real">
            <[[Config:Measure|Measure]] type="[[Measure#CrossValidation|CrossValidation]]" target=".05" />
            * for other types of modifiers see the datamodifiers subdirectory
            <[[Config:Modifier|Modifier]] type="[[Modifier#Noise|Noise]]" />
         </[[Config:Output|Output]]>
      -->

       <!-- 
      More complex examples of how you can use measures:

      * 5-fold crossvalidation (warning expensive on some model types!)
      <[[Config:Measure|Measure]] type="[[Measure#CrossValidation|CrossValidation]]" target=".001" use="on">
         <Option key="folds" value="5"/>
      </[[Config:Measure|Measure]]>   

      * Using a validation set, the size taken as 20% of the available samples
      <[[Config:Measure|Measure]] type="[[Measure#ValidationSet|ValidationSet]]" target=".001" errorFcn="meanAbsoluteError">
         <Option key="percentUsed" value="20"/>
      </[[Config:Measure|Measure]]>

      * Using a validation set defined in an external file (scattered data)
             <[[Config:Measure|Measure]] type="[[Measure#ValidationSet|ValidationSet]]" target=".001">
            * the validation set come from a file
            <Option key="type" value="file"/>
            * the test data is scattered data so we need a scattered sample evaluator
            to load the data and evaluate the points. The filename is taken from the
            <[[Config:ScatteredDataFile|ScatteredDataFile]]> tag in the simulator xml file.
            Optionally you can specify an option with key "id" to specify a specifc
            dataset if there is more than one choice.
            <[[Config:SampleEvaluator|SampleEvaluator]]
            type="ibbt.sumo.sampleevaluators.datasets.ScatteredDatasetSampleEvaluator"/>
                     </[[Config:Measure|Measure]]>

      * Used for testing optimization problems
         * Calculates the (relative) error between the current minimum and a known minimum.
           Often one uses this just as a stopping criterion for benchmarking problems.
         * trueValue: a known global minimum
      <[[Config:Measure|Measure]] type="[[Measure#TestMinimum|TestMinimum]]" errorFcn="relativeError" trueValue="-5.0" target="0.1" use="on" />   
      -->
   </[[Config:Run|Run]]>