Difference between revisions of "Outputs"

From SUMOwiki
Jump to navigationJump to search
 
(22 intermediate revisions by 6 users not shown)
Line 1: Line 1:
There are three levels on which you can configure the way the different outputs in the simulator file are modelled.
+
There are three levels on which you can configure the way the different outputs in the simulator file are modeled.
  
== Default behaviour, all outputs modelled ==
+
== Default behavior, all outputs modeled ==
  
If no Outputs tag is defined in the configuration file, all outputs are modelled and evaluated with the default CrossValidation and MinMax (see [Measures]) measure and target accuracy 0.001. Complex outputs modelled directly.
+
If no Outputs tag is defined in the configuration file, all outputs are modeled and evaluated with the [[Measures|measure]] specified by the plan or the run. Complex outputs are modeled directly.  The default error function is rootRelativeSquareError.m.
  
== Default behaviour, selected outputs modelled ==
+
== Default behavior, selected outputs modeled ==
  
To change this default behaviour, one must specify an Outputs tag in the Run configuration. If you want to keep the default modelling behaviour of the toolbox (CrossValidation and MinMax, 0.001 accuracy), but you don't want to model each output, you only have to add an Output subelement for each output that you want modelled.
+
To change this default behavior, you must specify an <Outputs> tag in the Run configuration. Inside you specify an <Output> tag for each output you want to model.  As an example, the Academic2DTwice example has two outputs: 'out' and 'outinverse'. If you only want to model 'outinverse' you specify:
  
An example of such a configuration for the Academic2DTwice test function is:
+
<source lang="xml">
 +
<Outputs>
 +
<Output name="outinverse" />
 +
</Outputs>
 +
</source>
  
 +
If on the other hand you want to model both you can delete the <Outputs> tag all together and fall back to the default behavior, or you can put:
  
<pre><nowiki>
+
<source lang="xml">
 
<Outputs>
 
<Outputs>
 
<Output name="out" />
 
<Output name="out" />
 
<Output name="outinverse" />
 
<Output name="outinverse" />
 
</Outputs>
 
</Outputs>
</nowiki></pre>
+
</source>
  
 +
== Custom behavior for each output ==
  
== Custom behaviour for each output ==
+
=== Component customization ===
  
=== Component Customization ===
+
If you also want to change or fine-tune the behavior of the toolbox for each output separately, you can add subelements to each Output tag to customize the toolbox for that particular output. This allows you to use different sample selectors and/or model builders for each output, change the default measure, or combine multiple measures.
 
 
If you also want to change or fine-tune the behaviour of the toolbox for each output separately, you can add subelements to each Output tag to customize the toolbox for that particular output. This allows you to use different sample selectors and/or model builders for each output, change the default measure, or combine multiple measures.
 
  
 
Several examples of valid Output configurations can be found commented in default.xml.
 
Several examples of valid Output configurations can be found commented in default.xml.
Line 30: Line 34:
 
Here is an example of an output configuration for the Academic2DTwice test function:
 
Here is an example of an output configuration for the Academic2DTwice test function:
  
<pre><nowiki>
+
<source lang="xml">
 
<Outputs>
 
<Outputs>
 
<Output name="out">
 
<Output name="out">
<SampleSelector>gradient</SampleSelector>
+
<SequentialDesign>lola</SequentialDesign>
<Measure type="CrossValidation" target=".0001" use="on" />
+
<Measure type="CrossValidation" target=".01" errorFcn="meanSquareError" use="on" />
<AdaptiveModelBuilder>svm</AdaptiveModelBuilder>
+
<ModelBuilder>lssvmps</ModelBuilder>
 
</Output>
 
</Output>
 
 
 
<Output name="outinverse">
 
<Output name="outinverse">
<SampleSelector>grid</SampleSelector>
+
<SequentialDesign>delaunay</SequentialDesign>
<Measure type="CrossValidation" target=".05" use="on" />
+
<Measure type="CrossValidation" target=".05" errorFcn="maxRelativeError" use="on" />
 
<Measure type="MinMax" />
 
<Measure type="MinMax" />
 
</Output>
 
</Output>
 
</Outputs>
 
</Outputs>
</nowiki></pre>
+
</source>
  
This configuration models the first Output named "out" using the CrossValidation measure, but with increased target accuracy, and uses the gradient sample selector to select new samples. It builds models for this output using the svm model builder. The second output, named "outinverse", uses CrossValidation in combination with the MinMax measure, and uses the grid sample selector to select new samples for this output. It builds models using the model builder defined higher up in the xml hierarchy. As you can see, you can add multiple measures to one Output, so that each measure has something to say in deciding the accuracy of the models. However, you can only select one SampleSelector or AdaptiveModelBuilder.
+
This configuration models the first Output named "out" using the CrossValidation [[Measures|measure]] (using the meanSquareError function, and a target of 0.1) with the LOLA sample selector to select new samples. It builds models for this output using the lssvmps model builder. The second output, named "outinverse", uses CrossValidation in combination with the MinMax measure, and uses the delaunay sample selector to select new samples for this output. It builds models using the model builder defined higher up in the xml hierarchy. As you can see, you can add multiple measures to one Output, so that each measure has something to say in deciding the accuracy of the models. However, you can only select one SequentialDesign or odelBuilder.
  
If an Output element does not contain any Measure elements, that output is modelled using the default configuration, which is the CrossValidation measure and target accuracy 0.001.
+
For information on how multiple measures are handled and the configuration options of each measure, see [[Measures]].
 
 
For more information on how multiple measures are handled and the configuration options of each measure, see [[Measures]].
 
  
 
=== Complex handling ===
 
=== Complex handling ===
  
by default, a complex output is treated as such and is passed in its original form to all the components of the toolbox. However, some components, do not support complex numbers directly, and are therefore incompatible with the default setting.
+
By default, a complex output is treated as such and is passed in its original form to all the components of the toolbox. However, some components, do not support complex numbers directly and are therefore incompatible with the default setting (they will give an error). In order to get these components to work with complex outputs, the outputs will have to be pre-processed. This is done by changing the complexHandling attribute of the Output. There are 3 valid values for this attribute:
 
 
The following components are at this moment incapable of handling complex outputs:
 
* Model builders using neural networks (annbatch, anngenetic, ...)
 
* Model builders using (LS)SVM (svmbatch, svmgenetic, lssvmgenetic, ...)
 
* Model builders using RBF neural networks (rbfnnbatch, rbfnngenetic, ...)
 
* Model builders using Splines (splines, splinesgenetic, ...)
 
*
 
 
 
In order to get these components to work with complex outputs, the outputs will have to be pre-processed. This is done by changing the complexHandling attribute of the Output. There are 3 valid values for this attribute:
 
  
 
#'''complex''': Complex outputs are treated as is. Will not work with components in the aforementioned list.
 
#'''complex''': Complex outputs are treated as is. Will not work with components in the aforementioned list.
#'''split''': Complex numbers are split in real and imaginary parts, and each part is modelled separately. This means that two models will be built, and the toolbox will have twice the work it would normally have. Also any correlation between the real and imaginary part is lost.
+
#'''split''': Complex numbers are split in real and imaginary parts, and each part is modeled separately. This means that two models will be built, and the toolbox will have twice the work it would normally have. Also any correlation between the real and imaginary part is lost.
#'''real''': Only the real part of the complex number is modelled, the imaginary part is discarded.
+
#'''real''': Only the real part of the complex number is modeled, the imaginary part is discarded.
#'''imaginary''': Only the imaginary part of the complex number is modelled, the real part is discarded. The imaginary part is treated as one real output.
+
#'''imaginary''': Only the imaginary part of the complex number is modeled, the real part is discarded. The imaginary part is treated as one real output.
#'''modulus''': The modulus of the complex number is modelled instead of the original number. Since the modulus is a real number, it can be modelled using all the components available.
+
#'''modulus''': The modulus of the complex number is modeled instead of the original number. Since the modulus is a real number, it can be modeled using all the components available.
  
 
A full-blown example for the InductivePosts test function that uses all the options mentioned in this article can be found below:
 
A full-blown example for the InductivePosts test function that uses all the options mentioned in this article can be found below:
  
<pre><nowiki>
+
<source lang="xml">
<!-- Model outputs S11 and S12 using the gradient sample selector (overwrites the beforementioned setting).
+
<!-- Model outputs S11 and S12 using the LOLA sample selector (overwrites the beforementioned setting).
 
Uses cross-validation for model evaluation. Both complex outputs are treated as two separate real numbers
 
Uses cross-validation for model evaluation. Both complex outputs are treated as two separate real numbers
with no relation to each other, resulting in total in 4 reals that will be modelled for these two complex outputs. -->
+
with no relation to each other, resulting in total in 4 reals that will be modeled for these two complex outputs. -->
 
 
 
<Output name="S11,S12" complexHandling="real">
 
<Output name="S11,S12" complexHandling="real">
<SampleSelector>gradient</SampleSelector>
+
<SequentialDesign>lola</SequentialDesign>
 
<Measure type="CrossValidation" target=".01" />
 
<Measure type="CrossValidation" target=".01" />
 
</Output>
 
</Output>
  
<!-- Model output S12 as a complex number using the grid sample selector. Models are evaluated using the TestSamples measure.
+
<!-- Model output S12 as a complex number using the error sample selector. Models are evaluated using the ValidationSet measure.
 
The CrossValidation accuracy of the model is also calculated for comparison purposes, but isn't used in the modeling algorithm. -->
 
The CrossValidation accuracy of the model is also calculated for comparison purposes, but isn't used in the modeling algorithm. -->
 
 
 
<Output name="S12" complexHandling="complex">
 
<Output name="S12" complexHandling="complex">
<SampleSelector>error</SampleSelector>
+
<SequentialDesign>error</SequentialDesign>
<AdaptiveModelBuilder>svmps</AdaptiveModelBuilder>
+
<ModelBuilder>svmps</ModelBuilder>
 
<Measure type="ValidationSet" target=".05" />
 
<Measure type="ValidationSet" target=".05" />
 
<Measure type="CrossValidation" target=".05" use="off" />
 
<Measure type="CrossValidation" target=".05" use="off" />
Line 98: Line 91:
 
<Measure type="CrossValidation" target=".05" />
 
<Measure type="CrossValidation" target=".05" />
 
</Output>
 
</Output>
</nowiki></pre>
+
</source>
 +
 
 +
== Multi-Output Modeling ==
 +
 
 +
See the page [[Running#Models_with_multiple_outputs]].

Latest revision as of 09:52, 28 February 2014

There are three levels on which you can configure the way the different outputs in the simulator file are modeled.

Default behavior, all outputs modeled

If no Outputs tag is defined in the configuration file, all outputs are modeled and evaluated with the measure specified by the plan or the run. Complex outputs are modeled directly. The default error function is rootRelativeSquareError.m.

Default behavior, selected outputs modeled

To change this default behavior, you must specify an <Outputs> tag in the Run configuration. Inside you specify an <Output> tag for each output you want to model. As an example, the Academic2DTwice example has two outputs: 'out' and 'outinverse'. If you only want to model 'outinverse' you specify:

<Outputs>
	<Output name="outinverse" />
</Outputs>

If on the other hand you want to model both you can delete the <Outputs> tag all together and fall back to the default behavior, or you can put:

<Outputs>
	<Output name="out" />
	<Output name="outinverse" />
</Outputs>

Custom behavior for each output

Component customization

If you also want to change or fine-tune the behavior of the toolbox for each output separately, you can add subelements to each Output tag to customize the toolbox for that particular output. This allows you to use different sample selectors and/or model builders for each output, change the default measure, or combine multiple measures.

Several examples of valid Output configurations can be found commented in default.xml.

Here is an example of an output configuration for the Academic2DTwice test function:

<Outputs>
	<Output name="out">
		<SequentialDesign>lola</SequentialDesign>
		<Measure type="CrossValidation" target=".01" errorFcn="meanSquareError" use="on" />
		<ModelBuilder>lssvmps</ModelBuilder>
	</Output>
	
	<Output name="outinverse">
		<SequentialDesign>delaunay</SequentialDesign>
		<Measure type="CrossValidation" target=".05" errorFcn="maxRelativeError" use="on" />
		<Measure type="MinMax" />
	</Output>
</Outputs>

This configuration models the first Output named "out" using the CrossValidation measure (using the meanSquareError function, and a target of 0.1) with the LOLA sample selector to select new samples. It builds models for this output using the lssvmps model builder. The second output, named "outinverse", uses CrossValidation in combination with the MinMax measure, and uses the delaunay sample selector to select new samples for this output. It builds models using the model builder defined higher up in the xml hierarchy. As you can see, you can add multiple measures to one Output, so that each measure has something to say in deciding the accuracy of the models. However, you can only select one SequentialDesign or odelBuilder.

For information on how multiple measures are handled and the configuration options of each measure, see Measures.

Complex handling

By default, a complex output is treated as such and is passed in its original form to all the components of the toolbox. However, some components, do not support complex numbers directly and are therefore incompatible with the default setting (they will give an error). In order to get these components to work with complex outputs, the outputs will have to be pre-processed. This is done by changing the complexHandling attribute of the Output. There are 3 valid values for this attribute:

  1. complex: Complex outputs are treated as is. Will not work with components in the aforementioned list.
  2. split: Complex numbers are split in real and imaginary parts, and each part is modeled separately. This means that two models will be built, and the toolbox will have twice the work it would normally have. Also any correlation between the real and imaginary part is lost.
  3. real: Only the real part of the complex number is modeled, the imaginary part is discarded.
  4. imaginary: Only the imaginary part of the complex number is modeled, the real part is discarded. The imaginary part is treated as one real output.
  5. modulus: The modulus of the complex number is modeled instead of the original number. Since the modulus is a real number, it can be modeled using all the components available.

A full-blown example for the InductivePosts test function that uses all the options mentioned in this article can be found below:

<!-- Model outputs S11 and S12 using the LOLA sample selector (overwrites the beforementioned setting).
Uses cross-validation for model evaluation. Both complex outputs are treated as two separate real numbers
with no relation to each other, resulting in total in 4 reals that will be modeled for these two complex outputs. -->
				
<Output name="S11,S12" complexHandling="real">
	<SequentialDesign>lola</SequentialDesign>
	<Measure type="CrossValidation" target=".01" />
</Output>

<!-- Model output S12 as a complex number using the error sample selector. Models are evaluated using the ValidationSet measure.
The CrossValidation accuracy of the model is also calculated for comparison purposes, but isn't used in the modeling algorithm. -->
				
<Output name="S12" complexHandling="complex">
	<SequentialDesign>error</SequentialDesign>
	<ModelBuilder>svmps</ModelBuilder>
	<Measure type="ValidationSet" target=".05" />
	<Measure type="CrossValidation" target=".05" use="off" />
</Output>

<!-- Model the modulus of complex output S22 using cross-validation and the default model builder and sample selector. -->

<Output name="S22" complexHandling="modulus">
	<Measure type="CrossValidation" target=".05" />
</Output>

Multi-Output Modeling

See the page Running#Models_with_multiple_outputs.