Difference between revisions of "Config:DataSource"

From SUMOwiki
Jump to navigationJump to search
m (Javdrher moved page Config:SampleEvaluator to Config:DataSource: Component was renamed)
 
(16 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 +
This page lists the various SampleEvaluators used by the SUMO Toolbox. To find out more about the data formats and how to define your own data generating code go [[Interfacing_with_the_toolbox|here]].
 +
 +
 +
'''Generated for SUMO toolbox version 7.0'''.
 +
''We are well aware that documentation is not always complete and possibly even out of date in some cases. We try to document everything as best we can but much is limited by available time and manpower.  We are are a university research group after all. The most up to date documentation can always be found (if not here) in the default.xml configuration file and, of course, in the source files.  If something is unclear please dont hesitate to [[Reporting problems|ask]].''
 
== SampleEvaluator ==
 
== SampleEvaluator ==
  
=== LocalSampleEvaluator ===
+
=== local ===
 
+
Use this if you data generator is a native executable, shell script, or java class
Evaluate samples locally using a native executable/script or a java class
 
 
<source xmlns:saxon="http://icl.com/saxon" lang="xml">
 
<source xmlns:saxon="http://icl.com/saxon" lang="xml">
<[[Config:SampleEvaluator|SampleEvaluator]] id="local" type="ibbt.sumo.SampleEvaluators.LocalSampleEvaluator">
+
<[[Config:SampleEvaluator|SampleEvaluator]] id="local" type="ibbt.sumo.sampleevaluators.LocalSampleEvaluator">
 
   <!-- Maximum number of times to resubmit a point (e.g., in case something went wrong) -->
 
   <!-- Maximum number of times to resubmit a point (e.g., in case something went wrong) -->
 
   <Option key="maxResubmissions" value="1"/>
 
   <Option key="maxResubmissions" value="1"/>
Line 12: Line 16:
 
   <Option key="sampleTimeout" value="-1"/>
 
   <Option key="sampleTimeout" value="-1"/>
 
   <!-- Can be set to "java" for java executables, to "external" for platform-specific
 
   <!-- Can be set to "java" for java executables, to "external" for platform-specific
         binaries and to nothing at all for auto-detection.-->
+
         binaries/scripts and to nothing at all for auto-detection.-->
 
   <Option key="simulatorType" value=""/>
 
   <Option key="simulatorType" value=""/>
 
   <!-- Can be set to a number of seconds, if one simulator evaluation exceeds this timeout,
 
   <!-- Can be set to a number of seconds, if one simulator evaluation exceeds this timeout,
Line 21: Line 25:
 
</[[Config:SampleEvaluator|SampleEvaluator]]>
 
</[[Config:SampleEvaluator|SampleEvaluator]]>
 
</source>
 
</source>
 +
=== matlabOld ===
 +
Evaluate samples using a matlab script (ie. your simulator is a matlab script). The evaluation is handled via the Java side of the toolbox.
  
=== MatlabSampleEvaluator ===
 
 
Evaluate samples using a matlab script (ie. your simulator is a matlab script)
 
 
<source xmlns:saxon="http://icl.com/saxon" lang="xml">
 
<source xmlns:saxon="http://icl.com/saxon" lang="xml">
<[[Config:SampleEvaluator|SampleEvaluator]] id="matlab" type="ibbt.sumo.SampleEvaluators.matlab.MatlabSampleEvaluator">
+
<[[Config:SampleEvaluator|SampleEvaluator]] id="matlab" type="ibbt.sumo.sampleevaluators.matlab.MatlabSampleEvaluator">
 
   <!-- Maximum number of times to resubmit a point (e.g., in case something went wrong) -->
 
   <!-- Maximum number of times to resubmit a point (e.g., in case something went wrong) -->
 
   <Option key="maxResubmissions" value="1"/>
 
   <Option key="maxResubmissions" value="1"/>
Line 35: Line 38:
 
</source>
 
</source>
  
=== MatlabClassSampleEvaluator ===
+
=== matlab ===
 +
Evaluate samples using a matlab script '''without''' using Java. This is the default evaluator of Matlab m-files.
  
Evaluate samples using a matlab class
 
 
<source xmlns:saxon="http://icl.com/saxon" lang="xml">
 
<source xmlns:saxon="http://icl.com/saxon" lang="xml">
<[[Config:SampleEvaluator|SampleEvaluator]] id="matlabClass" type="ibbt.sumo.SampleEvaluators.matlab.MatlabClassSampleEvaluator">
+
<SampleEvaluator id="matlabDirect" type="MatlabDirectSampleEvaluator"/>
  <!-- Maximum number of times to resubmit a point (e.g., in case something went wrong) -->
 
  <Option key="maxResubmissions" value="1"/>
 
  <!-- If a sample takes longer than "sampleTimeout*average evaluation time of one sample"
 
        seconds to evaluate it is removed from the pending list (set to -1 to disable) -->
 
  <Option key="sampleTimeout" value="-1"/>
 
</[[Config:SampleEvaluator|SampleEvaluator]]>
 
 
</source>
 
</source>
  
=== GriddedDatasetSampleEvaluator ===
+
=== griddedDataset ===
 +
Evaluate samples using a gridded dataset. This data format does not include any inputs, but lists only outputs and only work for a uniform grid of data points. The order in which the output values are given determine their location in the grid.
 +
 
 +
For example, if you want to define a 3-dimensional dataset with grid size 2x3x2 on the [-1,1] domain, you must provide the outputs for the samples in the following order:
 +
 
 +
<code><pre>
 +
value at [-1, -1, -1]
 +
value at [-1, -1,  1]
 +
value at [-1,  0, -1]
 +
value at [-1,  0,  1]
 +
value at [-1,  1, -1]
 +
value at [-1,  1,  1]
 +
value at [ 1, -1, -1]
 +
value at [ 1, -1,  1]
 +
value at [ 1,  0, -1]
 +
value at [ 1,  0,  1]
 +
value at [ 1,  1, -1]
 +
value at [ 1,  1,  1]
 +
</pre></code>
  
Evaluate samples using a gridded dataset
 
 
<source xmlns:saxon="http://icl.com/saxon" lang="xml">
 
<source xmlns:saxon="http://icl.com/saxon" lang="xml">
<[[Config:SampleEvaluator|SampleEvaluator]] id="gridded" type="ibbt.sumo.SampleEvaluators.datasets.GriddedDatasetSampleEvaluator">
+
<[[Config:SampleEvaluator|SampleEvaluator]] id="griddedDataset" type="ibbt.sumo.sampleevaluators.datasets.GriddedDatasetSampleEvaluator">
 
   <!-- Using an ID you can specify which dataset from the simulator file to use -->
 
   <!-- Using an ID you can specify which dataset from the simulator file to use -->
 
   <!-- <Option key="id" value="someDataset"/> -->
 
   <!-- <Option key="id" value="someDataset"/> -->
Line 58: Line 72:
 
</source>
 
</source>
  
=== ScatteredDatasetSampleEvaluator ===
+
=== scatteredDataset ===
Evaluate samples using a scattered dataset
+
Evaluate samples using a scattered dataset. Each row of the dataset represents a data point. If the dimensionality of the problem is D (i.e. there are D inputs), the first D columns represent the inputs and the remaining columns the outputs. Note that complex number need to given as separate columns.
 +
 
 
<source xmlns:saxon="http://icl.com/saxon" lang="xml">
 
<source xmlns:saxon="http://icl.com/saxon" lang="xml">
<[[Config:SampleEvaluator|SampleEvaluator]] id="scattered" type="ibbt.sumo.SampleEvaluators.datasets.ScatteredDatasetSampleEvaluator">
+
<[[Config:SampleEvaluator|SampleEvaluator]] id="scatteredDataset" type="ibbt.sumo.sampleevaluators.datasets.ScatteredDatasetSampleEvaluator">
 
   <!-- Using an ID you can specify which dataset from the simulator file to use -->
 
   <!-- Using an ID you can specify which dataset from the simulator file to use -->
 
   <!-- <Option key="id" value="someDataset"/> -->
 
   <!-- <Option key="id" value="someDataset"/> -->
Line 70: Line 85:
 
Evaluate samples on a SGE administered cluster through a remote, ssh reachable frontnode
 
Evaluate samples on a SGE administered cluster through a remote, ssh reachable frontnode
 
<source xmlns:saxon="http://icl.com/saxon" lang="xml">
 
<source xmlns:saxon="http://icl.com/saxon" lang="xml">
<[[Config:SampleEvaluator|SampleEvaluator]] id="calcua" type="ibbt.sumo.SampleEvaluators.sge.RemoteSGESampleEvaluator">
+
<[[Config:SampleEvaluator|SampleEvaluator]] id="calcua" type="ibbt.sumo.sampleevaluators.distributed.sge.RemoteSGESampleEvaluator">
 
   <!-- Maximum number of times to resubmit a point (e.g., in case something went wrong) -->
 
   <!-- Maximum number of times to resubmit a point (e.g., in case something went wrong) -->
 
   <Option key="maxResubmissions" value="1"/>
 
   <Option key="maxResubmissions" value="1"/>
Line 78: Line 93:
 
    
 
    
 
   <!--The platform specs of the cluster-->
 
   <!--The platform specs of the cluster-->
   <[[Config:Executable|Executable]] platform="unix" arch="x86"/>
+
   <[[Config:Executable|Executable]] platform="linux" arch="x86_64"/>
   <[[Config:Backend|Backend]] id="remoteSGE" type="ibbt.sumo.SampleEvaluators.sge.RemoteSGEBackend">
+
   <[[Config:Backend|Backend]] id="remoteSGE" type="ibbt.sumo.sampleevaluators.distributed.sge.RemoteSGEBackend">
 
       <!--ssh user name to login on the front node-->
 
       <!--ssh user name to login on the front node-->
 
       <Option key="user" value="dgorisse"/>
 
       <Option key="user" value="dgorisse"/>
Line 89: Line 104:
 
       <Option key="pollInterval" value="20"/>
 
       <Option key="pollInterval" value="20"/>
 
       <!--queues we can submit to-->
 
       <!--queues we can submit to-->
       <Option key="queues" value="testqueue,normalprio,myrinet,cmpl_wis-inf"/>
+
       <Option key="queues" value="all.q,fast.q"/>
 
       <!--check for faster queues (more slots available) every xx seconds-->
 
       <!--check for faster queues (more slots available) every xx seconds-->
 
       <Option key="queueRevisionRate" value="10"/>
 
       <Option key="queueRevisionRate" value="10"/>
 
       <!--script on the frontnode that sets up the necessary shell environment-->
 
       <!--script on the frontnode that sets up the necessary shell environment-->
 
       <Option key="environmentCommand" value=". ~/.profile;"/>
 
       <Option key="environmentCommand" value=". ~/.profile;"/>
  </[[Config:Backend|Backend]]>
 
</[[Config:SampleEvaluator|SampleEvaluator]]>
 
</source>
 
=== begrid ===
 
Evaluate samples on a LCG administered grid through a remote, ssh reachable frontnode
 
<source xmlns:saxon="http://icl.com/saxon" lang="xml">
 
<[[Config:SampleEvaluator|SampleEvaluator]] id="begrid" type="ibbt.sumo.SampleEvaluators.edg.RemoteEDGSampleEvaluator">
 
  <!-- Maximum number of times to resubmit a point (e.g., in case something went wrong) -->
 
  <Option key="maxResubmissions" value="1"/>
 
  <!-- If a sample takes longer than "sampleTimeout*average evaluation time of one sample"
 
        seconds to evaluate it is removed from the pending list (set to -1 to disable) -->
 
  <Option key="sampleTimeout" value="-1"/>
 
 
  <[[Config:Executable|Executable]] platform="unix" arch="x86"/>
 
  <[[Config:Backend|Backend]] id="remoteEDG" type="ibbt.sumo.SampleEvaluators.edg.RemoteEDGBackend">
 
      <Option key="user" value="dgorissen"/>
 
      <Option key="frontNode" value="ui01.cmi.ua.ac.be"/>
 
      <Option key="remoteDirectory" value="/home/edp/dgorissen/JobOutput"/>
 
      <Option key="pollInterval" value="25"/>
 
      <!--Virtual Organization to use-->
 
      <Option key="vo" value="betest"/>
 
      <!--script on the frontnode that sets up the necessary shell environment-->
 
      <Option key="environmentCommand" value=". /etc/profile;"/>
 
 
   </[[Config:Backend|Backend]]>
 
   </[[Config:Backend|Backend]]>
 
</[[Config:SampleEvaluator|SampleEvaluator]]>
 
</[[Config:SampleEvaluator|SampleEvaluator]]>
 
</source>
 
</source>

Latest revision as of 16:20, 27 February 2014

This page lists the various SampleEvaluators used by the SUMO Toolbox. To find out more about the data formats and how to define your own data generating code go here.


Generated for SUMO toolbox version 7.0. We are well aware that documentation is not always complete and possibly even out of date in some cases. We try to document everything as best we can but much is limited by available time and manpower. We are are a university research group after all. The most up to date documentation can always be found (if not here) in the default.xml configuration file and, of course, in the source files. If something is unclear please dont hesitate to ask.

SampleEvaluator

local

Use this if you data generator is a native executable, shell script, or java class

<[[Config:SampleEvaluator|SampleEvaluator]] id="local" type="ibbt.sumo.sampleevaluators.LocalSampleEvaluator">
   <!-- Maximum number of times to resubmit a point (e.g., in case something went wrong) -->
   <Option key="maxResubmissions" value="1"/>
   <!-- If a sample takes longer than "sampleTimeout*average evaluation time of one sample" 
   seconds to evaluate it is removed from the pending list (set to -1 to disable) -->
   <Option key="sampleTimeout" value="-1"/>
   <!-- Can be set to "java" for java executables, to "external" for platform-specific
        binaries/scripts and to nothing at all for auto-detection.-->
   <Option key="simulatorType" value=""/>
   <!-- Can be set to a number of seconds, if one simulator evaluation exceeds this timeout,
        the simulation is aborted -->
   <!-- <Option key="timeout" value="12"/> -->
   <!-- Number of samples to evaluate concurrently, useful if you have a dual or multi-core machine -->
   <Option key="threadCount" value="1"/>
</[[Config:SampleEvaluator|SampleEvaluator]]>

matlabOld

Evaluate samples using a matlab script (ie. your simulator is a matlab script). The evaluation is handled via the Java side of the toolbox.

<[[Config:SampleEvaluator|SampleEvaluator]] id="matlab" type="ibbt.sumo.sampleevaluators.matlab.MatlabSampleEvaluator">
   <!-- Maximum number of times to resubmit a point (e.g., in case something went wrong) -->
   <Option key="maxResubmissions" value="1"/>
   <!-- If a sample takes longer than "sampleTimeout*average evaluation time of one sample" 
   seconds to evaluate it is removed from the pending list (set to -1 to disable) -->
   <Option key="sampleTimeout" value="-1"/>
</[[Config:SampleEvaluator|SampleEvaluator]]>

matlab

Evaluate samples using a matlab script without using Java. This is the default evaluator of Matlab m-files.

	<SampleEvaluator id="matlabDirect" type="MatlabDirectSampleEvaluator"/>

griddedDataset

Evaluate samples using a gridded dataset. This data format does not include any inputs, but lists only outputs and only work for a uniform grid of data points. The order in which the output values are given determine their location in the grid.

For example, if you want to define a 3-dimensional dataset with grid size 2x3x2 on the [-1,1] domain, you must provide the outputs for the samples in the following order:

value at [-1, -1, -1]
value at [-1, -1,  1]
value at [-1,  0, -1]
value at [-1,  0,  1]
value at [-1,  1, -1]
value at [-1,  1,  1]
value at [ 1, -1, -1]
value at [ 1, -1,  1]
value at [ 1,  0, -1]
value at [ 1,  0,  1]
value at [ 1,  1, -1]
value at [ 1,  1,  1]
<[[Config:SampleEvaluator|SampleEvaluator]] id="griddedDataset" type="ibbt.sumo.sampleevaluators.datasets.GriddedDatasetSampleEvaluator">
   <!-- Using an ID you can specify which dataset from the simulator file to use -->
   <!-- <Option key="id" value="someDataset"/> -->
</[[Config:SampleEvaluator|SampleEvaluator]]>

scatteredDataset

Evaluate samples using a scattered dataset. Each row of the dataset represents a data point. If the dimensionality of the problem is D (i.e. there are D inputs), the first D columns represent the inputs and the remaining columns the outputs. Note that complex number need to given as separate columns.

<[[Config:SampleEvaluator|SampleEvaluator]] id="scatteredDataset" type="ibbt.sumo.sampleevaluators.datasets.ScatteredDatasetSampleEvaluator">
   <!-- Using an ID you can specify which dataset from the simulator file to use -->
   <!-- <Option key="id" value="someDataset"/> -->
</[[Config:SampleEvaluator|SampleEvaluator]]>

calcua

Evaluate samples on a SGE administered cluster through a remote, ssh reachable frontnode

<[[Config:SampleEvaluator|SampleEvaluator]] id="calcua" type="ibbt.sumo.sampleevaluators.distributed.sge.RemoteSGESampleEvaluator">
   <!-- Maximum number of times to resubmit a point (e.g., in case something went wrong) -->
   <Option key="maxResubmissions" value="1"/>
   <!-- If a sample takes longer than "sampleTimeout*average evaluation time of one sample" 
        seconds to evaluate it is removed from the pending list (set to -1 to disable) -->
   <Option key="sampleTimeout" value="-1"/>
   
   <!--The platform specs of the cluster-->
   <[[Config:Executable|Executable]] platform="linux" arch="x86_64"/>
   <[[Config:Backend|Backend]] id="remoteSGE" type="ibbt.sumo.sampleevaluators.distributed.sge.RemoteSGEBackend">
      <!--ssh user name to login on the front node-->
      <Option key="user" value="dgorisse"/>
      <!--Submissions happen from this front node, you need to have key-based ssh authentication-->
      <Option key="frontNode" value="submit.calcua.ua.ac.be"/>
      <!--Directory on the front node where input/output files, dependencies, etc. are stored-->
      <Option key="remoteDirectory" value="/storeA/users/dgorisse/output"/>
      <!--poll for result files every xx seconds-->
      <Option key="pollInterval" value="20"/>
      <!--queues we can submit to-->
      <Option key="queues" value="all.q,fast.q"/>
      <!--check for faster queues (more slots available) every xx seconds-->
      <Option key="queueRevisionRate" value="10"/>
      <!--script on the frontnode that sets up the necessary shell environment-->
      <Option key="environmentCommand" value=". ~/.profile;"/>
   </[[Config:Backend|Backend]]>
</[[Config:SampleEvaluator|SampleEvaluator]]>