http://sumowiki.intec.ugent.be/api.php?action=feedcontributions&user=Sprashant&feedformat=atomSUMOwiki - User contributions [en]2019-08-26T00:48:50ZUser contributionsMediaWiki 1.19.20+dfsg-2.3http://sumowiki.intec.ugent.be/Useful_LinksUseful Links2015-06-17T07:58:31Z<p>Sprashant: /* Journal Links */</p>
<hr />
<div>=== Related publications ===<br />
<br />
See the [[Related publications]] page.<br />
<br />
=== Related projects ===<br />
<br />
A list of projects with similar ideas and scope.<br />
* [http://www.cs.rtu.lv/jekabsons/regression.html] Matlab regression software<br />
* [http://sites.google.com/site/gptips4matlab GTIPS] Matlab symbolic regression software using genetic programming<br />
* [http://www.optiy.eu OptiY]<br />
* [http://www.nutechsolutions.com/prod_cv_analytics.asp ClearVu Analytics]<br />
* [https://sites.google.com/site/felipeacviana/ Surrogates Toolbox]: Another matlab surrogate modeling toolbox<br />
* [http://www.evolved-analytics.com Evolved-Analytics]: DataModeler package.<br />
* [http://www.muneda.com/ MunEDA]: MunEDA provides leading EDA software technology for analysis, modelling and optimization of yield and performance of analog, mixed-signal and digital designs.<br />
* [http://www.lsoptsupport.com/faqs/setting-parameters-for-metamodel-based-optimization-strategies LS-OPT: Functionality similar to the SUMO-Toolbox in LS-OPT]<br />
* [http://mucm.aston.ac.uk/MUCM/MUCMToolkit/index.php?page=MetaHomePage.html MUCH Toolkit]: A toolbox for sensitivity analysis using surrogate models<br />
* [http://home.mit.bme.hu/~kollar/topics/fdident.html FDIDENT: frequency domain identification toolbox]<br />
* [http://www.dmoz.org/Science/Math/Statistics/Software/Regression_and_Curve_Fitting/: Regression software on Open Directory]<br />
* [http://www.infiniscale.com/ Infiniscale]: TechModeler, TechAnalyzer: automatic model generation tools for EM applications<br />
* [http://www.vrand.com/visualDOC.html visualDOC]: VisualDOC, a design optimization tool<br />
* [http://www.salford-systems.com/mars.php MARS]: A spline based modeling tool<br />
* [http://www.phoenix-int.com/software/phx_modelcenter.php Modelcenter]: datamining and modeling tool<br />
* [http://www.simulia.com/products/isight2.html iSIGHT]: datamining and modeling tool<br />
* [http://www.tmpinc.com/datascape_overview.html Datascape]: datamining and modeling tool<br />
* [http://www.friendship-systems.com/ Friendship systems]: Tools for modeling ship hulls parametrically and performing the modeling calculations (Equilibrium tool)<br />
* [http://www.csse.monash.edu.au/%7Edavida/nimrod/nimrodg.htm Nimrod/G]: Execute parameter sweeps on the grid<br />
* [http://www-sop.inria.fr/oasis/ProActive/ ProActive]: Java grid library <br />
* [http://www.csse.monash.edu.au/%7Edavida/nimrod/nimrodo.htm Nimrod/O]: Grid-enabled optimization toolkit <br />
* [http://www.ece.northwestern.edu/OTC/ NEOS]: Distributed optimization toolkit <br />
* [http://software.sci.utah.edu/scirun.html SciRun]: Problem Solving Environment (PSE), for simulation, modeling, and visualization of scientific problems. <br />
* [http://www.esteco.it/ ModeFRONTIER]: environment dedicated to the set up of design assessment chains and efficient investigation of the design space. <br />
* [http://www.gridbus.org/ Gridbus]: metascheduler for the grid <br />
* [http://icl.cs.utk.edu/netsolve/overview/ Gridsolve/Netsolve]: grid enabled scientific computation toolbox <br />
* [http://www.dtreg.com/ DTREG]: a powerful statistical analysis program that generates classification and regression trees and Support Vector Machine models that can be used to predict parameter values. <br />
* [http://www.soton.ac.uk/%7Epbn/MDO/ MDO]: A collection of MDO links <br />
* [http://www.research.att.com/~njas/gosset/index.html GOSSET]: A general purpose program for designing experiments <br />
* [http://www.cs.sandia.gov/DAKOTA/ DAKOTA]: Design Analysis Kit for Optimization and Terascale Applications<br />
* [http://www.cs.cf.ac.uk/gridprojects/dipso/ DIPSO]: Wide-Area Distributed Problem Solving (DIPSO) <br />
* [http://www.geodise.org GEODISE]: Grid Enabled Optimisation and Design Search for Engineering<br />
* [http://www.fast.u-psud.fr/ezyfit/ EZfit] : Free curve fitting toolbox for matlab<br />
* [http://www.ians.uni-stuttgart.de/spinterp/about.html SGIT] : A Sparse grid interpolation toolbox<br />
* [http://www.csie.ntu.edu.tw/~yien/quickrbf/quickstart.php QuickRBF] : an RBF fitting library (native)<br />
* [http://www.farfieldtechnology.com/products/toolbox/ FastRBF] : another RBF fitting library (matlab)<br />
* [http://promethee.irsn.org PROMETHEE project: easy parametric modeling]<br />
* [http://simlab.jrc.ec.europa.eu/ Sensitivity Analysis library]<br />
* Gaussian Process Matlab code<br />
** [http://www.gaussianprocess.org/gpml/code/matlab/doc/ Code] based on Rasmussen's book<br />
** [http://www.cs.man.ac.uk/~neill/gp/ Gaussian Process Software]<br />
** [http://www.ios.htwg-konstanz.de/joomla_mof/index.php?option=com_content&view=article&id=48:polyreg-polynomial-gaussian-process-regression&catid=36:code&Itemid=81 GP] using polynomial covariance functions<br />
* [http://shark-project.sourceforge.net/ Shark Machine Learning Library - An open source machine learning bundle]<br />
<br />
=== Related labs ===<br />
<br />
A list of some of the labs/researchers with similar ideas and scope.<br />
<br />
* [http://aerospace.engin.umich.edu/index.html Department of Aerospace Engineering at the University of Michigan]<br />
* [http://www.mae.ufl.edu/~mdo/ The Structural and Multidisciplinary Optimization Group at the University of Florida]<br />
* [http://web.engr.oregonstate.edu/~tgd/ School of Electrical Engineering and Computer Science, Oregon State U]<br />
* [http://www.soton.ac.uk/engineering/research/groups/ced.page Southampton University Computational Engineering and Design Group]<br />
* [http://edog.mne.psu.edu/ Engineering Design & Optimization Group (Penn state)]<br />
* [http://nd.edu/~ame/ Aerospace and Mechanical Engineering] [http://www.gano.name/shawn/ Homepage]<br />
* [http://shyylab.engin.umich.edu/research/design-optimization Computational Thermo-Fluids Group]<br />
* [http://www.ensc.sfu.ca/~gwa5/index.htm Product Design and Optimization Laboratory (PDOL)]<br />
* [http://www.cerfacs.fr/4-25708-Home.php Computational Fluid Dynamics (CFD) group at CERFACS]<br />
* [http://webuser.uni-weimar.de/~roos1/ Dirk Roos]<br />
* [http://www.cerfacs.fr/~duchaine/HTML/research.htm Florent Duchaine]<br />
* http://www.nlr.nl/ National aerospace lab]<br />
<br />
=== Data sets - Simulation code ===<br />
<br />
A list of publicly available datasets and simulation codes, useful for testing.<br />
<br />
* [http://www.mat.univie.ac.at/~neum/stat.html Statistics links] : A nice collection of data fitting and analysis codes<br />
* [http://en.wikipedia.org/wiki/Classic_data_sets Wikipedia Classic Datasets]<br />
* [http://www.uni-koeln.de/themen/Statistik/data/rousseeuw/ The ROUSSEEUW datasets]<br />
* [ftp://ftp.sas.com/pub/neural/dojo/dojo.html Donoho-Johnstone Benchmarks] <br />
* [http://people.scs.fsu.edu/~burkardt/datasets/datasets.html datasets]<br />
* [http://www-syscom.univ-mlv.fr/~vignat/Signal/Space/index.html Spice like simulator for matlab]<br />
* [http://www.itl.nist.gov/div898/strd/general/dataarchive.html Nist dataset archive]<br />
* [http://homes.esat.kuleuven.be/~smc/daisy/daisydata.html Daisy datasets]<br />
* [http://lib.stat.cmu.edu/datasets/ Statlib dataset archive]<br />
* [http://www.iau.dtu.dk/nnbook/systems.html Datasets from the book "Neural networks for the modeling and control of dynamic systems.]<br />
* [http://www.gpc.de/e_poses.html A simulation environment for production and transport, logistics and automation systems]<br />
* [http://shumway.physics.asu.edu/codes.html Nanostructure Simulation and Modeling Programs]<br />
* [http://pedsim.silmaril.org/ A Modular, Distributed Pedestrian Crowd Simulation System]<br />
* [http://g95.sourceforge.net/g95_status.html Fortran simulation codes]<br />
* [http://www.genie.ac.uk/ Grid ENabled Integrated Earth system model]<br />
* [http://gcmd.nasa.gov/KeywordSearch/Home.do?Portal=GCMD&MetadataType=0 Nasa datasets]<br />
* [http://funapp.cs.bilkent.edu.tr/DataSets/ Function approximation repository]<br />
* [http://www.idsia.ch/~andrea/Andrea_Rizzoli_Home_Page/Sim_Tools.html A Collection of Modeling and Simulation Resources on the Internet]<br />
* [http://www.grc.nasa.gov/WWW/K-12/freesoftware_page.htm Free simulation software from Nasa]<br />
* [http://opensees.berkeley.edu/index.php Earthquake simulation]<br />
* [http://www.pdl.cmu.edu/DiskSim/ Harddisk simulator]<br />
* [http://spib.rice.edu/spib/mtn_top.html Mountain top radar data]<br />
* [http://www.statsci.org/datasets.html Statsci dataset repository]<br />
* [http://statwww.epfl.ch/davison/BMA/Data4BMA/ dataset repository]<br />
* [http://astrostatistics.psu.edu/datasets/ dataset repository]<br />
* [http://www.maths.uq.edu.au/CEToolBox/ problems from the Cross Entropy toolbox]<br />
* [http://www.cs.waikato.ac.nz/~ml/weka/index_datasets.html Weka datasets]<br />
* [http://www.ailab.si/orange/datasets.asp?Inst=on&Atts=on&Class=on&Values=on&Description=on&sort=Data+Set dataset repository]<br />
* [http://www.itee.uq.edu.au/%7Emarcusg/msg.html Max Set of Gaussians Landscape Generator]<br />
* [http://www.mathworks.fr/matlabcentral/fileexchange/loadAuthor.do?objectId=364966&objectType=author Collection of useful Matlab scripts by Thomas Abrahamsson ]<br />
* [http://www.mathworks.com/matlabcentral/fileexchange/loadFile.do?objectId=10731&objectType=FILE# A Matlab toolbox for Linear Structural Dynamics Analysis]<br />
* [http://webscripts.softpedia.com/script/Scientific-Engineering-Ruby/Controls-and-Systems-Modeling/StructDyn-32656.html Matlab simulators for control and systems modeling]<br />
* [http://www.ae.uiuc.edu/m-selig/ads.html UIUC Airfoil Data Site]<br />
* [http://public.ca.sandia.gov/TNF/abstract.html TNF workshop data archives]<br />
* [http://www.gaussianprocess.org Code for kriging models]<br />
* [http://www.comsol.be/ Comsol multiphysics] package<br />
<br />
=== Predefined functions ===<br />
<br />
* [http://www.cs.cmu.edu/afs/cs/project/jair/pub/volume24/ortizboyer05a-html/node6.html Continuous benchmark problems]<br />
* [http://www2.imm.dtu.dk/~km/Test_ex_forms/test_ex.html Optimization test functions]<br />
* [http://www.geatbx.com/docu/fcnindex-01.html Optimization test functions]<br />
* [http://www.ewh.ieee.org/soc/es/May2001/14/Begin.htm Optimization test functions]<br />
* [http://www.it.lut.fi/ip/evo/functions/functions.html Optimization test functions]<br />
* [http://www.cs.colostate.edu/~genitor/functions.html Optimization test functions]<br />
* [http://www-optima.amp.i.kyoto-u.ac.jp/member/student/hedar/Hedar_files/TestGO_files/Page364.htm Optimization test functions]<br />
* [http://www.it.lut.fi/ip/evo/functions/functions.html Functions with multiple global optima]<br />
* [http://www.mat.univie.ac.at/~neum/glopt/moretest More test set]<br />
* [http://www.mat.univie.ac.at/~neum/software/dixon.tar.gz Dixon-Szegö test set]<br />
* [http://titan.princeton.edu/2010-10-11/TestProblems/ Handbook of Test Problems for local and global optimization]<br />
<br />
=== Optimization ===<br />
<br />
* [http://ab-initio.mit.edu/wiki/index.php/NLopt NLopt is a free/open-source library for nonlinear optimization]<br />
* [http://www.tik.ee.ethz.ch/sop/pisa/ PISA optimization framework]<br />
* [http://www.hvass-labs.org/projects/swarmops/ swarmOps]<br />
* [http://www.icsi.berkeley.edu/~storn/code.html Differential Evolution]<br />
* [http://control.ee.ethz.ch/~joloef/wiki/pmwiki.php YALMIP] High-level optimization problem solver, uses external solvers as backend<br />
* [http://www.mat.univie.ac.at/~neum/glopt/software_g.html Global optimization software]<br />
* [http://www.gerad.ca/NOMAD/Abramson/nomadm.html Mesh-Adaptive Direct Search (MADS) software in Matlab]<br />
* [http://www-rocq.inria.fr/~gilbert/modulopt/modulopt.html Modulopt]: fortran implementations and some matlab<br />
* [http://www.geodise.org/documentation/OptionsMatlab/html/ MatlabOptions] Matlab interface to the design and optimization package (OPTIONS).<br />
<br />
=== Various links ===<br />
<br />
* [http://videolectures.net/Top/Computer_Science/Machine_Learning/ An excellent collection of machine learning lecture videos]<br />
* [http://home.online.no/~pjacklam/matlab/software/util/fullindex.html A collection of useful Matlab scripts]<br />
* [http://home.online.no/~pjacklam/matlab/doc/mtt/ MATLAB array manipulation tips and tricks]<br />
* [http://www.adaptivebox.net/research/bookmark/psocodes_link.html Particle Swarm implementations]<br />
* [http://mloss.org A great selection of open source machine learning software]<br />
* [http://stommel.tamu.edu/~baum/toolboxes.html A list of Matlab toolboxes]<br />
* [http://mdoboard.proboards59.com/ ISSMO-REASON: Research and Engineering Applications in Structural Optimization Network]<br />
* [http://www.nafems.org/about/ NAFEMS]<br />
* [http://www.kat-net.net/ The European Coordinating Action on Key Aerodynamic Technologies]<br />
* [http://www.altairhyperworks.co.uk/Default.aspx Altair Hyperworks]<br />
* [http://www.ifte.de/english/research/index.html IFTE]<br />
* [http://www.optiy.eu/Features.html OptyI]<br />
* [http://www.technet-alliance.com/ Technet Alliance]<br />
<br />
=== Conference links ===<br />
<br />
* [http://www.conferencealerts.com Conference alerts]<br />
* [http://www.conferencealerts.com/engineer.htm Engineering conferences]<br />
* [http://www.conferencealerts.com/ai.htm AI Conference alerts]<br />
* [http://www.ieee.org/web/conferences/search/index.html IEEE conferences]<br />
* [http://www.aiaa.org/content.cfm?pageid=1 AIAA Conferences]<br />
* [http://www.wikicfp.com/ Wiki of conferences]<br />
* [http://users.jyu.fi/~miettine/lista.html#Conferences Kaisa's conferences]<br />
* [http://www.conference-service.com/conferences/neural-networks.html AI Conferences]<br />
* [http://ieee-cis.org/conferences/co_sponsorship_1/ IEEE CIS conferences]<br />
* [http://www.makhfi.com/events.htm Neural Network Events]<br />
* [http://openresearch.org/mw/index.php?title=Upcoming_deadlines&field=Machine+learning Open research AI conferences]<br />
* [http://ppsn2014.ijs.si International Conference on Parallel Problem Solving from Nature]<br />
* [http://icannga.com International Conference on Adaptive and Natural Computing Algorithms]<br />
* [http://www.engopt.org/ EngOpt]<br />
* [http://www.sigevo.org/gecco-2014/ Genetic and Evolutionary Computation Conference]<br />
* [http://iccs2014.ivec.org/ International Conference on Computational Science]<br />
* [http://www.iccsa.org International Conference on Computational Science and Applications]<br />
<br />
=== Journal Links ===<br />
<br />
*[http://www.linklings.net/tomacs/charter.html ACM Transactions on Modeling and Computer Simulation]<br />
*[http://www.elsevier.com/wps/find/journaldescription.cws_home/622330/description#description Simulation Modelling Practice and Theory]<br />
*[http://www.elsevier.com/wps/find/journaldescription.cws_home/422911/description#description Advances in Engineering Software]<br />
*[http://journaltool.asme.org/Content/JournalDescriptions.cfm?journalId=12 Journal of Mechanical Design ]<br />
*[http://journaltool.asme.org/Content/JournalDescriptions.cfm?journalId=3&Journal=JCISE Journal of Computing and Information Science in Engineering]<br />
*[http://www.elsevier.com/wps/find/journaldescription.cws_home/975/description#description Engineering Applications of Artificial Intelligence]<br />
*[http://www.elsevier.com/wps/find/journaldescription.cws_home/622240/description#description Advanced Engineering Informatics]<br />
*[http://www.springer.com/computer/information+systems/journal/366 Engineering with Computers]<br />
*[http://www.springer.com/computer/mathematics/journal/521 Neural Computing and Applications]<br />
*[http://www.aiaa.org/content.cfm?pageid=322&lupubid=2 AIAA Journal]<br />
*[http://journals.cambridge.org/action/displayJournal?jid=aie Artificial Intelligence for Engineering Design, Analysis and Manufacturing ]<br />
*[http://jmlr.csail.mit.edu/ Journal of Machine Learning Research]<br />
*[http://www.elsevier.com/wps/find/journaldescription.cws_home/505645/description#description Computer Methods in Applied Mechanics and Engineering]<br />
*[http://www.elsevier.com/wps/find/journaldescription.cws_home/524998/description#description Applied Mathematical Modelling]<br />
*[http://www.siam.org/journals/sisc.php SIAM Journal on Scientific Computing]<br />
*[http://www.iop.org/EJ/journal/-page=scope/0266-5611 Inverse Problems]<br />
*[http://www.tandf.co.uk/journals/titles/17415977.asp Inverse problems in science and engineering]<br />
*[http://ieeexplore.ieee.org/xpl/RecentIssue.jsp?punumber=3468 IEEE Transactions on Systems, Man and Cybernetics, Part A]<br />
*[http://ieeexplore.ieee.org/xpl/RecentIssue.jsp?punumber=3477 IEEE Transactions on Systems, Man and Cybernetics, Part B]<br />
*[http://ieeexplore.ieee.org/xpl/RecentIssue.jsp?punumber=5326 IEEE Transactions on Systems, Man and Cybernetics, Part C]<br />
*[http://http://ieeexplore.ieee.org/xpl/RecentIssue.jsp?punumber=4235 IEEE Transactions on Evolutionary Computation]<br />
*[http://www.mitpressjournals.org/loi/evco Evolutionary Computation]<br />
*[http://www.techscience.com/cmes/aims_scope.html Computer Modeling in Engineering & Sciences]<br />
*[http://www.informaworld.com/smpp/title~db=all~content=t713723652~tab=summary Journal of Experimental & Theoretical Artificial Intelligence]<br />
*[http://www.elsevier.com/locate/jocs Journal of Computational Science]<br />
*[http://www.macs.hw.ac.uk/~ml355/journals.htm A comprehensive list of journals related to evolutionary algorithms/optimization and related fields]</div>Sprashanthttp://sumowiki.intec.ugent.be/FAQFAQ2014-03-05T08:46:18Z<p>Sprashant: /* What about classification problems? */</p>
<hr />
<div>== General ==<br />
<br />
=== What is a global surrogate model? ===<br />
<br />
A global [http://en.wikipedia.org/wiki/Surrogate_model surrogate model] is a mathematical model that mimics the behavior of a computationally expensive simulation code over '''the complete parameter space''' as accurately as possible, using as little data points as possible. So note that optimization is not the primary goal, although it can be done as a post-processing step. Global surrogate models are useful for:<br />
<br />
* design space exploration, to get a ''feel'' of how the different parameters behave<br />
* sensitivity analysis<br />
* ''what-if'' analysis<br />
* prototyping<br />
* visualization<br />
* ...<br />
<br />
In addition they are a cheap way to model large scale systems, multiple global surrogate models can be chained together in a model cascade.<br />
<br />
See also the [[About]] page.<br />
<br />
=== What about surrogate driven optimization? ===<br />
<br />
When coining the term '''surrogate driven optimization''' most people associate it with trust-region strategies and simple polynomial models. These frameworks first construct a local surrogate which is optimized to find an optimum. Afterwards, a move limit strategy decides how the local surrogate is scaled and/or moved through the input space. Subsequently the surrogate is rebuild and optimized. I.e. the surrogate zooms in to the global optimum. For instance the [http://www.cs.sandia.gov/DAKOTA/ DAKOTA] Toolbox implements such strategies where the surrogate construction is separated from optimization.<br />
<br />
Such a framework was earlier implemented in the SUMO Toolbox but was deprecated as it didn't fit the philosophy and design of the toolbox. <br />
<br />
Instead another, equally powerful, approach was taken. The current optimization framework is in fact a sampling selection strategy that balances local and global search. In other words, it balances between exploring the input space and exploiting the information the surrogate gives us.<br />
<br />
A configuration example can be found [[Config:SampleSelector#expectedImprovement|here]].<br />
<br />
=== What is (adaptive) sampling? Why is it used? ===<br />
<br />
In classical Design of Experiments you need to specify the design of your experiment up-front. Or in other words, you have to say up-front how many data points you need and how they should be distributed. Two examples are Central Composite Designs and Latin Hypercube designs. However, if your data is expensive to generate (e.g., an expensive simulation code) it is not clear how many points are needed up-front. Instead data points are selected adaptively, only a couple at a time. This process of incrementally selecting new data points in regions that are the most interesting is called adaptive sampling, sequential design, or active learning. Of course the sampling process needs to start from somewhere so the very first set of points is selected based on a fixed, classic experimental design. See also [[Running#Understanding_the_control_flow]].<br />
SUMO provides a number of different sampling algorithms: [[SampleSelector]]<br />
<br />
Of course sometimes you dont want to do sampling. For example if you have a fixed dataset you just want to load all the data in one go and model that. For how to do this see [[FAQ#How_do_I_turn_off_adaptive_sampling_.28run_the_toolbox_for_a_fixed_set_of_samples.29.3F]].<br />
<br />
=== What about dynamical, time dependent data? ===<br />
<br />
The original design and purpose was to tackle static input-output systems, where there is no memory. Just a complex mapping that must be learnt and approximated. Of course you can take a fixed time interval and apply the toolbox but that typically is not a desired solution. Usually you are interested in time series prediction, e.g., given a set of output values from time t=0 to t=k, predict what happens at time t=k+1,k+2,...<br />
<br />
The toolbox was originally not intended for this purpose. However, it is quite easy to add support for recurrent models. Automatic generation of dynamical models would involve adding a new model type (just like you would add a new regression technique) or require adapting an existing one. For example it would not be too much work to adapt the ANN or SVM models to support dynamic problems. The only extra work besides that would be to add a new [[Measures|Measure]] that can evaluate the fidelity of the models' prediction.<br />
<br />
Naturally though, you would be unable to use sample selection (since it makes no sense in those problems). Unless of course there is a specialized need for it. In that case you would add a new [[SampleSelector]].<br />
<br />
For more information on this topic [[Contact]] us.<br />
<br />
=== What about classification problems? ===<br />
<br />
The main focus of the SUMO Toolbox is on regression/function approximation. However, the framework for hyperparameter optimization, model selection, etc. can also be used for classification. Starting from version 6.3 a demo file is included in the distribution that shows how this works on the well known two spiral test problem. It is possible to specify a run as a classification problem by setting the 'classificationMode' and 'numberOfClasses' option in ContextConfig in the configuration file. Classification models from WEKA are also available in SUMO. Please refer to the default configuration file for the explanation on usage of WEKA model types available through SUMO. The LOLA-Voronoi sample selection scheme also supports classification, and its usage is documented in the default configuration file as well.<br />
<br />
=== Does SUMO support discrete inputs/outputs ===<br />
<br />
Not, if you mean in a smart way. There is a way to flag an input/output as discrete but it is not used anywhere. It is on the wishlist but we have not been able to get to it yet. Discrete inputs are just handled as if they were continuous. Depending on how many levels there are and if there is an ordering this may work ok or not work at all. You could of course add your own model type that can handle these :) As for discrete outputs see [[FAQ#What_about_classification_problems.3F]].<br />
<br />
=== Can the toolbox drive my simulation code directly? ===<br />
<br />
Yes it can. See the [[Interfacing with the toolbox]] page.<br />
<br />
=== What is the difference between the M3-Toolbox and the SUMO-Toolbox? ===<br />
<br />
The SUMO toolbox is a complete, feature-full framework for automatically generating approximation models and performing adaptive sampling. In contrast, the M3-Toolbox was more of a proof-of-principle.<br />
<br />
=== What happened to the M3-Toolbox? ===<br />
<br />
The M3 Toolbox project has been discontinued (Fall 2007) and superseded by the SUMO Toolbox. Please contact tom.dhaene@ua.ac.be for any inquiries and requests about the M3 Toolbox.<br />
<br />
=== How can I stay up to date with the latest news? ===<br />
<br />
To stay up to date with the latest news and releases, we also recommend subscribing to our newsletter [http://www.sumo.intec.ugent.be here]. Traffic will be kept to a minimum (1 message every 2-3 months) and you can unsubscribe at any time.<br />
<br />
You can also follow our blog: [http://sumolab.blogspot.com/ http://sumolab.blogspot.com/].<br />
<br />
=== What is the roadmap for the future? ===<br />
<br />
There is no explicit roadmap since much depends on where our research leads us, what feedback we get, which problems we are working on, etc. However, to get an idea of features to come you can always check the [[Whats new]] page.<br />
<br />
You can also follow our blog: [http://sumolab.blogspot.com/ http://sumolab.blogspot.com/].<br />
<br />
=== Will there be an R/Scilab/Octave/Sage/.. version? ===<br />
<br />
At the start of the project we considered moving from Matlab to one of the available open source alternatives. However, after much discussion we decided against this for several reasons, including:<br />
<br />
* Existing experience and know-how of the development team<br />
* The widespread use of the Matlab platform in the target application domains<br />
* The quality and amount of available Matlab documentation<br />
* The quality and number of Matlab toolboxes<br />
* Support for object orientation (inheritance, polymorphism, etc.)<br />
* Many well documented interfacing options (especially the seamless integration with Java)<br />
<br />
Matlab, as a proprietary platform, definitely has its problems and deficiencies but the number of advanced algorithms and available toolboxes make it a very attractive platform. Equally important is the fact that every function is properly documented, tested, and includes examples, tutorials, and in some cases GUI tools. A lot of things would have been a lot harder and/or time consuming to implement on one of the other platforms. Add to that the fact that many engineers (particularly in aerospace) already use Matlab quite heavily. Thus given our situation, goals, and resources at the time, Matlab was the best choice for us. <br />
<br />
The other platforms remain on our radar however, and we do look into them from time to time. Though, with our limited resources porting to one of those platforms is not (yet) cost effective.<br />
<br />
=== What are collaboration options? ===<br />
<br />
We will gladly help out with any SUMO-Toolbox related questions or problems. However, since we are a university research group the most interesting goal for us is to work towards some joint publication (e.g., we can help with the modeling of your problem). Alternatively, it is always nice if we could use your data/problem (fully referenced and/or anonymized if necessary of course) as an example application during a conference presentation or in a PhD thesis.<br />
<br />
The most interesting case is if your problem involves sample selection and modeling. This means you have some simulation code or script to drive and you want an accurate model while minimizing the number of data points. In this case, in order for us to optimally help you it would be easiest if we could run your simulation code (or script) locally or access it remotely. Else its difficult to give good recommendations about what settings to use.<br />
<br />
If this is not possible (e.g., expensive, proprietary or secret modeling code) or if your problem does not involve sample selection, you can send us a fixed data set that is representative of your problem. Again, this may be fully anonymized and will be kept confidential of course.<br />
<br />
In either case (code or dataset) remember:<br />
<br />
* the data file should be an ASCII file in column format (each row containing one data point) (see also [[Interfacing_with_the_toolbox]])<br />
* include a short description of your data:<br />
** number of inputs and number of outputs<br />
** the range of each input (or scaled to [-1 1] if you do not wish to disclose this)<br />
** if the outputs are real or complex valued<br />
** how noisy the data is or if it is completely deterministic (computer simulation) (please also see: [[FAQ#My_data_contains_noise_can_the_SUMO-Toolbox_help_me.3F]]).<br />
** if possible the expected range of each output (or scaled if you do not wish to disclose this)<br />
** if possible the names of each input/output + a short description of what they mean<br />
** any further insight you have about the data, expected behavior, expected importance of each input, etc.<br />
<br />
If you have any further questions or comments related to this please [[Contact]] us.<br />
<br />
=== Can you help me model my problem? ===<br />
<br />
Please see the previous question: [[FAQ#What_are_collaboration_options.3F]]<br />
<br />
== Installation and Configuration ==<br />
<br />
=== What is the relationship between Matlab and Java? ===<br />
<br />
Many people do not know this, but your Matlab installation automatically includes a Java virtual machine. By default, Matlab seamlessly integrates with Java, allowing you to create Java objects from the command line (e.g., 's = java.lang.String'). It is possible to disable java support but in order to use the SUMO Toolbox it should not be. To check if Java is enabled you can use the 'usejava' command.<br />
<br />
=== What is Java, why do I need it, do I have to install it, etc. ? ===<br />
<br />
The short answer is: no, dont worry about it. The long answer is: Some of the code of the SUMO Toolbox is written in [http://en.wikipedia.org/wiki/Java_(programming_language) Java], since it makes a lot more sense in many situations and is a proper programming language instead of a scripting language like Matlab. Since Matlab automatically includes a JVM to run Java code there is nothing you need to do or worry about (see the previous FAQ entry). Unless its not working of course, in that case see [[FAQ#When_running_the_toolbox_you_get_something_like_.27.3F.3F.3F_Undefined_variable_.22ibbt.22_or_class_.22ibbt.sumo.config.ContextConfig.setRootDirectory.22.27]].<br />
<br />
=== What is XML? ===<br />
<br />
XML stands for eXtensible Markup Language and is related to HTML (= the stuff web pages are written in). The first thing you have to understand is that '''does not do anything'''. Honest. Many engineers are not used to it and think it is some complicated computer programming language-stuff-thingy. This is of course not the case (we ignore some of the fancy stuff you can do with it for now). XML is a markup language meaning, it provides some rules how you can annotate or structure existing text.<br />
<br />
The way SUMO uses XML is really simple and there is not much to understand (for more information on how SUMO uses XML go this [[Config:ToolboxConfiguration#Interpreting_the_configuration_file|page]]). <br />
<br />
<br />
First some simple terminology. Take the following example:<br />
<br />
<source lang="xml"><br />
<Foo attr="bar">bla bla bla</Foo> <br />
</source><br />
<br />
Here we have '''a tag''' called ''Foo'' containing text ''bla bla bla''. The tag Foo also has an '''attribute''' ''attr'' with value ''bar''. '<Foo>' is what we call the '''opening tag''', and '</Foo>' is the '''closing tag'''. Each time you open a tag you must close it again. How you name the tags or attributes it totally up to you, you choose :)<br />
<br />
Lets take a more interesting example. Here we have used XML to represent information about a receipe for pancakes:<br />
<br />
<source lang="xml"><br />
<recipe category="dessert"><br />
<title>Pancakes</title><br />
<author>sumo@intec.ugent.be</author><br />
<date>Wed, 14 Jun 95</date><br />
<description><br />
Good old fashioned pancakes.<br />
</description><br />
<ingredients><br />
<item><br />
<amount>3</amount><br />
<type>eggs</type><br />
</item><br />
<br />
<item><br />
<amount>0.5 tablespoon</amount><br />
<type>salt</type><br />
</item><br />
...<br />
</ingredients><br />
<preparation><br />
...<br />
</preparation><br />
</recipe><br />
</source><br />
<br />
So basically, you see that XML is just a way to structure, order, and group information. Thats it! So SUMO basically uses it to store and structure configuration options. And this works well due to the nice hierarchical nature of XML.<br />
<br />
If you understand this there is nothing else to it in order to be able to understand the SUMO configuration files. If you need more information see the tutorial here: [http://www.w3schools.com/XML/xml_whatis.asp http://www.w3schools.com/XML/xml_whatis.asp]. You can also have a look at the wikipedia page here: [http://en.wikipedia.org/wiki/XML http://en.wikipedia.org/wiki/XML]<br />
<br />
=== Why does SUMO use XML? ===<br />
<br />
XML is the de facto standard way of structuring information. This ranges from spreadsheet files (Microsoft Excel for example), to configuration data, to scientific data, ... There are even whole database systems based solely on XML. So basically, its an intuitive way to structure data and it is used everywhere. This makes that there are a very large number of libraries and programming languages available that can parse, and handle XML easily. That means less work for the programmer. Then of course there is stuff like XSLT, XQuery, etc that makes life even easier.<br />
So basically, it would not make sense for SUMO to use any other format :). For more information on how SUMO uses XML go this [[Config:ToolboxConfiguration#Interpreting_the_configuration_file|page]].<br />
<br />
=== I get an error that SUMO is not yet activated ===<br />
<br />
Make sure you installed the activation file that was mailed to you as is explained in the [[Installation]] instructions. Also double check your system meets the [[System requirements]] and that [http://www.sumowiki.intec.ugent.be/index.php/FAQ#When_running_the_toolbox_you_get_something_like_.27.3F.3F.3F_Undefined_variable_.22ibbt.22_or_class_.22ibbt.sumo.config.ContextConfig.setRootDirectory.22.27|java java is enabled]. To fully verify that the activation file installation is correct ensure that the file ContextConfig.class is present in the directory ''<SUMO installation directory>/bin/java/ibbt/sumo/config''.<br />
<br />
Please note that more flexible research licenses are available if it is possible to [[FAQ#What_are_collaboration_options.3F|collaborate in any way]].<br />
<br />
== Upgrading ==<br />
<br />
=== How do I upgrade to a newer version? ===<br />
<br />
Delete your old <code><SUMO-Toolbox-directory></code> completely and replace it by the new one. Install the new activation file / extension pack as before (see [[Installation]]), start Matlab and make sure the default run works. To port your old configuration files to the new version: make a copy of default.xml (from the new version) and copy over your custom changes (from the old version) one by one. This should prevent any weirdness if the XML structure has changed between releases.<br />
<br />
If you had a valid activation file for the previous version, just [[Contact]] us (giving your SUMOlab website username) and we will send you a new activation file. Note that to update an activation file you must first unzip a copy of the toolbox to a new directory and install the activation file as if it was the very first time. Upgrading of an activation file without performing a new toolbox install is (unfortunately) not (yet) supported.<br />
<br />
== Using ==<br />
<br />
=== I have no idea how to use the toolbox, what should I do? ===<br />
<br />
See: [[Running#Getting_started]]<br />
<br />
=== I want to try one of the different examples ===<br />
<br />
See [[Running#Running_different_examples]].<br />
<br />
=== I want to model my own problem ===<br />
<br />
See : [[Adding an example]].<br />
<br />
=== I want to contribute some data/patch/documentation/... ===<br />
<br />
See : [[Contributing]].<br />
<br />
=== How do I interface with the SUMO Toolbox? ===<br />
<br />
See : [[Interfacing with the toolbox]].<br />
<br />
=== What configuration options (model type, sample selection algorithm, ...) should I use for my problem? ===<br />
<br />
See [[General_guidelines]].<br />
<br />
=== Ok, I generated a model, what can I do with it? ===<br />
<br />
See: [[Using a model]].<br />
<br />
=== How can I share a model created by the SUMO Toolbox? ===<br />
<br />
See : [[Using a model#Model_portability| Model portability]].<br />
<br />
=== I dont like the final model generated by SUMO how do I improve it? ===<br />
<br />
Before you start the modeling you should really ask youself this question: ''What properties do I want to see in the final model?'' You have to think about what for you constitutes a good model and what constitutes a poor model. Then you should rank those properties depending on how important you find them. Examples are:<br />
<br />
* accuracy in the training data<br />
** is it important that the error in the training data is exactly 0, or do you prefer some smoothing<br />
* accuracy outside the training data<br />
** this is the validation or test error, how important is proper generalization (usually this is very important)<br />
* what does accuracy mean to you? a low maximum error, a low average error, both, ...<br />
* smoothness<br />
** should your model be perfectly smooth or is it acceptable that you have a few small ripples here and there for example<br />
* are some regions of the response more important than others?<br />
** for example you may want to be certain that the minima/maxima are captured very accurately but everything in between is less important<br />
* are there particular special features that your model should have<br />
** for example, capture underlying poles or discontinuities correctly<br />
* extrapolation capability<br />
* ...<br />
<br />
It is important to note that often these criteria may be conflicting. The classical example is fitting noisy data: the lower your training error the higher your testing error. A natural approach is to combine multiple criteria, see [[Multi-Objective Modeling]].<br />
<br />
Once you have decided on a set of requirements the question is then, can the SUMO-Toolbox produce a model that meets them? In SUMO model generation is driven by one or more [[Measures]]. So you should choose the combination of [[Measures]] that most closely match your requirements. Of course we can not provide a Measure for every single property, but it is very straightforward to [[Add_Measure|add your own Measure]].<br />
<br />
Now, lets say you have chosen what you think are the best Measures but you are still not happy with the final model. Reasons could be:<br />
<br />
* you need more modeling iterations or you need to build more models per iteration (see [[Running#Understanding_the_control_flow]]). This will result in a more extensive search of the model parameter space, but will take longer to run.<br />
* you should switch to a different model parameter optimization algorithm (e.g., for example instead of the Pattern Search variant, try the Genetic Algorithm variant of your AdaptiveModelBuilder.)<br />
* the model type you are using is not ideally suited to your data<br />
* there simply is not enough data, use a larger initial design or perform more sampling iterations to get more information per dimension<br />
* maybe the sample distribution is causing troubles for your model (e.g., Kriging can have problems with clustered data). In that case it could be worthwhile to choose a different sample selection algorithm.<br />
* the range of your response variable is not ideal (for example, neural networks have trouble modeling data if the range of the outputs is very very small)<br />
<br />
You may also refer to the following [[General_guidelines]]. Finally, of course it may be that your problem is simply a very difficult one and does not approximate well. But, still you should at least get something satisfactory.<br />
<br />
If you are having these kinds of problems, please [[Reporting_problems|let us know]] and we will gladly help out.<br />
<br />
=== My data contains noise can the SUMO-Toolbox help me? ===<br />
<br />
The original purpose of the SUMO-Toolbox was for it to be used in conjunction with computer simulations. Since these are fully deterministic you do not have to worry about noise in the data and all the problems it causes. However, the methods in the toolbox are general fitting methods that work on noisy data as well. So yes, the toolbox can be used with noisy data, but you will just have to be more careful about how you apply the methods and how you perform model selection. Its only when you use the toolbox with a noisy simulation engine that a few special options may need to be set. In that case [[Contact]] us for more information.<br />
<br />
Note though, that the toolbox is not a statistical package, if you have noisy data and you need noise estimation algorithms, kernel smoothing algorithms, etc. you should look towards other tools.<br />
<br />
=== What is the difference between a ModelBuilder and a ModelFactory? ===<br />
<br />
See [[Add Model Type]].<br />
<br />
=== Why are the Neural Networks so slow? ===<br />
<br />
The ANN models are an extremely powerful model type that give very good results in many problems. However, they are quite slow to use. There are some things you can do:<br />
<br />
* use trainlm or trainscg instead of the default training function trainbr. trainbr gives very good, smooth results but is slower to use. If results with trainlm are not good enough, try using msereg as a performance function.<br />
* try setting the training goal (= the SSE to reach during training) to a small positive number (e.g., 1e-5) instead of 0.<br />
* check that the output range of your problem is not very small. If your response data lies between 10e-5 and 10e-9 for example it will be very hard for the neural net to learn it. In that case rescale your data to a more sane range.<br />
* switch from ANN to one of the other neural network modelers: fanngenetic or nanngenetic. These are a lot faster than the default backend based on the [http://www.mathworks.com/products/neuralnet/ Matlab Neural Network Toolbox]. However, the accuracy is usually not as good.<br />
* If you are using [[Measures#CrossValidation| CrossValidation]] try to switch to a different measure since CrossValidation is very expensive to use. CrossValidation is used by default if you have not defined a [[Measures| measure]] yourself. When using one of the neural network model types, try to use a different measure if you can. For example, our tests have shown that minimizing the sum of [[Measures#SampleError| SampleError]] and [[Measures#LRMMeasure| LRMMeasure]] can give equal or even better results than CrossValidation, while being much cheaper (see [[Multi-Objective Modeling]] for how to combine multiple measures). See also the comments in <code>default.xml</code> for examples.<br />
* Finally, as with any model type things will slow down if you have many dimensions or very large amounts of data. If that is the case, try some dimensionality reduction or subsampling techniques.<br />
<br />
See also [[FAQ#How_can_I_make_the_toolbox_run_faster.3F]]<br />
<br />
=== How can I make the toolbox run faster? ===<br />
<br />
There are a number of things you can do to speed things up. These are listed below. Remember though that the main reason the toolbox may seem to be slow is due to the many models being built as part of the hyperparameter optimization. Please make sure you fully understand the [[Running#Understanding_the_control_flow|control flow described here]] before trying more advanced options.<br />
<br />
* First of all check that your virus scanner is not interfering with Matlab. If McAfee or any other program wants to scan every file SUMO generates this really slows things down and your computer becomes unusable.<br />
<br />
* Turn off the plotting of models in [[Config:ContextConfig#PlotOptions| ContextConfig]], you can always generate plots from the saved mat files<br />
<br />
* This is an important one. For most model builders there is an option "maxFunEals", "maxIterations", or equivalent. Change this value to change the maximum number of models built between 2 sampling iterations. The higher this number, the slower, but the better the models ''may'' be. Equivalently, for the Genetic model builders reduce the population size and the number of generations.<br />
<br />
* If you are using [[Measures#CrossValidation]] see if you can avoid it and use one of the other measures or a combination of measures (see [[Multi-Objective Modeling]]<br />
<br />
* If you are using a very dense [[Measures#ValidationSet]] as your Measure, this means that every single model will be evaluated on that data set. For some models like RBF, Kriging, SVM, this can slow things down.<br />
<br />
* Disable some, or even all of the [[Config:ContextConfig#Profiling| profilers]] or disable the output handlers that draw charts. For example, you might use the following configuration for the profilers:<br />
<br />
<source lang="xml"><br />
<Profiling><br />
<Profiler name=".*share.*|.*ensemble.*|.*Level.*" enabled="true"><br />
<Output type="toImage"/><br />
<Output type="toFile"/><br />
</Profiler><br />
<br />
<Profiler name=".*" enabled="true"><br />
<Output type="toFile"/><br />
</Profiler><br />
</Profiling><br />
</source><br />
<br />
The ".*" means match any one or more characters ([http://java.sun.com/j2se/1.5.0/docs/api/java/util/regex/Pattern.html see here for the full list of supported wildcards]). Thus in this example all the profilers that have "share", "ensemble", or "Level" in their name shoud be enabled and should be saved as a text file (toFile) AND as an image file (toImage). All the other profilers should be saved just to file. The idea is to only save to image what you want as an image since image generation is expensive. If you do this or switch off image generation completely you will see everything run much faster.<br />
<br />
* Decrease the logging granularity, a log level of FINE (the default is FINEST or ALL) is more then granular enough. Setting it to FINE, INFO, or even WARNING should speed things up.<br />
<br />
* If you have a multi-core/multi-cpu machine:<br />
** if you have the Matlab Parallel Computing Toolbox, try setting the parallelMode option to true in [[Config:ContextConfig]]. Now all model training occurs in parallel. This may give unexpected errors in some cases so beware when using.<br />
** if you are using a native executable or script as the sample evaluator set the threadCount variable in [[Config:SampleEvaluator#LocalSampleEvaluator| LocalSampleEvaluator]] equal to the number of cores/CPUs (only do this if it is ok to start multiple instances of your simulation script in parallel!)<br />
<br />
* Dont use the Min-Max measure, it can slow things down. See also [[FAQ#How_do_I_force_the_output_of_the_model_to_lie_in_a_certain_range]]<br />
<br />
* If you are using neural networks see [[FAQ#Why_are_the_Neural_Networks_so_slow.3F]]<br />
<br />
* If you are having problems with very slow or seemingly hanging runs:<br />
** Do a run inside the [http://www.mathworks.com/access/helpdesk/help/techdoc/index.html?/access/helpdesk/help/techdoc/matlab_env/f9-17018.html&http://www.google.be/search?client=firefox-a&rls=org.mozilla%3Aen-US%3Aofficial&channel=s&hl=nl&q=matlab+profiler&meta=&btnG=Google+zoeken Matlab profiler] and see where most time is spent.<br />
<br />
** Monitor CPU and physical/virtual memory usage while the SUMO toolbox is running and see if you notice anything strange. <br />
<br />
* Also note that by default Matlab only allocates about 117 MB memory space for the Java Virtual Machine. If you would like to increase this limit (which you should) please follow the instructions [http://www.mathworks.com/support/solutions/data/1-18I2C.html?solution=1-18I2C here]. See also the general memory instructions [http://www.mathworks.com/support/tech-notes/1100/1106.html here].<br />
<br />
To check if your SUMO run has hanged, monitor your log file (with the level set at least to FINE). If you see no changes for about 30 minutes the toolbox will probably have stalled. [[Reporting problems| report the problems here]].<br />
<br />
Such problems are hard to identify and fix so it is best to work towards a reproducible test case if you think you found a performance or scalability issue.<br />
<br />
=== How do I build models with more than one output ===<br />
<br />
Sometimes you have multiple responses that you want to model at once. See [[Running#Models_with_multiple_outputs]]<br />
<br />
=== How do I turn off adaptive sampling (run the toolbox for a fixed set of samples)? ===<br />
<br />
See : [[Adaptive Modeling Mode]].<br />
<br />
=== How do I change the error function (relative error, RMSE, ...)? ===<br />
<br />
The [[Measures| <Measure>]] tag specifies the algorithm to use to assign models a score, e.g., [[Measures#CrossValidation| CrossValidation]]. It is also possible to specify which '''error function''' to use, in the measure. The default error function is '<code>rootRelativeSquareError</code>'.<br />
<br />
Say you want to use [[Measures#CrossValidation| CrossValidation]] with the maximum absolute error, then you would put:<br />
<br />
<source lang="xml"><br />
<Measure type="CrossValidation" target="0.001" errorFcn="maxAbsoluteError"/><br />
</source><br />
<br />
On the other hand, if you wanted to use the [[Measures#ValidationSet| ValidationSet]] measure with a relative root-mean-square error you would put:<br />
<br />
<source lang="xml"><br />
<Measure type="ValidationSet" target="0.001" errorFcn="relativeRms"/><br />
</source><br />
<br />
The default error function is '<code>rootRelativeSquareError</code>'. These error functions can be found in the <code>src/matlab/tools/errorFunctions</code> directory. You are free to modify them and add your own. Remember that the choice of error function is very important! Make sure you think well about it. Also see [[Multi-Objective Modeling]].<br />
<br />
=== How do I enable more profilers? ===<br />
<br />
Go to the [[Config:ContextConfig#Profiling| <Profiling>]] tag and put <code>"<nowiki>.*</nowiki>"</code> as the regular expression. See also the next question.<br />
<br />
=== What regular expressions can I use to filter profilers? ===<br />
<br />
See the syntax [http://java.sun.com/j2se/1.5.0/docs/api/java/util/regex/Pattern.html here].<br />
<br />
=== How can I ensure deterministic results? ===<br />
<br />
See : [[Random state]].<br />
<br />
=== How do I get a simple closed-form model (symbolic expression)? ===<br />
<br />
See : [[Using a model]].<br />
<br />
=== How do I enable the Heterogenous evolution to automatically select the best model type? ===<br />
<br />
Simply use the [[Config:AdaptiveModelBuilder#heterogenetic| heterogenetic modelbuilder]] as you would any other.<br />
<br />
=== What is the combineOutputs option? ===<br />
<br />
See [[Running#Models_with_multiple_outputs]]<br />
<br />
=== What error function should I use? ===<br />
<br />
The default error function is the Root Relative Square Error (RRSE). On the other hand meanRelativeError may be more intuitive but in that case you have to be careful if you have function values close to zero since in that case the relative error explodes or even gives infinity. You could also use one of the combined relative error functions (contain a +1 in the denominator to account for small values) but then you get something between a relative and absolute error (=> hard to interpret).<br />
<br />
So to be sure an absolute error seems the safest bet (like the RMSE), however in that case you have to come up with sensible accuracy targets and realize that you will build models that try to fit the regions of high absolute value better than the low ones.<br />
<br />
Picking an error function is a very tricky business and many people do not realize this. Which one is best for you and what targets you use ultimately depends on your application and on what kind of model you want. There is no general answer.<br />
<br />
A recommended read is [http://www.springerlink.com/content/24104526223221u3/ is this paper]. See also the page on [[Multi-Objective Modeling]].<br />
<br />
=== I just want to generate an initial design (no sampling, no modeling) ===<br />
<br />
Do a regular SUMO run, except set the 'maxModelingIterations' in the SUMO tag to 0. The resulting run will only generate (and evaluate) the initial design and save it to samples.txt in the output directory.<br />
<br />
=== How do I start a run with the samples of of a previous run, or with a custom initial design? ===<br />
<br />
Use a Dataset design component, for example:<br />
<br />
<source lang="xml"><br />
<InitialDesign type="DatasetDesign"><br />
<Option key="file" value="/path/to/the/file/containing/the/points.txt"/><br />
</InitialDesign><br />
</source><br />
<br />
The points of a previous run can be found in the samples.txt file in the output directory of the run you want to continue.<br />
<br />
As a sidenote, remark you can start the toolbox with *data points* of a previous run, but not with the *models* of a previous run.<br />
<br />
=== What is a level plot? ===<br />
<br />
A level plot is a plot that shows how the error histogram changes as the best model improves. An example is:<br />
<gallery><br />
Image:levelplot.png<br />
</gallery><br />
Level plots only work if you have a separate dataset (test set) that the model can be checked against. See the comments in default.xml for how to enable level plots.<br />
<br />
<br />
<br />
=== How do I force the output of the model to lie in a certain range ===<br />
<br />
See [[Measures#MinMax]].<br />
<br />
=== My problem is high dimensional and has a lot of input parameters (more than 10). Can I use SUMO? ===<br />
<br />
That depends. Remember that the main focus of SUMO is to generate accurate 'global' models. If you want to do sampling the practical dimensionality is limited to around 6-8 (though it depends on the problem and how cheap the simulations are!). Since the more dimensions the more space you need to fill. At that point you need to see if you can extend the models with domain specific knowledge (to improve performance) or apply a dimensionality reduction method ([[FAQ#Can_the_toolbox_tell_me_which_are_the_most_important_inputs_.28.3D_variable_selection.29.3F|see the next question]]). On the other hand, if you don't need to do sample selection but you have a fixed dataset which you want to model. Then the performance on high dimensional data just depends on the model type. For examples SVM type models are independent of the dimension and thus can always be applied. Though things like feature selection are always recommended.<br />
<br />
=== Can the toolbox tell me which are the most important inputs (= variable selection)? ===<br />
<br />
When tackling high dimensional problems a crucial question is "Are all my input parameters relevant?". Normally domain knowledge would answer this question but this is not always straightforward. In those cases a whole set of algorithms exist for doing dimensionality reduction (= feature selection). Support for some of these algorithms may eventually make it into the toolbox but are not currently implemented. That is a whole PhD thesis on its own. However, if a model type provides functions for input relevance determination the toolbox can leverage this. For example, the LS-SVM model available in the toolbox supports Automatic Relevance Determination (ARD). This means that if you use the SUMO Toolbox to generate an LS-SVM model, you can call the function ''ARD()'' on the model and it will give you a list of the inputs it thinks are most important.<br />
<br />
=== Should I use a Matlab script or a shell script for interfacing with my simulation code? ===<br />
<br />
When you want to link SUMO with an external simulation engine (ADS Momentum, SPECTRE, FEBIO, SWAT, ...) you need a [http://en.wikipedia.org/wiki/Shell_script shell script] (or executable) that can take the requested points from SUMO, setup the simulation engine (e.g., set necessary input files), calls the simulator for all the requested points, reads the output (e.g., one or more output files), and returns the results to SUMO (see [[Interfacing with the toolbox]]).<br />
<br />
Which one you choose (matlab script + [[Config:SampleEvaluator#matlab|Matlab Sample Evaluator]], or shell script/executable with [[Config:SampleEvaluator#local|Local Sample Evaluator]] is basically a matter of preference, take whatever is easiest for you.<br />
<br />
HOWEVER, there is one important consideration: Matlab does not support threads so this means that if you use a matlab script to interface with the simulation engine, simulations and modeling will happen sequentially, NOT in parallel. This means the modeling code will sit around waiting, doing nothing, until the simulation(s) have finished. If your simulation code takes a long time to run this is not very efficient.<br />
<br />
On the other hand, using a shell script/executable, does allow the modeling and simulation to occur in parallel (at least if you wrote your interface script in such a way that it can be run multiple times in parallel, i.e., no shared global directories or variables that can cause [http://en.wikipedia.org/wiki/Race_condition race conditions]).<br />
<br />
As a sidenote, note that if you already put work into a Matlab script, it is still possible to use a shell script, by writing a shell script that starts Matlab (using -nodisplay or -nojvm options), executes your script (using the -r option), and exits Matlab again. Of course it is not very elegant and adds some overhead but depending on your situation it may be worth it.<br />
<br />
<br />
<br />
=== How can I look at the internal structure of a SUMO model ===<br />
<br />
See [[Using_a_model#Available_methods]].<br />
<br />
=== Is there any design documentation available? ===<br />
<br />
An in depth overview of the rationale and philosophy, including a treatment of the software architecture underlying the SUMO Toolbox is available in the form of a PhD dissertation. A copy of this dissertation [http://www.sumo.intec.ugent.be/?q=system/files/2010_04_PhD_DirkGorissen.pdf is available here].<br />
<br />
== Troubleshooting ==<br />
<br />
=== I have a problem and I want to report it ===<br />
<br />
See : [[Reporting problems]].<br />
<br />
===I am getting a java out of memory error, what happened?===<br />
Datasets are loaded through java. This means that the java heap space is used for storing the data. If you try to load a huge dataset (> 50MB), you might experience problems with the maximum heap size. You can solve this by raising the heap size as described on the following webpage:<br />
[http://www.mathworks.com/support/solutions/data/1-18I2C.html]<br />
<br />
=== I sometimes get flat models when using rational functions ===<br />
<br />
First make sure the model is indeed flat, and does not just appear so on the plot. You can verify this by looking at the output axis range and making sure it is within reasonable bounds. When there are poles in the model, the axis range is sometimes stretched to make it possible to plot the high values around the pole, causing the rest of the model to appear flat. If the model contains poles, refer to the next question for the solution.<br />
<br />
The [[Config:AdaptiveModelBuilder#rational| RationalModel]] tries to do a least squares fit, based on which monomials are allowed in numerator and denominator. We have experienced that some models just find a flat model as the best least squares fit. There are two causes for this:<br />
<br />
* The number of sample points is few, and the model parameters (as explained [[Model types explained#PolynomialModel|here]]) force the model to use only a very small set of degrees of freedom. The solution in this case is to increase the minimum percentage bound in the RationalFactory section of your configuration file: change the <code>"percentBounds"</code> option to <code>"60,100"</code>, <code>"80,100"</code>, or even <code>"100,100"</code>. A setting of <code>"100,100"</code> will force the polynomial models to always exactly interpolate. However, note that this does not scale very well with the number of samples (to counter this you can set <code>"maxDegrees"</code>). If, after increasing the <code>"percentBounds"</code> you still get weird, spiky, models you simply need more samples or you should switch to a different model type.<br />
* Another possibility is that given a set of monomial degrees, the flat function is just the best possible least squares fit. In that case you simply need to wait for more samples.<br />
* The measure you are using is not accurately estimating the true error, try a different measure or error function. Note that a maximum relative error is dangerous to use since a the 0-function (= a flat model) has a lower maximum relative error than a function which overshoots the true behavior in some places but is otherwise correct.<br />
<br />
=== When using rational functions I sometimes get 'spikes' (poles) in my model ===<br />
<br />
When the denominator polynomial of a rational model has zeros inside the domain, the model will tend to infinity near these points. In most cases these models will only be recognized as being `the best' for a short period of time. As more samples get selected these models get replaced by better ones and the spikes should disappear.<br />
<br />
So, it is possible that a rational model with 'spikes' (caused by poles inside the domain) will be selected as best model. This may or may not be an issue, depending on what you want to use the model for. If it doesn't matter that the model is very inaccurate at one particular, small spot (near the pole), you can use the model with the pole and it should perform properly.<br />
<br />
However, if the model should have a reasonable error on the entire domain, several methods are available to reduce the chance of getting poles or remove the possibility altogether. The possible solutions are:<br />
<br />
* Simply wait for more data, usually spikes disappear (but not always).<br />
* Lower the maximum of the <code>"percentBounds"</code> option in the RationalFactory section of your configuration file. For example, say you have 500 data points and if the maximum of the <code>"percentBounds"</code> option is set to 100 percent it means the degrees of the polynomials in the rational function can go up to 500. If you set the maximum of the <code>"percentBounds"</code> option to 10, on the other hand, the maximum degree is set at 50 (= 10 percent of 500). You can also use the <code>"maxDegrees"</code> option to set an absolute bound.<br />
* If you roughly know the output range your data should have, an easy way to eliminate poles is to use the [[Measures#MinMax| MinMax]] [[Measures| Measure]] together with your current measure ([[Measures#CrossValidation| CrossValidation]] by default). This will cause models whose response falls outside the min-max bounds to be penalized extra, thus spikes should disappear.<br />
* Use a different model type (RBF, ANN, SVM,...), as spikes are a typical problem of rational functions.<br />
* Increase the population size if using the genetic version<br />
* Try using the [[SampleSelector#RationalPoleSuppressionSampleSelector| RationalPoleSuppressionSampleSelector]], it was designed to get rid of this problem more quickly, but it only selects one sample at the time.<br />
<br />
However, these solutions may not still not suffice in some cases. The underlying reason is that the order selection algorithm contains quite a lot of randomness, making it prone to over-fitting. This issue is being worked on but will take some time. Automatic order selection is not an easy problem<br />
<br />
=== There is no noise in my data yet the rational functions don't interpolate ===<br />
<br />
[[FAQ#I sometimes get flat models when using rational functions |see this question]].<br />
<br />
=== When loading a model from disk I get "Warning: Class ':all:' is an unknown object class. Object 'model' of this class has been converted to a structure." ===<br />
<br />
You are trying to load a model file without the SUMO Toolbox in your Matlab path. Make sure the toolbox is in your Matlab path. <br />
<br />
In short: Start Matlab, run <code><SUMO-Toolbox-directory>/startup.m</code> (to ensure the toolbox is in your path) and then try to load your model.<br />
<br />
=== When running the SUMO Toolbox you get an error like "No component with id 'annpso' of type 'adaptive model builder' found in config file." ===<br />
<br />
This means you have specified to use a component with a certain id (in this case an AdaptiveModelBuilder component with id 'annpso') but a component with that id does not exist further down in the configuration file (in this particular case 'annpso' does not exist but 'anngenetic' or 'ann' does, as a quick search through the configuration file will show). So make sure you only declare components which have a definition lower down. So see which components are available, simply scroll down the configuration file and see which id's are specified. Please also refer to the [[Toolbox configuration#Declarations and Definitions | Declarations and Definitions]] page.<br />
<br />
=== When using NANN models I sometimes get "Runtime error in matrix library, Choldc failed. Matrix not positive definite" ===<br />
<br />
This is a problem in the mex implementation of the [http://www.iau.dtu.dk/research/control/nnsysid.html NNSYSID] toolbox. Simply delete the mex files, the Matlab implementation will be used and this will not cause any problems.<br />
<br />
=== When using FANN models I sometimes get "Invalid MEX-file createFann.mexa64, libfann.so.2: cannot open shared object file: No such file or directory." ===<br />
<br />
This means Matlab cannot find the [http://leenissen.dk/fann/ FANN] library itself to link to dynamically. Make sure the FANN libraries (stored in src/matlab/contrib/fann/src/.libs/) are in your library path, e.g., on unix systems, make sure they are included in LD_LIBRARY_PATH.<br />
<br />
=== Undeﬁned function or method ’createFann’ for input arguments of type ’double’. ===<br />
<br />
See [[FAQ#When_using_FANN_models_I_sometimes_get_.22Invalid_MEX-file_createFann.mexa64.2C_libfann.so.2:_cannot_open_shared_object_file:_No_such_file_or_directory..22]]<br />
<br />
=== When trying to use SVM models I get 'Error during fitness evaluation: Error using ==> svmtrain at 170, Group must be a vector' ===<br />
<br />
You forgot to build the SVM mex files for your platform. For windows they are pre-compiled for you, on other systems you have to compile them yourself with the makefile.<br />
<br />
=== When running the toolbox you get something like '??? Undefined variable "ibbt" or class "ibbt.sumo.config.ContextConfig.setRootDirectory"' ===<br />
<br />
First see [[FAQ#What_is_the_relationship_between_Matlab_and_Java.3F | this FAQ entry]].<br />
<br />
This means Matlab cannot find the needed Java classes. This typically means that you forgot to run 'startup' (to set the path correctly) before running the toolbox (using 'go'). So make sure you always run 'startup' before running 'go' and that both commands are always executed in the toolbox root directory.<br />
<br />
If you did run 'startup' correctly and you are still getting an error, check that Java is properly enabled:<br />
<br />
# typing 'usejava jvm' should return 1 <br />
# typing 's = java.lang.String', this should ''not'' give an error<br />
# typing 'version('-java')' should return at least version 1.5.0<br />
<br />
If (1) returns 0, then the jvm of your Matlab installation is not enabled. Check your Matlab installation or startup parameters (did you start Matlab with -nojvm?)<br />
If (2) fails but (1) is ok, there is a very weird problem, check the Matlab documentation.<br />
If (3) returns a version before 1.5.0 you will have to upgrade Matlab to a newer version or force Matlab to use a custom, newer, jvm (See the Matlab docs for how to do this).<br />
<br />
=== You get errors related to ''gaoptimset'',''psoptimset'',''saoptimset'',''newff'' not being found or unknown ===<br />
<br />
You are trying to use a component of the SUMO toolbox that requires a Matlab toolbox that you do not have. See the [[System requirements]] for more information.<br />
<br />
=== After upgrading I get all kinds of weird errors or warnings when I run my XML files ===<br />
<br />
See [[FAQ#How_do_I_upgrade_to_a_newer_version.3F]]<br />
<br />
=== I get a warning about duplicate samples being selected, why is this? ===<br />
<br />
Sometimes, in special circumstances, multiple sample selectors may select the same sample at the same time. Even though in most cases this is detected and avoided, it can still happen when multiple outputs are modelled in one run, and each output is sampled by a different sample selector. These sample selectors may then accidentally choose the same new sample location.<br />
<br />
=== I sometimes see the error of the best model go up, shouldn't it decrease monotonically? ===<br />
<br />
There is no short answer here, it depends on the situation. Below 'single objective' refers to the case where during the hyperparameter optimization (= the modeling iteration) combineOutputs=false, and there is only a single measure set to 'on'. The other cases are classified as 'multi objective'. See also [[Multi-Objective Modeling]].<br />
<br />
# '''Sampling off'''<br />
## ''Single objective'': the error should always decrease monotonically, you should never see it rise. If it does [[reporting problems|report it as a bug]]<br />
## ''Multi objective'': There is a very small chance the error can temporarily decrease but it should be safe to ignore. In this case it is best to use a multi objective enabled modeling algorithm<br />
# '''Sampling on'''<br />
## ''Single objective'': inside each modeling iteration the error should always monotonically decrease. At each sampling iteration the best models are updated (to reflect the new data), thus there the best model score may increase, this is normal behavior(*). It is possible that the error increases for a short while, but as more samples come in it should decrease again. If this does not happen you are using a poor measure or poor hyperparameter optimization algorithm, or there is a problem with the modeling technique itself (e.g., clustering in the datapoints is causing numerical problems).<br />
## ''Multi objective'': Combination of 1.2 and 2.1.<br />
<br />
(*) This is normal if you are using a measure like cross validation that is less reliable on little data than on more data. However, in some cases you may wish to override this behavior if you are using a measure that is independent of the number of samples the model is trained with (e.g., a dense, external validation set). In this case you can force a monotonic decrease by setting the 'keepOldModels' option in the SUMO tag to true. Use with caution!<br />
<br />
=== At the end of a run I get Undefined variable "ibbt" or class "ibbt.sumo.util.JpegImagesToMovie.createMovie" ===<br />
<br />
This is normal, the warning printed out before the error explains why:<br />
<br />
''[WARNING] jmf.jar not found in the java classpath, movie creation may not work! Did you install the SUMO extension pack? Alternatively you can install the java media framwork from java.sun.com''<br />
<br />
By default, at the end of a run, the toolbox will try to generate a movie of all the intermediate model plots. To do this it requires the extension pack to be installed (you can download it from the SUMO lab website). So install the extension pack and you will no longer get the error. Alternatively you can simply set the "createMovie" option in the <SUMO> tag to "false".<br />
So note that there is nothing to worry about, everything has run correctly, it is just the movie creation that is failing.<br />
<br />
=== On startup I get the error "java.io.IOException: Couldn't get lock for output/SUMO-Toolbox.%g.%u.log" ===<br />
<br />
This error means that SUMO is unable to create the log file. Check the output directory exists and has the correct permissions. If your output directory is on a shared (network) drive this could also cause problems. Also make sure you are running the toolbox (calling 'go') from the toolbox root directory, and not in some toolbox sub directory! This is very important.<br />
<br />
If you still have problems you can override the default logfile name and location as follows:<br />
<br />
In the <FileHandler> tag inside the <Logging> tag add the following option:<br />
<br />
<code><br />
<Option key="Pattern" value="My_SUMO_Log_file.log"/><br />
</code><br />
<br />
This means that from now on the sumo log file will be saved as the file "My_SUMO_Log_file.log" in the SUMO root directory. You can use any path you like.<br />
For more information about this option see [http://java.sun.com/j2se/1.4.2/docs/api/java/util/logging/FileHandler.html the FileHandler Javadoc].<br />
<br />
=== The Toolbox crashes with "Too many open files" what should I do? ===<br />
<br />
This is a known bug, see [[Known_bugs#Version_6.1]].<br />
<br />
If this does not fix your problem then do the following:<br />
<br />
On Windows try increasing the limit in windows as dictated by the error message. Also, when you get the error, use the fopen("all") command to see which files are open and send us the list of filenames. Then we can maybe further help you debug the problem. Even better would be to use the Process Explorer utility [http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx available here]. When you get the error, dont shut down Matlab but start Process explorer and see which SUMO-Toolbox related files are open. If you then [[Reporting_problems|let us know]] we can further debug the problem.<br />
<br />
On Linux again don't shut down Matlab but:<br />
<br />
* open a new terminal window<br />
* type:<br />
<source lang="bash"><br />
lsof > openFiles.txt<br />
</source><br />
* Then [[Contact|send us]] the following information:<br />
** the file openFiles.txt <br />
** the exact Linux distribution you are using (Red Hat 10, CentOS 5, SUSE 11, etc).<br />
** the output of<br />
<source lang="bash"><br />
uname -a ; df -T ; mount<br />
</source><br />
<br />
As a temporary workaround you can try increasing the maximum number of open files ([http://www.linuxforums.org/forum/redhat-fedora-linux-help/64716-where-chnage-file-max-permanently.html see for example here]). We are currently debugging this issue.<br />
<br />
In general: to be safe it is always best to do a SUMO run from a clean Matlab startup, especially if the run is important or may take a long time.<br />
<br />
=== When using the LS-SVM models I get lots of warnings: "make sure lssvmFILE.x (lssvmFILE.exe) is in the current directory, change now to MATLAB implementation..." ===<br />
<br />
The LS-SVMs have a C implementation and a Matlab implementation. If you dont have the compiled mex files it will use the matlab implementation and give a warning. But everything will work properly. To get rid of the warnings, compile the mex files [[Installation#Windows|as described here]], this can be done very easily. Or simply comment out the lines that produce the output in the lssvmlab directory in src/matlab/contrib.<br />
<br />
=== I get an error "Undefined function or method 'trainlssvm' for input arguments of type 'cell'" ===<br />
<br />
You most likely forgot to [[Installation#Extension_pack|install the extension pack]].<br />
<br />
=== When running the SUMO-Toolbox under Linux, the [http://en.wikipedia.org/wiki/X_Window_System X server] suddenly restarts and I am logged out of my session ===<br />
<br />
Note that in Linux there is an explicit difference between the [http://en.wikipedia.org/wiki/Linux_kernel kernel] and the [http://en.wikipedia.org/wiki/X_Window_System X display server]. If the kernel crashes or panics your system completely freezes (you have to reset manually) or your computer does a full reboot. Luckily this is very rare. However, if you display server (X) crashes or restarts it means your operating system is still running fine, its just that you have to log in again since your graphical session has terminated. The FAQ entry is only for the latter. If you find your kernel is panicing or freezing, that is a more fundamental problem and you should contact your system admin.<br />
<br />
So what happens is that after a few seconds when the toolbox wants to plot the first model [http://en.wikipedia.org/wiki/X_Window_System X] crashes and you are suddenly presented with a login screen. The problem is not due to SUMO but rather to the Matlab - Display server interaction.<br />
<br />
What you should first do is set plotModels to false in the [[Config:ContextConfig]] tag, run again and see if the problem occurs again. If it does please [[Reporting_problems| report it]]. If the problem does not occur you can then try the following:<br />
<br />
* Log in as root (or use [http://en.wikipedia.org/wiki/Sudo sudo])<br />
* Edit the following configuration file using a text editor (pico, nano, vi, kwrite, gedit,...)<br />
<br />
<source lang="bash"><br />
/etc/X11/xorg.conf<br />
</source><br />
<br />
Note: the exact location of the xorg.conf file may vary on your system.<br />
<br />
* Look for the following line:<br />
<br />
<source lang="bash"><br />
Load "glx"<br />
</source><br />
<br />
* Comment it out by replacing it by:<br />
<br />
<source lang="bash"><br />
# Load "glx"<br />
</source><br />
<br />
* Then save the file, restart your X server (if you do not know how to do this simply reboot your computer)<br />
* Log in again, and try running the toolbox (making sure plotModels is set to true again). It should now work. If it still does not please [[Reporting_problems| report it]].<br />
<br />
Note:<br />
* this is just an empirical workaround, if you have a better idea please [[Contact|let us know]]<br />
* if you wish to debug further yourself please check the Xorg log files and those in /var/log<br />
* another possible workaround is to start matlab with the "-nodisplay" option. That could work as well.<br />
<br />
=== I get the error "Failed to close Matlab pool cleanly, error is Too many output arguments" ===<br />
<br />
This happens if you run the toolbox on Matlab version 2008a and you have the parallel computing toolbox installed. You can simply ignore this error message, it does not cause any problems. If you want to use SUMO with the parallel computing toolbox you will need Matlab 2008b.<br />
<br />
=== The toolbox seems to keep on running forever, when or how will it stop? ===<br />
<br />
The toolbox will keep on generating models and selecting data until one of the termination criteria has been reached. It is up to ''you'' to choose these targets carefully, so how low the toolbox runs simply depends on what targets you choose. Please see [[Running#Understanding_the_control_flow]].<br />
<br />
Of course choosing a-priori targets up front is not always easy and there is no real solution for this, except thinking well about what type of model you want (see [[FAQ#I_dont_like_the_final_model_generated_by_SUMO_how_do_I_improve_it.3F]]). In doubt you can always use a small value (or 0) and then simply quit the running toolbox using Ctrl-C when you think its been enough.<br />
<br />
While one could implement fancy, automatic stopping algorithms, their actual benefit is questionable.</div>Sprashanthttp://sumowiki.intec.ugent.be/Useful_LinksUseful Links2014-01-23T23:13:57Z<p>Sprashant: /* Conference links */</p>
<hr />
<div>=== Related publications ===<br />
<br />
See the [[Related publications]] page.<br />
<br />
=== Related projects ===<br />
<br />
A list of projects with similar ideas and scope.<br />
* [http://www.cs.rtu.lv/jekabsons/regression.html] Matlab regression software<br />
* [http://sites.google.com/site/gptips4matlab GTIPS] Matlab symbolic regression software using genetic programming<br />
* [http://www.optiy.eu OptiY]<br />
* [http://www.nutechsolutions.com/prod_cv_analytics.asp ClearVu Analytics]<br />
* [https://sites.google.com/site/felipeacviana/ Surrogates Toolbox]: Another matlab surrogate modeling toolbox<br />
* [http://www.evolved-analytics.com Evolved-Analytics]: DataModeler package.<br />
* [http://www.muneda.com/ MunEDA]: MunEDA provides leading EDA software technology for analysis, modelling and optimization of yield and performance of analog, mixed-signal and digital designs.<br />
* [http://www.lsoptsupport.com/faqs/setting-parameters-for-metamodel-based-optimization-strategies LS-OPT: Functionality similar to the SUMO-Toolbox in LS-OPT]<br />
* [http://mucm.aston.ac.uk/MUCM/MUCMToolkit/index.php?page=MetaHomePage.html MUCH Toolkit]: A toolbox for sensitivity analysis using surrogate models<br />
* [http://home.mit.bme.hu/~kollar/topics/fdident.html FDIDENT: frequency domain identification toolbox]<br />
* [http://www.dmoz.org/Science/Math/Statistics/Software/Regression_and_Curve_Fitting/: Regression software on Open Directory]<br />
* [http://www.infiniscale.com/ Infiniscale]: TechModeler, TechAnalyzer: automatic model generation tools for EM applications<br />
* [http://www.vrand.com/visualDOC.html visualDOC]: VisualDOC, a design optimization tool<br />
* [http://www.salford-systems.com/mars.php MARS]: A spline based modeling tool<br />
* [http://www.phoenix-int.com/software/phx_modelcenter.php Modelcenter]: datamining and modeling tool<br />
* [http://www.simulia.com/products/isight2.html iSIGHT]: datamining and modeling tool<br />
* [http://www.tmpinc.com/datascape_overview.html Datascape]: datamining and modeling tool<br />
* [http://www.friendship-systems.com/ Friendship systems]: Tools for modeling ship hulls parametrically and performing the modeling calculations (Equilibrium tool)<br />
* [http://www.csse.monash.edu.au/%7Edavida/nimrod/nimrodg.htm Nimrod/G]: Execute parameter sweeps on the grid<br />
* [http://www-sop.inria.fr/oasis/ProActive/ ProActive]: Java grid library <br />
* [http://www.csse.monash.edu.au/%7Edavida/nimrod/nimrodo.htm Nimrod/O]: Grid-enabled optimization toolkit <br />
* [http://www.ece.northwestern.edu/OTC/ NEOS]: Distributed optimization toolkit <br />
* [http://software.sci.utah.edu/scirun.html SciRun]: Problem Solving Environment (PSE), for simulation, modeling, and visualization of scientific problems. <br />
* [http://www.esteco.it/ ModeFRONTIER]: environment dedicated to the set up of design assessment chains and efficient investigation of the design space. <br />
* [http://www.gridbus.org/ Gridbus]: metascheduler for the grid <br />
* [http://icl.cs.utk.edu/netsolve/overview/ Gridsolve/Netsolve]: grid enabled scientific computation toolbox <br />
* [http://www.dtreg.com/ DTREG]: a powerful statistical analysis program that generates classification and regression trees and Support Vector Machine models that can be used to predict parameter values. <br />
* [http://www.soton.ac.uk/%7Epbn/MDO/ MDO]: A collection of MDO links <br />
* [http://www.research.att.com/~njas/gosset/index.html GOSSET]: A general purpose program for designing experiments <br />
* [http://www.cs.sandia.gov/DAKOTA/ DAKOTA]: Design Analysis Kit for Optimization and Terascale Applications<br />
* [http://www.cs.cf.ac.uk/gridprojects/dipso/ DIPSO]: Wide-Area Distributed Problem Solving (DIPSO) <br />
* [http://www.geodise.org GEODISE]: Grid Enabled Optimisation and Design Search for Engineering<br />
* [http://www.fast.u-psud.fr/ezyfit/ EZfit] : Free curve fitting toolbox for matlab<br />
* [http://www.ians.uni-stuttgart.de/spinterp/about.html SGIT] : A Sparse grid interpolation toolbox<br />
* [http://www.csie.ntu.edu.tw/~yien/quickrbf/quickstart.php QuickRBF] : an RBF fitting library (native)<br />
* [http://www.farfieldtechnology.com/products/toolbox/ FastRBF] : another RBF fitting library (matlab)<br />
* [http://promethee.irsn.org PROMETHEE project: easy parametric modeling]<br />
* [http://simlab.jrc.ec.europa.eu/ Sensitivity Analysis library]<br />
* Gaussian Process Matlab code<br />
** [http://www.gaussianprocess.org/gpml/code/matlab/doc/ Code] based on Rasmussen's book<br />
** [http://www.cs.man.ac.uk/~neill/gp/ Gaussian Process Software]<br />
** [http://www.ios.htwg-konstanz.de/joomla_mof/index.php?option=com_content&view=article&id=48:polyreg-polynomial-gaussian-process-regression&catid=36:code&Itemid=81 GP] using polynomial covariance functions<br />
* [http://shark-project.sourceforge.net/ Shark Machine Learning Library - An open source machine learning bundle]<br />
<br />
=== Related labs ===<br />
<br />
A list of some of the labs/researchers with similar ideas and scope.<br />
<br />
* [http://aerospace.engin.umich.edu/index.html Department of Aerospace Engineering at the University of Michigan]<br />
* [http://www.mae.ufl.edu/~mdo/ The Structural and Multidisciplinary Optimization Group at the University of Florida]<br />
* [http://web.engr.oregonstate.edu/~tgd/ School of Electrical Engineering and Computer Science, Oregon State U]<br />
* [http://www.soton.ac.uk/engineering/research/groups/ced.page Southampton University Computational Engineering and Design Group]<br />
* [http://edog.mne.psu.edu/ Engineering Design & Optimization Group (Penn state)]<br />
* [http://nd.edu/~ame/ Aerospace and Mechanical Engineering] [http://www.gano.name/shawn/ Homepage]<br />
* [http://shyylab.engin.umich.edu/research/design-optimization Computational Thermo-Fluids Group]<br />
* [http://www.ensc.sfu.ca/~gwa5/index.htm Product Design and Optimization Laboratory (PDOL)]<br />
* [http://www.cerfacs.fr/4-25708-Home.php Computational Fluid Dynamics (CFD) group at CERFACS]<br />
* [http://webuser.uni-weimar.de/~roos1/ Dirk Roos]<br />
* [http://www.cerfacs.fr/~duchaine/HTML/research.htm Florent Duchaine]<br />
* http://www.nlr.nl/ National aerospace lab]<br />
<br />
=== Data sets - Simulation code ===<br />
<br />
A list of publicly available datasets and simulation codes, useful for testing.<br />
<br />
* [http://www.mat.univie.ac.at/~neum/stat.html Statistics links] : A nice collection of data fitting and analysis codes<br />
* [http://en.wikipedia.org/wiki/Classic_data_sets Wikipedia Classic Datasets]<br />
* [http://www.uni-koeln.de/themen/Statistik/data/rousseeuw/ The ROUSSEEUW datasets]<br />
* [ftp://ftp.sas.com/pub/neural/dojo/dojo.html Donoho-Johnstone Benchmarks] <br />
* [http://people.scs.fsu.edu/~burkardt/datasets/datasets.html datasets]<br />
* [http://www-syscom.univ-mlv.fr/~vignat/Signal/Space/index.html Spice like simulator for matlab]<br />
* [http://www.itl.nist.gov/div898/strd/general/dataarchive.html Nist dataset archive]<br />
* [http://homes.esat.kuleuven.be/~smc/daisy/daisydata.html Daisy datasets]<br />
* [http://lib.stat.cmu.edu/datasets/ Statlib dataset archive]<br />
* [http://www.iau.dtu.dk/nnbook/systems.html Datasets from the book "Neural networks for the modeling and control of dynamic systems.]<br />
* [http://www.gpc.de/e_poses.html A simulation environment for production and transport, logistics and automation systems]<br />
* [http://shumway.physics.asu.edu/codes.html Nanostructure Simulation and Modeling Programs]<br />
* [http://pedsim.silmaril.org/ A Modular, Distributed Pedestrian Crowd Simulation System]<br />
* [http://g95.sourceforge.net/g95_status.html Fortran simulation codes]<br />
* [http://www.genie.ac.uk/ Grid ENabled Integrated Earth system model]<br />
* [http://gcmd.nasa.gov/KeywordSearch/Home.do?Portal=GCMD&MetadataType=0 Nasa datasets]<br />
* [http://funapp.cs.bilkent.edu.tr/DataSets/ Function approximation repository]<br />
* [http://www.idsia.ch/~andrea/Andrea_Rizzoli_Home_Page/Sim_Tools.html A Collection of Modeling and Simulation Resources on the Internet]<br />
* [http://www.grc.nasa.gov/WWW/K-12/freesoftware_page.htm Free simulation software from Nasa]<br />
* [http://opensees.berkeley.edu/index.php Earthquake simulation]<br />
* [http://www.pdl.cmu.edu/DiskSim/ Harddisk simulator]<br />
* [http://spib.rice.edu/spib/mtn_top.html Mountain top radar data]<br />
* [http://www.statsci.org/datasets.html Statsci dataset repository]<br />
* [http://statwww.epfl.ch/davison/BMA/Data4BMA/ dataset repository]<br />
* [http://astrostatistics.psu.edu/datasets/ dataset repository]<br />
* [http://www.maths.uq.edu.au/CEToolBox/ problems from the Cross Entropy toolbox]<br />
* [http://www.cs.waikato.ac.nz/~ml/weka/index_datasets.html Weka datasets]<br />
* [http://www.ailab.si/orange/datasets.asp?Inst=on&Atts=on&Class=on&Values=on&Description=on&sort=Data+Set dataset repository]<br />
* [http://www.itee.uq.edu.au/%7Emarcusg/msg.html Max Set of Gaussians Landscape Generator]<br />
* [http://www.mathworks.fr/matlabcentral/fileexchange/loadAuthor.do?objectId=364966&objectType=author Collection of useful Matlab scripts by Thomas Abrahamsson ]<br />
* [http://www.mathworks.com/matlabcentral/fileexchange/loadFile.do?objectId=10731&objectType=FILE# A Matlab toolbox for Linear Structural Dynamics Analysis]<br />
* [http://webscripts.softpedia.com/script/Scientific-Engineering-Ruby/Controls-and-Systems-Modeling/StructDyn-32656.html Matlab simulators for control and systems modeling]<br />
* [http://www.ae.uiuc.edu/m-selig/ads.html UIUC Airfoil Data Site]<br />
* [http://public.ca.sandia.gov/TNF/abstract.html TNF workshop data archives]<br />
* [http://www.gaussianprocess.org Code for kriging models]<br />
* [http://www.comsol.be/ Comsol multiphysics] package<br />
<br />
=== Predefined functions ===<br />
<br />
* [http://www.cs.cmu.edu/afs/cs/project/jair/pub/volume24/ortizboyer05a-html/node6.html Continuous benchmark problems]<br />
* [http://www2.imm.dtu.dk/~km/Test_ex_forms/test_ex.html Optimization test functions]<br />
* [http://www.geatbx.com/docu/fcnindex-01.html Optimization test functions]<br />
* [http://www.ewh.ieee.org/soc/es/May2001/14/Begin.htm Optimization test functions]<br />
* [http://www.it.lut.fi/ip/evo/functions/functions.html Optimization test functions]<br />
* [http://www.cs.colostate.edu/~genitor/functions.html Optimization test functions]<br />
* [http://www-optima.amp.i.kyoto-u.ac.jp/member/student/hedar/Hedar_files/TestGO_files/Page364.htm Optimization test functions]<br />
* [http://www.it.lut.fi/ip/evo/functions/functions.html Functions with multiple global optima]<br />
* [http://www.mat.univie.ac.at/~neum/glopt/moretest More test set]<br />
* [http://www.mat.univie.ac.at/~neum/software/dixon.tar.gz Dixon-Szegö test set]<br />
* [http://titan.princeton.edu/2010-10-11/TestProblems/ Handbook of Test Problems for local and global optimization]<br />
<br />
=== Optimization ===<br />
<br />
* [http://ab-initio.mit.edu/wiki/index.php/NLopt NLopt is a free/open-source library for nonlinear optimization]<br />
* [http://www.tik.ee.ethz.ch/sop/pisa/ PISA optimization framework]<br />
* [http://www.hvass-labs.org/projects/swarmops/ swarmOps]<br />
* [http://www.icsi.berkeley.edu/~storn/code.html Differential Evolution]<br />
* [http://control.ee.ethz.ch/~joloef/wiki/pmwiki.php YALMIP] High-level optimization problem solver, uses external solvers as backend<br />
* [http://www.mat.univie.ac.at/~neum/glopt/software_g.html Global optimization software]<br />
* [http://www.gerad.ca/NOMAD/Abramson/nomadm.html Mesh-Adaptive Direct Search (MADS) software in Matlab]<br />
* [http://www-rocq.inria.fr/~gilbert/modulopt/modulopt.html Modulopt]: fortran implementations and some matlab<br />
* [http://www.geodise.org/documentation/OptionsMatlab/html/ MatlabOptions] Matlab interface to the design and optimization package (OPTIONS).<br />
<br />
=== Various links ===<br />
<br />
* [http://videolectures.net/Top/Computer_Science/Machine_Learning/ An excellent collection of machine learning lecture videos]<br />
* [http://home.online.no/~pjacklam/matlab/software/util/fullindex.html A collection of useful Matlab scripts]<br />
* [http://home.online.no/~pjacklam/matlab/doc/mtt/ MATLAB array manipulation tips and tricks]<br />
* [http://www.adaptivebox.net/research/bookmark/psocodes_link.html Particle Swarm implementations]<br />
* [http://mloss.org A great selection of open source machine learning software]<br />
* [http://stommel.tamu.edu/~baum/toolboxes.html A list of Matlab toolboxes]<br />
* [http://mdoboard.proboards59.com/ ISSMO-REASON: Research and Engineering Applications in Structural Optimization Network]<br />
* [http://www.nafems.org/about/ NAFEMS]<br />
* [http://www.kat-net.net/ The European Coordinating Action on Key Aerodynamic Technologies]<br />
* [http://www.altairhyperworks.co.uk/Default.aspx Altair Hyperworks]<br />
* [http://www.ifte.de/english/research/index.html IFTE]<br />
* [http://www.optiy.eu/Features.html OptyI]<br />
* [http://www.technet-alliance.com/ Technet Alliance]<br />
<br />
=== Conference links ===<br />
<br />
* [http://www.conferencealerts.com Conference alerts]<br />
* [http://www.conferencealerts.com/engineer.htm Engineering conferences]<br />
* [http://www.conferencealerts.com/ai.htm AI Conference alerts]<br />
* [http://www.ieee.org/web/conferences/search/index.html IEEE conferences]<br />
* [http://www.aiaa.org/content.cfm?pageid=1 AIAA Conferences]<br />
* [http://www.wikicfp.com/ Wiki of conferences]<br />
* [http://users.jyu.fi/~miettine/lista.html#Conferences Kaisa's conferences]<br />
* [http://www.conference-service.com/conferences/neural-networks.html AI Conferences]<br />
* [http://ieee-cis.org/conferences/co_sponsorship_1/ IEEE CIS conferences]<br />
* [http://www.makhfi.com/events.htm Neural Network Events]<br />
* [http://openresearch.org/mw/index.php?title=Upcoming_deadlines&field=Machine+learning Open research AI conferences]<br />
* [http://ppsn2014.ijs.si International Conference on Parallel Problem Solving from Nature]<br />
* [http://icannga.com International Conference on Adaptive and Natural Computing Algorithms]<br />
* [http://www.engopt.org/ EngOpt]<br />
<br />
=== Journal Links ===<br />
<br />
*[http://www.linklings.net/tomacs/charter.html ACM Transactions on Modeling and Computer Simulation]<br />
*[http://www.elsevier.com/wps/find/journaldescription.cws_home/622330/description#description Simulation Modelling Practice and Theory]<br />
*[http://www.elsevier.com/wps/find/journaldescription.cws_home/422911/description#description Advances in Engineering Software]<br />
*[http://journaltool.asme.org/Content/JournalDescriptions.cfm?journalId=12 Journal of Mechanical Design ]<br />
*[http://journaltool.asme.org/Content/JournalDescriptions.cfm?journalId=3&Journal=JCISE Journal of Computing and Information Science in Engineering]<br />
*[http://www.elsevier.com/wps/find/journaldescription.cws_home/975/description#description Engineering Applications of Artificial Intelligence]<br />
*[http://www.elsevier.com/wps/find/journaldescription.cws_home/622240/description#description Advanced Engineering Informatics]<br />
*[http://www.springer.com/computer/information+systems/journal/366 Engineering with Computers]<br />
*[http://www.springer.com/computer/mathematics/journal/521 Neural Computing and Applications]<br />
*[http://www.aiaa.org/content.cfm?pageid=322&lupubid=2 AIAA Journal]<br />
*[http://journals.cambridge.org/action/displayJournal?jid=aie Artificial Intelligence for Engineering Design, Analysis and Manufacturing ]<br />
*[http://jmlr.csail.mit.edu/ Journal of Machine Learning Research]<br />
*[http://www.elsevier.com/wps/find/journaldescription.cws_home/505645/description#description Computer Methods in Applied Mechanics and Engineering]<br />
*[http://www.elsevier.com/wps/find/journaldescription.cws_home/524998/description#description Applied Mathematical Modelling]<br />
*[http://www.siam.org/journals/sisc.php SIAM Journal on Scientific Computing]<br />
*[http://www.iop.org/EJ/journal/-page=scope/0266-5611 Inverse Problems]<br />
*[http://www.tandf.co.uk/journals/titles/17415977.asp Inverse problems in science and engineering]<br />
*[http://ieeexplore.ieee.org/xpl/RecentIssue.jsp?punumber=3468 IEEE Transactions on Systems, Man and Cybernetics, Part A]<br />
*[http://ieeexplore.ieee.org/xpl/RecentIssue.jsp?punumber=3477 IEEE Transactions on Systems, Man and Cybernetics, Part B]<br />
*[http://ieeexplore.ieee.org/xpl/RecentIssue.jsp?punumber=5326 IEEE Transactions on Systems, Man and Cybernetics, Part C]<br />
*[http://www.techscience.com/cmes/aims_scope.html Computer Modeling in Engineering & Sciences]<br />
*[http://www.informaworld.com/smpp/title~db=all~content=t713723652~tab=summary Journal of Experimental & Theoretical Artificial Intelligence]<br />
*[http://www.elsevier.com/locate/jocs Journal of Computational Science]</div>Sprashanthttp://sumowiki.intec.ugent.be/Useful_LinksUseful Links2014-01-23T23:12:24Z<p>Sprashant: /* Conference links */</p>
<hr />
<div>=== Related publications ===<br />
<br />
See the [[Related publications]] page.<br />
<br />
=== Related projects ===<br />
<br />
A list of projects with similar ideas and scope.<br />
* [http://www.cs.rtu.lv/jekabsons/regression.html] Matlab regression software<br />
* [http://sites.google.com/site/gptips4matlab GTIPS] Matlab symbolic regression software using genetic programming<br />
* [http://www.optiy.eu OptiY]<br />
* [http://www.nutechsolutions.com/prod_cv_analytics.asp ClearVu Analytics]<br />
* [https://sites.google.com/site/felipeacviana/ Surrogates Toolbox]: Another matlab surrogate modeling toolbox<br />
* [http://www.evolved-analytics.com Evolved-Analytics]: DataModeler package.<br />
* [http://www.muneda.com/ MunEDA]: MunEDA provides leading EDA software technology for analysis, modelling and optimization of yield and performance of analog, mixed-signal and digital designs.<br />
* [http://www.lsoptsupport.com/faqs/setting-parameters-for-metamodel-based-optimization-strategies LS-OPT: Functionality similar to the SUMO-Toolbox in LS-OPT]<br />
* [http://mucm.aston.ac.uk/MUCM/MUCMToolkit/index.php?page=MetaHomePage.html MUCH Toolkit]: A toolbox for sensitivity analysis using surrogate models<br />
* [http://home.mit.bme.hu/~kollar/topics/fdident.html FDIDENT: frequency domain identification toolbox]<br />
* [http://www.dmoz.org/Science/Math/Statistics/Software/Regression_and_Curve_Fitting/: Regression software on Open Directory]<br />
* [http://www.infiniscale.com/ Infiniscale]: TechModeler, TechAnalyzer: automatic model generation tools for EM applications<br />
* [http://www.vrand.com/visualDOC.html visualDOC]: VisualDOC, a design optimization tool<br />
* [http://www.salford-systems.com/mars.php MARS]: A spline based modeling tool<br />
* [http://www.phoenix-int.com/software/phx_modelcenter.php Modelcenter]: datamining and modeling tool<br />
* [http://www.simulia.com/products/isight2.html iSIGHT]: datamining and modeling tool<br />
* [http://www.tmpinc.com/datascape_overview.html Datascape]: datamining and modeling tool<br />
* [http://www.friendship-systems.com/ Friendship systems]: Tools for modeling ship hulls parametrically and performing the modeling calculations (Equilibrium tool)<br />
* [http://www.csse.monash.edu.au/%7Edavida/nimrod/nimrodg.htm Nimrod/G]: Execute parameter sweeps on the grid<br />
* [http://www-sop.inria.fr/oasis/ProActive/ ProActive]: Java grid library <br />
* [http://www.csse.monash.edu.au/%7Edavida/nimrod/nimrodo.htm Nimrod/O]: Grid-enabled optimization toolkit <br />
* [http://www.ece.northwestern.edu/OTC/ NEOS]: Distributed optimization toolkit <br />
* [http://software.sci.utah.edu/scirun.html SciRun]: Problem Solving Environment (PSE), for simulation, modeling, and visualization of scientific problems. <br />
* [http://www.esteco.it/ ModeFRONTIER]: environment dedicated to the set up of design assessment chains and efficient investigation of the design space. <br />
* [http://www.gridbus.org/ Gridbus]: metascheduler for the grid <br />
* [http://icl.cs.utk.edu/netsolve/overview/ Gridsolve/Netsolve]: grid enabled scientific computation toolbox <br />
* [http://www.dtreg.com/ DTREG]: a powerful statistical analysis program that generates classification and regression trees and Support Vector Machine models that can be used to predict parameter values. <br />
* [http://www.soton.ac.uk/%7Epbn/MDO/ MDO]: A collection of MDO links <br />
* [http://www.research.att.com/~njas/gosset/index.html GOSSET]: A general purpose program for designing experiments <br />
* [http://www.cs.sandia.gov/DAKOTA/ DAKOTA]: Design Analysis Kit for Optimization and Terascale Applications<br />
* [http://www.cs.cf.ac.uk/gridprojects/dipso/ DIPSO]: Wide-Area Distributed Problem Solving (DIPSO) <br />
* [http://www.geodise.org GEODISE]: Grid Enabled Optimisation and Design Search for Engineering<br />
* [http://www.fast.u-psud.fr/ezyfit/ EZfit] : Free curve fitting toolbox for matlab<br />
* [http://www.ians.uni-stuttgart.de/spinterp/about.html SGIT] : A Sparse grid interpolation toolbox<br />
* [http://www.csie.ntu.edu.tw/~yien/quickrbf/quickstart.php QuickRBF] : an RBF fitting library (native)<br />
* [http://www.farfieldtechnology.com/products/toolbox/ FastRBF] : another RBF fitting library (matlab)<br />
* [http://promethee.irsn.org PROMETHEE project: easy parametric modeling]<br />
* [http://simlab.jrc.ec.europa.eu/ Sensitivity Analysis library]<br />
* Gaussian Process Matlab code<br />
** [http://www.gaussianprocess.org/gpml/code/matlab/doc/ Code] based on Rasmussen's book<br />
** [http://www.cs.man.ac.uk/~neill/gp/ Gaussian Process Software]<br />
** [http://www.ios.htwg-konstanz.de/joomla_mof/index.php?option=com_content&view=article&id=48:polyreg-polynomial-gaussian-process-regression&catid=36:code&Itemid=81 GP] using polynomial covariance functions<br />
* [http://shark-project.sourceforge.net/ Shark Machine Learning Library - An open source machine learning bundle]<br />
<br />
=== Related labs ===<br />
<br />
A list of some of the labs/researchers with similar ideas and scope.<br />
<br />
* [http://aerospace.engin.umich.edu/index.html Department of Aerospace Engineering at the University of Michigan]<br />
* [http://www.mae.ufl.edu/~mdo/ The Structural and Multidisciplinary Optimization Group at the University of Florida]<br />
* [http://web.engr.oregonstate.edu/~tgd/ School of Electrical Engineering and Computer Science, Oregon State U]<br />
* [http://www.soton.ac.uk/engineering/research/groups/ced.page Southampton University Computational Engineering and Design Group]<br />
* [http://edog.mne.psu.edu/ Engineering Design & Optimization Group (Penn state)]<br />
* [http://nd.edu/~ame/ Aerospace and Mechanical Engineering] [http://www.gano.name/shawn/ Homepage]<br />
* [http://shyylab.engin.umich.edu/research/design-optimization Computational Thermo-Fluids Group]<br />
* [http://www.ensc.sfu.ca/~gwa5/index.htm Product Design and Optimization Laboratory (PDOL)]<br />
* [http://www.cerfacs.fr/4-25708-Home.php Computational Fluid Dynamics (CFD) group at CERFACS]<br />
* [http://webuser.uni-weimar.de/~roos1/ Dirk Roos]<br />
* [http://www.cerfacs.fr/~duchaine/HTML/research.htm Florent Duchaine]<br />
* http://www.nlr.nl/ National aerospace lab]<br />
<br />
=== Data sets - Simulation code ===<br />
<br />
A list of publicly available datasets and simulation codes, useful for testing.<br />
<br />
* [http://www.mat.univie.ac.at/~neum/stat.html Statistics links] : A nice collection of data fitting and analysis codes<br />
* [http://en.wikipedia.org/wiki/Classic_data_sets Wikipedia Classic Datasets]<br />
* [http://www.uni-koeln.de/themen/Statistik/data/rousseeuw/ The ROUSSEEUW datasets]<br />
* [ftp://ftp.sas.com/pub/neural/dojo/dojo.html Donoho-Johnstone Benchmarks] <br />
* [http://people.scs.fsu.edu/~burkardt/datasets/datasets.html datasets]<br />
* [http://www-syscom.univ-mlv.fr/~vignat/Signal/Space/index.html Spice like simulator for matlab]<br />
* [http://www.itl.nist.gov/div898/strd/general/dataarchive.html Nist dataset archive]<br />
* [http://homes.esat.kuleuven.be/~smc/daisy/daisydata.html Daisy datasets]<br />
* [http://lib.stat.cmu.edu/datasets/ Statlib dataset archive]<br />
* [http://www.iau.dtu.dk/nnbook/systems.html Datasets from the book "Neural networks for the modeling and control of dynamic systems.]<br />
* [http://www.gpc.de/e_poses.html A simulation environment for production and transport, logistics and automation systems]<br />
* [http://shumway.physics.asu.edu/codes.html Nanostructure Simulation and Modeling Programs]<br />
* [http://pedsim.silmaril.org/ A Modular, Distributed Pedestrian Crowd Simulation System]<br />
* [http://g95.sourceforge.net/g95_status.html Fortran simulation codes]<br />
* [http://www.genie.ac.uk/ Grid ENabled Integrated Earth system model]<br />
* [http://gcmd.nasa.gov/KeywordSearch/Home.do?Portal=GCMD&MetadataType=0 Nasa datasets]<br />
* [http://funapp.cs.bilkent.edu.tr/DataSets/ Function approximation repository]<br />
* [http://www.idsia.ch/~andrea/Andrea_Rizzoli_Home_Page/Sim_Tools.html A Collection of Modeling and Simulation Resources on the Internet]<br />
* [http://www.grc.nasa.gov/WWW/K-12/freesoftware_page.htm Free simulation software from Nasa]<br />
* [http://opensees.berkeley.edu/index.php Earthquake simulation]<br />
* [http://www.pdl.cmu.edu/DiskSim/ Harddisk simulator]<br />
* [http://spib.rice.edu/spib/mtn_top.html Mountain top radar data]<br />
* [http://www.statsci.org/datasets.html Statsci dataset repository]<br />
* [http://statwww.epfl.ch/davison/BMA/Data4BMA/ dataset repository]<br />
* [http://astrostatistics.psu.edu/datasets/ dataset repository]<br />
* [http://www.maths.uq.edu.au/CEToolBox/ problems from the Cross Entropy toolbox]<br />
* [http://www.cs.waikato.ac.nz/~ml/weka/index_datasets.html Weka datasets]<br />
* [http://www.ailab.si/orange/datasets.asp?Inst=on&Atts=on&Class=on&Values=on&Description=on&sort=Data+Set dataset repository]<br />
* [http://www.itee.uq.edu.au/%7Emarcusg/msg.html Max Set of Gaussians Landscape Generator]<br />
* [http://www.mathworks.fr/matlabcentral/fileexchange/loadAuthor.do?objectId=364966&objectType=author Collection of useful Matlab scripts by Thomas Abrahamsson ]<br />
* [http://www.mathworks.com/matlabcentral/fileexchange/loadFile.do?objectId=10731&objectType=FILE# A Matlab toolbox for Linear Structural Dynamics Analysis]<br />
* [http://webscripts.softpedia.com/script/Scientific-Engineering-Ruby/Controls-and-Systems-Modeling/StructDyn-32656.html Matlab simulators for control and systems modeling]<br />
* [http://www.ae.uiuc.edu/m-selig/ads.html UIUC Airfoil Data Site]<br />
* [http://public.ca.sandia.gov/TNF/abstract.html TNF workshop data archives]<br />
* [http://www.gaussianprocess.org Code for kriging models]<br />
* [http://www.comsol.be/ Comsol multiphysics] package<br />
<br />
=== Predefined functions ===<br />
<br />
* [http://www.cs.cmu.edu/afs/cs/project/jair/pub/volume24/ortizboyer05a-html/node6.html Continuous benchmark problems]<br />
* [http://www2.imm.dtu.dk/~km/Test_ex_forms/test_ex.html Optimization test functions]<br />
* [http://www.geatbx.com/docu/fcnindex-01.html Optimization test functions]<br />
* [http://www.ewh.ieee.org/soc/es/May2001/14/Begin.htm Optimization test functions]<br />
* [http://www.it.lut.fi/ip/evo/functions/functions.html Optimization test functions]<br />
* [http://www.cs.colostate.edu/~genitor/functions.html Optimization test functions]<br />
* [http://www-optima.amp.i.kyoto-u.ac.jp/member/student/hedar/Hedar_files/TestGO_files/Page364.htm Optimization test functions]<br />
* [http://www.it.lut.fi/ip/evo/functions/functions.html Functions with multiple global optima]<br />
* [http://www.mat.univie.ac.at/~neum/glopt/moretest More test set]<br />
* [http://www.mat.univie.ac.at/~neum/software/dixon.tar.gz Dixon-Szegö test set]<br />
* [http://titan.princeton.edu/2010-10-11/TestProblems/ Handbook of Test Problems for local and global optimization]<br />
<br />
=== Optimization ===<br />
<br />
* [http://ab-initio.mit.edu/wiki/index.php/NLopt NLopt is a free/open-source library for nonlinear optimization]<br />
* [http://www.tik.ee.ethz.ch/sop/pisa/ PISA optimization framework]<br />
* [http://www.hvass-labs.org/projects/swarmops/ swarmOps]<br />
* [http://www.icsi.berkeley.edu/~storn/code.html Differential Evolution]<br />
* [http://control.ee.ethz.ch/~joloef/wiki/pmwiki.php YALMIP] High-level optimization problem solver, uses external solvers as backend<br />
* [http://www.mat.univie.ac.at/~neum/glopt/software_g.html Global optimization software]<br />
* [http://www.gerad.ca/NOMAD/Abramson/nomadm.html Mesh-Adaptive Direct Search (MADS) software in Matlab]<br />
* [http://www-rocq.inria.fr/~gilbert/modulopt/modulopt.html Modulopt]: fortran implementations and some matlab<br />
* [http://www.geodise.org/documentation/OptionsMatlab/html/ MatlabOptions] Matlab interface to the design and optimization package (OPTIONS).<br />
<br />
=== Various links ===<br />
<br />
* [http://videolectures.net/Top/Computer_Science/Machine_Learning/ An excellent collection of machine learning lecture videos]<br />
* [http://home.online.no/~pjacklam/matlab/software/util/fullindex.html A collection of useful Matlab scripts]<br />
* [http://home.online.no/~pjacklam/matlab/doc/mtt/ MATLAB array manipulation tips and tricks]<br />
* [http://www.adaptivebox.net/research/bookmark/psocodes_link.html Particle Swarm implementations]<br />
* [http://mloss.org A great selection of open source machine learning software]<br />
* [http://stommel.tamu.edu/~baum/toolboxes.html A list of Matlab toolboxes]<br />
* [http://mdoboard.proboards59.com/ ISSMO-REASON: Research and Engineering Applications in Structural Optimization Network]<br />
* [http://www.nafems.org/about/ NAFEMS]<br />
* [http://www.kat-net.net/ The European Coordinating Action on Key Aerodynamic Technologies]<br />
* [http://www.altairhyperworks.co.uk/Default.aspx Altair Hyperworks]<br />
* [http://www.ifte.de/english/research/index.html IFTE]<br />
* [http://www.optiy.eu/Features.html OptyI]<br />
* [http://www.technet-alliance.com/ Technet Alliance]<br />
<br />
=== Conference links ===<br />
<br />
* [http://www.conferencealerts.com Conference alerts]<br />
* [http://www.conferencealerts.com/engineer.htm Engineering conferences]<br />
* [http://www.conferencealerts.com/ai.htm AI Conference alerts]<br />
* [http://www.ieee.org/web/conferences/search/index.html IEEE conferences]<br />
* [http://www.aiaa.org/content.cfm?pageid=1 AIAA Conferences]<br />
* [http://www.wikicfp.com/ Wiki of conferences]<br />
* [http://users.jyu.fi/~miettine/lista.html#Conferences Kaisa's conferences]<br />
* [http://www.conference-service.com/conferences/neural-networks.html AI Conferences]<br />
* [http://ieee-cis.org/conferences/co_sponsorship_1/ IEEE CIS conferences]<br />
* [http://www.makhfi.com/events.htm Neural Network Events]<br />
* [http://openresearch.org/mw/index.php?title=Upcoming_deadlines&field=Machine+learning Open research AI conferences]<br />
* [http://ppsn2014.ijs.si International Conference on Parallel Problem Solving from Nature]<br />
* [http://icannga.com International Conference on Adaptive and Natural Computing Algorithms]<br />
<br />
=== Journal Links ===<br />
<br />
*[http://www.linklings.net/tomacs/charter.html ACM Transactions on Modeling and Computer Simulation]<br />
*[http://www.elsevier.com/wps/find/journaldescription.cws_home/622330/description#description Simulation Modelling Practice and Theory]<br />
*[http://www.elsevier.com/wps/find/journaldescription.cws_home/422911/description#description Advances in Engineering Software]<br />
*[http://journaltool.asme.org/Content/JournalDescriptions.cfm?journalId=12 Journal of Mechanical Design ]<br />
*[http://journaltool.asme.org/Content/JournalDescriptions.cfm?journalId=3&Journal=JCISE Journal of Computing and Information Science in Engineering]<br />
*[http://www.elsevier.com/wps/find/journaldescription.cws_home/975/description#description Engineering Applications of Artificial Intelligence]<br />
*[http://www.elsevier.com/wps/find/journaldescription.cws_home/622240/description#description Advanced Engineering Informatics]<br />
*[http://www.springer.com/computer/information+systems/journal/366 Engineering with Computers]<br />
*[http://www.springer.com/computer/mathematics/journal/521 Neural Computing and Applications]<br />
*[http://www.aiaa.org/content.cfm?pageid=322&lupubid=2 AIAA Journal]<br />
*[http://journals.cambridge.org/action/displayJournal?jid=aie Artificial Intelligence for Engineering Design, Analysis and Manufacturing ]<br />
*[http://jmlr.csail.mit.edu/ Journal of Machine Learning Research]<br />
*[http://www.elsevier.com/wps/find/journaldescription.cws_home/505645/description#description Computer Methods in Applied Mechanics and Engineering]<br />
*[http://www.elsevier.com/wps/find/journaldescription.cws_home/524998/description#description Applied Mathematical Modelling]<br />
*[http://www.siam.org/journals/sisc.php SIAM Journal on Scientific Computing]<br />
*[http://www.iop.org/EJ/journal/-page=scope/0266-5611 Inverse Problems]<br />
*[http://www.tandf.co.uk/journals/titles/17415977.asp Inverse problems in science and engineering]<br />
*[http://ieeexplore.ieee.org/xpl/RecentIssue.jsp?punumber=3468 IEEE Transactions on Systems, Man and Cybernetics, Part A]<br />
*[http://ieeexplore.ieee.org/xpl/RecentIssue.jsp?punumber=3477 IEEE Transactions on Systems, Man and Cybernetics, Part B]<br />
*[http://ieeexplore.ieee.org/xpl/RecentIssue.jsp?punumber=5326 IEEE Transactions on Systems, Man and Cybernetics, Part C]<br />
*[http://www.techscience.com/cmes/aims_scope.html Computer Modeling in Engineering & Sciences]<br />
*[http://www.informaworld.com/smpp/title~db=all~content=t713723652~tab=summary Journal of Experimental & Theoretical Artificial Intelligence]<br />
*[http://www.elsevier.com/locate/jocs Journal of Computational Science]</div>Sprashanthttp://sumowiki.intec.ugent.be/AboutAbout2014-01-08T11:37:09Z<p>Sprashant: /* Features */</p>
<hr />
<div>== History ==<br />
In 2004, research within the (former) COMS research group, led by professor [http://www.sumo.intec.ugent.be/?q=tomd Tom Dhaene], was focused on developing efficient, adaptive and accurate algorithms for polynomial and rational modeling of linear time-invariant (LTI) systems. This work resulted in a set of Matlab scripts that were used as a testing ground for new ideas and concepts. Research progressed, and with time these scripts were re-worked and refactored into one coherent Matlab toolbox, tentatively named the Multivariate MetaModeling (M3) Toolbox. The first public release of the toolbox (v2.0) occurred in November 2006. In October 2007, the development of the M3 Toolbox was discontinued.<br />
<br />
[[Image:Sumo.jpg|150 px|right|blindDACE Toolbox]]<br />
In April 2008, the first public release of the Surrogate Modeling (SUMO) Toolbox (v5.0) occurred.<br />
<br />
For a list of changes since then refer to the [[Changelog]] and [[Whats new]] pages.<br />
<br />
== Intended use ==<br />
<br />
=== Global Surrogate Models ===<br />
The SUMO Toolbox was originally designed to solve the following problem:<br />
<br />
<center>''Automatically generate a highly accurate surrogate model (= a regression model) for a computational expensive simulation code<br />
<br>requiring as little data points and as little user-interaction as possible.''</center><br />
<br />
In addition the toolbox provides powerful, adaptive algorithms and a whole suite of model types for<br />
* data fitting problems (regression, function approximation, curve fitting)<br />
* response surface modeling (RSM)<br />
* scattered data interpolation<br />
* model selection<br />
* Design Of Experiments (DoE)<br />
* model parameter optimization, e.g., finding the optimal neural network topology, SVM kernel parameters, rational function order, etc. (= hyperparameter optimization)<br />
* iterative adaptive sample selection (also known as sequential design or active learning)<br />
<br />
Note that the SUMO toolbox is able to drive the simulation code directly.<br />
<br />
For domain experts or engineers the SUMO Toolbox provides a flexible, pluggable platform to which the response surface modeling task can be delegated. For researchers in surrogate modeling it provides a common framework to implement, test and benchmark new modeling and sampling algorithms.<br />
<br />
See the Wikipedia [http://en.wikipedia.org/wiki/Surrogate_model surrogate model] page to find out more.<br />
<br />
=== Surrogate Driven Optimization ===<br />
While the main focus of the SUMO Toolbox is to create accurate global surrogate models, it can be used for other goals too.<br />
<br />
For instance, the toolbox can be used to create consecutive local surrogate models for optimization purposes. The information obtained from the local surrogate models is used to guide the adaptive sampling process to the global optimum.<br />
<br />
A good sample strategy for surrogate driven optimization seeks a balance between local search and global search, or refining the surrogate model and finding the optimum.<br />
Such a sample strategy is implemented (akin to (Super)EGO), see the different [[Sample_Selectors#expectedImprovement|sample selectors]] for more information.<br />
<br />
=== Dynamic systems or Time series prediction ===<br />
<br />
See [[FAQ#What_about_dynamical.2C_time_dependent_data.3F]].<br />
<br />
=== Classification ===<br />
<br />
See [[FAQ#What_about_classification_problems.3F]].<br />
<br />
== Application range ==<br />
The SUMO Toolbox has already been applied successfully to a wide range of problems from domains as diverse as aerodynamics, geology, metallurgy, electro-magnetics (EM), electronics, engineering and economics. The SUMO Toolbox can be applied to any situation where the problem can be described as a function that maps a set of inputs onto a set of outputs. We generally refer to this function as the [[Simulator]].<br />
<br />
<br />
[[Image:sumotask.png|center|SUMO-Toolbox : Generating an approximation for a reference model]]<br />
<br />
Across the different problems to which we have applied the toolbox, the input dimension has ranged from 1 to 130 and the output dimension from 1 to 70 (including both complex and real valued outputs). The number of data points has ranged from as little as 15 to as many as 100000.<br />
<br />
== Design goals ==<br />
<br />
The SUMO Toolbox was designed with a number of goals in mind:<br />
<br />
* A flexible tool that integrates different modeling methods and does not tie the user down to one particular set of problems. Reliance on domain specific features should be avoided.<br />
<br />
* The focus should be on adaptivity, i.e., relieving the burden on the domain expert as much as possible. Given a simulation model, the software should produce an accurate surrogate model with minimal user interaction. This also includes easily integrating with the existing design environment.<br />
<br />
* At the same time keeping in mind that there is no such thing as a `one-size-fits-all'. Different problems need to be modeled differently and require different a priori process knowledge. Therefore the software should be modular and easily extensible to new methods.<br />
<br />
* Engineers or domain experts do not tend to trust a black box system that generates models but is unclear about the reasons why a particular model should be preferred. Therefore an important design goal was that the expert user should be able to have full manual control over the modeling process if necessary. In addition the toolbox should support fine grain logging and profiling capabilities so its modeling and sampling decisions can be retraced.<br />
<br />
Given this design philosophy, the toolbox can cater to both the researchers working on novel surrogate modeling techniques as well as to the engineers who need the surrogate model as part of their design process. For the former, the toolbox provides a common platform on which to deploy, test, and compare new modeling algorithms and sampling techniques. For the latter, the software functions as a highly configurable and flexible component to which surrogate model construction can be delegated, easing the burden of the user and enhancing productivity.<br />
<br />
== Features ==<br />
The main features of the toolbox are listed below. For an overview of recent changes see the [[Whats new]] page. A detailed list of changes can be found in the [[Changelog]].<br />
<br />
{| class="wikitable" style="text-align:left" border="0" cellpadding="5" cellspacing="0"<br />
! Implementation Language <br />
| Matlab, Java, and where applicable C, C++<br />
|- <br />
! Design patterns<br />
| Fully object oriented, with the focus on clean design and encapsulation.<br />
|- <br />
! Minimum Requirements<br />
| See the [[system requirements]] page<br />
|-<br />
! Supported data sources*<br />
| Local executable/script, simulation engine, Java class, Matlab script, dataset (txt file) (see [[Interfacing with the toolbox]])<br />
|-<br />
! Supported data types<br />
| Supports multi-dimensional inputs and outputs. Outputs can be any combination of real/complex.<br />
|-<br />
! Supported problem types<br />
| Regression ([[FAQ#What_about_classification_problems.3F|classification]], [[FAQ#What_about_dynamical.2C_time_dependent_data.3F|time series prediction]])<br />
|-<br />
! Configuration<br />
| Extensively configurable through one main [[FAQ#What_is_XML.3F|XML]] configuration file.<br />
|-<br />
! Flexibility<br />
| Virtually every component of the modeling process can be configured, replaced or extended by a user specific, custom implementation<br />
|-<br />
! Predefined accuracy<br />
| The toolbox will run until the user required accuracy has been reached, the maximum number of samples has been exceeded or a timeout has occurred<br />
|-<br />
! Model Types*<br />
| Out of the box support for:<br />
* Polynomial/Rational functions<br />
* Feedforward Neural Networks, 3 implementations<br />
** One based on the [http://www.mathworks.com/products/neuralnet/ Matlab Neural Network toolbox]<br />
** One based on the [http://leenissen.dk/fann/ Fast Artificial Neural Network Library (FANN)]<br />
** One based on the [http://www.iau.dtu.dk/research/control/nnsysid.html NNSYSID Toolbox]<br />
* Radial Basis Function (RBF) Models<br />
* RBF Neural Networks<br />
* Gaussian Process Models (based on [http://www.GaussianProcess.org/gpml/code GPML])<br />
* Kriging Models (two custom implementations)<br />
* Blind Kriging Models<br />
* Smoothing spline models<br />
* Support Vector Machines (SVM)<br />
** Least Squares SVM (based on [http://www.esat.kuleuven.ac.be/sista/lssvmlab/ LS-SVMlab])<br />
** epsilon-SVM (based on [http://www.csie.ntu.edu.tw/~cjlin/libsvm/ LIBSVM] or [http://svmlight.joachims.org/ SVMlight])<br />
** nu-SVM (based on [http://www.csie.ntu.edu.tw/~cjlin/libsvm/ LIBSVM])<br />
* Support for model types from the [http://www.cs.waikato.ac.nz/ml/weka/ WEKA] data mining library <br />
** Classification, regression and clustering algorithms present in WEKA can be used simply by adding entries in the configuration. Please refer to the 'weka' ModelBuilder in the demo configuration file. A list of algorithms can be found [http://wiki.pentaho.com/display/DATAMINING/Data+Mining+Algorithms+and+Tools+in+Weka here].<br />
|-<br />
! Model parameter optimization algorithms*<br />
| Pattern Search, EOG, Simulated Annealing, Genetic Algorithm, BGFS, DIRECT, Particle Swarm Optimization (PSO), NSGA-II ...<br />
|-<br />
! Sample selection algorithms (=sequential design, active learning)*<br />
| Random, error-based, density-based, gradient-based, and many different hybrids<br />
|-<br />
! Experimental design*<br />
| Latin Hypercube Sampling, Central Composite, Box-Behnken, random, user defined, full factorial<br />
|-<br />
! Model selection measures*<br />
| Validation set, cross-validation, leave-one-out, model difference, AIC (also in a multi-objective context, see [[Multi-Objective Modeling]])<br />
|-<br />
! Sample Evaluation*<br />
| On the local machine (taking advantage of multi-core CPUs) or in parallel on a cluster/grid<br />
|-<br />
! Supported distributed middlewares*<br />
| [http://gridengine.sunsource.net/ Sun Grid Engine], LCG Grid middleware (both accessed through a SSH accessible frontnode)<br />
|-<br />
! Logging<br />
| Extensive logging to enable close monitoring of the modeling process. Logging granularity is fully configurable and log streams can be easily redirected (to file, console, a remote machine, ...).<br />
|-<br />
! Profiling*<br />
| Extensive profiling framework for easy gathering (and plotting) of modeling metrics (average sample evaluation time, hyperparameter optimization trace, ...)<br />
|-<br />
! Easy tracking of modeling progress<br />
| Automatic storing of best models and their plots. Ability to automatically generate a movie of the sequence of plots.<br />
|-<br />
! Model browser GUI<br />
| A graphical tool is available to easily visualize high dimensional models and browse through data ([[Model Visualization GUI|more information here]])<br />
|-<br />
! Available test problems*<br />
| Out of the box support for many built-in functions (Ackley, Camel Back, Goldstein-Price, ...) and datasets (Abalone, Boston Housing, FishLength, ...) from various application domains. Including a number of datasets (and some simulation code) from electronics. In total over 50 examples are available.<br />
|-<br />
! License<br />
| [[License terms]]<br />
|}<br />
<br />
<nowiki>*</nowiki> Custom implementations can easily be added<br />
<br />
== Screenshots ==<br />
A number of screenshots to give a feel of the SUMO Toolbox. Note these screenshots do not necessarily reflect the latest toolbox version.<br />
<br />
<gallery><br />
Image:octagon.png<br />
Image:metamodel-sumo-hourglass.png<br />
Image:SUMO_Toolbox1.png<br />
Image:SUMO_Toolbox2.png<br />
Image:SUMO_Toolbox3.png<br />
Image:SUMO_Toolbox4.png<br />
Image:ISCSampleSelector1.png<br />
Image:ISCSampleSelector2.png<br />
Image:SUMO_Gui1.png<br />
Image:SUMO_Gui2.png<br />
Image:Contour1.png<br />
Image:TwoDim1.png<br />
Image:TwoDim2.png<br />
Image:ThreeDim1.png<br />
Image:ThreeDim2.png<br />
Image:ThreeDim3.png<br />
Image:FEBioTrekEI.png<br />
Image:FEBioTrekFunc.png<br />
</gallery><br />
<br />
== Movies ==<br />
<br />
[[Image:youtube-logo.jpg|right|70px|link=http://www.youtube.com/sumolab|]] A number of video clips generated by or related to the SUMO Toolbox [http://www.youtube.com/sumolab can be found at our YouTube channel]. Feel free to make suggestions or leave comments.<br />
<br />
Note these movies do not necessarily reflect the latest toolbox version. Improvements and/or interface adjustments may have been made since then.<br />
<br />
== Documentation ==<br />
<br />
An in depth overview of the rationale and philosophy, including a treatment of the software architecture underlying the SUMO Toolbox is available in the form of a PhD dissertation. A copy of this dissertation [http://www.sumo.intec.ugent.be/?q=system/files/2010_04_PhD_DirkGorissen.pdf is available here].<br />
<br />
In addition the following poster and presentation give a high level overview:<br />
<br />
* Poster: [[Media:SUMO_poster.pdf|SUMO poster]]<br />
* Presentation: [[Media:SUMO_presentation.pdf|SUMO slides]]<br />
<br />
To stay up to date with the latest news and releases, we also recommend subscribing to the [http://www.sumo.intec.ugent.be SUMO newsletter]. <br />
Traffic will be kept to a minimum and you can unsubscribe at any time.<br />
<br />
A blog covering related research can be found here [http://sumolab.blogspot.com http://sumolab.blogspot.com].<br />
<br />
== Citations ==<br />
<br />
See [[Citing|Citing the toolbox]].</div>Sprashanthttp://sumowiki.intec.ugent.be/AboutAbout2014-01-08T11:17:03Z<p>Sprashant: /* Features */</p>
<hr />
<div>== History ==<br />
In 2004, research within the (former) COMS research group, led by professor [http://www.sumo.intec.ugent.be/?q=tomd Tom Dhaene], was focused on developing efficient, adaptive and accurate algorithms for polynomial and rational modeling of linear time-invariant (LTI) systems. This work resulted in a set of Matlab scripts that were used as a testing ground for new ideas and concepts. Research progressed, and with time these scripts were re-worked and refactored into one coherent Matlab toolbox, tentatively named the Multivariate MetaModeling (M3) Toolbox. The first public release of the toolbox (v2.0) occurred in November 2006. In October 2007, the development of the M3 Toolbox was discontinued.<br />
<br />
[[Image:Sumo.jpg|150 px|right|blindDACE Toolbox]]<br />
In April 2008, the first public release of the Surrogate Modeling (SUMO) Toolbox (v5.0) occurred.<br />
<br />
For a list of changes since then refer to the [[Changelog]] and [[Whats new]] pages.<br />
<br />
== Intended use ==<br />
<br />
=== Global Surrogate Models ===<br />
The SUMO Toolbox was originally designed to solve the following problem:<br />
<br />
<center>''Automatically generate a highly accurate surrogate model (= a regression model) for a computational expensive simulation code<br />
<br>requiring as little data points and as little user-interaction as possible.''</center><br />
<br />
In addition the toolbox provides powerful, adaptive algorithms and a whole suite of model types for<br />
* data fitting problems (regression, function approximation, curve fitting)<br />
* response surface modeling (RSM)<br />
* scattered data interpolation<br />
* model selection<br />
* Design Of Experiments (DoE)<br />
* model parameter optimization, e.g., finding the optimal neural network topology, SVM kernel parameters, rational function order, etc. (= hyperparameter optimization)<br />
* iterative adaptive sample selection (also known as sequential design or active learning)<br />
<br />
Note that the SUMO toolbox is able to drive the simulation code directly.<br />
<br />
For domain experts or engineers the SUMO Toolbox provides a flexible, pluggable platform to which the response surface modeling task can be delegated. For researchers in surrogate modeling it provides a common framework to implement, test and benchmark new modeling and sampling algorithms.<br />
<br />
See the Wikipedia [http://en.wikipedia.org/wiki/Surrogate_model surrogate model] page to find out more.<br />
<br />
=== Surrogate Driven Optimization ===<br />
While the main focus of the SUMO Toolbox is to create accurate global surrogate models, it can be used for other goals too.<br />
<br />
For instance, the toolbox can be used to create consecutive local surrogate models for optimization purposes. The information obtained from the local surrogate models is used to guide the adaptive sampling process to the global optimum.<br />
<br />
A good sample strategy for surrogate driven optimization seeks a balance between local search and global search, or refining the surrogate model and finding the optimum.<br />
Such a sample strategy is implemented (akin to (Super)EGO), see the different [[Sample_Selectors#expectedImprovement|sample selectors]] for more information.<br />
<br />
=== Dynamic systems or Time series prediction ===<br />
<br />
See [[FAQ#What_about_dynamical.2C_time_dependent_data.3F]].<br />
<br />
=== Classification ===<br />
<br />
See [[FAQ#What_about_classification_problems.3F]].<br />
<br />
== Application range ==<br />
The SUMO Toolbox has already been applied successfully to a wide range of problems from domains as diverse as aerodynamics, geology, metallurgy, electro-magnetics (EM), electronics, engineering and economics. The SUMO Toolbox can be applied to any situation where the problem can be described as a function that maps a set of inputs onto a set of outputs. We generally refer to this function as the [[Simulator]].<br />
<br />
<br />
[[Image:sumotask.png|center|SUMO-Toolbox : Generating an approximation for a reference model]]<br />
<br />
Across the different problems to which we have applied the toolbox, the input dimension has ranged from 1 to 130 and the output dimension from 1 to 70 (including both complex and real valued outputs). The number of data points has ranged from as little as 15 to as many as 100000.<br />
<br />
== Design goals ==<br />
<br />
The SUMO Toolbox was designed with a number of goals in mind:<br />
<br />
* A flexible tool that integrates different modeling methods and does not tie the user down to one particular set of problems. Reliance on domain specific features should be avoided.<br />
<br />
* The focus should be on adaptivity, i.e., relieving the burden on the domain expert as much as possible. Given a simulation model, the software should produce an accurate surrogate model with minimal user interaction. This also includes easily integrating with the existing design environment.<br />
<br />
* At the same time keeping in mind that there is no such thing as a `one-size-fits-all'. Different problems need to be modeled differently and require different a priori process knowledge. Therefore the software should be modular and easily extensible to new methods.<br />
<br />
* Engineers or domain experts do not tend to trust a black box system that generates models but is unclear about the reasons why a particular model should be preferred. Therefore an important design goal was that the expert user should be able to have full manual control over the modeling process if necessary. In addition the toolbox should support fine grain logging and profiling capabilities so its modeling and sampling decisions can be retraced.<br />
<br />
Given this design philosophy, the toolbox can cater to both the researchers working on novel surrogate modeling techniques as well as to the engineers who need the surrogate model as part of their design process. For the former, the toolbox provides a common platform on which to deploy, test, and compare new modeling algorithms and sampling techniques. For the latter, the software functions as a highly configurable and flexible component to which surrogate model construction can be delegated, easing the burden of the user and enhancing productivity.<br />
<br />
== Features ==<br />
The main features of the toolbox are listed below. For an overview of recent changes see the [[Whats new]] page. A detailed list of changes can be found in the [[Changelog]].<br />
<br />
{| class="wikitable" style="text-align:left" border="0" cellpadding="5" cellspacing="0"<br />
! Implementation Language <br />
| Matlab, Java, and where applicable C, C++<br />
|- <br />
! Design patterns<br />
| Fully object oriented, with the focus on clean design and encapsulation.<br />
|- <br />
! Minimum Requirements<br />
| See the [[system requirements]] page<br />
|-<br />
! Supported data sources*<br />
| Local executable/script, simulation engine, Java class, Matlab script, dataset (txt file) (see [[Interfacing with the toolbox]])<br />
|-<br />
! Supported data types<br />
| Supports multi-dimensional inputs and outputs. Outputs can be any combination of real/complex.<br />
|-<br />
! Supported problem types<br />
| Regression ([[FAQ#What_about_classification_problems.3F|classification]], [[FAQ#What_about_dynamical.2C_time_dependent_data.3F|time series prediction]])<br />
|-<br />
! Configuration<br />
| Extensively configurable through one main [[FAQ#What_is_XML.3F|XML]] configuration file.<br />
|-<br />
! Flexibility<br />
| Virtually every component of the modeling process can be configured, replaced or extended by a user specific, custom implementation<br />
|-<br />
! Predefined accuracy<br />
| The toolbox will run until the user required accuracy has been reached, the maximum number of samples has been exceeded or a timeout has occurred<br />
|-<br />
! Model Types*<br />
| Out of the box support for:<br />
* Polynomial/Rational functions<br />
* Feedforward Neural Networks, 3 implementations<br />
** One based on the [http://www.mathworks.com/products/neuralnet/ Matlab Neural Network toolbox]<br />
** One based on the [http://leenissen.dk/fann/ Fast Artificial Neural Network Library (FANN)]<br />
** One based on the [http://www.iau.dtu.dk/research/control/nnsysid.html NNSYSID Toolbox]<br />
* Radial Basis Function (RBF) Models<br />
* RBF Neural Networks<br />
* Gaussian Process Models (based on [http://www.GaussianProcess.org/gpml/code GPML])<br />
* Kriging Models (two custom implementations)<br />
* Blind Kriging Models<br />
* Smoothing spline models<br />
* Support Vector Machines (SVM)<br />
** Least Squares SVM (based on [http://www.esat.kuleuven.ac.be/sista/lssvmlab/ LS-SVMlab])<br />
** epsilon-SVM (based on [http://www.csie.ntu.edu.tw/~cjlin/libsvm/ LIBSVM] or [http://svmlight.joachims.org/ SVMlight])<br />
** nu-SVM (based on [http://www.csie.ntu.edu.tw/~cjlin/libsvm/ LIBSVM])<br />
* Support for model types from the [http://www.cs.waikato.ac.nz/ml/weka/ WEKA] data mining library <br />
|-<br />
! Model parameter optimization algorithms*<br />
| Pattern Search, EOG, Simulated Annealing, Genetic Algorithm, BGFS, DIRECT, Particle Swarm Optimization (PSO), NSGA-II ...<br />
|-<br />
! Sample selection algorithms (=sequential design, active learning)*<br />
| Random, error-based, density-based, gradient-based, and many different hybrids<br />
|-<br />
! Experimental design*<br />
| Latin Hypercube Sampling, Central Composite, Box-Behnken, random, user defined, full factorial<br />
|-<br />
! Model selection measures*<br />
| Validation set, cross-validation, leave-one-out, model difference, AIC (also in a multi-objective context, see [[Multi-Objective Modeling]])<br />
|-<br />
! Sample Evaluation*<br />
| On the local machine (taking advantage of multi-core CPUs) or in parallel on a cluster/grid<br />
|-<br />
! Supported distributed middlewares*<br />
| [http://gridengine.sunsource.net/ Sun Grid Engine], LCG Grid middleware (both accessed through a SSH accessible frontnode)<br />
|-<br />
! Logging<br />
| Extensive logging to enable close monitoring of the modeling process. Logging granularity is fully configurable and log streams can be easily redirected (to file, console, a remote machine, ...).<br />
|-<br />
! Profiling*<br />
| Extensive profiling framework for easy gathering (and plotting) of modeling metrics (average sample evaluation time, hyperparameter optimization trace, ...)<br />
|-<br />
! Easy tracking of modeling progress<br />
| Automatic storing of best models and their plots. Ability to automatically generate a movie of the sequence of plots.<br />
|-<br />
! Model browser GUI<br />
| A graphical tool is available to easily visualize high dimensional models and browse through data ([[Model Visualization GUI|more information here]])<br />
|-<br />
! Available test problems*<br />
| Out of the box support for many built-in functions (Ackley, Camel Back, Goldstein-Price, ...) and datasets (Abalone, Boston Housing, FishLength, ...) from various application domains. Including a number of datasets (and some simulation code) from electronics. In total over 50 examples are available.<br />
|-<br />
! License<br />
| [[License terms]]<br />
|}<br />
<br />
<nowiki>*</nowiki> Custom implementations can easily be added<br />
<br />
== Screenshots ==<br />
A number of screenshots to give a feel of the SUMO Toolbox. Note these screenshots do not necessarily reflect the latest toolbox version.<br />
<br />
<gallery><br />
Image:octagon.png<br />
Image:metamodel-sumo-hourglass.png<br />
Image:SUMO_Toolbox1.png<br />
Image:SUMO_Toolbox2.png<br />
Image:SUMO_Toolbox3.png<br />
Image:SUMO_Toolbox4.png<br />
Image:ISCSampleSelector1.png<br />
Image:ISCSampleSelector2.png<br />
Image:SUMO_Gui1.png<br />
Image:SUMO_Gui2.png<br />
Image:Contour1.png<br />
Image:TwoDim1.png<br />
Image:TwoDim2.png<br />
Image:ThreeDim1.png<br />
Image:ThreeDim2.png<br />
Image:ThreeDim3.png<br />
Image:FEBioTrekEI.png<br />
Image:FEBioTrekFunc.png<br />
</gallery><br />
<br />
== Movies ==<br />
<br />
[[Image:youtube-logo.jpg|right|70px|link=http://www.youtube.com/sumolab|]] A number of video clips generated by or related to the SUMO Toolbox [http://www.youtube.com/sumolab can be found at our YouTube channel]. Feel free to make suggestions or leave comments.<br />
<br />
Note these movies do not necessarily reflect the latest toolbox version. Improvements and/or interface adjustments may have been made since then.<br />
<br />
== Documentation ==<br />
<br />
An in depth overview of the rationale and philosophy, including a treatment of the software architecture underlying the SUMO Toolbox is available in the form of a PhD dissertation. A copy of this dissertation [http://www.sumo.intec.ugent.be/?q=system/files/2010_04_PhD_DirkGorissen.pdf is available here].<br />
<br />
In addition the following poster and presentation give a high level overview:<br />
<br />
* Poster: [[Media:SUMO_poster.pdf|SUMO poster]]<br />
* Presentation: [[Media:SUMO_presentation.pdf|SUMO slides]]<br />
<br />
To stay up to date with the latest news and releases, we also recommend subscribing to the [http://www.sumo.intec.ugent.be SUMO newsletter]. <br />
Traffic will be kept to a minimum and you can unsubscribe at any time.<br />
<br />
A blog covering related research can be found here [http://sumolab.blogspot.com http://sumolab.blogspot.com].<br />
<br />
== Citations ==<br />
<br />
See [[Citing|Citing the toolbox]].</div>Sprashanthttp://sumowiki.intec.ugent.be/DownloadingDownloading2013-05-07T20:52:05Z<p>Sprashant: Updated SUMO download link</p>
<hr />
<div>== Obtaining the SUMO Toolbox ==<br />
<br />
The SUMO Toolbox is available as open source software for non-commercial use. For commercial use a license must be obtained. For details please see the the [[License terms]]. In both cases, we are always open to [[FAQ#What_are_collaboration_options.3F|some form of collaboration]]. If this is possible, please [[Contact|let us know]].<br />
<br />
Download instructions can be found [http://www.sumo.intec.ugent.be/SUMO#download on the SUMO lab website].<br />
<br />
== Nightly builds ==<br />
<br />
Nightly snapshots of our development tree are available [[Contact|on request]]. These contain the latest bugfixes, newest features but may also be unstable.<br />
<br />
== Installation ==<br />
<br />
See the [[Installation|installation instructions here]].<br />
<br />
== Latest Features ==<br />
<br />
To get an overview of what has changed in each version please consult the [[Whats new]] and [[Changelog]] pages.</div>Sprashanthttp://sumowiki.intec.ugent.be/Main_PageMain Page2013-05-07T13:38:47Z<p>Sprashant: Modified 'Members' link according to updated SUMO website</p>
<hr />
<div><div style="padding:0.7em 0em 0em 0em;"><br />
{{Largebox<br />
| Title = <div style="font-size:200%; padding:.1em;">SUrrogate MOdeling (SUMO) Toolbox</div><br />
| Contents =<br />
The Surrogate Modeling Toolbox (SUMO Toolbox) is a Matlab toolbox that automatically builds accurate [http://en.wikipedia.org/wiki/Surrogate_model surrogate models] (also known as metamodels or [http://en.wikipedia.org/wiki/Response_surface_methodology response surface models]) of a given data source (simulation code, data set, script, ...) within the accuracy and time constraints set by the user. In doing so the toolbox minimizes the number of data points (which it chooses automatically) since they are usually expensive. The toolbox tries to be as adaptive and autonomous as possible, requiring no user input besides some initial configuration.<br />
<br />
[[Image:metamodel-generation.png|500 px|center|SUMO-Toolbox : Generating an approximation for a reference model]]<br />
<br />
<br />
However, since there is no such thing as a ''one-size-fits-all'', the toolbox has been designed to be fully pluggable and extensible using standard [http://en.wikipedia.org/wiki/Object_orientation object oriented] design patterns. Implementations of the different components (model types, sampling strategies, model selection criteria, hyperparameter optimization algorithms,...) can be plugged-in, compared, or replaced by custom implementations. In this way the SUMO Toolbox provides a common platform to easily test and benchmark different sampling and approximation strategies while easily integrating in the engineering design process. [[About|more information....]]<br />
<br />
}}</div><br />
{{3widebox<br />
| Box1_title = [[About]]<br />
| Box1_contents = * [[About | General information]]<br />
* [[About#Design goals|Design goals]] <br />
* [[About#Features|Features]]<br />
* [[About#Screenshots|Screenshots]]<br />
* [[About#Movies|Movies]]<br />
* [[FAQ|Frequently Asked Questions - FAQ]]<br />
| Box2_title = [[Downloading|Download]] - [[Installation]]<br />
| Box2_contents = <br />
* [[System requirements]]<br />
* [[Downloading|Downloading the SUMO Toolbox]]<br />
* [[Installation|Installation instructions]]<br />
* [[License terms]]<br />
* [[Useful Links|Useful links]]<br />
| Box3_title = [[Running|Usage]]<br />
| Box3_contents = <br />
* [[Running#Getting_started|Getting started]]<br />
* [[Running|General usage]]<br />
* [[Configuration]]<br />
* [[Known bugs]]<br />
* [[Extending|Extending the toolbox]]<br />
* [[Tips|Useful tips]]<br />
}}<br />
<br />
{{2widebox<br />
| Box1_title = Open Positions<br />
| Box1_contents = We are constantly looking for highly motivated PhD candidates or post-docs, that have a strong interest in one of our [http://www.sumo.intec.ugent.be/?q=research research topics]. Our research area is highly multidisciplinary, and requires strong mathematical, physical and computer science knowledge.<br />
<br />
Please contact '''tom.dhaene@ugent.be'''<br />
<br />
| Box2_title = [[Feedback|Contact]]<br />
| Box2_contents = <br />
* [http://www.sumo.intec.ugent.be/node/35 Project members]<br />
* [[Feedback]]<br />
* [[Contributing]]<br />
* [[Reporting problems]]<br />
* [[Citing|Citing the toolbox]]<br />
* [[Image:blogger-logo.jpeg|left|15px|link=http://sumolab.blogspot.com Research blog]][http://sumolab.blogspot.com Research blog]<br />
}}<br />
<br />
* This website is under ongoing construction. Since it is a [http://en.wikipedia.org/wiki/Wiki Wiki], you can help by [[Special:Userlogin|contributing]]<br />
__NOEDITSECTION__ __NOTOC__</div>Sprashanthttp://sumowiki.intec.ugent.be/InstallationInstallation2012-11-23T14:19:10Z<p>Sprashant: /* Linux/Unix/OSX */</p>
<hr />
<div>== Introduction ==<br />
This page will walk you through the SUMO Toolbox installation. Please refer to the [[system requirements]] first. See the [[downloading]] section on how to download the toolbox.<br />
<br />
== Quick start ==<br />
<br />
Quick and dirty instructions:<br />
<br />
# Log into the SUMO lab website with the account information mailed to you and download the toolbox<br />
# Unzip the toolbox zip file, it will create a directory (= the toolbox installation directory)<br />
<!-- # Unzip the activation zip file '''INTO the toolbox installation directory''' (this file was mailed to you after you registered) --><br />
# Start Matlab<br />
# Go to the toolbox directory<br />
# Run '<code>startup</code>'<br />
# Run '<code>go</code>'<br />
<br />
== Basic Installation ==<br />
<br />
=== Toolbox ===<br />
Unzip the toolbox zip file to a directory somewhere on your harddisk, the full path of the SUMO Toolbox (including installation directory) will be referred to as the toolbox installation directory (e.g., c:\software\SUMO-Toolbox-6.3). Note that you do '''not''' have to put the toolbox in the Matlab installation directory, we actually advise against it since it can cause confusing errors.<br />
<br />
Once you have unzipped the toolbox zip file the directory structure looks like this:<br />
<br />
* <code><toolbox installation directory> ''(e.g., c:\software\SUMO-Toolbox-6.3)''</code><br />
** <code>bin/</code> : binaries, executable scripts, ...<br />
** <code>config/</code> : configuration files, location of <code>default.xml</code><br />
** <code>config/demo</code> : a couple of demo configuration files that may help you<br />
** <code>doc/</code> : some documentation<br />
** <code>doc/apidoc</code> : Javadoc and other api docs<br />
** <code>lib/</code> : required libraries (eg: dom4j)<br />
** <code>output/</code> : some output may be placed here (e.g., a global log file)<br />
** <code>src/</code> : all source code<br />
** <code>examples/</code> : project directories of different examples (you can test with these problems and use them as an example to [[Adding an example|add your own]])<br />
<br />
<!--<br />
=== Activation file ===<br />
<br />
Once you have received the activation file simply unzip it '''INTO''' in your toolbox installation directory. So place the zip file in the toolbox installation directory and unzip it there, it should place all files in the correct places (see also the README file in the activation zip). DO NOT unzip the activation file into its own directory somewhere else. Make sure you restart Matlab (if it was running) after you have done this.<br />
<br />
=== Extension pack ===<br />
<br />
There are a number of third party tools and modeling libraries that the SUMO Toolbox can use but that we cannot distribute together with the toolbox. These have been bundled in an extension pack. Only minor patches have been made to the original code to make them work better with SUMO (e.g., remove debug output). To install the extension pack, download the zip file, and unzip it INTO your toolbox installation directory. The files should be placed in the correct directories. Simply re-run 'startup' to make Matlab aware of the new files.<br />
<br />
If you download and/or use these files please respect their licenses (found in doc/licenses), '''THIS IS YOUR RESPONSIBILITY !!!'''.<br />
--><br />
=== Setup ===<br />
<br />
Setting up the toolbox is very easy. Start Matlab, navigate to the toolbox installation directory (not anywhere else, this is important!!) and run '<code>startup</code>'.<br />
<br />
=== Test run ===<br />
<br />
To ensure everything is working you can do a simple run of the toolbox with the default configuration. This means the toolbox will use the setting specified in <code><SUMO-Toolbox-installation-dir>/config/default.xml</code>.<br />
<br />
# Make sure that you are in the toolbox installation directory and you have run '<code>startup</code>' (see above)<br />
# Type '<code>go</code>' and press enter.<br />
# The toolbox will start to model the ''Academic2DTwice'' simulator. This simulator has 2 inputs and 2 outputs, and will be modeled using Kriging models, scored using [[Measures#CrossValidation| CrossValidation]], and samples selected using a combined sample selection method.<br />
# To see the exact settings used open <code>config/default.xml</code>. Feel free to edit this file and play around with the different options.<br />
<br />
The examples directory contains many example simulators that you can use to test the toolbox with. See [[Running#Running_different_examples]].<br />
<br />
== Ok, the test run works, now what? ==<br />
<br />
See [[Running]] page.<br />
<br />
== Problems ==<br />
<br />
See the [[reporting problems]] page.<br />
<br />
== Optional: Compiling libraries ==<br />
<br />
There are some alternative libraries and simulators available that have to be compiled for your specific platform. Instructions depend on your operating system. Ensure you have installed the extension pack before continuing.<br />
<br />
=== Linux/Unix/OSX ===<br />
<br />
# Ensure you have the following environment variables set:<br />
## <code>MATLABDIR=/path/to/your/matlab/installation</code><br />
## <code>JAVA_HOME=/path/to/your/SDK/installation</code><br />
# Ensure you have the usual build tools installed: gcc, g++, autotools, make, etc<br />
# From the command line shell (so NOT from inside Matlab): Go to the toolbox installation directory and type '<code>make</code>'. This will build everything for you (C/C++ files, SVM libraries, ...). If you only want to build certain packages simply '<code>make Package</code>' in the toolbox installation directory. <br />
## Note: if this is giving you problems, and you just want to compile the LS-SVMs you can try running makeLSSVM from inside Matlab (see the Windows instructions below)<br />
## For Matlab versions R2011a and above, in case your operating system is Mac OS X (version 10.6 - Snow Leopard upwards), and the C compiler being used is Xcode version 4.1 and above, you will need to install a patch from MathWorks, please refer to [http://www.mathworks.nl/support/solutions/en/data/1-FR6LXJ/ this]. You will also need to update the SDKROOT entry in mexopts.sh file (usually located in ~/.matlab/R2012b/mexopts.sh), according to its location in the filesystem as it has been moved in recent versions of OS X. For example, for OS X Lion the updated entry should be, <br />
###<code>SDKROOT='/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.7.sdk/'</code><br />
### It is also advisable to change the entry for <code>CC</code> from <code>gcc-4.2</code> to just <code>gcc</code>.<br />
## If you encounter the message "This process is attempting to exclude an item from Time Machine by path without administrator privileges. This is not supported." on Mac OS X while building the code, building it as a super-user by using the command '<code>sudo make</code>' instead of only '<code>make</code>' will solve the problem.<br />
# A complete list of available packages follows:<br />
<br />
{| style="margin: 1em auto 1em auto" border="1"<br />
|-<br />
! Package<br />
! Description<br />
! Requires extension pack<br />
|-<br />
| contrib<br />
| Builds the FANN, SVM (libsvm, LS-SVMlab) and NNSYSID libraries<br />
| Yes<br />
|-<br />
| cexamples<br />
| Builds the binaries for several C/C++ simulators<br />
| No<br />
|}<br />
<br />
Note: if you want to use FANN you will have to ensure Matlab knows of the FANN libraries. See [[FAQ#When_using_FANN_models_I_sometimes_get_.22Invalid_MEX-file_createFann.mexa64.2C_libfann.so.2:_cannot_open_shared_object_file:_No_such_file_or_directory..22 | the FAQ]].<br />
<br />
=== Windows ===<br />
<br />
# Compiling C/C++ codes (examples):<br />
## You will have to do this on your own using a C/C++ compiler of your choice: Dev-c++/Visual Studio/...<br />
# Compiling LS-SVM libraries:<br />
## In order to use the [http://www.esat.kuleuven.be/sista/lssvmlab/ LS-SVM] backend, you will have to compile the LS-SVM mex files (it will work if you dont but you will get a lot of warning messages about a missing CFile implementation).<br />
## This can be done using the built-in LCC compiler of matlab, by calling '<code>makeLSSVM</code>' from the Matlab command prompt (make sure the SUMO Toolbox is in your path)<br />
# Compiling ANN libraries:<br />
## In order to use the [http://leenissen.dk/fann/ FANN] backend, you will have to compile the FANN library and mex files.<br />
## So far nobody has yet got it to work under Windows, but don't let that stop you.<br />
# Compiling Java classes using Eclipse (only for SVN version of the Toolbox, on linux you can just use the make command)<br />
## open Eclipse<br />
## import ant buildfile<br />
### File->New->Other<br />
### Java->Java Project from Existing Ant Buildfile<br />
### Select build.xml in the SUMO root directory<br />
### click 'Finish'<br />
### Add external jars from Matlab install to buildpath if necessary<br />
## Open configure.m in the SUMO root directory and change bin/ to src/ for the java paths (the lines below ''% add java paths'')<br />
## Type in 'go' in the Matlab command line to see whether the compilation worked<br />
## If the SUMO Toolbox does not start, make sure the java virtual machine of matlab and eclipse are the same<br />
### Type in 'version('-java')' in the Matlab command line to find the jvm version used by matlab<br />
### In Eclipse, right click the SUMO-Toolbox project -> Properties -> Java Compiler<br />
### Change the Java Compiler used by Eclipse by ticking the 'Enable project specific settings' and selecting the same version of the Java Compiler as the Matlab jvm</div>Sprashanthttp://sumowiki.intec.ugent.be/InstallationInstallation2012-11-22T14:53:35Z<p>Sprashant: /* Linux/Unix/OSX */</p>
<hr />
<div>== Introduction ==<br />
This page will walk you through the SUMO Toolbox installation. Please refer to the [[system requirements]] first. See the [[downloading]] section on how to download the toolbox.<br />
<br />
== Quick start ==<br />
<br />
Quick and dirty instructions:<br />
<br />
# Log into the SUMO lab website with the account information mailed to you and download the toolbox<br />
# Unzip the toolbox zip file, it will create a directory (= the toolbox installation directory)<br />
<!-- # Unzip the activation zip file '''INTO the toolbox installation directory''' (this file was mailed to you after you registered) --><br />
# Start Matlab<br />
# Go to the toolbox directory<br />
# Run '<code>startup</code>'<br />
# Run '<code>go</code>'<br />
<br />
== Basic Installation ==<br />
<br />
=== Toolbox ===<br />
Unzip the toolbox zip file to a directory somewhere on your harddisk, the full path of the SUMO Toolbox (including installation directory) will be referred to as the toolbox installation directory (e.g., c:\software\SUMO-Toolbox-6.3). Note that you do '''not''' have to put the toolbox in the Matlab installation directory, we actually advise against it since it can cause confusing errors.<br />
<br />
Once you have unzipped the toolbox zip file the directory structure looks like this:<br />
<br />
* <code><toolbox installation directory> ''(e.g., c:\software\SUMO-Toolbox-6.3)''</code><br />
** <code>bin/</code> : binaries, executable scripts, ...<br />
** <code>config/</code> : configuration files, location of <code>default.xml</code><br />
** <code>config/demo</code> : a couple of demo configuration files that may help you<br />
** <code>doc/</code> : some documentation<br />
** <code>doc/apidoc</code> : Javadoc and other api docs<br />
** <code>lib/</code> : required libraries (eg: dom4j)<br />
** <code>output/</code> : some output may be placed here (e.g., a global log file)<br />
** <code>src/</code> : all source code<br />
** <code>examples/</code> : project directories of different examples (you can test with these problems and use them as an example to [[Adding an example|add your own]])<br />
<br />
<!--<br />
=== Activation file ===<br />
<br />
Once you have received the activation file simply unzip it '''INTO''' in your toolbox installation directory. So place the zip file in the toolbox installation directory and unzip it there, it should place all files in the correct places (see also the README file in the activation zip). DO NOT unzip the activation file into its own directory somewhere else. Make sure you restart Matlab (if it was running) after you have done this.<br />
<br />
=== Extension pack ===<br />
<br />
There are a number of third party tools and modeling libraries that the SUMO Toolbox can use but that we cannot distribute together with the toolbox. These have been bundled in an extension pack. Only minor patches have been made to the original code to make them work better with SUMO (e.g., remove debug output). To install the extension pack, download the zip file, and unzip it INTO your toolbox installation directory. The files should be placed in the correct directories. Simply re-run 'startup' to make Matlab aware of the new files.<br />
<br />
If you download and/or use these files please respect their licenses (found in doc/licenses), '''THIS IS YOUR RESPONSIBILITY !!!'''.<br />
--><br />
=== Setup ===<br />
<br />
Setting up the toolbox is very easy. Start Matlab, navigate to the toolbox installation directory (not anywhere else, this is important!!) and run '<code>startup</code>'.<br />
<br />
=== Test run ===<br />
<br />
To ensure everything is working you can do a simple run of the toolbox with the default configuration. This means the toolbox will use the setting specified in <code><SUMO-Toolbox-installation-dir>/config/default.xml</code>.<br />
<br />
# Make sure that you are in the toolbox installation directory and you have run '<code>startup</code>' (see above)<br />
# Type '<code>go</code>' and press enter.<br />
# The toolbox will start to model the ''Academic2DTwice'' simulator. This simulator has 2 inputs and 2 outputs, and will be modeled using Kriging models, scored using [[Measures#CrossValidation| CrossValidation]], and samples selected using a combined sample selection method.<br />
# To see the exact settings used open <code>config/default.xml</code>. Feel free to edit this file and play around with the different options.<br />
<br />
The examples directory contains many example simulators that you can use to test the toolbox with. See [[Running#Running_different_examples]].<br />
<br />
== Ok, the test run works, now what? ==<br />
<br />
See [[Running]] page.<br />
<br />
== Problems ==<br />
<br />
See the [[reporting problems]] page.<br />
<br />
== Optional: Compiling libraries ==<br />
<br />
There are some alternative libraries and simulators available that have to be compiled for your specific platform. Instructions depend on your operating system. Ensure you have installed the extension pack before continuing.<br />
<br />
=== Linux/Unix/OSX ===<br />
<br />
# Ensure you have the following environment variables set:<br />
## <code>MATLABDIR=/path/to/your/matlab/installation</code><br />
## <code>JAVA_HOME=/path/to/your/SDK/installation</code><br />
# Ensure you have the usual build tools installed: gcc, g++, autotools, make, etc<br />
# From the command line shell (so NOT from inside Matlab): Go to the toolbox installation directory and type '<code>make</code>'. This will build everything for you (C/C++ files, SVM libraries, ...). If you only want to build certain packages simply '<code>make Package</code>' in the toolbox installation directory. <br />
## Note: if this is giving you problems, and you just want to compile the LS-SVMs you can try running makeLSSVM from inside Matlab (see the Windows instructions below)<br />
## For Matlab versions R2011a and above, in case your operating system is Mac OS X (version 10.6 - Snow Leopard upwards), and the C compiler being used is Xcode version 4.1 and above, you will need to install a patch from MathWorks, please refer to [http://www.mathworks.nl/support/solutions/en/data/1-FR6LXJ/ this]. You will also need to update the SDKROOT entry in mexopts.sh file (usually located in ~/.matlab/R2012b/mexopts.sh), according to its location in the filesystem as it has been moved in recent versions of OS X. For example, for OS X Lion the updated entry should be, <br />
###<code>SDKROOT='/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.7.sdk/'</code> <br />
## If you encounter the message "This process is attempting to exclude an item from Time Machine by path without administrator privileges. This is not supported." on Mac OS X while building the code, building it as a super-user by using the command '<code>sudo make</code>' instead of only '<code>make</code>' will solve the problem.<br />
# A complete list of available packages follows:<br />
<br />
{| style="margin: 1em auto 1em auto" border="1"<br />
|-<br />
! Package<br />
! Description<br />
! Requires extension pack<br />
|-<br />
| contrib<br />
| Builds the FANN, SVM (libsvm, LS-SVMlab) and NNSYSID libraries<br />
| Yes<br />
|-<br />
| cexamples<br />
| Builds the binaries for several C/C++ simulators<br />
| No<br />
|}<br />
<br />
Note: if you want to use FANN you will have to ensure Matlab knows of the FANN libraries. See [[FAQ#When_using_FANN_models_I_sometimes_get_.22Invalid_MEX-file_createFann.mexa64.2C_libfann.so.2:_cannot_open_shared_object_file:_No_such_file_or_directory..22 | the FAQ]].<br />
<br />
=== Windows ===<br />
<br />
# Compiling C/C++ codes (examples):<br />
## You will have to do this on your own using a C/C++ compiler of your choice: Dev-c++/Visual Studio/...<br />
# Compiling LS-SVM libraries:<br />
## In order to use the [http://www.esat.kuleuven.be/sista/lssvmlab/ LS-SVM] backend, you will have to compile the LS-SVM mex files (it will work if you dont but you will get a lot of warning messages about a missing CFile implementation).<br />
## This can be done using the built-in LCC compiler of matlab, by calling '<code>makeLSSVM</code>' from the Matlab command prompt (make sure the SUMO Toolbox is in your path)<br />
# Compiling ANN libraries:<br />
## In order to use the [http://leenissen.dk/fann/ FANN] backend, you will have to compile the FANN library and mex files.<br />
## So far nobody has yet got it to work under Windows, but don't let that stop you.<br />
# Compiling Java classes using Eclipse (only for SVN version of the Toolbox, on linux you can just use the make command)<br />
## open Eclipse<br />
## import ant buildfile<br />
### File->New->Other<br />
### Java->Java Project from Existing Ant Buildfile<br />
### Select build.xml in the SUMO root directory<br />
### click 'Finish'<br />
### Add external jars from Matlab install to buildpath if necessary<br />
## Open configure.m in the SUMO root directory and change bin/ to src/ for the java paths (the lines below ''% add java paths'')<br />
## Type in 'go' in the Matlab command line to see whether the compilation worked<br />
## If the SUMO Toolbox does not start, make sure the java virtual machine of matlab and eclipse are the same<br />
### Type in 'version('-java')' in the Matlab command line to find the jvm version used by matlab<br />
### In Eclipse, right click the SUMO-Toolbox project -> Properties -> Java Compiler<br />
### Change the Java Compiler used by Eclipse by ticking the 'Enable project specific settings' and selecting the same version of the Java Compiler as the Matlab jvm</div>Sprashanthttp://sumowiki.intec.ugent.be/InstallationInstallation2012-10-12T07:49:38Z<p>Sprashant: /* Linux/Unix/OSX */</p>
<hr />
<div>== Introduction ==<br />
This page will walk you through the SUMO Toolbox installation. Please refer to the [[system requirements]] first. See the [[downloading]] section on how to download the toolbox.<br />
<br />
== Quick start ==<br />
<br />
Quick and dirty instructions:<br />
<br />
# Log into the SUMO lab website with the account information mailed to you and download the toolbox<br />
# Unzip the toolbox zip file, it will create a directory (= the toolbox installation directory)<br />
<!-- # Unzip the activation zip file '''INTO the toolbox installation directory''' (this file was mailed to you after you registered) --><br />
# Start Matlab<br />
# Go to the toolbox directory<br />
# Run '<code>startup</code>'<br />
# Run '<code>go</code>'<br />
<br />
== Basic Installation ==<br />
<br />
=== Toolbox ===<br />
Unzip the toolbox zip file to a directory somewhere on your harddisk, the full path of the SUMO Toolbox (including installation directory) will be referred to as the toolbox installation directory (e.g., c:\software\SUMO-Toolbox-6.3). Note that you do '''not''' have to put the toolbox in the Matlab installation directory, we actually advise against it since it can cause confusing errors.<br />
<br />
Once you have unzipped the toolbox zip file the directory structure looks like this:<br />
<br />
* <code><toolbox installation directory> ''(e.g., c:\software\SUMO-Toolbox-6.3)''</code><br />
** <code>bin/</code> : binaries, executable scripts, ...<br />
** <code>config/</code> : configuration files, location of <code>default.xml</code><br />
** <code>config/demo</code> : a couple of demo configuration files that may help you<br />
** <code>doc/</code> : some documentation<br />
** <code>doc/apidoc</code> : Javadoc and other api docs<br />
** <code>lib/</code> : required libraries (eg: dom4j)<br />
** <code>output/</code> : some output may be placed here (e.g., a global log file)<br />
** <code>src/</code> : all source code<br />
** <code>examples/</code> : project directories of different examples (you can test with these problems and use them as an example to [[Adding an example|add your own]])<br />
<br />
<!--<br />
=== Activation file ===<br />
<br />
Once you have received the activation file simply unzip it '''INTO''' in your toolbox installation directory. So place the zip file in the toolbox installation directory and unzip it there, it should place all files in the correct places (see also the README file in the activation zip). DO NOT unzip the activation file into its own directory somewhere else. Make sure you restart Matlab (if it was running) after you have done this.<br />
<br />
=== Extension pack ===<br />
<br />
There are a number of third party tools and modeling libraries that the SUMO Toolbox can use but that we cannot distribute together with the toolbox. These have been bundled in an extension pack. Only minor patches have been made to the original code to make them work better with SUMO (e.g., remove debug output). To install the extension pack, download the zip file, and unzip it INTO your toolbox installation directory. The files should be placed in the correct directories. Simply re-run 'startup' to make Matlab aware of the new files.<br />
<br />
If you download and/or use these files please respect their licenses (found in doc/licenses), '''THIS IS YOUR RESPONSIBILITY !!!'''.<br />
--><br />
=== Setup ===<br />
<br />
Setting up the toolbox is very easy. Start Matlab, navigate to the toolbox installation directory (not anywhere else, this is important!!) and run '<code>startup</code>'.<br />
<br />
=== Test run ===<br />
<br />
To ensure everything is working you can do a simple run of the toolbox with the default configuration. This means the toolbox will use the setting specified in <code><SUMO-Toolbox-installation-dir>/config/default.xml</code>.<br />
<br />
# Make sure that you are in the toolbox installation directory and you have run '<code>startup</code>' (see above)<br />
# Type '<code>go</code>' and press enter.<br />
# The toolbox will start to model the ''Academic2DTwice'' simulator. This simulator has 2 inputs and 2 outputs, and will be modeled using Kriging models, scored using [[Measures#CrossValidation| CrossValidation]], and samples selected using a combined sample selection method.<br />
# To see the exact settings used open <code>config/default.xml</code>. Feel free to edit this file and play around with the different options.<br />
<br />
The examples directory contains many example simulators that you can use to test the toolbox with. See [[Running#Running_different_examples]].<br />
<br />
== Ok, the test run works, now what? ==<br />
<br />
See [[Running]] page.<br />
<br />
== Problems ==<br />
<br />
See the [[reporting problems]] page.<br />
<br />
== Optional: Compiling libraries ==<br />
<br />
There are some alternative libraries and simulators available that have to be compiled for your specific platform. Instructions depend on your operating system. Ensure you have installed the extension pack before continuing.<br />
<br />
=== Linux/Unix/OSX ===<br />
<br />
# Ensure you have the following environment variables set:<br />
## <code>MATLABDIR=/path/to/your/matlab/installation</code><br />
## <code>JAVA_HOME=/path/to/your/SDK/installation</code><br />
# Ensure you have the usual build tools installed: gcc, g++, autotools, make, etc<br />
# From the command line shell (so NOT from inside Matlab): Go to the toolbox installation directory and type '<code>make</code>'. This will build everything for you (C/C++ files, SVM libraries, ...). If you only want to build certain packages simply '<code>make Package</code>' in the toolbox installation directory. <br />
## Note: if this is giving you problems, and you just want to compile the LS-SVMs you can try running makeLSSVM from inside Matlab (see the Windows instructions below)<br />
## For Matlab versions R2011a and above, in case your operating system is Mac OS X (version 10.6 - Snow Leopard upwards), and the C compiler being used is Xcode version 4.1 and above, you will need to install a patch from MathWorks, please refer to [http://www.mathworks.nl/support/solutions/en/data/1-FR6LXJ/ this]. You will also need to update the SDKROOT entry in mexopts.sh file (usually located in ~/.matlab/R2012b/mexopts.sh), according to its location in the filesystem as it has been moved in recent versions of OS X. For example, for OS X Lion the updated entry should be, <br />
###<code>SDKROOT='/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.7.sdk/'</code> <br />
# A complete list of available packages follows:<br />
<br />
{| style="margin: 1em auto 1em auto" border="1"<br />
|-<br />
! Package<br />
! Description<br />
! Requires extension pack<br />
|-<br />
| contrib<br />
| Builds the FANN, SVM (libsvm, LS-SVMlab) and NNSYSID libraries<br />
| Yes<br />
|-<br />
| cexamples<br />
| Builds the binaries for several C/C++ simulators<br />
| No<br />
|}<br />
<br />
Note: if you want to use FANN you will have to ensure Matlab knows of the FANN libraries. See [[FAQ#When_using_FANN_models_I_sometimes_get_.22Invalid_MEX-file_createFann.mexa64.2C_libfann.so.2:_cannot_open_shared_object_file:_No_such_file_or_directory..22 | the FAQ]].<br />
<br />
=== Windows ===<br />
<br />
# Compiling C/C++ codes (examples):<br />
## You will have to do this on your own using a C/C++ compiler of your choice: Dev-c++/Visual Studio/...<br />
# Compiling LS-SVM libraries:<br />
## In order to use the [http://www.esat.kuleuven.be/sista/lssvmlab/ LS-SVM] backend, you will have to compile the LS-SVM mex files (it will work if you dont but you will get a lot of warning messages about a missing CFile implementation).<br />
## This can be done using the built-in LCC compiler of matlab, by calling '<code>makeLSSVM</code>' from the Matlab command prompt (make sure the SUMO Toolbox is in your path)<br />
# Compiling ANN libraries:<br />
## In order to use the [http://leenissen.dk/fann/ FANN] backend, you will have to compile the FANN library and mex files.<br />
## So far nobody has yet got it to work under Windows, but don't let that stop you.<br />
# Compiling Java classes using Eclipse (only for SVN version of the Toolbox, on linux you can just use the make command)<br />
## open Eclipse<br />
## import ant buildfile<br />
### File->New->Other<br />
### Java->Java Project from Existing Ant Buildfile<br />
### Select build.xml in the SUMO root directory<br />
### click 'Finish'<br />
### Add external jars from Matlab install to buildpath if necessary<br />
## Open configure.m in the SUMO root directory and change bin/ to src/ for the java paths (the lines below ''% add java paths'')<br />
## Type in 'go' in the Matlab command line to see whether the compilation worked<br />
## If the SUMO Toolbox does not start, make sure the java virtual machine of matlab and eclipse are the same<br />
### Type in 'version('-java')' in the Matlab command line to find the jvm version used by matlab<br />
### In Eclipse, right click the SUMO-Toolbox project -> Properties -> Java Compiler<br />
### Change the Java Compiler used by Eclipse by ticking the 'Enable project specific settings' and selecting the same version of the Java Compiler as the Matlab jvm</div>Sprashant