Each entry to a benchmark should contain source code, a shell script named run.sh that will run the program to be benchmarked and that will examine the output of run.sh and write out the results to a file named benchmark.dat. The benchmark.dat file describes the results for each of the metrics of the benchmark as well as other important data such as the name of the algorithm used, the name and version of the code, the date, and the name of the contributor to the benchmark. The file has a very simple format. Each line contains a key value pair. The key and value are seperated by whitespace. Everything on the line after the first whitespace is considered the value. Here is an example:
force_calls 431 force_calls_max 1139 force_calls_min 197 nfailed 0 code Eon code_version r1789 date 15 Oct 2012 contributor Sam Chill
The required keys in the benchmark.dat file are defined by each benchmark. Additionally the keys code, code_version, date, and contributor should be specified.
Entries may be submitted via email to Graeme Henkelman.
This website, the output from the benchmark runs, and the code used to run each benchmark is stored in a subversion repository. Read-only access to the repository is provided with the command:
svn checkout http://optbench.org/svn/benchmarksIf you have an account, you can checkout and modify the repository like so:
svn checkout svn+ssh://username@optbench.org/svn/benchmarks
In the top level directory there are two folders
bin/ website/
The bin directory has some helper programs and the website is in the website directory. In the website directory there are files that end in .tmpl these are template files that get rendered to html. To build the website one runs "make" in that directory and the templates get turned into html.
Each suite of benchmarks has its own subdirectory and each benchmark has its own directory in the folder of the suite it belongs to. For example all of the minimization benchmarks are in the website/minimization folder. A minimization benchmark named "Lennard-Jones 38" is in the website/minimization/lj38 folder. Each entry in the benchmark (a run of a code with particular settings) has its own folder in the benchmark folder: for example website/minimization/lj38/eon-cg. All of the important information for each benchmark entry is stored in a file named benchmark.dat. So for this example the file is located at website/minimization/lj38/eon-cg/benchmark.dat.
When the website template files gets rendered by running "make", each template gets passed data from all of the benchmark.dat files in its suite. Once a template file has been written for a benchmark it does not need to be modified to add new entries. The template is designed to loop over all the entries in each benchmark.