Optimization Services benchmark Language (OSbL)

Click to see or download the OSbL XML Schema -> OSbL.xsd [prototype under development]

OSbL (benchmark) is a specification of benchmark information on each optimization service. OSbL is the third and last piece of information (along with OSeL and OSpL) that the OS registry keeps.

The availability of more than one service (especially solvers) for many classes of problems makes the OS registry an obvious choice as a benchmarking facility. It can potentially be useful both in choosing a solver for a particular application and in comparing solvers generally. One certain thing is that the benchmarking should be independent of any claims or statistics made by individual service providers.

There are significant barriers to achieving these potentials, however, which motivate some derived research. Someone who has developed a new model, but who is not sure which of the several applicable solver services to apply, is often advised that the only way to be sure is to carry out some test runs on typical problem instances. The straightforward way to do this is to send several test instances to each candidate solver service. But benchmarking on only a few related instances can be misleading. Furthermore, if different services are not on comparable machines under comparable conditions, the results may say little about the relative efficiency of the solver algorithms. The results may say more about the reliability of the solver services, but even so they may be distorted by differences in the memory available on the workstations devoted to different solver services, or by differences in limits imposed by the providers of services. There is not necessarily any obvious way to compensate for the differences between runs, moreover, because in general each solver service may be selected by the OS registry according to the load at the time a job is submitted. Deciding on an appropriate benchmarking methodology is related to other concurrent researches at NEOS and the NEOS benchmark solver at:  http://www-neos.mcs.anl.gov/neos/solvers/MULTI:Benchmark/AMPL.html
The Performance World Forum from GAMS World at http://www.gamsworld.org/performance/  is also good site for discussion and dissemination of information and tools about all aspects of performance testing of solvers for mathematical programming problems. Possible collaboration on Optimization Services benchmarking can be established with these concurrent project.

For all the aforementioned issues, OSbL uses a relatively safe rating system based on a set of performance scores on the base of 100 as shown in the OSbL schema in the following figure.

Links:

Click to see or download the OSbL XML Schema -> OSbL.xsd