Optimization Services benchmark Language (OSbL)
The availability of more than one service (especially solvers) for many classes of problems makes the OS registry an obvious choice as a benchmarking facility. It can potentially be useful both in choosing a solver for a particular application and in comparing solvers generally. One certain thing is that the benchmarking should be independent of any claims or statistics made by individual service providers.
There are significant barriers to achieving these potentials, however, which motivate some derived research. Someone who has developed a new model, but who is not sure which of the several applicable solver services to apply, is often advised that the only way to be sure is to carry out some test runs on typical problem instances. The straightforward way to do this is to send several test instances to each candidate solver service. But benchmarking on only a few related instances can be misleading. Furthermore, if different services are not on comparable machines under comparable conditions, the results may say little about the relative efficiency of the solver algorithms. The results may say more about the reliability of the solver services, but even so they may be distorted by differences in the memory available on the workstations devoted to different solver services, or by differences in limits imposed by the providers of services. There is not necessarily any obvious way to compensate for the differences between runs, moreover, because in general each solver service may be selected by the OS registry according to the load at the time a job is submitted. Deciding on an appropriate benchmarking methodology is related to other concurrent researches at NEOS and the NEOS benchmark solver at: http://www-neos.mcs.anl.gov/neos/solvers/MULTI:Benchmark/AMPL.html
For all the aforementioned issues, OSbL uses a relatively safe rating system based on a set of performance scores on the base of 100 as shown in the OSbL schema in the following figure.
|© Optimization Services (OS) 2006 | All rights reserved.|