This page contains details about benchmarks of COIN-OR solvers with GAMS interface vs. other GAMS solvers.
Benchmarks were made by creating trace-files using the GAMS trace option and processing them with the PAVER server.
NEW:
- 19.07.2008: benchmarking CBC single threaded vs. multithreaded version (running 2 threads).
- 06.07.2008: updated benchmarks of MIP solvers and of GAMS BCH using a newer CBC version.
- 01.06.2008: benchmarking MIP solvers CPLEX, CBC, GLPK, SCIP on LINLib.
- 19.05.2008: benchmarking GAMS BCH-facility on the FCNetLib (CPLEX, SCIP, CBC).
- 28.01.2008: LP solvers 4 years ago and today (CLP, GLPK, IPOPT(today)).
- 07.01.2008: Comparision with updated MIP solvers (CPLEX 11, GLPK 4.25).
- 07.01.2008: Comparision with updated LP solvers (CPLEX 11, GLPK 4.24).
- 27.09.2007: Comparision of DICOPT stop rules 0 to 3
- 28.09.2007: Comparision of SBB with CONOPT and default and high memnodes
- 28.09.2007: Updated comparision of SBB, AlphaECP, DICOPT, BONMIN.
- 01.10.2007: Updated comparision of CPLEX, CLP, GLPK, IPOPT on LPs.
Content:
- General Notes (README)
- Solvers
- LP Benchmarks
- MIP Benchmarks
- NLP Benchmarks
- MINLP Benchmarks
- Solvers are always run on each model with a timelimit of 1 hour and without iteration or domain violation limit.
- The PerformanceTools scripts behind the PAVER server currently contain a bug that make PAVER omit the "locally optimal" row/column in the solver square summary for discrete models (MIP, MINLP).
This also leads to wrong sum-ups in the "total"-row/column.
Resource time summary and performance profiles should be correct.
- LINLib results were produced on a machine with Intel Core2 2.40 GHz, 2 GB RAM, and running under Linux 2.6.18.
- MINLPLib results were produced on a machine with Intel Pentium IV 3 GHz, 1 GB RAM, and running under Linux 2.6.11.
We used commercial solvers as they are included in GAMS 22.5 and open-source solvers as their were available on 21.8.2007.
For results after 02.09.2007, the solver version is indicated together with the results, if different.
In detail, the following solvers were used:
The following trunk revisions were used:
CBC rev 770, CLP rev 1092, CGL rev 520, CoinUtils rev 837, OSI rev 1073
GAMS/CBC uses the OSI/CLP interface and CbcSolver routines which are also used by the CBC-standalone version.
Additionally the following options are changed:
- Dual and primal optimality tolerances are set to 1e-7.
GAMS/GLPK uses the OSI/GLPK interface. MIPs are solved with GLPKs advanced Branch and Cut solver (lpx_intopt).
Additionally the following options are set or changed:
- Dual and primal optimality tolerances are set to 1e-7.
- Scaling is set to "geometric mean scaling, then equilibrium scaling".
- The primal simplex algorithm is used.
- LP presolve is turned on.
- Textbook pricing is used.
- Backtracking by best projection heuristic is used.
- The integer tolerance is 1e-5.
- All cutgenerators are turned on.
The stable branch revision 1067 was used. Linear solver is MA27.
The GAMS/IPOPT interface changes the defaults of the following IPOPT options:
- mu_strategy is set to adaptive
- bound_relax_factor is set to 0.0
SBB = Simple Branch and Bound
Since we also deal with nonconvex models, we have set acceptnonopt to 1.
DICOPT = Discrete and Continuous Optimizer
DICOPT was run with standard options, which means that DICOPT stops if the NLP solution is worsening or the major iteration limit of 20 is reached.
It will be interesting to see how things will change with a change in these options.
AlphaECP 1.30 = Extended Cutting Plane
BARON 7.8.1 = Branch And Reduce Optimization Navigator
The trunk revision 681 was used.
The trunk revision 127 was used.
We run on all LPs from the LINLib which have at least 50000 nonzeros (categories medium, large, and huge).
This are 118 models.
02.09.2007:
- results of the following comparision:
- CPLEX-0: CPLEX with default parameters (simplex algorithm).
- CPLEX-1: CPLEX with "lpmethod 4" (barrier algorithm).
- CBC-0: CLP with default parameters.
- GLPK-0: GLPK with default parameters (see above).
- IPOPT-0: IPOPT with default parameters (see above).
- IPOPT-1: IPOPT with mehrotra_algorithm, jac_c_constant, jac_d_constant, hessian_constant turned on.
- results of the following comparision (as above, but less solvers):
- CPLEX-0: CPLEX with default parameters (simplex algorithm).
- CBC-0: CLP with default parameters.
- GLPK-0: GLPK with default parameters (see above).
- IPOPT-1: IPOPT with mehrotra_algorithm, jac_c_constant, jac_d_constant, hessian_constant turned on.
- results of the following comparision:
- IPOPT-0: IPOPT with default parameters (see above).
- IPOPT-1: IPOPT with mehrotra_algorithm, jac_c_constant, jac_d_constant, hessian_constant turned on.
01.10.2007:
- results of the following comparision:
- CPLEX-0: CPLEX with default parameters (simplex algorithm).
- CBC-0: CLP with default parameters.
- GLPK-0: GLPK 4.22 with default parameters (see above), but steepest depth pivot choice algorithm.
- GLPK-4.20: GLPK 4.20 with default parameters (see above).
- IPOPT-1: IPOPT with mehrotra_algorithm, jac_c_constant, jac_d_constant, hessian_constant turned on.
07.01.2008:
- results of the following comparision:
- 1-CPLEX-0: CPLEX 11 with default parameters (simplex algorithm).
- 2-CPLEX-1: CPLEX 11 with "lpmethod 4" (barrier algorithm).
- 3-CPLEX-0: CPLEX 10.2 with default parameters (simplex algorithm).
- CBC-0: CLP with default parameters.
- GLPK-0: GLPK 4.22 with default parameters (see above), but steepest depth pivot choice algorithm.
- GLPK-4.20: GLPK 4.20 with default parameters (see above).
- IPOPT-0: IPOPT with default parameters.
- IPOPT-1: IPOPT with mehrotra_algorithm, jac_c_constant, jac_d_constant, hessian_constant turned on.
28.01.2008:
- results of the following comparision:
- CBC2004: CLP from GAMS 21.4 (2004) with default parameters
- CBC2008: CLP 2.0 (2008) with default parameters
- GLPK2004: GLPK 4.5 from GAMS 21.4 (2004) with default parameters
- GLPK2008: GLPK 4.25 (2008) with default parameters
- IPOPT2008: IPOPT (2008) with mehrotra_algorithm, jac_c_constant, jac_d_constant, hessian_constant turned on.
We run on all MIPs from the LINLib.
This are 125 models.
19.07.2008:
- results of the following comparision on the 125 mip models from the LINLib running
- CBC-1: Cbc 2.1 (stable version from approx. 24.06.08) compiled in singlethreading mode
- CBC-2: Cbc 2.1 (stable version rev. 1004 from approx. 15.07.08) compiled in multithreading mode and run with 2 threads
gap tolerance = 0%
default solver parameters
machine: Core2 Duo T7500, 2 GB RAM, Linux 2.4.16, GCC 4.2.1
06.07.2008:
- results of the following comparision on the 125 mip models from the LINLib:
- CPLEX 11.0 (as in GAMS 22.6)
- SCIP 1.0
- CBC 2.1 (stable version from approx. 24.06.08)
- GLPK 4.26
gap tolerance = 0%
default solver parameters
machine: Core2 Duo T7500, 2 GB RAM, Linux 2.4.16, GCC 4.2.1
- results of benchmarking MIP solvers on the GAMS FCNetLib (Fixed Cost Network Flow Library)
analysing the effect of dicuts computed via the GAMS BCH facility.
The following solvers were used:
- CPLEX 11.0 (as in GAMS 22.6)
- SCIP 1.0
- CBC 2.1 (stable/2.1 from approx. 24.06.08)
01.06.2008:
- results of the following comparision:
- CPLEX 11.0 (as in GAMS 22.6)
- SCIP 1.0
- CBC 2.1 (stable version from approx. 25.05.08)
- GLPK 4.26
gap tolerance = 0%
default solver parameters
machine: Core2 Duo T7500, 2 GB RAM, Linux 2.4.16, GCC 4.2.1
19.05.2008:
- results of benchmarking MIP solvers on the GAMS FCNetLib (Fixed Cost Network Flow Library)
analysing the effect of dicuts computed via the GAMS BCH facility.
The following solvers were used:
- CPLEX 11.0 (as in GAMS 22.6)
- SCIP 1.0
- CBC 2.1 (stable/2.1 revision 946)
07.01.2008:
- results of the following comparision
(we run with a gap tolerance of 0.01%):
- 1-CPLEX-1: CPLEX 11 with default parameters.
- 2-CPLEX-1: CPLEX 10.2 with default parameters.
- CBC-1: CBC 2.0 with default parameters.
- GLPK-1: GLPK 4.25 with default parameters (see above).
02.09.2007:
- results of the following comparision
(we run with a gap tolerance of 0.01%, except for GLPK which does not have a MIP gap tolerance yet):
- CPLEX-1: CPLEX with default parameters.
- CBC-1: CBC with default parameters.
- GLPK-1: GLPK with default parameters (see above).
NLP Benchmarks
We run on all NLPs from the GlobalLib.
This are 379 models.
02.09.2007:
- results of the following comparision:
- CONOPT-1: CONOPT with parameters rtredg=1e-6, rtnwmi=1e-6, and rtnwma=1.01e-6.
- IPOPT-1: IPOPT with parameters tol=1e-6 and acceptable_tol=1e-6.
- KNITRO-1: KNITRO with parameters feastol=1e-6 and opttol=1e-6.
Next we have run all QQPs from the GlobalLib with atmost 1000 variables.
This are 162 models.
The gap tolerance is set to 1%, the feasibility tolerance to 1e-4.
02.09.2007:
- results of the following comparision:
- BARON-1: BARON.
- LaGO-1: LaGO with violation subdivision.
MINLP Benchmarks
We run on a selected set of MINLPs from the MINLPLib.
The selection aims to not overrepresent a type of model that one solver might be very good at (e.g., AlphaECP is very good on the models fo{7,8,9}*, m7*, no7*, and o{7,8,9}*, so we took only some of them).
Also we limited to models with atmost 1000 variables.
This gives a set of 167 models.
We run with a gap tolerance of 1%.
02.09.2007:
Solver A vs. Solver B:
- results of the following comparision on all 161 models without semicontinuous variables and SOS constraints (since these are not supported by GAMS/Bonmin yet):
- SBB-1: SBB with CONOPT as subsolver.
- AlphaECP-1: AlphaECP with CPLEX as subsolver.
- DICOPT-1: DICOPT with CONOPT and CPLEX as subsolver.
- BONMIN-1: BONMIN.
- results of the following comparision on all 40 convex models:
- SBB-1: SBB with CONOPT as subsolver.
- AlphaECP-1: AlphaECP with CPLEX as subsolver.
- DICOPT-1: DICOPT with CONOPT and CPLEX as subsolver.
- BONMIN-1: BONMIN.
- results of the following comparision on all 149 models which BARON can handle (not SOS, no semicontinuous variables, no functions like sin, cos, ...):
- BARON
- LaGO-1: LaGO with violation subdivision and "unfixed discrete" as node selection type.
- BONMIN-1: BONMIN.
- results of the following comparision on all 40 convex models (no SOS, semicontinuous variables, or functions like sin, cos, ...):
- BARON
- LaGO-1: LaGO with violation subdivision and "unfixed discrete" as node selection type.
- BONMIN-1: BONMIN.
- results of the following comparision on all 149 models which BARON can handle (not SOS, no semicontinuous variables, no functions like sin, cos, ...):
- BARON
- LaGO-1: LaGO with violation subdivision and "unfixed discrete" as node selection type.
- results of the following comparision on all 40 convex models (no SOS, semicontinuous variables, or functions like sin, cos, ...):
- BARON
- AlphaECP-1: AlphaECP with CPLEX as subsolver.
- BONMIN-1: BONMIN.
- DICOPT-1: DICOPT with CONOPT and CPLEX as subsolver.
- LaGO-1: LaGO with violation subdivision and "unfixed discrete" as node selection type.
- SBB-1: SBB with CONOPT as subsolver.
27.09.2007:
- results of the following comparision on all 167 models:
- DICOPT-4: DICOPT with CONOPT and CPLEX as subsolvers and options maxcycles 10000 and stop 0
- DICOPT-5: DICOPT with CONOPT and CPLEX as subsolvers and options maxcycles 10000 and stop 1
- DICOPT-6: DICOPT with CONOPT and CPLEX as subsolvers and options maxcycles 10000 and stop 2
- DICOPT-7: DICOPT with CONOPT and CPLEX as subsolvers and options maxcycles 10000 and stop 3
28.09.2007:
- results of the following comparision on all 167 models:
- SBB-1: SBB with CONOPT
- SBB-7: SBB with CONOPT and memnodes 100000
- results of the following comparision on all 161 models without semicontinuous variables and SOS constraints (since these are not supported by GAMS/Bonmin yet):
- SBB-7: SBB with CONOPT as subsolver and memnodes 10000.
- AlphaECP-1: AlphaECP with CPLEX as subsolver.
- DICOPT-5: DICOPT with CONOPT and CPLEX as subsolver and options maxcycles 10000 and stop 1 (stop on crossover)
- BONMIN-1: BONMIN.
- results of the following comparision on all 40 convex models:
- SBB-7: SBB with CONOPT as subsolver and memnodes 10000.
- AlphaECP-1: AlphaECP with CPLEX as subsolver.
- DICOPT-5: DICOPT with CONOPT and CPLEX as subsolver and options maxcycles 10000 and stop 1 (stop on crossover)
- BONMIN-1: BONMIN.
02.09.2007:
Influence of NLP subsolver on SBB performance:
- results of the following comparision:
- SBB-1: SBB with CONOPT as subsolver. CONOPT option Rtmaxv is set to 1e+20.
- SBB-2: SBB with IPOPT as subsolver and IPOPT option expect_infeasible_problem set to yes.
- SBB-3: SBB with IPOPT as subsolver and IPOPT options expect_infeasible_problem and warm_start_init_point set to yes.
- SBB-4: SBB with CONOPT and IPOPT in a infeasseq as subsolver: If CONOPT reports a subproblem as infeasible and we are at atmost depth 50 of the Branch-and-Bound tree, SBB calls IPOPT. IPOPT option expect_infeasible_problem is set to yes.
- SBB-5: SBB with IPOPT as subsolver. IPOPT option expect_infeasible_problem set to yes and print_level set to 0.
- results of the following comparision on all 40 convex models:
- SBB-1: SBB with CONOPT as subsolver. CONOPT option Rtmaxv is set to 1e+20.
- SBB-2: SBB with IPOPT as subsolver and IPOPT option expect_infeasible_problem set to yes.
- SBB-3: SBB with IPOPT as subsolver and IPOPT options expect_infeasible_problem and warm_start_init_point set to yes.
- SBB-4: SBB with CONOPT and IPOPT in a infeasseq as subsolver: If CONOPT reports a subproblem as infeasible and we are at atmost depth 50 of the Branch-and-Bound tree, SBB calls IPOPT. IPOPT option expect_infeasible_problem is set to yes.
- results of the following comparision:
- SBB-1: SBB with CONOPT as subsolver. CONOPT option Rtmaxv is set to 1e+20.
- SBB-5: SBB with IPOPT as subsolver. IPOPT option expect_infeasible_problem set to yes and print_level set to 0.
- SBB-4: SBB with CONOPT and IPOPT in a infeasseq as subsolver: If CONOPT reports a subproblem as infeasible and we are at atmost depth 50 of the Branch-and-Bound tree, SBB calls IPOPT. IPOPT option expect_infeasible_problem is set to yes.
- results of the following comparision on all 40 convex models::
- SBB-1: SBB with CONOPT as subsolver. CONOPT option Rtmaxv is set to 1e+20.
- SBB-5: SBB with IPOPT as subsolver. IPOPT option expect_infeasible_problem set to yes and print_level set to 0.
- SBB-4: SBB with CONOPT and IPOPT in a infeasseq as subsolver: If CONOPT reports a subproblem as infeasible and we are at atmost depth 50 of the Branch-and-Bound tree, SBB calls IPOPT. IPOPT option expect_infeasible_problem is set to yes.
Influence of NLP and MIP subsolver on DICOPT performance:
- results of the following comparision:
- DICOPT-1: DICOPT with CPLEX and CONOPT as subsolver.
- DICOPT-2: DICOPT with CPLEX and IPOPT as subsolver.
- results of the following comparision:
- DICOPT-1: DICOPT with CPLEX and CONOPT as subsolver.
- DICOPT-3: DICOPT with CBC and CONOPT as subsolver.
Influence of MIP subsolver on AlphaECP performance:
- results of the following comparision:
- AlphaECP-1: AlphaECP with CPLEX as subsolver.
- AlphaECP-2: AlphaECP with CBC as subsolver.