Improve performance of HiGHS interface

[BenChampion]

Summary

Make the pyomo interface to HiGHS faster.

Rationale

Following a HiGHS user report, we've noticed that the pyomo interface to HiGHS can be slow. In particular, it seems to be slower than the interface to cbc by a significant margin.

Description

One potential solution is for pyomo to write out a file and read it into HiGHS. (My understanding is that this is how the pyomo cbc interface works already.) Some experimentation suggests this could be much faster.

Although this seems inelegant, I'm not sure of the exact drawbacks.

Additional information

From reading other related issues, this may relate to #1030?

[mrmundt]

@BenChampion - Oh no, adding more to #1030 ?! The list never ends..

Jk, thanks for the report. A lot of our different interfaces write files to then be read into solvers, so it's a fairly well-established paradigm.

[BenChampion]

Just collecting various discussions into a single summary (mostly for myself, so I don't forget!)

Thanks are due to @michaelbynum for much of this! I take responsibility for any errors, of course.

Root cause

We think that this is due to the HiGHS interface (Highs) being "persistent", with no "direct" interface available.

The "persistent" paradigm targets the use case of a sequence of solves with only relatively minor changes to the model between solves. As such, it has additional overhead for change-tracking data structures, in exchange for not having to reload the model for every minor change.

If the true use case is solving a sequence of completely different models, this additional overhead doesn't gain us anything, and the "direct" approach is more appropriate.

Fix proposal

Make a HighsDirect solver interface, analogous to the GurobiDirect interface.

Additional discussion

The HiGHS team doesn't have the pyomo expertise necessary to implement a fix. Nevertheless, this is very much something we'd like to see on behalf of our users via pyomo (an open-source modelling language to go along with an open-source solver!)

Remaining question

Why doesn't the Cbc interface seem to have as much overhead, even though it's a PersistentSolver too?

[michaelbynum]

Which Cbc interface are you referring to?

[michaelbynum]

Good summary by the way. Thanks for that!

[BenChampion]

Which Cbc interface are you referring to?

Ah, I think this answers/invalidates my remaining question. I now see there's another interface, CBC (all caps) that inherits from OptSolver. It seems to be the one you get when you do pyo.SolverFactory("cbc"), while the first one I spotted (Cbc) seems to be the pyo.SolverFactory("appsi_cbc") one.

It'll be the pyo.SolverFactory("cbc") one, which I'm guessing isn't persistent. My mistake! That is consistent with the performance difference.