Repair Sub-Agent

[ElliotGestrin]

When generating complex code (such as C++) or when using a more lightweight LLM, the models will likely cause minor issues, e.g. declaring variables with the wrong type or similar easily-corrected flaws. Currently, OpenEvolve will simply evaluate this flawed program, likely scoring it 0, and add it to the population. This will then (possibly) be used as a parent later, which hopefully would fix the compilation issue.

Instead, it could make sense to attempt to correct the flaws immediately, before evaluating the child and adding it to the population. This PR enables such functionality, through an optional repair sub-agent.

The evaluator can raise an EvaluatorRepairRequest (defined in evaluation_results.py) which wil be caught and trigger a repair agent, if one is available. This can be done anywhere in the evaluator and allows for passing error messages and similar to the repair agent, along with a default score to assign if the repair itself fails (or is disabled).

After each repair attempt, the evaluator will automatically be called again and the repaired child will be evaluated. This could trigger another repair request if needed, and this cycle can be looped a configurable number of times.

To enable the repair subagent, one configures the new config.yaml keys:

evaluator:
  repair_on_failure: true
  max_repair_attempts: 2
  repair_diff_based: false  # true for SEARCH/REPLACE diffs, false for full rewrite

llm:
  repair_models:  # optional — falls back to evaluator_models, then models
    - name: "your-repair-model"
      weight: 1.0

Setting max_repair_attempts: 0 or repair_on_failure: false (defaults) disables the repair sub-agent. A separate LLM (likely smaller) can be used for the repair process, and new prompts are available (repair_diff_user.txt and repair_full_rewrite_user.txt) to configure the repair behaviour.

The repair changelogs will be stored and can be visualized in a new tab in the visualizer, if this is enabled.

Happy to discuss and adapt this implementation to better fit the project. Thanks for the great work!

[CLAassistant]

All committers have signed the CLA.

[ElliotGestrin]

Even if the main PR content (the repair agent) is rejected, the last minor determinism change should probably be included. Currently, setting a seed appears to be able to cause non-determinism between machines, due to hashing of sets.