Add dynamic router for manual model selection #504

ryanhoangt · 2025-09-26T14:49:46Z

This PR is to:

Add serialization/deserialization support for all existing router implementations
Add support for switching model mid convo

Agent Server images for this PR

• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server

Variants & Base Images

Variant	Base Image	Docs / Tags
golang	`golang:1.21-bookworm`	Link
java	`eclipse-temurin:17-jdk`	Link
python	`nikolaik/python-nodejs:python3.12-nodejs22`	Link

Pull (multi-arch manifest)

docker pull ghcr.io/openhands/agent-server:df4aa57-python

Run

docker run -it --rm \
  -p 8000:8000 \
  --name agent-server-df4aa57-python \
  ghcr.io/openhands/agent-server:df4aa57-python

All tags pushed for this build

ghcr.io/openhands/agent-server:df4aa57-golang
ghcr.io/openhands/agent-server:v1.0.0a3_golang_tag_1.21-bookworm_binary
ghcr.io/openhands/agent-server:df4aa57-java
ghcr.io/openhands/agent-server:v1.0.0a3_eclipse-temurin_tag_17-jdk_binary
ghcr.io/openhands/agent-server:df4aa57-python
ghcr.io/openhands/agent-server:v1.0.0a3_nikolaik_s_python-nodejs_tag_python3.12-nodejs22_binary

The df4aa57 tag is a multi-arch manifest (amd64/arm64); your client pulls the right arch automatically.

openhands/sdk/llm/llm.py

openhands/sdk/agent/base.py

github-actions · 2025-10-10T13:59:33Z

Coverage Report •

File	Stmts	Miss	Cover	Missing
openhands-sdk/openhands/sdk
__init__.py	19	2	89%	56–57
openhands-sdk/openhands/sdk/agent
base.py	163	16	90%	146, 152, 170–172, 221, 229–230, 261, 307, 314, 327, 364–365, 375–376
openhands-sdk/openhands/sdk/context/condenser
llm_summarizing_condenser.py	45	25	44%	22, 29, 32–34, 37–38, 41, 43, 45–49, 52, 55, 57, 64, 66, 71–75, 77
openhands-sdk/openhands/sdk/conversation
base.py	60	13	78%	95, 106, 160, 162, 170, 173–174, 176, 178, 184–187
title_utils.py	49	38	22%	41–42, 48–51, 53–54, 56, 73–74, 76, 78, 82, 84, 121, 124, 127, 130–131, 133, 135–136, 138–140, 153–156, 181, 183–184, 187–190, 193
openhands-sdk/openhands/sdk/conversation/impl
local_conversation.py	159	46	71%	129–130, 150, 155, 166, 183, 191–193, 197–198, 248–250, 257, 276–277, 279, 283–285, 293, 295, 297, 301, 303–305, 307, 309, 315–316, 329–330, 332, 334, 338–341, 363, 365–367, 384, 386
remote_conversation.py	351	239	31%	47–59, 66–69, 89–94, 97–101, 104–108, 111–113, 115–118, 121–124, 127–128, 130–142, 145–150, 167–171, 173, 177, 179–180, 182–185, 187, 193, 195, 197–199, 201–203, 207, 209–212, 216, 221–222, 224, 227, 236–237, 240–241, 254–256, 259–260, 264, 266–267, 270, 273–275, 279, 281, 283–285, 288–290, 295–297, 299, 304, 309, 314–317, 320, 329, 337–340, 343, 348–349, 354–358, 363–367, 372–375, 378, 382–383, 387, 391, 394, 434–438, 440–441, 453, 456, 458–460, 463, 466, 468, 471, 474–475, 478–479, 482–485, 487, 490, 493, 499, 502, 504–505, 509, 514, 519–521, 527, 533–535, 538, 543, 550–551, 558, 560–564, 567–568, 577, 585, 590–592, 594, 597, 599–600, 616, 623, 629–630, 633, 635–639, 641–644, 647–650
openhands-sdk/openhands/sdk/llm
llm.py	451	172	61%	276, 286, 294, 298, 303, 307, 319, 323–325, 329–330, 341, 343, 347, 364, 388, 398, 403, 407, 412, 423, 438, 459–460, 463, 467, 479, 484–487, 494, 497, 505–510, 513, 528, 532–534, 536–537, 542–543, 545, 552, 555–557, 619–622, 627–632, 636–637, 646–648, 651–652, 689–690, 736, 756, 794, 810, 813–815, 818–826, 830–832, 835, 838–840, 847–848, 857, 864–866, 870, 872–877, 879–896, 899–903, 905–906, 912–921, 934, 944–945, 950–951, 955, 957, 966–967, 971, 976–977, 982, 996, 1017, 1021, 1036, 1042–1043, 1062–1063, 1071, 1096–1098, 1102, 1116, 1121
message.py	247	112	54%	43, 48, 50, 67–69, 71–74, 76, 97, 99, 104, 177, 181, 198–203, 241, 248, 251, 268, 286, 289, 292, 301, 305–307, 323, 339–345, 359, 361–362, 364–371, 374, 387, 389, 391–396, 404–405, 407–408, 418–419, 423–427, 430–433, 435–436, 439, 444–446, 453, 455, 472, 486, 502, 527–529, 531–532, 536–539, 543–545, 548–552, 554–556, 558, 566–567, 584–585
openhands-sdk/openhands/sdk/llm/router
base.py	73	42	42%	47–48, 51, 55, 58, 63, 78–79, 81, 84, 110–112, 116, 122–124, 127–128, 130, 151, 153–155, 161, 164–165, 171–174, 177–178, 183–185, 191, 194–195, 197–198, 203
openhands-sdk/openhands/sdk/llm/router/impl
dynamic.py	26	13	50%	39–40, 43, 57–59, 61–64, 69–70, 74
openhands-tools/openhands/tools/preset
planning.py	43	30	30%	63–64, 66–67, 69–71, 73, 84–86, 88, 93–95, 97–102, 113, 115, 132, 137, 153, 156, 164, 166, 179
TOTAL	8301	3623	56%

examples/01_standalone_sdk/19_llm_routing.py

xingyaoww

Overall looks great

examples/01_standalone_sdk/25_llm_manual_switch.py

xingyaoww · 2025-10-13T14:52:37Z

openhands-sdk/openhands/sdk/llm/router/base.py

@@ -30,11 +31,12 @@ class RouterLLM(LLM):
    """

    router_name: str = Field(default="base_router", description="Name of the router")
-    llms_for_routing: dict[str, LLM] = Field(
+    llms_for_routing: dict[str, LLMBase] = Field(


Let's confirm this will be persisted correctly

Let's use an immutable mapping for this:

from pydantic import BaseModel, ConfigDict
from types import MappingProxyType

class MyModel(BaseModel):
model_config = ConfigDict(frozen=True)
data: dict[str, int]

def __init__(self, **kwargs): super().__init__(**kwargs) object.__setattr__(self, "data", MappingProxyType(self.data))

m = MyModel(data={"a": 1})
m.data["a"] = 2 # ❌ TypeError: 'mappingproxy' object does not support item assignment

I updated the example to assert for the correct active model and add immutability for llms_for_routing!

enyst · 2025-10-20T11:59:13Z

examples/01_standalone_sdk/25_llm_manual_switch.py

+conversation.run()
+
+# Switch back to primary model for complex task
+print("Switching back to claude for complex reasoning...")


I wonder how would the user know in advance that they'll need to switch?

I think we were designing the DynamicRouter as a way for the user to "manually select model name from drop down"?

And further down the road, we should by default initialize LLM as DynamicRouter in our GUI/CLI if we have UI for switching model on the fly

enyst · 2025-10-20T12:00:32Z

openhands-sdk/openhands/sdk/llm/llm.py



-class LLM(BaseModel, RetryMixin, NonNativeToolCallingMixin):
+class LLMBase(DiscriminatedUnionMixin, RetryMixin, NonNativeToolCallingMixin):


Could you please explain, why is LLM a DiscriminatedUnionMixin?

We use it to distinguish between router classes and normal LLM when deserializing, without it we'll need an explicit kind field.

enyst · 2025-10-20T12:02:13Z

openhands-sdk/openhands/sdk/llm/llm.py

            message.function_calling_enabled = self.is_function_calling_active()
+            message.extended_thinking_enabled = get_features(
+                self.model
+            ).supports_extended_thinking


Sorry, is this a different issue than the PR is about?

Ah for trajectories with a single LLM, we don't need this field, but it requires when using an extending thinking LLM like Sonnet 4.5 with another non-thinking LLM. I ran into the error earlier when testing and adding this field to Message seems to fix it.

enyst · 2025-10-20T12:06:48Z

openhands-sdk/openhands/sdk/llm/router/base.py

+    @model_validator(mode="after")
+    def make_immutable(self):
+        object.__setattr__(
+            self, "llms_for_routing", MappingProxyType(self.llms_for_routing)


Why do we expressly make them immutable when... they weren't?
Do I understand correctly, that will mean that the user cannot decide at runtime to just switch to another LLM they pick up, let's say, from openhands provider? (where they already had an API key)

Actually my earlier implementation did allow user to add a new LLM config to the router during runtime, but I remember @xingyaoww said it might lead to some weird issues if we modify configs like that, ideally we want immutability when possible. I think it boils down to what "UX" we want to achieve, whether to pre-define all LLMs in advance or add a new LLM during the session. After thinking a bit more, I feel like asking users to set some LLMs they prefer to use in advance (e.g. a set of Sonnet 4.5, Codex and GLM-4.6) and switching between them later seem to be more convenient and not too rigid, I imagine myself also too lazy to manually input LLM configs within sessions many times 🤔

enyst · 2025-10-20T12:23:18Z

openhands-sdk/openhands/sdk/llm/router/base.py

+            persisted = self.__class__.model_validate(persisted_dict)
+
+        # Check classes match exactly
+        if type(persisted) is not type(self):


TBH I don't know, I don't really like that we use this code, which "sets things in stone", when what we'd like to implement in this PR is precisely flexibility? It seems a bit contradictory, and it's getting too complex maybe... I'm following up

enyst

Thank you for the work on this, @ryanhoangt !

I'm not sure that this way is the best way, though, and I'd appreciate to think it over a bit. It seems that this PR:

requires the user to declare ahead of time a "DynamicRouter", otherwise they're stuck with what they started
they can only switch between two?
it seems to add quite a lot of complexity to LLM class and its persistence, to work around the fact that we froze the LLM, if I understand correctly? Please correct me if this is wrong.

TBH I'm not sure about this, and I'd appreciate to look a bit at the alternative.

@xingyaoww I understand we want freeze for evals, but then maybe we could have an INLINE_CONVERSATIONS for evals? In that case, we store it with the conversation and enforce its restoration as-it-was.

I looked into what it implies to switch LLM, and it seems relatively straightforward, if:

we don't do the enforced freeze for any LLM parameters
it's very related to something like this, LLM profiles. Profiles are persisted on disk, just like any LLM, and it seems to me that it becomes totally possible to choose to persist into the Conversation the full LLM, or only its profile reference: then the LLM Registry will provide the full LLM. Similarly, it can provide it for switching.

My thinking is, we could look into using that mechanism to switch LLM on the fly. We could use those profiles registered in the registry.

What if we try to do it that way? WDYT?

blacksmith-sh · 2025-11-26T13:00:43Z