fix(tokenizer): apply config-build fallback to offset tokenizer too

[hallerite]

Problem

#72 wired _load_fast_tokenizer_directly into _load_tokenizer_via_auto so load_tokenizer survives model-config build failures (e.g. HF RoPE validation rejecting nested rope_parameters for poolside/Laguna-XS.2).

But _get_offset_tokenizer was calling AutoTokenizer.from_pretrained directly to keep the fastokens patch out of this path — bypassing the fallback entirely.

That meant every hand-coded renderer (LagunaXS2, Qwen35, ...) still crashed on the first rollout for Laguna-family models, with the same KeyError #72 was meant to fix:

LagunaXS2Renderer.render
 → emit_text_segments
 → attribute_text_segments
 → _get_offset_tokenizer
 → AutoTokenizer.from_pretrained
 → HF RoPE validator
 → KeyError("Missing required keys in `rope_parameters` for 'rope_type'='default': {'rope_theta'}")

This bypass is also why disabling renderers made the symptom go away on hosted-rl: the offset-tokenizer path is only hit when a hand-coded renderer is in use.

Fix

Route the offset-tokenizer load through _load_tokenizer_via_auto. Same vanilla path (no fastokens patching, since that helper doesn't apply it), but now the _load_fast_tokenizer_directly fallback runs when the model config build fails.

Verification

Reproduced end-to-end against poolside/Laguna-XS.2 with prime-rl + reverse-text:

• Without this patch — every rollout aborts with ModelError() -> KeyError("Missing required keys in 'rope_parameters' for 'rope_type'='default': {'rope_theta'}").
• With this patch — rollouts succeed; first two RL steps complete cleanly:
  • Step 0: reward 0.3553, 97.9 tok/sample
  • Step 1: reward 0.3116, 110.8 tok/sample

(The run then OOMs on the trainer side at step 2, which is an unrelated single-GPU FSDP capacity issue with the 256-expert Laguna model.)

Follow-up worth a separate issue

For hand-coded renderers, the bulk of message-body tokenization goes through attribute_text_segments → vanilla offset tokenizer, not through the fastokens-patched main tokenizer. fastokens only helps decode + the few _encode calls that don't need offsets. Worth considering whether to skip fastokens entirely for hand-coded renderers (load_tokenizer(..., use_fastokens=False)) and avoid keeping two tokenizer copies in memory. Not in scope for this fix.

🤖 Generated with Claude Code

Note

Apply config-build fallback to _get_offset_tokenizer in renderer base

Routes tokenizer instantiation in _get_offset_tokenizer through _load_tokenizer_via_auto instead of calling AutoTokenizer.from_pretrained directly, bringing it in line with the main tokenizer loading path. This ensures the config-build fallback is available when loading offset tokenizers, while still requiring the result to be a fast tokenizer with offset_mapping support.

^{Macroscope summarized f0435d4.}

---

Note

Low Risk
Single call-site change in tokenizer loading for offset attribution; aligns with an existing tested fallback path and does not alter fastokens or main load_tokenizer behavior.

Overview
_get_offset_tokenizer no longer calls AutoTokenizer.from_pretrained directly. It now loads the vanilla, offset-capable tokenizer through _load_tokenizer_via_auto, same trust/revision kwargs as before.

That keeps this path without fastokens (still not load_tokenizer) but applies the config-build fallback used elsewhere: when HF fails building the model config (e.g. Laguna rope_parameters / missing rope_theta), loading can succeed via tokenizer.json instead. Hand-coded renderers that depend on attribute_text_segments / offset mapping were still crashing on those models because the offset path skipped that fallback.

^{Reviewed by Cursor Bugbot for commit f0435d4. Bugbot is set up for automated code reviews on this repo. Configure here.}

[macroscopeapp]

Approvability

Verdict: Approved

Small fix that applies an existing fallback pattern to the offset tokenizer loading path. The change replaces a direct AutoTokenizer.from_pretrained call with an internal wrapper that adds error recovery for edge cases while keeping the happy path unchanged.

^{You can customize Macroscope's approvability policy. Learn more.}