Fix: Eval hash mismatch due to parameter truncation in DB storage#1523
Open
rlundeen2 wants to merge 4 commits intoAzure:mainfrom
Open
Fix: Eval hash mismatch due to parameter truncation in DB storage#1523rlundeen2 wants to merge 4 commits intoAzure:mainfrom
rlundeen2 wants to merge 4 commits intoAzure:mainfrom
Conversation
jsong468
reviewed
Mar 19, 2026
Contributor
There was a problem hiding this comment.
Could you re-run the configuring scenarios notebook?
Contributor
Author
There was a problem hiding this comment.
There is another bug because scenarios aren't setting the underlying model so it's defaulting to "gpt-4o" and the hash is different. I want scenarios to grab this from the registry so there isn't a mismatch. But for now the notebook doesn't update. I'd like to tackle with a future PR
Store eval_hash inside ComponentIdentifier serialization (to_dict/from_dict) so it survives DB round-trips without recomputation from truncated params. - ComponentIdentifier: added stored_eval_hash field and KEY_EVAL_HASH - EvaluationIdentifier: uses stored_eval_hash when available - ScenarioResultEntry/ScoreEntry/AttackResultEntry: compute eval_hash before truncation - atomic_attack.py: same fix for enriched identifier persistence - Tests: round-trip, double round-trip, and regression tests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
rlundeen2
force-pushed
the
users/rlundeen/2026_03_19_eval_hash_bug
branch
from
409d1b7 to
7084a21
Compare
rlundeen2
commented
Mar 20, 2026
| #: Evaluation hash preserved from DB round-trip. Computed before truncation and | ||
| #: stored alongside the identity so that EvaluationIdentifier can use it directly | ||
| #: instead of recomputing from potentially truncated params. | ||
| stored_eval_hash: Optional[str] = field(default=None, init=False, compare=False) |
Contributor
Author
There was a problem hiding this comment.
want to rename to eval_hash
rlundeen2
commented
Mar 20, 2026
| ComponentIdentifier: The identity with ``stored_eval_hash`` set. | ||
| """ | ||
| identifier = super().get_identifier() | ||
| if identifier.stored_eval_hash is None: |
Contributor
Author
There was a problem hiding this comment.
I think we can make this better
rlundeen2
commented
Mar 20, 2026
| object.__setattr__(identifier, "stored_eval_hash", eval_hash) | ||
| return identifier | ||
|
|
||
| def get_eval_hash(self) -> str: |
Contributor
Author
There was a problem hiding this comment.
I think we should get rid of get_eval_hash
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bug: Running
await printer.print_summary_async(scenario_result)in 1_configuring_scenarios.ipynb prints "official evaluation has not been run yet for this specific configuration" — even when evals have been run.Root cause: Long scorer params (e.g., system prompt templates) are truncated to 80 characters when stored in the DB via
ComponentIdentifier.to_dict(max_value_length=80). Theidentity .hashis correctly preserved through the round-trip, buteval_hashis recomputed from the truncated params by EvaluationIdentifier, producing a different hash than what was stored during the eval run. This causes the metrics lookup to fail silently.Fix: Store eval_hash inside the ComponentIdentifier serialization (to_dict/from_dict) so it survives DB round-trips without recomputation from truncated params.
No DB schema migration needed — eval_hash is stored inside the existing JSON columns. Old data without it falls back to recomputation (same as prior behavior).