Improve classification quality with self-critique, majority voting, and cross-project consistency#2841
Open
migeed-z wants to merge 2 commits intofacebook:mainfrom
Open
Improve classification quality with self-critique, majority voting, and cross-project consistency#2841migeed-z wants to merge 2 commits intofacebook:mainfrom
migeed-z wants to merge 2 commits intofacebook:mainfrom
Conversation
Summary: I noticed when looking at the classifier output for facebook#2764 that the "verdict" formatting needed to be fixed. Two fixes: 1. formatter.py: Add _format_reason() to render JSON reason dicts as labeled readable sections (e.g. "**Spec check:** ...", "**Reasoning:** ...") 2. llm_client.py: Ensure reason is always a string by serializing dict values, so downstream code handles it consistently. Reviewed By: grievejia Differential Revision: D97422229
…nd cross-project consistency Summary: The primer classifier has been producing inconsistent results across runs — the same primer diff can be classified as 'improvement' in one run and 'regression' in another. This was observed on real PRs like facebook#2839 (altair TypeVar iterability) and facebook#2764 (overload resolution, 60+ projects). Three changes to improve reliability: 1. **Self-critique pass (Pass 1.5)**: After Pass 1 produces reasoning, a new pass checks it for factual errors — e.g., claiming dicts are not iterable, incorrect inheritance claims, wrong TypeVar constraint analysis. This catches hallucinations before they reach the verdict pass. Tested on PR facebook#2839 where it correctly identified that both constraints of `_C` (list and TypedDict) are iterable. 2. **Majority voting on verdict (Pass 2)**: Instead of a single verdict call, makes 5 independent calls and takes the majority. This reduces non-determinism where the same reasoning could be classified either way. Vote distribution is logged for transparency. 3. **Cross-project consistency enforcement**: After classifying all projects independently, groups them by error kind and enforces majority verdict within each group. This prevents the classifier from saying 'overload resolution improved' for one project and 'overload resolution regressed' for another with the same pattern. Also upgrades the default Anthropic model from claude-opus-4-20250514 to claude-opus-4-6 for better Pass 1 reasoning quality. Differential Revision: D97571454
yangdanny97
approved these changes
Mar 22, 2026
Contributor
yangdanny97
left a comment
yangdanny97
left a comment
There was a problem hiding this comment.
Review automatically exported from Phabricator review in Meta.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
The primer classifier has been producing inconsistent results across runs — the same primer diff can be classified as 'improvement' in one run and 'regression' in another. This was observed on real PRs like #2839 (altair TypeVar iterability) and #2764 (overload resolution, 60+ projects).
Three changes to improve reliability:
Self-critique pass (Pass 1.5): After Pass 1 produces reasoning, a new pass checks it for factual errors — e.g., claiming dicts are not iterable, incorrect inheritance claims, wrong TypeVar constraint analysis. This catches hallucinations before they reach the verdict pass. Tested on PR Avoid pinning solver variables too early when RHS is a union (#2839) #2839 where it correctly identified that both constraints of
_C(list and TypedDict) are iterable.Majority voting on verdict (Pass 2): Instead of a single verdict call, makes 5 independent calls and takes the majority. This reduces non-determinism where the same reasoning could be classified either way. Vote distribution is logged for transparency.
Cross-project consistency enforcement: After classifying all projects independently, groups them by error kind and enforces majority verdict within each group. This prevents the classifier from saying 'overload resolution improved' for one project and 'overload resolution regressed' for another with the same pattern.
Also upgrades the default Anthropic model from claude-opus-4-20250514 to claude-opus-4-6 for better Pass 1 reasoning quality.
Differential Revision: D97571454