MAINT: Allow custom Likert system prompt and scale#1514
Open
jsong468 wants to merge 2 commits intoAzure:mainfrom
Open
MAINT: Allow custom Likert system prompt and scale#1514jsong468 wants to merge 2 commits intoAzure:mainfrom
jsong468 wants to merge 2 commits intoAzure:mainfrom
Conversation
jbolor21
reviewed
Mar 18, 2026
| chat_target (PromptChatTarget): The chat target to use for scoring. | ||
| likert_scale (LikertScalePaths): The Likert scale configuration to use for scoring. | ||
| likert_scale (Optional[LikertScalePaths]): The Likert scale configuration to use for scoring. | ||
| custom_likert_path (Optional[Path]): Path to a custom YAML file containing the Likert scale definition. |
Contributor
There was a problem hiding this comment.
maybe this is cleaner here but wondering why we can't use the generic self ask scorer since you can add custom scales already there? this can take custom scales & scale definitions as a yaml or string
jbolor21
reviewed
Mar 18, 2026
| ) | ||
| # Only load the default system prompt template if a custom one wasn't already | ||
| # set via custom_system_prompt_path in __init__. | ||
| if not hasattr(self, "_scoring_instructions_template"): |
Contributor
There was a problem hiding this comment.
nit: can we set to None & avoid using hasattr()?
ie if self._scoring_instructions_template is None:
or pass in as a parameter?
jbolor21
reviewed
Mar 18, 2026
| # The system prompt should come from the custom template, not the default one | ||
| assert "Custom prompt for test_cat" in scorer._system_prompt | ||
| assert "from 1 to 5" in scorer._system_prompt | ||
|
|
Contributor
There was a problem hiding this comment.
nit: maybe add unit tests for all the conditions of the scale/yaml - ie both likert & custom provided gives error, test missing both parameters gives error, file not found, non yaml/malformed?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a follow up to #1444 that allow users to create and use their own yamls for system prompt and Likert scales instead of being forced to choose one of the LikertScalePaths. New validation functions are added to ensure the system prompt and scales are of the right format.
Tests and Documentation
Additional unit tests added.