Fix Custom Evaluators section in AGENTCORE_EVALUATIONS_GUIDE.md

[abhilashbalachandran]

Description

The Custom Evaluators code example had incorrect API parameters that cause ParamValidationError and ValidationException when used against the actual CreateEvaluator API.

What was wrong:

• Used flat parameters (evaluationLevel, inferenceConfig, instructions, ratingScale) — the API requires these nested under evaluatorConfig.llmAsAJudge
• Used {{input}} and {{output}} as placeholders — the API expects single-brace placeholders from a fixed set: {context}, {assistant_turn}, {tool_turn}, {available_tools} (varies by evaluation level)
• Rating scale used {"type": "NUMERIC", "min": 0.0, "max": 1.0} — the API expects {"numerical": [{"value": ..., "label": ..., "definition": ...}]}

What was fixed:

• Corrected the boto3 create_evaluator example to use the nested evaluatorConfig → llmAsAJudge structure
• Added a placeholder reference table documenting valid placeholders per evaluation level
• Added Starter Toolkit SDK example and JSON config file format
• Added example for adding a custom evaluator to an online config via update_online_evaluation_config
• Added note about the auto-appended standardization prompt

Verified by:

• Created a custom evaluator using the corrected API structure — succeeded
• Added it to a live online evaluation config — succeeded
• Confirmed evaluation results with scores and explanations appeared in CloudWatch Logs

References:

AWS Docs — Create evaluator
AWS boto3 sdk for bedrock agentcore control: create_evaluator