chore: add E2E guardrails parity tests [AL-288]#733
Open
apetraru-uipath wants to merge 2 commits intofeat/guardrails_decoratorsfrom
Open
chore: add E2E guardrails parity tests [AL-288]#733apetraru-uipath wants to merge 2 commits intofeat/guardrails_decoratorsfrom
apetraru-uipath wants to merge 2 commits intofeat/guardrails_decoratorsfrom
Conversation
Add stage parameter to pii_detection_guardrail and prompt_injection_guardrail decorators, matching the flexibility already available in deterministic_guardrail. pii_detection_guardrail defaults to PRE_AND_POST; prompt_injection_guardrail defaults to PRE and raises pydantic.ValidationError for POST or PRE_AND_POST (prompt injection is input-only). Fix _apply_llm_input_guardrail and _apply_guardrail_to_message_list: previously concatenated the full conversation history and tried to replace the joined text inside a single message, which silently failed in any multi-turn conversation. Now evaluates only the last HumanMessage (PRE/input) or last AIMessage (POST/output) — semantically correct and replacement works reliably. Replace _extract_text_from_messages with focused helpers: _get_last_human_message, _get_last_ai_message, _extract_message_text, _apply_message_text_modification. Add target_type parameter to _apply_guardrail_to_message_list so input and output graph-scope wrappers target the correct message type. PII decorator, joke-agent-decorator sample, BlockAction uses AgentRuntimeError. Fix _wrap_llm_with_guardrail: factory functions returning BaseChatModel were not wrapped (fell through StateGraph/dict branches). Also fix Pydantic setattr block on UiPathChat by using __class__ swap to a dynamic subclass instead of monkey-patching invoke/ainvoke. Fix BlockAction swallowed by bare except: split try/except so only guardrail API errors are suppressed; action exceptions (AgentRuntimeError) now propagate. Fix CompiledStateGraph not recognised: add _wrap_compiled_graph_with_guardrail and handle CompiledStateGraph return type from factory functions. Fix mypy errors in decorators.py: typed list[BaseMessage], CompiledStateGraph type params, type: ignore[valid-type, misc] for dynamic subclass, and type: ignore[method-assign] for CompiledStateGraph method patching. Add prompt_injection_guardrail decorator: _create_prompt_injection_guardrail, _apply_prompt_injection_guardrail, public prompt_injection_guardrail function; exported from guardrails/__init__.py; stacked on create_llm() in joke-agent-decorator/graph.py with BlockAction to block on detection. Reformat decorators.py for consistency. Middleware cleanup: delete monolithic middleware.py (duplicate of middlewares/ split files); update guardrails/__init__.py to import from .middlewares; update joke-agent/graph.py to use new split-file API (tool_names -> tools, optional scopes on PromptInjection, unconditional POST filter comment). Revert renames: restore LoggingSeverityLevel as proper int Enum (ERROR, INFO, WARNING, DEBUG) in actions.py; remove PromptInjectionValidatorType from enums.py; fix pii_detection.py docstring to use Entity+PIIDetectionEntity; export LoggingSeverityLevel from guardrails/__init__.py; update both samples to use LoggingSeverityLevel instead of AgentGuardrailSeverityLevel. Manual refinements: updates to actions.py, decorators.py, enums.py, models.py, middlewares/pii_detection.py, guardrails/__init__.py, and joke-agent/graph.py. Remove joke-agent/.agent/REQUIRED_STRUCTURE.md; further manual edits to joke-agent/graph.py. Refactor decorators.py into decorators/ package (pii.py, prompt_injection.py, deterministic.py, _base.py) with tool-level guardrail support: - Split monolithic decorators.py into decorators/ subpackage - Add _wrap_tool_with_guardrail using __class__ swap (Pydantic-safe) - Add deterministic_guardrail decorator (TOOL scope, local rules, no API call) - Extend pii_guardrail to support BaseTool and optional tools= kwarg - Extend _detect_scope to return GuardrailScope.TOOL for BaseTool instances - Export deterministic_guardrail and RuleFunction from guardrails/__init__.py - Update joke-agent-decorator/graph.py to demonstrate all three decorator types on analyze_joke_syntax tool (3x @deterministic_guardrail + @pii_guardrail) - Add local CustomFilterAction to joke-agent-decorator sample Tool guardrail fixes and sample updates: - _base.py: unwrap LangGraph tool-call envelope (args) for rule evaluation; rewrap modified args so super().invoke() receives valid input; handle ToolMessage/Command in _extract_output for POST-stage deterministic rules. - joke-agent-decorator: Agent PII uses LogAction(WARNING) with custom message; README aligned with current guardrails and verification scenarios. Deps and lockfiles: pyproject.toml updates (root and joke-agent-decorator); remove samples/joke-agent-decorator/uv.lock and samples/joke-agent/uv.lock; uv.lock at repo root updated. Refactor _base.py: remove unnecessary casts in _evaluate_rules; catch only ValueError (JSONDecodeError is subclass); extract _apply_guardrail_to_message_list, _apply_guardrail_to_input_messages, _apply_guardrail_to_output_messages and use them in _wrap_stategraph_with_guardrail, _wrap_compiled_graph_with_guardrail, and _wrap_function_with_guardrail to reduce cognitive complexity. Made-with: Cursor Enable enabled_for_evals override across decorators and middlewares for PII, prompt injection, and deterministic guardrails (default true, user-overridable), plus docs/sample updates for the new parameter. Fix mypy errors in pii_detection.py and prompt_injection.py: rebind guardrail to a typed non-optional local variable so mypy can narrow the type inside nested class closures. Made-with: Cursor Fix _wrap_compiled_graph_with_guardrail: output guardrail (POST stage) was never applied — invoke/ainvoke discarded the graph output without running _apply_guardrail_to_output_messages. Now captures the output and evaluates it, matching the behaviour of _wrap_stategraph_with_guardrail. Update pyproject.toml and uv.lock. Fix double POST guardrail application on StructuredTool: remove ainvoke override from _GuardedTool in deterministic.py and pii_detection.py. StructuredTool.ainvoke delegates to self.invoke via run_in_executor, so the guardrail chain in invoke already runs once; the ainvoke override caused a second POST application, producing "words++++" instead of "words++". Fix LogAction double-logging: replace print()+logger.log() with a single logger.log() call using the guardrail name as context prefix. Fix _evaluate_rules violation message: include guardrail name instead of positional index so errors read "Rule <name> detected violation" rather than "Rule 1 detected violation". Pass guardrail_name through _apply_pre/_apply_post call sites in deterministic.py. Fix AgentRuntimeError swallowed in guardrail middleware/decorator except handlers: add explicit except AgentRuntimeError: raise before the generic except Exception in pii_detection decorator (_apply_pre, _apply_post), pii_detection middleware (_wrap_tool_call_func, _check_messages), and prompt_injection middleware (_check_messages) so BlockAction errors propagate instead of being silently logged.
…entation [AL-288] (#732) Co-authored-by: Claude <noreply@anthropic.com>
apetraru-uipath
changed the title
apetraru-uipath
force-pushed
the
feat/guardrails_decorators
branch
from
fc9705d to
f2e1211
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed?
E2E parity test suite verifying that the middleware API (
create_agent(middleware=[...])) and the decorator API (@pii_detection_guardrail,@prompt_injection_guardrail,@deterministic_guardrail) produce identical runtime behaviour for every guardrail scenario.New files:
tests/cli/mocks/parity_agent_decorator.py— joke agent with 8 guardrails configured via decoratorstests/cli/mocks/parity_agent_middleware.py— same agent and same 8 guardrails configured via middlewaretests/cli/test_guardrails_in_langgraph.py— 6 scenarios × 2 flavors = 12 parametrized test runsHow has this been tested?
uv run pytest tests/cli/test_guardrails_in_langgraph.py— 12/12 pass.Tests mock
UiPathChatOpenAI.ainvokeandGuardrailsService.evaluate_guardrail; no real API calls. Blocking tests assert all 5 fields ofexc.__cause__.error_info(code,title,detail,category,status).Scenarios covered (× 2 flavors each):
test_happy_pathtest_agent_pii_blockBlockAction→AgentRuntimeErrortest_llm_prompt_injection_blockBlockActionbefore LLM is ever called (llm_call_count == 0)test_tool_pii_blockBlockActionbefore tool runstest_tool_deterministic_word_filterToolMessagehas no "donkey"test_tool_deterministic_length_blockAgentRuntimeErrorAre there any breaking changes?