MAINT: enable strict mypy checking and fix violations by tejas0077 · Pull Request #1515 · Azure/PyRIT

tejas0077 · 2026-03-18T05:52:54Z

Related to #720

The @retry decorator from tenacity lacks type stubs, causing mypy
to report an untyped-decorator error when running strict mode:

pyrit/common/net_utility.py:85: error: Untyped decorator makes function
"make_request_and_raise_if_error_async" untyped [untyped-decorator]

Added # type: ignore[untyped-decorator] comment to suppress the error
until tenacity ships proper type stubs.

@romanlutz — tagging you as you filed the original issue.

Tests and Documentation

No new tests or documentation required for this change.

Verified fix by running:
python -m mypy pyrit/common/ --strict

Result: Success: no issues found in 20 source files

…nsparency and governance Signed-off-by: Tejas Saubhage <tsaubhage0007@gmail.com>

Signed-off-by: Tejas Saubhage <tsaubhage0007@gmail.com>

@Retry

Added type: ignore[untyped-decorator] for tenacity @Retry decorator which lacks type stubs, resolving strict mypy check failure. Related to Azure#720

romanlutz · 2026-03-18T11:08:00Z

Is that the only typing error if you turn on strict typing? I'm not sure if you meant to fix more or want me to review now.

- Remove unused type: ignore comment in net_utility.py - Cast blob_stream.readall() to bytes in storage_io.py - Cast blob_properties.size > 0 to bool in storage_io.py python -m mypy pyrit/common/ --strict -> Success: no issues found in 20 source files python -m mypy pyrit/models/ --strict -> Success: no issues found in 26 source files

tejas0077 · 2026-03-18T14:03:27Z

Hi @romanlutz, I have fixed all strict mypy errors in both common and models modules.
Changes made:
pyrit/common/net_utility.py Removed unused type: ignore comment as tenacity now has proper type stubs in this environment.
pyrit/models/storage_io.py Fixed two no-any-return errors by casting blob_stream.readall() to bytes and blob_properties.size > 0 to bool.
Verification:
python -m mypy pyrit/common/ --strict → Success: no issues found in 20 source files
python -m mypy pyrit/models/ --strict → Success: no issues found in 26 source files
Ready for review!

- pyrit/auth/azure_auth.py: cast token_provider() to str, add type: ignore[no-any-return] for get_bearer_token_provider - pyrit/embedding/openai_text_embedding.py: remove unused type: ignore comment - pyrit/score/printer/console_scorer_printer.py: remove 6 unused type: ignore comments - pyrit/prompt_target/openai/openai_tts_target.py: remove 5 unused type: ignore comments - pyrit/prompt_target/openai/openai_completion_target.py: remove unused type: ignore comment - pyrit/prompt_target/hugging_face/hugging_face_chat_target.py: remove 2 unused type: ignore comments python -m mypy pyrit/ --strict -> Success: no issues found in 422 source files

tejas0077 · 2026-03-18T14:12:05Z

Hi @romanlutz, I have now gone further and fixed all strict mypy errors across the entire pyrit codebase, not just common and models.
Additional files fixed:
pyrit/auth/azure_auth.py -cast token_provider() return to str, added type: ignore[no-any-return] for get_bearer_token_provider which returns Any from the azure-identity library
pyrit/embedding/openai_text_embedding.py -removed unused type: ignore comment
pyrit/score/printer/console_scorer_printer.py -removed 6 unused type: ignore comments
pyrit/prompt_target/openai/openai_tts_target.py -removed 5 unused type: ignore comments
pyrit/prompt_target/openai/openai_completion_target.py -removed unused type: ignore comment
pyrit/prompt_target/hugging_face/hugging_face_chat_target.py -removed 2 unused type: ignore comments
Final verification:
python -m mypy pyrit/ --strict - Success: no issues found in 422 source files
Ready for review!

pyrit/datasets/harm_definition/ai_governance_failure.yaml

romanlutz · 2026-03-18T14:41:52Z

If it's fixed across the entire codebase you may want to set the strict setting to true in the toml file

- Enable strict = true in pyproject.toml (per reviewer romanlutz suggestion) - Fix 170 mypy strict errors across 61 files - Key patterns used: - Optional[T] annotations where = None defaults existed - assert x is not None guards before attribute access - or '' / or [] / or 0 fallbacks where semantically safe - cast() for typed dict .pop() returns - type: ignore[arg-type] inside lambdas where _try_register guards None - Added _client property on OpenAITarget for non-optional client access - Added memory property on PromptNormalizer for non-optional memory access

tejas0077 · 2026-03-18T17:26:29Z

Hi @romanlutz, all review comments have been addressed:
Enabled strict = true in pyproject.toml as suggested
Fixed all 170 strict mypy errors across 61 files (python -m mypy pyrit/ --strict → no issues found in 422 source files)
Removed the stray YAML files (ai_governance_failure.yaml, ai_supply_chain.yaml, ai_system_transparency.yaml) that were accidentally included from another branch
Pre-commit passing

Copilot

Pull request overview

This PR tightens type-safety across PyRIT by enabling global strict mypy checks and applying a broad set of typing-related fixes (optionals, casts, and targeted ignores) across core runtime components (memory, targets, scorers, datasets, UI RPC, and backend routes).

Changes:

Enable mypy strict mode in pyproject.toml and adjust code to satisfy strict typing across the repository.
Add runtime/type-narrowing guards (assertions) and fix various Optional-related issues (e.g., Optional[...] annotations, casts, safer MIME/extension handling).
Refactor OpenAI target call sites to use a unified _client accessor and remove some no-longer-needed type ignores.

Reviewed changes

Copilot reviewed 66 out of 66 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
pyproject.toml	Turns on global strict mypy settings.
pyrit/common/net_utility.py	Tightens httpx client kwargs typing/casts; related to Tenacity decorator typing.
pyrit/common/display_response.py	Normalizes formatting and adds storage IO guards for notebook image display.
pyrit/common/data_url_converter.py	Handles missing file extensions safely.
pyrit/ui/rpc.py	Adds asserts and adjusts `wait_for_score` to return `Optional[Score]`.
pyrit/ui/rpc_client.py	Adds asserts for RPC client internal fields.
pyrit/prompt_target/rpc_client.py	Adds asserts for RPC client internal fields.
pyrit/prompt_target/text_target.py	Changes CSV import default sequencing behavior for `MessagePiece`.
pyrit/prompt_target/prompt_shield_target.py	Adds required-value asserts and defaults API version.
pyrit/prompt_target/websocket_copilot_target.py	Hardens MIME type validation when checking image paths.
pyrit/prompt_target/openai/openai_target.py	Adds `_client` accessor and request_piece asserts; updates typing on helpers.
pyrit/prompt_target/openai/openai_chat_target.py	Uses `_client` for chat completions.
pyrit/prompt_target/openai/openai_completion_target.py	Uses `_client` for completions.
pyrit/prompt_target/openai/openai_image_target.py	Uses `_client` for image generation/edit.
pyrit/prompt_target/openai/openai_response_target.py	Uses `_client` for Responses API calls.
pyrit/prompt_target/openai/openai_tts_target.py	Uses `_client` for TTS calls and removes arg-type ignores.
pyrit/prompt_target/openai/openai_video_target.py	Uses `_client` for video calls; adds assertion for missing text piece.
pyrit/prompt_target/hugging_face/hugging_face_chat_target.py	Removes some ignores, guards Optional model_id usage, and tightens typing.
pyrit/prompt_normalizer/prompt_normalizer.py	Introduces `memory` property and replaces direct `_memory` access; adds return-value ignore.
pyrit/prompt_normalizer/normalizer_request.py	Makes converter configuration lists Optional.
pyrit/prompt_converter/template_segment_converter.py	Makes template parameters handling robust to Optional.
pyrit/prompt_converter/denylist_converter.py	Makes `denylist` optional in signature.
pyrit/prompt_converter/codechameleon_converter.py	Improves Optional check for encrypt_function.
pyrit/prompt_converter/add_text_image_converter.py	Defaults MIME type when detection fails.
pyrit/prompt_converter/add_image_text_converter.py	Defaults MIME type when detection fails.
pyrit/prompt_converter/azure_speech_text_to_audio_converter.py	Avoids returning None for audio path.
pyrit/score/scorer.py	Fixes Optional typing on registry behavior and adds targeted assignment ignores.
pyrit/score/printer/console_scorer_printer.py	Removes `no-any-return` ignores around Fore.* returns.
pyrit/score/human/human_in_the_loop_gradio.py	Asserts score presence after RPC call.
pyrit/score/true_false/self_ask_true_false_scorer.py	Adds assertion after loading YAML.
pyrit/score/true_false/prompt_shield_scorer.py	Ensures boolean parsing returns `list[bool]`.
pyrit/score/float_scale/self_ask_likert_scorer.py	Adds additional Likert scale enum entries.
pyrit/score/float_scale/azure_content_filter_scorer.py	Makes `update_registry_behavior` Optional in annotation.
pyrit/setup/initializers/components/scorers.py	Adds arg-type ignores and adjusts registration typing/formatting.
pyrit/setup/initializers/airt.py	Avoids passing Optional model names by defaulting to empty string.
pyrit/scenario/scenarios/foundry/red_team_agent.py	Avoids passing Optional endpoint into auth helper; adds ignore for kwargs typing.
pyrit/scenario/scenarios/airt/scam.py	Uses fallback empty list for seed_groups typing.
pyrit/scenario/scenarios/airt/psychosocial.py	Uses fallback empty list for seed_groups typing.
pyrit/scenario/scenarios/airt/jailbreak.py	Makes `jailbreak_names` Optional and uses fallback empty list for seed_groups.
pyrit/scenario/scenarios/airt/leakage.py	Avoids passing Optional endpoint into auth helper.
pyrit/scenario/scenarios/airt/cyber.py	Avoids passing Optional endpoint into auth helper.
pyrit/scenario/scenarios/airt/content_harms.py	Avoids passing Optional endpoint into auth helper.
pyrit/models/data_type_serializer.py	Makes internal fields Optional and changes storage IO selection/guards.
pyrit/models/message_piece.py	Adds type ignore when setting `id = None`.
pyrit/models/storage_io.py	Ensures Azure blob read returns `bytes` and normalizes boolean return.
pyrit/models/seeds/seed_prompt.py	Adds ignore on `data_type` assignment typing.
pyrit/models/seeds/seed_dataset.py	Normalizes optional path fields to strings when constructing simulated conversation seeds.
pyrit/memory/memory_interface.py	Makes key attributes Optional; adjusts serialization and schema reflection behavior.
pyrit/memory/memory_models.py	Adds targeted ignores and normalizes sequence defaulting.
pyrit/memory/central_memory.py	Makes singleton memory instance Optional.
pyrit/memory/sqlite_memory.py	Adds engine initialization checks before table operations.
pyrit/memory/azure_sql_memory.py	Adds engine/token guards and optional-expiry handling.
pyrit/backend/routes/media.py	Uses Optional-safe results_path when establishing allowed media root.
pyrit/backend/routes/version.py	Makes engine access optional-safe, but introduces a redundant conditional.
pyrit/cli/frontend_core.py	Tightens typing for dummy `termcolor.cprint`.
pyrit/auth/azure_auth.py	Normalizes token return types and adds ignore for provider callable typing.
pyrit/embedding/openai_text_embedding.py	Removes an unnecessary arg-type ignore for AsyncOpenAI init.
pyrit/executor/workflow/xpia.py	Adds asserts for callback and memory initialization.
pyrit/executor/promptgen/fuzzer/fuzzer.py	Asserts `next_message` is present before building requests.
pyrit/executor/attack/printer/markdown_printer.py	Refactors strategy identifier handling for Optional safety.
pyrit/executor/attack/multi_turn/tree_of_attacks.py	Removes a cast for metadata access and adds a scorer assert.
pyrit/analytics/result_analysis.py	Refactors strategy identifier handling for Optional safety.
pyrit/show_versions.py	Tightens deps_info typing and variable naming.
pyrit/datasets/seed_datasets/remote/vlsu_multimodal_dataset.py	Skips incomplete examples; hardens category fields; adjusts cache path building/guards.
pyrit/datasets/seed_datasets/remote/harmbench_multimodal_dataset.py	Adjusts cache path building/guards for image caching.

Comments suppressed due to low confidence (4)

pyrit/prompt_target/text_target.py:83

When sequence is missing from the CSV, this now forces sequence=0. MessagePiece already has a default (sequence=-1), so using 0 changes ordering semantics (unknown vs first message). Consider omitting the argument when absent or defaulting to -1 to preserve the model's default meaning.
pyrit/setup/initializers/components/scorers.py:157
This _try_register call is split across lines in a way that hurts readability and makes the # type: ignore hard to associate with the correct expression. Consider formatting the call with one argument per line (or parentheses) and, instead of # type: ignore[arg-type], narrow gpt4o after the non-None check (e.g., via cast/assert) so the scorer constructors receive a non-optional PromptChatTarget.
pyrit/backend/routes/media.py:96
allowed_root = Path(memory.results_path or "").resolve() makes the traversal guard ineffective if results_path is unset (empty string resolves to the current working directory). For safety, treat missing/empty results_path as an initialization error and return 500 instead of defaulting to "".

    try:
        memory = CentralMemory.get_memory_instance()
        allowed_root = Path(memory.results_path or "").resolve()
    except Exception as exc:
        raise HTTPException(status_code=500, detail="Memory not initialized; cannot determine results path.") from exc

    # Path traversal guard
    if not requested.is_relative_to(allowed_root):
        raise HTTPException(status_code=403, detail="Access denied: path is outside the allowed results directory.")

pyrit/prompt_target/openai/openai_target.py:503

There are two identical asserts for request_piece in the ContentFilterFinishReasonError handler. This is dead duplication and should be reduced to a single assert (or reuse the request_piece variable created just above).

You can also share your feedback on Copilot code review. Take the survey.

pyrit/prompt_normalizer/prompt_normalizer.py

        # handling empty responses message list and None responses
        if not responses or not any(responses):
-            return None
+            return None  # type: ignore[return-value]



pyrit/datasets/seed_datasets/remote/vlsu_multimodal_dataset.py

@@ -245,8 +247,10 @@ async def _fetch_and_save_image_async(self, image_url: str, group_id: str) -> st
        serializer = data_serializer_factory(category="seed-prompt-entries", data_type="image_path", extension="png")

        # Return existing path if image already exists
-        serializer.value = str(serializer._memory.results_path + serializer.data_sub_directory + f"/{filename}")
+        serializer.value = str((serializer._memory.results_path or "") + serializer.data_sub_directory + f"/{filename}")
        try:
+            assert serializer._memory.results_storage_io is not None
+            assert serializer._memory.results_storage_io is not None
            if await serializer._memory.results_storage_io.path_exists(serializer.value):


pyrit/models/data_type_serializer.py

        file_path = await self.get_data_filename(file_name=output_filename)
+        assert self._memory.results_storage_io is not None, "Storage IO not initialized"
+        assert self._memory.results_storage_io is not None, "Storage IO not initialized"
        await self._memory.results_storage_io.write_file(file_path, data)


pyrit/common/net_utility.py

@@ -10,18 +10,18 @@



pyrit/datasets/seed_datasets/remote/harmbench_multimodal_dataset.py

+        serializer.value = str((serializer._memory.results_path or "") + serializer.data_sub_directory + f"/{filename}")
        try:
+            assert serializer._memory.results_storage_io is not None
+            assert serializer._memory.results_storage_io is not None
            if await serializer._memory.results_storage_io.path_exists(serializer.value):


pyrit/backend/routes/version.py

-        if memory.engine.url.database:
-            db_name = memory.engine.url.database.split("?")[0]
+        if memory.engine is not None and memory.engine.url.database:
+            db_name = memory.engine.url.database.split("?")[0] if memory.engine.url.database else None if memory.engine.url.database else None


pyrit/models/data_type_serializer.py

        ticks = int(time.time() * 1_000_000)
-        results_path = self._memory.results_path
+        results_path = self._memory.results_path or ""
        file_name = file_name if file_name else str(ticks)

        if self._is_azure_storage_url(results_path):
            full_data_directory_path = results_path + self.data_sub_directory
            self._file_path = full_data_directory_path + f"/{file_name}.{self.file_extension}"
        else:
            full_data_directory_path = results_path + self.data_sub_directory
+            assert self._memory.results_storage_io is not None
            await self._memory.results_storage_io.create_directory_if_not_exists(Path(full_data_directory_path))
            self._file_path = Path(full_data_directory_path, f"{file_name}.{self.file_extension}")


pyrit/common/display_response.py

+            assert memory.results_storage_io is not None, "Storage IO not initialized"
+            assert memory.results_storage_io is not None, "Storage IO not initialized"


- Replace duplicate assert statements with RuntimeError raises - Use DB_DATA_PATH as safe fallback instead of empty string - Validate results_path and results_storage_io before use - Simplify redundant nested conditional in version.py

tejas0077 · 2026-03-19T07:37:52Z

Hi @romanlutz, addressed all Copilot review comments:

Replaced duplicate assert statements with RuntimeError raises
(asserts can be stripped under python -O)
Used DB_DATA_PATH as safe fallback instead of empty string in
data_type_serializer.py to prevent root path writes
Validated results_path and results_storage_io before use in
harmbench and vlsu dataset serializers
Simplified redundant nested conditional in version.py

mypy still passes: python -m mypy pyrit/ --strict → no issues found in 422 source files

Also need to investigate the pre-commit failure on macOS -will check that next.

- copilot_authenticator.py: assert username/password not None before page.fill, remove unused type: ignore - _banner.py: rename role variable to char_role to fix type narrowing conflict - audio_transcript_scorer.py: cast bool expression to bool for no-any-return - azure_blob_storage_target.py: assert client not None before get_blob_client - xpia.py: assert response not None before get_value - context_compliance.py: assert response not None before get_value - tree_of_attacks.py: assert response not None before get_piece/get_value - prompt_normalizer.py: change return type to Optional[Message] python -m mypy pyrit/ --strict -> Success: no issues found in 425 source files

tejas0077 · 2026-03-19T08:03:03Z

Hi @romanlutz, I have now fixed all strict mypy errors across the entire pyrit codebase including the cascading errors from the Optional[Message] return type change.
Final files fixed in this update:
pyrit/auth/copilot_authenticator.py - assert username and password not None before page.fill, removed unused type: ignore
pyrit/cli/_banner.py - renamed role variable to char_role to fix type narrowing conflict with loop variable
pyrit/score/audio_transcript_scorer.py - cast bool expression to bool for no-any-return
pyrit/prompt_target/azure_blob_storage_target.py -assert client not None before get_blob_client
pyrit/executor/workflow/xpia.py -assert response not None before get_value
pyrit/executor/attack/single_turn/context_compliance.py - assert response not None before get_value
pyrit/executor/attack/multi_turn/tree_of_attacks.py -assert response not None before get_piece and get_value
pyrit/prompt_normalizer/prompt_normalizer.py -changed return type to Optional[Message]
Final verification:
python -m mypy pyrit/ --strict -Success: no issues found in 425 source files
Ready for review!

romanlutz · 2026-03-19T13:30:43Z

pyrit/auth/copilot_authenticator.py


                logger.info("Waiting for password input...")
                await page.wait_for_selector("#i0118", timeout=self._elements_timeout)
+                assert self._password is not None, "Password is not set"


We have been avoiding asserts in product code. There are a lot of them in this PR. Is there a way to avoid them? I suspect it's done here to rule out certain types. We should at least explain if it's possible for the assertion to get triggered.

tejas0077 · 2026-03-19T13:41:29Z

Good call. I used assert as the quickest way to narrow Optional types for mypy, but you're right that they're unsafe in production since python -O strips them. I'll replace them with proper if x is None: raise ValueError(...) checks throughout. These are all invariant guards that shouldn't trigger in practice, but explicit raises are the right pattern here. Will push shortly

tejas0077 · 2026-03-19T13:48:51Z

Fixed — replaced all 67 assert x is not None guards with explicit if x is None: raise ValueError(...) checks so they're not stripped under python -O. mypy still passes: no issues found in 425 source files.

Tejas Saubhage added 3 commits March 13, 2026 00:19

feat: add ISO 42001 aligned harm definitions for AI supply chain, tra…

336fa28

…nsparency and governance Signed-off-by: Tejas Saubhage <tsaubhage0007@gmail.com>

feat: register ISO 42001 harm scales in LikertScalePaths enum

cbdc28a

Signed-off-by: Tejas Saubhage <tsaubhage0007@gmail.com>

maint: fix untyped decorator mypy error in net_utility.py

f3df706

Added type: ignore[untyped-decorator] for tenacity @Retry decorator which lacks type stubs, resolving strict mypy check failure. Related to Azure#720

romanlutz reviewed Mar 18, 2026

View reviewed changes

pyrit/datasets/harm_definition/ai_governance_failure.yaml Outdated Show resolved Hide resolved

Tejas Saubhage added 2 commits March 18, 2026 13:10

maint: remove stray yaml files accidentally included from another branch

d7362be

romanlutz changed the title ~~MAINT: fix untyped decorator mypy error in net_utility.py~~ MAINT: enable strict mypy checking and fix violations Mar 19, 2026

romanlutz requested a review from Copilot March 19, 2026 05:58

Merge branch 'main' into maint/fix-mypy-type-definitions

429de21

Copilot AI reviewed Mar 19, 2026

View reviewed changes

romanlutz reviewed Mar 19, 2026

View reviewed changes

maint: replace assert guards with explicit if/raise for python -O safety

a229059

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MAINT: enable strict mypy checking and fix violations#1515

MAINT: enable strict mypy checking and fix violations#1515
tejas0077 wants to merge 11 commits intoAzure:mainfrom
tejas0077:maint/fix-mypy-type-definitions

tejas0077 commented Mar 18, 2026 •

edited by romanlutz

Loading

Uh oh!

romanlutz commented Mar 18, 2026

Uh oh!

tejas0077 commented Mar 18, 2026 •

edited

Loading

Uh oh!

tejas0077 commented Mar 18, 2026

Uh oh!

Uh oh!

romanlutz commented Mar 18, 2026

Uh oh!

tejas0077 commented Mar 18, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

tejas0077 commented Mar 19, 2026 •

edited

Loading

Uh oh!

tejas0077 commented Mar 19, 2026 •

edited

Loading

Uh oh!

romanlutz Mar 19, 2026

Uh oh!

tejas0077 commented Mar 19, 2026

Uh oh!

tejas0077 commented Mar 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		assert memory.results_storage_io is not None, "Storage IO not initialized"
		assert memory.results_storage_io is not None, "Storage IO not initialized"

Conversation

tejas0077 commented Mar 18, 2026 • edited by romanlutz Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tests and Documentation

Uh oh!

romanlutz commented Mar 18, 2026

Uh oh!

tejas0077 commented Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tejas0077 commented Mar 18, 2026

Uh oh!

Uh oh!

romanlutz commented Mar 18, 2026

Uh oh!

tejas0077 commented Mar 18, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

tejas0077 commented Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tejas0077 commented Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

romanlutz Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

tejas0077 commented Mar 19, 2026

Uh oh!

tejas0077 commented Mar 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tejas0077 commented Mar 18, 2026 •

edited by romanlutz

Loading

tejas0077 commented Mar 18, 2026 •

edited

Loading

tejas0077 commented Mar 19, 2026 •

edited

Loading

tejas0077 commented Mar 19, 2026 •

edited

Loading