Skip to content

Harden Python NVMe tests to fail gracefully on malformed JSON#27

Draft
Copilot wants to merge 12 commits into
masterfrom
copilot/make-e2e-tests-more-robust
Draft

Harden Python NVMe tests to fail gracefully on malformed JSON#27
Copilot wants to merge 12 commits into
masterfrom
copilot/make-e2e-tests-more-robust

Conversation

Copy link
Copy Markdown

Copilot AI commented Jun 2, 2026

Several tests assumed valid JSON command output and directly indexed nested fields. When output was malformed or schema-shifted, failures were noisy (JSONDecodeError/KeyError/AttributeError) instead of actionable test failures.

  • Centralized JSON parsing + access guards (base test class)

    • Added parse_json_output(...) in tests/nvme_test.py to convert decode/type problems into explicit self.fail(...) messages with command context.
    • Added json_get(..., required=True) to enforce required keys and report missing-field failures clearly.
  • Replaced fragile direct JSON access in common helpers

    • Updated core helpers in tests/nvme_test.py (get_ctrl_id, get_nsid_list, get_max_ns, get_lba_format_size, get_id_*_field_value, namespace list checks) to use guarded parsing/access.
    • Preserved existing functional behavior while making schema errors fail fast with precise diagnostics.
  • Hardened JSON shape checks in format/copy test paths

    • Updated tests/nvme_format_test.py to validate lbafs presence/type and per-entry structure before use.
    • Updated tests/nvme_copy_test.py to validate elbafs presence/type and entry shape before reading pif/iterating.
  • Backward-compatible create-ns output handling

    • Kept support for both JSON and legacy text output in _get_created_nsid(...), with clearer failure text when neither format yields nsid.
json_output = self.parse_json_output(result.stdout, "nvme id-ns")
lbafs = self.json_get(json_output, "lbafs", context="nvme id-ns", required=True)
self.assertIsInstance(lbafs, list, f"ERROR : ... got {type(lbafs).__name__}")

Add two mechanisms to run end-to-end tests against a binary other than
the one built from the current source tree:

1. NVME_BIN environment variable (highest priority):
     NVME_BIN=/usr/bin/nvme python3 tests/tap_runner.py ...

2. -Dnvme-bin meson option (for meson-driven runs):
     meson setup .build -Dnvme-bin=/usr/bin/nvme
     meson test -C .build

The NVME_BIN env var is also explicitly set by meson when running tests,
pointing to the build-tree binary by default (or the -Dnvme-bin value
when specified). This makes the binary selection deterministic regardless
of PATH.

Priority order: NVME_BIN env var > config.json nvme_bin > build-tree binary

Update the README to document all three selection mechanisms.
Copilot AI assigned Copilot and igaw Jun 2, 2026
Copilot AI changed the title tests: support NVME_BIN env var and nvme-bin meson option for e2e tests tests: support external nvme binary via NVME_BIN env var and -Dnvme-bin meson option Jun 2, 2026
Copilot finished work on behalf of igaw June 2, 2026 07:18
Copilot AI requested a review from igaw June 2, 2026 07:18
Copilot AI changed the title tests: support external nvme binary via NVME_BIN env var and -Dnvme-bin meson option tests: tolerate legacy create-ns output across nvme-cli versions Jun 2, 2026
Copilot finished work on behalf of igaw June 2, 2026 07:33
Copilot finished work on behalf of igaw June 2, 2026 08:29
Copilot AI changed the title tests: tolerate legacy create-ns output across nvme-cli versions Harden Python NVMe tests to fail gracefully on malformed JSON Jun 2, 2026
Copilot finished work on behalf of igaw June 2, 2026 08:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants