Refactor integration tests for multiple discovery modes

[bburda]

Summary

Refactor integration test suite to support running the same tests across different discovery modes (runtime, manifest-only, hybrid). Currently we have separate test files for different modes which leads to code duplication and inconsistent coverage.

---

Proposed solution

1. Create a parameterized test base class that can run against different discovery configurations:

   class DiscoveryModeTestBase(unittest.TestCase):
       discovery_mode = None  # Set by subclass or pytest parametrize
       expected_capabilities = {}  # What features are available in this mode

2. Define capability flags per discovery mode:

   DISCOVERY_CAPABILITIES = {
       "runtime": {
           "dynamic_nodes": True,
           "configurations": True,  # Requires running nodes
           "operations": True,
           "faults": True,
           "data_read": True,
           "data_write": True,
       },
       "manifest_only": {
           "dynamic_nodes": False,
           "configurations": False,  # No running nodes
           "operations": False,      # Can list, but not execute
           "faults": False,
           "data_read": False,
           "data_write": False,
       },
       "hybrid": {
           # Manifest structure + runtime data where available
       }
   }

3. Use skip decorators based on capabilities:

   @requires_capability("configurations")
   def test_get_configuration_value(self):
       # Only runs if discovery mode supports configurations
       ...
   
   @requires_capability("operations")
   def test_execute_operation(self):
       # Only runs if discovery mode supports operation execution
       ...

4. Consolidate test files:

   • Merge test_integration.test.py and test_discovery_manifest.test.py
   • Use pytest parametrize or separate launch files per mode
   • Single source of truth for API contract tests

5. Launch file structure:

   test/
     test_api.test.py           # Unified test file
     launch/
       test_runtime.launch.py   # Runtime discovery mode
       test_manifest.launch.py  # Manifest-only mode
       test_hybrid.launch.py    # Hybrid mode
   

---

Additional context

• Current pain points:

  • test_integration.test.py focuses on runtime mode
  • test_discovery_manifest.test.py duplicates many tests for manifest mode
  • Inconsistent expected status codes between modes

• Benefits:

  • Single set of API contract tests
  • Clear documentation of what works in each mode
  • Easier to add new discovery modes
  • Reduced maintenance burden

• Implementation notes:

  • Consider using pytest-launch-testing for better parametrization
  • Each test should document which capability it requires
  • Tests that work in all modes should not require any capabilities