[Feature Request] Specify and implement redacted reports

[mbuechse]

Applicants for SCS-0004 integrator certification have to prove that they have set up / have been operating compliant cloud environments. These environments can be quite sensitive, and their owners could have reasonable reservations regarding the sharing of our report files, because they might contain sensitive information.

• (loosely) specify what MUST, SHOULD, MAY, MUST NOT to be contained in redacted report vs. a full report
• similarly for Tempest!
• add a command-line switch to scs-compliance-test.py that requests for the report to be redacted
• add a similar option to scs-test-runner.py (probably as a subject-specific option in config.toml)
• provide an official means of redacting tempest logs/output

[mbuechse]

I would even go so far as to structurally split the report into a non-sensitive section and a sensitive section. And I would change the overall format of the report to be more testcase-centric, which means that we should upgrade the current testsuite to provide non-sensitive testcase-specific log output.

[mbuechse]

I think if a testcase needs to add messages to the report besides the result (PASS/FAIL/ABORT), then these messages belong in stdout (not stderr). It is part of the normal operation, the normal output of the testcase.

The main difficulty here is programming-wise: we have many testcases that are implemented by the same function with some parameter (e.g., test presence of some image, test presence of some flavor, test presence of some service). The function doesn't know the name of the testcase. So either this name has to be supplied to the function as a parameter, or the function cannot directly print to stdout. Either way, the general type of a testcase function becomes more complicated...

[mbuechse]

Well, it is actually not too hard if we leverage the power of dynamic typing in Python.

$ Tests/iaas/openstack_test.py -c scs2 scs-0102-prop-os_purpose 2> /dev/null
scs-0102-prop-os_purpose: property os_purpose not set or not correct for image(s): Ubuntu Minimal 24.04, Ubuntu Minimal 22.04, Debian 13, Debian 12, AlmaLinux 10, Ubuntu 24.04, Ubuntu 22.04, Debian 11, Ubuntu Minimal 20.04, Ubuntu 20.04, AlmaLinux 9, AlmaLinux 8, PSKE Ubuntu 22.04 20241117, PSKE Ubuntu 22.04 20241002, Flatcar Container Linux 3975.2.2, CirrOS 0.6.1, PSKE Ubuntu 22.04 20240701, Flatcar Container Linux 3815.2.5, Fedora 37
scs-0102-prop-os_purpose: FAIL

So now we have two lines beginning with scs-0102-prop-os_purpose: , one containing a message and the other one containing the result. It wouldn't be too hard now for the main test script to parse this stdout, and to associate the message with the testcase.

I'm also just realizing that the testcases with the parametric functions I mentioned above (presence of some flavor/image/service) don't even need to output any message, because the testcases are so specific. If the testcase that checks for presence of flavor SCS-4V-16 fails, then that flavor is missing, and what else is there to know?

[mbuechse]

@fkr When one of the image-metadata testcases fails, it produces a list of all public images that need correcting for the testcase to pass (for instance: "property X missing in images Y, Z"). Would this list (Y, Z) already be sensitive information? I suppose it could be so construed?

[fkr]

@fkr When one of the image-metadata testcases fails, it produces a list of all public images that need correcting for the testcase to pass (for instance: "property X missing in images Y, Z"). Would this list (Y, Z) already be sensitive information? I suppose it could be so construed?

yes, those could be images who leak infos due to their naming.

[mbuechse]

@fkr I suspected as much. I guess we are on the safe side if the redacted report is merely a list of passed testcases (plus checked_at date and subject moniker). Either the list is complete, then fine, or it isn't, but then we don't need to know if the missing testcases have failed, aborted, or not even tried.