[Feature request] Conformance Statement Overview and Details - Ability to show latest result instead of latest execution

[olw-medcom]

Hey there,

Currently it seems that when I click "Manage tests" on an organization and view their conformance statements, I see the latest execution and not the latest results.
As an example, these two sessions on the same test case

Shows like this on the conformance overview/details:

As you can see, since their run the test case has actually changed name and the conformance statement shows the latest execution, and not the latest failure.

It would be nice to have some sort of toggle both on the conformance statement overview page and on the details page to see the latest success or failure run instead of latest execution run.

Furthermore if the title of the test suite / test case also changed accordingly to the latest result run, that would be awesome. That way we can quickly glance how far along an organizaiton is in their conformance.

All this is currently possible by going into the Session Dashboard of course, but it's just a tad troublesome.

[olw-medcom]

Something like this (just to give an idea)

[costas80]

Hmm, I'm not sure I get this fully. Let me elaborate.

As you can see, since their run the test case has actually changed name and the conformance statement shows the latest execution, and not the latest failure.

The conformance statements indeed always show the latest run. If you failed in a previous test session and then executed another test and stopped it in the middle (or if the admin terminated it) then the status of the test for the statement is incomplete. The previous failure in this case is historical really and of no consequence to the statement's status.

In your subsequent screenshots where you add a "Latest run" option, this is already the case and applies everywhere. I think what you're going for is a way to ignore test sessions that are finished as incomplete? Note that it's not pending/active test sessions that get listed in the statement, its completed test sessions that did not produce a definitive result (either stopped by the user, terminated by you, or existed via exit step with an explicit incomplete status).

Furthermore if the title of the test suite / test case also changed accordingly to the latest result run, that would be awesome. That way we can quickly glance how far along an organizaiton is in their conformance.

I don't think I understand this point either. When a test case/suite changes name you will always see the latest name in the conformance statement page. The session dashboard always shows the names as they were at the time of execution (i.e., this is a snapshot if you may). The same applies if anything else changed name as well (specification, organisation etc.).

[costas80]

Ok, checking your issue title again ...

Conformance Statement Overview and Details - Ability to show latest result instead of latest execution

... I think the misunderstanding is that an "incomplete" test session is in fact a result (results are not only failures and successes). In other words as long as a test session was executed and completed it is considered to be a result towards the conformance status, regardless of whether a definitive outcome (success or failure) was produced.

[olw-medcom]

Right, that's exactly it. We would like to see the latest outcome, and not incompletions.

Furthermore

I don't think I understand this point either. When a test case/suite changes name you will always see the latest name in the conformance statement page. The session dashboard always shows the names as they were at the time of execution (i.e., this is a snapshot if you may). The same applies if anything else changed name as well (specification, organisation etc.).

If you look at the two first pictures I posted, you can see that the test case was just called "MSH Send SBDH Acknowledgement Test" whereas the picture below the first one, shows the latest name which "v1.1 - MSH Send SBDH Acknowledgement Test".

[costas80]

Right, that's exactly it. We would like to see the latest outcome, and not incompletions.

This is not something we could do on the statement detail page. Here we only record the latest completed session related to the test case in question, and the conformance status is calculated to take that into account. This concept is rooted into the conformance calculation so it would be very difficult to make this a sort of preference (e.g. "ignore incomplete tests for the conformance status").

Having said this however, maybe I could propose a half-way solution that is "good enough" for your use case. Currently each test case in the conformance statement provides options including "View test session":

We could provide an additional option here as "View all test sessions" that would take you to the session dashboard and show you all test sessions related to the relevant test case. Like this you would directly see the session history for the test case in question without needing to apply specific filters. Would this work for you?

The thing is that I'm reluctant to show that the conformance statement somehow has a toggle to ignore completed test sessions with an inconclusive result. This is not the case, and adding a UI shortcut for this could lead to follow-up impressions (not necessarily by you :)) that you can configure how the conformance status is calculated (taking into account such sessions or not).

If you look at the two first pictures I posted, you can see that the test case was just called "MSH Send SBDH Acknowledgement Test" whereas the picture below the first one, shows the latest name which "v1.1 - MSH Send SBDH Acknowledgement Test".

Yes I see. So if I follow correctly, your idea is that if we would show the previous completed/failed test case on the conformance statement detail page, the test case name would be the one from when the test was executed. I see your point, but I'm similarly reluctant on this. The conformance statement page always shows the latest information about everything. If we were to show the test case name at the time of execution here, then this would "magically" change if you run a new test session. I'm thinking that maybe an alternative here is to indicate that the test case had a different name when executed (something like a tooltip). What do you think?

On this last point, keep in mind as well that not all changes are things you would want to highlight so much. I know several cases where test developers have fixed simple typos in names, and would not want to highlight this so much. I would definitely want to avoid some sort of metadata update options where you choose whether you want to highlight changes in names or whether these should be applied "silently".

[olw-medcom]

We could provide an additional option here as "View all test sessions" that would take you to the session dashboard and show you all test sessions related to the relevant test case

I think this would be a good meet-you-in-the-middle solution! I would enojy being able to show the team this functionality to gain a quick glance at how an organization is performing 😊

I would definitely want to avoid some sort of metadata update options where you choose whether you want to highlight changes in names or whether these should be applied "silently".

I understand, we can work around it, we just need to be aware of it and utilize the "reset test case history" more often.

[costas80]

I think this would be a good meet-you-in-the-middle solution! I would enojy being able to show the team this functionality to gain a quick glance at how an organization is performing 😊

Deal! So from this I'm keeping the feature of a shortcut option in the conformance statement page to take you to the session dashboard and show you all tests related to the specific test case. As usual I'll let you know when this is available to pull.

[olw-medcom]

Thank you @costas80, as always! I appreciate our correspondence ^_^

[costas80]

The conformance statement detail page now includes an update that changes the previous View test session option for a test case to View test sessions:

This option will now take you to the session dashboard to view the latest test session (as was previously the case), but will also list the other test sessions related to this test case.

I trust this covers you @olw-medcom no?