Creating macros to compare GNU output to UUTILS output #9499
Description
When implementing the integration tests for the stty saved state parser feature I added integration tests using the expected_result macro that is similar to the ucmd macro but runs it on the GNU binaries.
These were incredibly useful because they surfaces three different issues that I would not have been aware of including:
- Some platforms have different numbers of control flags
- When an invalid argument or other error message comes up the --help suggestion is printed.
It would be great to add as many of these comparisons as possible to the integration tests to provide more visibility into the differences. We could help facilitate that by having a helper function that you provide the command to and as part of the macro it runs both the GNU version and the UUTILS version and validates that the stdout, stderr and exit code match.
One issue that is blocking our usage of the GNU comparisons for the exact error messaging is that the help message in the uutils implementation is:
Try '/path/to/stty --help' instead of Try 'stty --help' or on the MacOS CI runner its Try 'gstty --help
I'm not entirely sure if thats intended since its actually useful to have the full path of the binary thats being run.
I made a helper function to account for this difference but it would be good to have this as a shared macro that all of the other integration tests can use and start adding it to as many integration tests as we can to surface differences between the two implementations. This is a great area for a new contributor to get involved in since its only additive to the existing testing and it shouldn't change any existing testing behaviour just refactoring it to make it easier to use and read.