Differential testing#132
Conversation
…w in correct order
…ed output equivalence
actyp
left a comment
There was a problem hiding this comment.
Thank you for making the changes so quickly!
They are looking good overall, but we need to focus on the alphabet handling.
I think we should follow the way that TestRunner is implemented. If we want to be totally symmetric we can do the following:
-
[like
TestRunner] Create aDiffTesterto be our general diff testing interface -
[like
TestRunnerEnabler] Create aDiffTesterEnablerto provide the necessary configs (LearnerConfig, DiffTesterConfig) to theDiffTesterStandard -
[like
TestRunnerStandard] Rename the currentDiffTestertoDiffTesterStandardthat implements the interface.
The DiffTesterStandard constructor should take the DiffTesterEnabler and some AlphabetBuilder as inputs.
Since we need an Alphabet<String> for our use case, I think this presents a nice opportunity to use the AlphabetBuilderTransformer (PSF class) as the argument constructor possibly like this:
constructor(..., AlphabetBuilderTransformer<I, String> ...)...
This way the user should provide the implementation of the transformation from their symbols to String.
- [like
TestRunnerBuilder] We also need to create aDiffTesterBuilderand add it as an argument to the CommandLineParser constructor.
Let me know if you have any questions.
Overview
This PR implements a differential oracle for comparing two Mealy machine models, and a report generator for the results.
New classes
DifferentialOracle: uses an BFS algorithm over the product state space to find divergences between two Mealy machine models. Returns a list ofDivergenceRecordinstances.DivergenceRecord: contains a witness sequence and the two differing outputs, for a single divergence.DifferentialReport: generates a test file and a report file from the found divergences.Tests
DifferentialOracleTest: contains unit tests for the core BFS algorithm.DifferentialReportTest: contains unit tests for the report writer.Test resources
16 .dot files used for the unit tests. Some are basic hand-crafted models and some are taken from the dtls-fuzzer repo (https://github.com/assist-project/dtls-fuzzer/tree/master/experiments/results/clients).