Skip to content

Differential testing#132

Open
mave9159 wants to merge 91 commits into
protocol-fuzzing:mainfrom
mave9159:differential-testing
Open

Differential testing#132
mave9159 wants to merge 91 commits into
protocol-fuzzing:mainfrom
mave9159:differential-testing

Conversation

@mave9159
Copy link
Copy Markdown

Overview

This PR implements a differential oracle for comparing two Mealy machine models, and a report generator for the results.

New classes

  • DifferentialOracle: uses an BFS algorithm over the product state space to find divergences between two Mealy machine models. Returns a list of DivergenceRecord instances.
  • DivergenceRecord: contains a witness sequence and the two differing outputs, for a single divergence.
  • DifferentialReport: generates a test file and a report file from the found divergences.

Tests

  • DifferentialOracleTest: contains unit tests for the core BFS algorithm.
  • DifferentialReportTest: contains unit tests for the report writer.

Test resources

16 .dot files used for the unit tests. Some are basic hand-crafted models and some are taken from the dtls-fuzzer repo (https://github.com/assist-project/dtls-fuzzer/tree/master/experiments/results/clients).

mave9159 added 30 commits April 9, 2026 14:44
Copy link
Copy Markdown
Member

@actyp actyp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for making the changes so quickly!
They are looking good overall, but we need to focus on the alphabet handling.

I think we should follow the way that TestRunner is implemented. If we want to be totally symmetric we can do the following:

  1. [like TestRunner] Create a DiffTester to be our general diff testing interface

  2. [like TestRunnerEnabler] Create a DiffTesterEnabler to provide the necessary configs (LearnerConfig, DiffTesterConfig) to the DiffTesterStandard

  3. [like TestRunnerStandard] Rename the current DiffTester to DiffTesterStandard that implements the interface.

The DiffTesterStandard constructor should take the DiffTesterEnabler and some AlphabetBuilder as inputs.

Since we need an Alphabet<String> for our use case, I think this presents a nice opportunity to use the AlphabetBuilderTransformer (PSF class) as the argument constructor possibly like this:

constructor(..., AlphabetBuilderTransformer<I, String> ...)...

This way the user should provide the implementation of the transformation from their symbols to String.

  1. [like TestRunnerBuilder] We also need to create a DiffTesterBuilder and add it as an argument to the CommandLineParser constructor.

Let me know if you have any questions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants