Test-Suite-Generation-AVM

Modified from the AVMFramework (Alternating Variable Method). Documentation: http://avmframework.org. The repository is cloned into avmf/ directory.

This repository contains the implementation of an analysis of test suite generation based on the AVM framework for the CSI 5186 – AI-enabled Software Verification and Testing course, Assignment 2, Autumn 2025. Report

The project is developed by Fernando Nogueira and Kelvin Mock.

Components

Analytical Tasks

1. Analyze how the Alternating Variable Method (AVM) work, by comparing with existing metaheuristics like Hill Climbing or Simulated Annealing algorithms.
2. Visualize the flow of several example codes which is supposed to be tested with generated test cases provided by the AVM Framework.
   • classify method of the Triangle class
   • in consideration of TriangleBranchTargetObjectiveFunction
3. Provide a test suite for a particular code example (from the AVM reposiotry) by its branch coverage.
   • classify method of the Triangle class.
4. Provide a smallest test suite for a code segment firstly by its statement coverage, then an analysis of its branch coverage thereafter.

   Add (int a, int b) {
   if (b > a) {
   b = b - a
   Print b
   }
   if (a > b) {
   b = a - b
   Print b
   }
   if (a = b) {
   if (a = 0) {
   Print ’0’
   }
   }
   }

5. Provide a test suite for another code example from the AVM repository by its statement coverage, with an analysis of uncovered branches.
   • intersect method of the Line class
6. A comparative analysis on the fitness function - exploring the types of metrics of different candidate test inputs, i.e., approach level and branch distance.
7. Explore the importance of normalizing metrics.

Implementation Tasks

8. Implement and integrate a new local search algorithm into the AVMf test-generation process, by modifying GenerateInputData.java to adapt to our solution, and compare its effectiveness against the default AVM.
9. (Optionally) Export results to a CSV or JSON file for analysis with Python thereafter.

How it looks like

Our Chosen Algorithm!!!

Execution Guides

(Note: Guidelines below were summarized by GitHub Copilot - Claude Sonnet 4 model)

Development Workspace Setup

Because this repository is a multi-project setup containing a cloned repository, we need Test-Suite-Generation-AVM.code-workspace file at the root level of the project such that your IDE recognizes the Java classpath configurations.

Build System: Gradle Wrapper

This project uses the Gradle Wrapper to ensure consistent builds across different environments. The wrapper automatically downloads and uses the correct Gradle version (4.9) specified for this project.

Gradle Wrapper Files

• gradlew: Unix/Linux/macOS script
• gradlew.bat: Windows script
• gradle/wrapper/gradle-wrapper.jar: Wrapper implementation
• gradle/wrapper/gradle-wrapper.properties: Configuration (specifies Gradle 4.9)

Why Use the Gradle Wrapper?

• ✅ Version Consistency: Everyone uses Gradle 4.9 (avoiding compatibility issues)
• ✅ No Installation Required: Automatically downloads Gradle if not present
• ✅ Cross-Platform: Works on Windows, Linux, macOS
• ✅ Reproducible Builds: Same results across environments
• ✅ CI/CD Friendly: Build servers don't need Gradle pre-installed

Recommended Commands

Windows (PowerShell/CMD):

# Build the project
gradlew build

# Run the main program
gradlew runGenerateInputData -PprogramArgs="Triangle 1T"

# Clean build artifacts
gradlew clean

# Run tests
gradlew test

Unix/Linux/macOS:

# Build the project
./gradlew build

# Run the main program
./gradlew runGenerateInputData -PprogramArgs="Triangle 1T"

# Clean build artifacts
./gradlew clean

# Run tests
./gradlew test

💡 Tip: Always use gradlew/./gradlew instead of gradle to ensure you're using the project's intended Gradle version.

Building the Project

1. Navigate to the avmf/ directory:

   cd avmf/

2. Build the project using the Gradle Wrapper (recommended):

   # Windows
   gradlew build
   
   # Unix/Linux/macOS
   ./gradlew build

   Or alternatively with Maven:

   mvn package

Running the GenerateInputData Program

Issue Resolution

If you encounter the error:

Error: Could not find or load main class org.avmframework.examples.GenerateInputData
Caused by: java.lang.ClassNotFoundException: org.avmframework.examples.GenerateInputData

This occurs because:

• The project needs to be built first
• Dependencies (like Apache Commons Math) need to be included in the classpath

Solution

Option 1: Using Gradle Wrapper (Recommended) A custom Gradle task has been added to build.gradle to properly run the GenerateInputData class with all dependencies:

# Windows
gradlew runGenerateInputData -PprogramArgs="testobject branch [search]"

# Unix/Linux/macOS  
./gradlew runGenerateInputData -PprogramArgs="testobject branch [search]"

Option 2: Using Maven The exec-maven-plugin has been configured in pom.xml to run the GenerateInputData class:

mvn exec:java "-Dexec.args=testobject branch [search]"

Note: If you get Java version errors with Maven, the pom.xml has been updated to use Java 8 instead of Java 7.

Usage Examples

Using Gradle:

# Basic usage - Triangle test object with branch 1T
gradlew runGenerateInputData -PprogramArgs="Triangle 1T"

# Different test objects and branches
gradlew runGenerateInputData -PprogramArgs="Line 3T"
gradlew runGenerateInputData -PprogramArgs="Calendar 5F"

# Using different search algorithms
gradlew runGenerateInputData -PprogramArgs="Triangle 1T GeometricSearch"
gradlew runGenerateInputData -PprogramArgs="Line 2F LatticeSearch"
gradlew runGenerateInputData -PprogramArgs="Triangle 3T IteratedPatternSearch"

Using Maven:

# Basic usage - Triangle test object with branch 1T
mvn exec:java "-Dexec.args=Triangle 1T"

# Different test objects and branches
mvn exec:java "-Dexec.args=Line 3T"
mvn exec:java "-Dexec.args=Calendar 5F"

# Using different search algorithms
mvn exec:java "-Dexec.args=Triangle 1T GeometricSearch"
mvn exec:java "-Dexec.args=Line 2F LatticeSearch"
mvn exec:java "-Dexec.args=Triangle 3T IteratedPatternSearch"

Parameters

• testobject: One of Calendar, Line, or Triangle
• branch: Branch ID format X(T|F) where X is a branching node number
  • Calendar: branches 1T/F to 23T/F
  • Line: branches 1T/F to 7T/F
  • Triangle: branches 1T/F to 8T/F
• search (optional): Search algorithm to use
  • IteratedPatternSearch (default)
  • GeometricSearch
  • LatticeSearch

Expected Output from AVMFramework

The program will output:

• Best solution (input values that achieve the target branch)
• Best objective value (Approach Level and Branch Distance)
• Number of objective function evaluations
• Running time

Example output:

Best solution: [326, 205, 112]
Best objective value: Approach Level=0, Branch Distance=0.0
Number of objective function evaluations: 1 (unique: 1)
Running time: 3ms

Setup Python

1. Run uv install to setup uv for the project.
2. Run uv venv to setup a virtual environment for Python.
3. Activate the virtual environment with .venv/Scripts/activate/.
4. Run uv sync to install all necessary dependencies stated in "pyproject.toml" file.

Control Flow Graph

Get "Graphxiz" for plotting control flow graphs, using the following methods:

1. Chocolatey: run choco install graphviz.
2. winget: run winget install graphviz.

Procedures for the Project

1. Run generate_cfg.py to compile a control flow graph for the classify method of the Triangle class.

• You may expect 2 files: PNG and PDF.

2. Under the avmf\ sub-directory, run GenerateInputData.java according to the above guidelines with your chosen build method. Ensure a "results.csv" to be exported with statistics about the outcomes of the program's run.
3. Run parse_algo_results.py to extract insights and comparisons from the CSV file.

• Note: Run uv run -- python parse_algo_results.py with "uv".