MX-17306 Cross shard analysis #1
Conversation
There was a problem hiding this comment.
Pull request overview
This PR introduces cross-shard analysis functionality for MultiversX blockchain logs. The implementation provides a framework for parsing node logs and analyzing cross-shard miniblock execution to validate proper ordering without gaps or duplications.
Key changes include:
- Implementation of a log parsing framework using the Aho-Corasick algorithm for efficient pattern matching
- Cross-shard analysis tools for tracking miniblocks across shards and epochs
- PDF report generation for timeline visualization of miniblock and header data
Reviewed changes
Copilot reviewed 24 out of 28 changed files in this pull request and generated 46 comments.
Show a summary per file
| File | Description |
|---|---|
| requirements.txt | Adds pyahocorasick dependency for pattern matching |
| requirements-dev.txt | Adds development dependencies (autopep8, flake8, pytest, pyright) |
| pyrightconfig.json | Configures Pyright type checker settings |
| .pre-commit-config.yaml | Sets up pre-commit hooks for code quality |
| .flake8 | Configures flake8 linter to ignore line length and bare except warnings |
| .gitignore | Excludes JSON and PDF output files from version control |
| multiversx_logs_parser_tools/aho_corasik_parser.py | Implements Aho-Corasick pattern matching for log parsing |
| multiversx_logs_parser_tools/entry_parser.py | Parses individual log entries into structured components |
| multiversx_logs_parser_tools/node_logs_checker.py | Provides node-level log processing framework |
| multiversx_logs_parser_tools/archive_handler.py | Handles processing of zipped log archives |
| multiversx_logs_parser_tools/helpers.py | Provides utility functions for path validation and dict manipulation |
| multiversx_cross_shard_analysis/constants.py | Defines enums, mappings, and color schemes for miniblock states |
| multiversx_cross_shard_analysis/decode_reserved.py | Decodes protobuf-encoded reserved fields from miniblock headers |
| multiversx_cross_shard_analysis/header_structures.py | Structures for managing and organizing blockchain header data |
| multiversx_cross_shard_analysis/miniblock_data.py | Processes miniblock data and generates report structures |
| multiversx_cross_shard_analysis/header_analysis_parser.py | Parses header-related log entries |
| multiversx_cross_shard_analysis/header_analysis_checker.py | Performs header analysis checks on node logs |
| multiversx_cross_shard_analysis/header_analysis_archive_handler.py | Orchestrates header analysis across archived logs |
| multiversx_cross_shard_analysis/gather_data.py | Main entry point for data gathering and report generation |
| multiversx_cross_shard_analysis/miniblocks_timeline_report.py | Generates PDF timeline reports for miniblocks |
| multiversx_cross_shard_analysis/miniblocks_round_report.py | Generates PDF reports showing miniblocks per round by shard |
| multiversx_cross_shard_analysis/headers_timeline_report.py | Generates PDF timeline reports for headers by nonce |
| multiversx_cross_shard_analysis/test_miniblocks.py | Test cases for miniblock processing functionality |
| multiversx_cross_shard_analysis/test_decode_reserved.py | Test cases for reserved field decoding |
| .vscode/ltex.dictionary.en-US.txt | Custom dictionary entries for spell checking |
| README.md | Documentation for installation and usage |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| for header in node_data.header_dictionary[header_status]: | ||
| shard_id = get_shard_id(header) | ||
| added = False | ||
| if header_status == 'commited_headers': |
There was a problem hiding this comment.
The word "commited" is a misspelling. The correct spelling is "committed" (with double 't').
| header_data.add_proposed_header(header_exec_result) | ||
| shard_data = ShardData() | ||
| shard_data.add_node(header_data) | ||
| assert shard_data.parsed_headers[0].header_dictionary['commited_headers'][0] == header_exec_result |
There was a problem hiding this comment.
The word "commited" is a misspelling. The correct spelling is "committed" (with double 't').
|
|
||
| mentioned_headers = { | ||
| "origin_shard_proposed_headers": "20ec12", | ||
| "origin_shard_commited_headers": "20ec12", |
There was a problem hiding this comment.
The word "commited" is a misspelling. The correct spelling is "committed" (with double 't').
| from .aho_corasik_parser import AhoCorasickParser | ||
| except Exception: | ||
| # Fallback when running the script directly (not as a package) | ||
| from aho_corasik_parser import AhoCorasickParser |
There was a problem hiding this comment.
The import uses "aho_corasik_parser" which is a misspelling. The correct spelling is "aho_corasick_parser" to match the Aho-Corasick algorithm name.
| from .aho_corasik_parser import AhoCorasickParser | |
| except Exception: | |
| # Fallback when running the script directly (not as a package) | |
| from aho_corasik_parser import AhoCorasickParser | |
| from .aho_corasick_parser import AhoCorasickParser | |
| except Exception: | |
| # Fallback when running the script directly (not as a package) | |
| from aho_corasick_parser import AhoCorasickParser |
| def reset(self): | ||
| self.header_dictionary = { | ||
| 'proposed_headers': [], | ||
| 'commited_headers': [] |
There was a problem hiding this comment.
The word "commited" is a misspelling. The correct spelling is "committed" (with double 't').
| def get_color_for_state(self, mention_type: str, tx_count: int, header: dict[str, Any]) -> Colors: | ||
| reserved = header.get('reserved', {}) | ||
| if reserved == {}: | ||
| reserved = get_default_decoded_data(tx_count=tx_count) |
There was a problem hiding this comment.
Variable reserved is not used.
| @@ -0,0 +1,187 @@ | |||
| from enum import Enum | |||
There was a problem hiding this comment.
Import of 'Enum' is not used.
| from enum import Enum |
|
|
||
| class AhoCorasickParser(ABC): | ||
| def __init__(self): | ||
| self.initialize_checker() |
There was a problem hiding this comment.
This call to AhoCorasickParser.initialize_checker in an initialization method is overridden by HeaderAnalysisParser.initialize_checker.
| self.initialize_checker() |
| class NodeLogsChecker(Generic[P]): | ||
| def __init__(self, parser_cls: Type[P], args: argparse.Namespace): | ||
| self.parser: P = parser_cls() | ||
| self.initialize_checker(args) |
There was a problem hiding this comment.
This call to NodeLogsChecker.initialize_checker in an initialization method is overridden by HeaderAnalysisChecker.initialize_checker.
No description provided.