CG0562

[SFJohnson24]

This PR resolves the issues currrently reported with CG0562. Mainly: the authors need to filter only the date portion and not datetime for the record count. I added regex handling to this operator to support this and the rule below can be found that implements this. I also added logic for grouping based on a wildcard column to appropriately process the -- in the column name
Datasets.json
Datasets.xlsx
Rule_underscores.json
CORE-Report-2025-12-01T15-16-44.xlsx

This pull request adds support for applying regex transformations to grouping columns in the record_count operation, allowing users to group records based on extracted patterns (such as dates from datetime strings). The changes include updates to the operation logic, schema, documentation, and comprehensive unit tests to ensure correct behavior for regex-based grouping, including support for grouping aliases and filters.

Record Count Operation Enhancements

• Added support for a new regex parameter in the record_count operation, enabling transformation of grouping column values using a regex pattern before grouping. This allows, for example, grouping by just the date portion of a datetime string. (cdisc_rules_engine/operations/record_count.py, cdisc_rules_engine/models/operation_params.py, cdisc_rules_engine/utilities/rule_processor.py, resources/schema/Operations.json) [1] [2] [3] [4] [5]

• Implemented helper methods _get_grouping_for_operations, _get_regex_grouped_counts, and _apply_regex_to_grouping_columns in record_count.py to handle regex transformation and grouping logic robustly, including proper handling of grouping aliases and filters. (cdisc_rules_engine/operations/record_count.py) [1] [2]

Schema and Documentation Updates

• Updated the operations schema (Operations.json) and documentation (Operations.md) to describe the new regex parameter and provide examples of how to use regex-based grouping in YAML operation definitions. (resources/schema/Operations.json, resources/schema/Operations.md) [1] [2] [3]

Unit Test Coverage

• Added extensive unit tests for the new regex grouping feature, covering scenarios with and without grouping aliases, and with filters, ensuring correctness and robustness of the implementation. (tests/unit/test_operations/test_record_count.py)

Codebase Consistency

• Refactored and updated related code to consistently resolve variable names with domain wildcards and ensure correct merging and grouping behavior in the base operation logic. (cdisc_rules_engine/operations/base_operation.py) [1] [2]

These changes collectively make the record_count operation more flexible and powerful for data analysis involving grouped record counts with transformed grouping keys.

[RamilCDISC]

We have all the test cases here with pandasdataset only. Could you please add some cases using DASK.

[SFJohnson24]

added

[RamilCDISC]

The PR adds the regex option for record count operation, to select a specific part of a value to compare with. The PR was validated by:

1. Validating the PR for any unwanted code or comments.
2. Validating the PR logic in context with the AC.
3. Ensuring all the unit and regression testing pass.
4. Ensuring all related testing is updated.
5. Ensuring the updated testing covers cases for both pandas and DASK implementations.
6. Running manual testing using dev editor for positive dataset.
7. Running manual testing using dev editor for negative dataset.
8. Ensuring test cases for the regex matching.