Skip to content

Python package: cdisc-rules-engine and Rules is a raw list #1744

@bkharade

Description

@bkharade

Standard

CDISC SDTM/SDTMIG

Reference Rule ID(s)

All

Conformance Rule ID(s) (if published in CORE)

CORE-

JIRA Ticket

https://jira.cdisc.org/projects/CORERULES/issues/CORERULES-

CLI Command Used (if applicable)

python core.py -s SDTM -v 3.4 -d /path/to/datasets

Rule YAML

NA

Attach any sample test data file(s)

NA

Attach any output report and/or log file(s)

Looking for rules cache in: C:\Users\X\PythonProject\cdisc-rules\cache
Using AE dataset from: C:\Users\X\PythonProject\cdisc-rules\ae.xpt
Successfully loaded rules cache
sdtmig/3-4
Found 1 rules for SDTMIG 3.4
Successfully loaded AE dataset with 6 records
Columns: STUDYID, DOMAIN, USUBJID, AESEQ, AETERM, AEDECOD, AEBODSYS, AESER, AEACN, AEACNOTH, AEREL, AERELNST, AESHOSP, AESLIFE, AECONTRT, AETOXGR, AESTDTC, AEENDTC, AESTDY, AEENDY, AEENRTPT, AEENTPT
[['CORE-000351', 'CORE-000793', 'CORE-000252', 'CORE-000791', 'CORE-000094', 'CORE-000012', 'CORE-000358', 'CORE-000672', 'CORE-000742', 'CORE-000744', 'CORE-000201', 'CORE-000236', 'CORE-000370', 'CORE-000328', 'CORE-000087', 'CORE-000025', 'CORE-000156', 'CORE-000305', 'CORE-000101', 'CORE-000510', 'CORE-000714', 'CORE-000929', 'CORE-000303', 'CORE-000527', 'CORE-000261', 'CORE-000890', 'CORE-000131', 'CORE-000256', 'CORE-000864', 'CORE-000689', 'CORE-000505', 'CORE-000017', 'CORE-000571', 'CORE-000027', 'CORE-000015', 'CORE-000221', 'CORE-000535', 'CORE-000522', 'CORE-000188', 'CORE-000718', 'CORE-000268', 'CORE-000206', 'CORE-000517', 'CORE-000128', 'CORE-000222', 'CORE-000384', 'CORE-000258', 'CORE-000127', 'CORE-000867', 'CORE-000553', 'CORE-001043', 'CORE-000658', 'CORE-001081', 'CORE-000159', 'CORE-000913', 'CORE-000243', 'CORE-000200', 'CORE-000123', 'CORE-000739', 'CORE-000174', 'CORE-000355', 'CORE-000534', 'CORE-000889', 'CORE-000699', 'CORE-000289', 'CORE-000176', 'CORE-000863', 'CORE-000726', 'CORE-000716', 'CORE-000321', 'CORE-000841', 'CORE-000150', 'CORE-000953', 'CORE-000253', 'CORE-000137', 'CORE-000706', 'CORE-000077', 'CORE-000352', 'CORE-000088', 'CORE-000642', 'CORE-000318', 'CORE-000365', 'CORE-000700', 'CORE-000518', 'CORE-000103', 'CORE-000581', 'CORE-000220', 'CORE-000324', 'CORE-000195', 'CORE-000792', 'CORE-000707', 'CORE-000308', 'CORE-000542', 'CORE-000020', 'CORE-000230', 'CORE-000138', 'CORE-000139', 'CORE-000362', 'CORE-000711', 'CORE-000126', 'CORE-000091', 'CORE-000084', 'CORE-000334', 'CORE-000779', 'CORE-000232', 'CORE-000133', 'CORE-000033', 'CORE-000184', 'CORE-000153', 'CORE-000580', 'CORE-000865', 'CORE-000597', 'CORE-000540', 'CORE-000076', 'CORE-000271', 'CORE-000090', 'CORE-000228', 'CORE-000657', 'CORE-000914', 'CORE-000502', 'CORE-000270', 'CORE-000575', 'CORE-000679', 'CORE-000709', 'CORE-000182', 'CORE-000892', 'CORE-000168', 'CORE-000142', 'CORE-000147', 'CORE-000157', 'CORE-000254', 'CORE-000238', 'CORE-000104', 'CORE-000100', 'CORE-000765', 'CORE-000023', 'CORE-000251', 'CORE-000686', 'CORE-000572', 'CORE-000148', 'CORE-000290', 'CORE-000124', 'CORE-000016', 'CORE-000092', 'CORE-000728', 'CORE-000080', 'CORE-000761', 'CORE-000218', 'CORE-000901', 'CORE-000298', 'CORE-000539', 'CORE-000746', 'CORE-000146', 'CORE-000085', 'CORE-001078', 'CORE-000018', 'CORE-000006', 'CORE-000004', 'CORE-000096', 'CORE-000179', 'CORE-000117', 'CORE-000643', 'CORE-000529', 'CORE-000135', 'CORE-000129', 'CORE-000242', 'CORE-000118', 'CORE-000866', 'CORE-000202', 'CORE-000207', 'CORE-000122', 'CORE-000106', 'CORE-000198', 'CORE-000538', 'CORE-000234', 'CORE-000741', 'CORE-000108', 'CORE-000787', 'CORE-000007', 'CORE-000097', 'CORE-000010', 'CORE-000745', 'CORE-000130', 'CORE-000181', 'CORE-000541', 'CORE-000267', 'CORE-000260', 'CORE-000598', 'CORE-000354', 'CORE-000028', 'CORE-000862', 'CORE-000177', 'CORE-000014', 'CORE-000266', 'CORE-000219', 'CORE-000187', 'CORE-000021', 'CORE-000026', 'CORE-000079', 'CORE-000235', 'CORE-000203', 'CORE-000264', 'CORE-000723', 'CORE-000136', 'CORE-000337', 'CORE-000024', 'CORE-001080', 'CORE-000102', 'CORE-000310', 'CORE-000291', 'CORE-000185', 'CORE-000225', 'CORE-000093', 'CORE-000720', 'CORE-000240', 'CORE-000743', 'CORE-000778', 'CORE-000237', 'CORE-000152', 'CORE-000231', 'CORE-000457', 'CORE-000269', 'CORE-000659', 'CORE-000086', 'CORE-000701', 'CORE-000244', 'CORE-000747', 'CORE-000041', 'CORE-000248', 'CORE-000785', 'CORE-000149', 'CORE-000656', 'CORE-000582', 'CORE-000484', 'CORE-000160', 'CORE-000196', 'CORE-000095', 'CORE-000197', 'CORE-000001', 'CORE-000217', 'CORE-000173', 'CORE-000005', 'CORE-000292', 'CORE-001082', 'CORE-000705', 'CORE-000226', 'CORE-000143', 'CORE-000852', 'CORE-000550', 'CORE-000710', 'CORE-000579', 'CORE-000259', 'CORE-000786', 'CORE-000376', 'CORE-000009', 'CORE-000361', 'CORE-000089', 'CORE-000293', 'CORE-000144', 'CORE-000655', 'CORE-000099', 'CORE-000916', 'CORE-000167', 'CORE-000774', 'CORE-000729', 'CORE-000178', 'CORE-000141', 'CORE-000262', 'CORE-000098', 'CORE-000158', 'CORE-000155', 'CORE-000002', 'CORE-000952', 'CORE-000183', 'CORE-000685', 'CORE-000145', 'CORE-000853', 'CORE-000229', 'CORE-000132', 'CORE-000784', 'CORE-000719', 'CORE-000013', 'CORE-000151', 'CORE-000708', 'CORE-000169', 'CORE-000241', 'CORE-000011', 'CORE-000008', 'CORE-000594', 'CORE-000357', 'CORE-000165', 'CORE-000175', 'CORE-000164', 'CORE-000081', 'CORE-000299', 'CORE-000302', 'CORE-000134', 'CORE-000356', 'CORE-000163', 'CORE-000161', 'CORE-000616', 'CORE-000078', 'CORE-000776', 'CORE-000294', 'CORE-000199', 'CORE-000125', 'CORE-000233', 'CORE-000783', 'CORE-000034', 'CORE-000544', 'CORE-000250', 'CORE-000363', 'CORE-000172', 'CORE-000227', 'CORE-000075', 'CORE-000712', 'CORE-000239', 'CORE-000030', 'CORE-000249', 'CORE-000763', 'CORE-000003', 'CORE-000204', 'CORE-000272', 'CORE-000732', 'CORE-000140', 'CORE-000154', 'CORE-000186', 'CORE-000552', 'CORE-000019', 'CORE-000717', 'CORE-000022', 'CORE-000171', 'CORE-000162', 'CORE-000180', 'CORE-000105', 'CORE-000846', 'CORE-000193', 'CORE-000374', 'CORE-000224', 'CORE-000713', 'CORE-000210', 'CORE-000111', 'CORE-000121', 'CORE-000297', 'CORE-000083', 'CORE-000192', 'CORE-000213', 'CORE-000114', 'CORE-000885', 'CORE-000216', 'CORE-000113', 'CORE-000777', 'CORE-000223', 'CORE-000073', 'CORE-000057', 'CORE-000208', 'CORE-000886', 'CORE-000045', 'CORE-000191', 'CORE-000072', 'CORE-001034', 'CORE-000046', 'CORE-000758', 'CORE-000247', 'CORE-000736', 'CORE-000295', 'CORE-000109', 'CORE-000120', 'CORE-000115', 'CORE-000112', 'CORE-000296', 'CORE-000215', 'CORE-000119', 'CORE-000212', 'CORE-000845', 'CORE-000170', 'CORE-000915', 'CORE-000927', 'CORE-000049', 'CORE-000074', 'CORE-000211', 'CORE-000107', 'CORE-000214', 'CORE-000110', 'CORE-000844', 'CORE-000209', 'CORE-000246', 'CORE-000047', 'CORE-000082', 'CORE-000245', 'CORE-000760', 'CORE-000050', 'CORE-000031', 'CORE-000037', 'CORE-000043', 'CORE-000070', 'CORE-000068', 'CORE-000055', 'CORE-000065', 'CORE-000054', 'CORE-000032', 'CORE-000766', 'CORE-000029', 'CORE-000036', 'CORE-000042', 'CORE-000056', 'CORE-000038', 'CORE-000064', 'CORE-000116', 'CORE-000069', 'CORE-000071', 'CORE-000051', 'CORE-000039', 'CORE-000040', 'CORE-000066', 'CORE-000035', 'CORE-000044', 'CORE-000190', 'CORE-000060', 'CORE-000052', 'CORE-000189', 'CORE-000166', 'CORE-000061', 'CORE-000767', 'CORE-000067', 'CORE-000048', 'CORE-000880', 'CORE-000058', 'CORE-000757', 'CORE-000059']]
Found 1 rules applicable to AE domain
Error: 'list' object has no attribute 'get'
Traceback (most recent call last):
File "C:\Users\X\PythonProject\cdisc-rules\test-cdisc.py", line 88, in main
print(f"\nProcessing rule {idx+1}/{len(ae_rules)}: {rule.get('core_id')} - {rule.get('description')[:80]}...")
^^^^^^^^
AttributeError: 'list' object has no attribute 'get'

Process finished with exit code 0

Expected output

rules should be a list of dictionaries with the following keys:

core_id (e.g. "CORE-000252")
domains (e.g. {'Include': ['DM'], 'Exclude': []} or {'Include': ['ALL']})
author
reference
sensitivity
executability
description
authorities
standards
classes
rule_type
conditions
actions
datasets
output_variables

Any Additional Information

I have installed cdisc-rules-engine using pip install cdisc-rules-engine on a windows machine and in addition to installing the library, I have also downloaded the rules cache (found in the resources/cache folder of this repository) and stored them in my project.

I tried to execute the steps provided at https://pypi.org/project/cdisc-rules-engine/#description

Step 1: Load the Rules
The rules can be loaded into an in-memory cache by doing the following:

import os
import pathlib
import pickle

from multiprocessing.managers import SyncManager
from cdisc_rules_engine.services.cache import InMemoryCacheService

class CacheManager(SyncManager):
pass

If you're working from a terminal you may need to
use SyncManager directly rather than define CacheManager
CacheManager.register("InMemoryCacheService", InMemoryCacheService)

def load_rules_cache(path_to_rules_cache):
cache_path = pathlib.Path(path_to_rules_cache)
manager = CacheManager()
manager.start()
cache = manager.InMemoryCacheService()

files = next(os.walk(cache_path), (None, None, []))[2]

for fname in files:
    with open(cache_path / fname, "rb") as f:
        cache.add_all(pickle.load(f))

return cache

Rules in this cache can also be accessed by standard and version using the get_rules_cache_key function.

from cdisc_rules_engine.utilities.utils import get_rules_cache_key

cache = load_rules_cache("path/to/rules/cache")

Note that the standard version is separated by a dash, not a period
cache_key_prefix = get_rules_cache_key("sdtmig", "3-4")
rules = cache.get_all_by_prefix(cache_key_prefix)
print(rules)

I was expecting rules will be a list of dictionaries as per documentation but I receive [['CORE-000351', 'CORE-000793', 'CORE-000252', 'CORE-000791']] (big list just shortened for discussion).

Can someone help me here if I am missing any settings.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No fields configured for Bug.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions