Fix sp_nrt_ldf_dimensional_data_postprocessing: stop early-RETURN guard from misclassifying filtered ldf_uids, harmonize INNER→LEFT JOIN#839
Open
nullflux wants to merge 2 commits into
Open
Conversation
…LDF_DIMENSIONAL_DATA
The early-RETURN guard at lines 136-158 computed @backfill_list against
#LDF_META_DATA, conflating data_type-whitelist filtering with truly missing
NRT metadata rows. Any ldf_uid with data_type NOT IN ('ST','CV','LIST_ST')
caused the entire batch to abort. Guard now checks
nrt_odse_state_defined_field_metadata directly so it only fires for genuinely
missing source rows. Also harmonized the #LDF_DATA step (line 648) to use
LEFT JOIN on nrt_srte_LDF_PAGE_SET, matching the #LDF_META_DATA step at
line 108; nrt_ldf_data.ldf_page_id can legitimately be NULL and the prior
INNER JOIN silently dropped those rows.
DataDrivenUnitTests-compatible test that seeds nrt_ldf_data with three ldf_uids - one SUB (data_type whitelist filtered) plus a CV and ST - then EXECs sp_nrt_ldf_dimensional_data_postprocessing and asserts LDF_DIMENSIONAL_DATA receives the two valid rows. Pre-fix, the SUB row triggered the early-RETURN guard and LDF_DIMENSIONAL_DATA stayed empty. Uses an isolated 97xxxxxx UID namespace to avoid colliding with fixture or baseline rows.
nullflux
added a commit
that referenced
this pull request
May 20, 2026
nullflux
added a commit
that referenced
this pull request
May 20, 2026
…PR routines in orchestrator Two changes: 1. Extend bug #5b's patient_id-NULL fix to the 10 multi-condition nrt_investigation variants (22000010-22000100) and the Tetanus variant (22000200). The original cherry-pick (bb7cec9) only set patient_id on the foundation Inv (20000100); the Tier 3 variants left it NULL, so the same DELETE-by-sentinel-PATIENT_UID cascade that blocks HEPATITIS_DATAMART would block any condition-datamart SP whose query path goes through F_PAGE_CASE -> D_PATIENT lookup. 2. New orchestrator step 2.5 apply_pending_pr_routines pulls each pending-PR routine (PRs #837, #839, #840) from its bug branch via `git show` and applies it to the freshly-reset DB before fixtures. Without this the orchestrator aborts at step 8 on ldf_answers_tetanus.sql (bug #7 early-RETURN guard fires on a SUB data_type ldf_uid, and bug #8 SUBSTRING fires on the tetanus datamart SP). Remove a branch from the list once its PR merges and the baseline image is refreshed. End-to-end coverage uplift vs. coverage_tier_3.md baseline: HEPATITIS_DATAMART 0 -> 1 (5b) COVID_CASE_DATAMART 0 -> 1 (5b on 22000070) BMIRD_STREP_PNEUMO_DATAMART 0 -> 1 (5b on 22000100) F_PAGE_CASE 1 -> 6 (multi-condition + 5b) LDF_DIMENSIONAL_DATA 0 -> 5 (PR #839 bug-7) MORB_RPT_USER_COMMENT 0 -> 1 (PR #837 bug-3) Still 0 (separate per-condition blockers, not 5b-related): HEPATITIS_CASE, HEP100, F_STD_PAGE_CASE, STD_HIV_DATAMART, TB_DATAMART, VAR_DATAMART, LDF_TETANUS, LDF_HEPATITIS — see coverage_hep_datamart_investigation.md for per-condition diagnoses. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
ericbuckley
approved these changes
May 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
sp_nrt_ldf_dimensional_data_postprocessingnever populatedLDF_DIMENSIONAL_DATAfor fixtures with valid LDF answers. Two independent defects, both fixed here:Early-RETURN guard misclassification (lines 136-158).
@backfill_listwas computed against#LDF_META_DATA, which had already been filtered by adata_type IN ('ST','CV','LIST_ST')whitelist. Anyldf_uidwith a data_type outside the whitelist (e.g.SUB) showed up as "missing" from the metadata set, tripped the guard, and aborted the batch. Guard now readsnrt_odse_state_defined_field_metadatadirectly so it only fires for genuinely missing source rows.INNER vs LEFT JOIN inconsistency (line 648). The
#LDF_DATAstep joinednrt_srte_LDF_PAGE_SETwithINNER JOIN, but the metadata step at line 108 usesLEFT JOINon the same relationship. Sincenrt_ldf_data.ldf_page_idcan legitimately be NULL, the INNER side was silently dropping rows. Harmonized to LEFT JOIN.Verified locally: pre-fix the SP exited at step 2 logging "Missing NRT Record" with
LDF_DIMENSIONAL_DATAat 0; post-fix it runs toSP_COMPLETE, step 25 inserts 5 rows, and the table has 5 rows.Related Issue
APP-471
Additional Notes
A
testData/unitfixture is included (isolated97xxxxxxUID namespace) that seeds three ldf_uids — a SUB, a CV, and an ST — and asserts the two non-SUB rows land inLDF_DIMENSIONAL_DATAwhile SUB is filtered cleanly through the no-longer-overzealous guard.Unblocks the per-condition LDF tables (LDF_TETANUS, LDF_HEPATITIS, etc.) for downstream datamart SPs. The unguarded SUBSTRING that fires when those tables are still empty is handled separately in #840
Checklist