feat: add MIG hierarchy materialized view and diff view #237

hf-kklein · 2026-01-07T14:24:04Z

Add SQL materialized view to flatten MIG hierarchies into a single queryable table, and a diff view to compare MIG versions.

New files:

materialize_mig_view.sql: Recursive CTE to flatten hierarchy
migview.py: MigHierarchyMaterialized class and create_mig_view()
create_mig_diff_view.sql: SQL view for version comparison
mig_diff_view.py: MigDiffLine class and create_mig_diff_view()
test_mig_views.py: Tests for hierarchy and diff views

🤖 Generated with Claude Code

Design for adding SQLModel support for Message Implementation Guides, mirroring the existing AHB SQLModel implementation patterns. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Add SQL model classes for MIGs mirroring the existing AHB SQLModel implementation patterns: - MigCode, MigDataElement, MigDataElementGroup, MigSegment, MigSegmentGroup, MigSegmentGroupLink, MessageImplementationGuide - from_model() and to_model() conversion methods - Position fields for stable list ordering - Self-referential SegmentGroup relationship via link table - SQL-only fields: gueltig_von, gueltig_bis, edifact_format_version Includes roundtrip tests verifying XML -> Pydantic -> SQL -> Pydantic equality for all example MIG files. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Update SQL Models section to include MIG support with code example. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Apply black/isort formatting and fix line-too-long pylint error. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Pylint 4.0.4 incorrectly reports that Relationship fields (which are lists at runtime) don't have an 'append' member. This is a false positive specific to SQLModel's typing. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Add SQL materialized view to flatten MIG hierarchies into a single queryable table, and a diff view to compare MIG versions. New files: - materialize_mig_view.sql: Recursive CTE to flatten hierarchy - migview.py: MigHierarchyMaterialized class and create_mig_view() - create_mig_diff_view.sql: SQL view for version comparison - mig_diff_view.py: MigDiffLine class and create_mig_diff_view() - test_mig_views.py: Tests for hierarchy and diff views 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Resolve conflicts after squash merge of feat/mig-sqlmodels into main. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Compares COMDIS MIG between FV2410 and FV2504 to demonstrate the diff view functionality with a small, readable diff. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Adds PRICAT comparison (FV2410 vs FV2504) with ~108 diff entries to demonstrate the diff view works with more complex MIGs. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Copilot

Pull request overview

This PR adds SQL materialized view functionality for Message Implementation Guides (MIGs), closely following the established pattern from the existing AHB (Anwendungshandbuch) views. The implementation enables flattening hierarchical MIG structures into queryable tables and comparing MIG versions across different EDIFACT format versions.

Key Changes

Adds recursive SQL CTE to flatten MIG hierarchies into a materialized view
Implements diff view to identify added, deleted, and modified rows between MIG versions
Provides helper functions to create databases and populate views with comprehensive test coverage

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
unittests/test_mig_views.py	Comprehensive test suite covering hierarchy views, diff views, validity dates, and snapshot tests with example and private submodule MIG files
unittests/snapshots/test_mig_views.ambr	Snapshot test data for COMDIS and PRICAT diff comparisons between FV2410 and FV2504
src/fundamend/sqlmodels/migview.py	Python module defining MigHierarchyMaterialized model and helper functions to create/populate the materialized view
src/fundamend/sqlmodels/mig_diff_view.py	Python module defining MigDiffLine model and create_mig_diff_view function for version comparison
src/fundamend/sqlmodels/materialize_mig_view.sql	Complex recursive CTE SQL script that flattens segment groups, segments, data elements, and codes into a single queryable table
src/fundamend/sqlmodels/create_mig_diff_view.sql	SQL view definition for comparing MIG versions using FULL OUTER JOIN pattern to identify differences
src/fundamend/sqlmodels/init.py	Module exports updated to include new MIG view classes and functions

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

unittests/test_mig_views.py

Copilot · 2026-01-07T17:04:44Z

src/fundamend/sqlmodels/mig_diff_view.py

+"""
+This module contains the SQLModel class for the MIG diff view and a function to create it.
+The view allows comparing two MIG versions to find rows that were added, deleted, or modified.
+"""
+
+# pylint: disable=duplicate-code
+# This module intentionally follows the same patterns as ahb_diff_view.py
+
+import logging
+from pathlib import Path
+from typing import Optional
+
+import sqlalchemy
+from efoli import EdifactFormat, EdifactFormatVersion
+from sqlmodel import Field, Session, SQLModel
+
+from fundamend.sqlmodels.internals import _execute_bare_sql
+
+_logger = logging.getLogger(__name__)
+
+
+def _check_mig_hierarchy_exists_and_has_data(session: Session) -> None:
+    """Check if mig_hierarchy_materialized exists and has data, logging warnings if not."""
+    try:
+        result = session.execute(sqlalchemy.text("SELECT COUNT(*) FROM mig_hierarchy_materialized"))
+        count = result.scalar()
+        if count == 0:
+            _logger.warning(
+                "mig_hierarchy_materialized exists but is empty. "
+                "The v_mig_diff view will not return any results. "
+                "Make sure to call create_mig_view() after populating the database."
+            )
+    except sqlalchemy.exc.OperationalError:
+        _logger.warning(
+            "mig_hierarchy_materialized does not exist. "
+            "The v_mig_diff view requires mig_hierarchy_materialized to be created first. "
+            "Call create_mig_view() before create_mig_diff_view()."
+        )
+
+
+def create_mig_diff_view(session: Session) -> None:
+    """
+    Create a view for comparing MIG versions.
+    This assumes that create_mig_view (materialize_mig_view.sql) has already been called.
+    """
+    _check_mig_hierarchy_exists_and_has_data(session)
+    _execute_bare_sql(session=session, path_to_sql_commands=Path(__file__).parent / "create_mig_diff_view.sql")
+    _logger.info("Created view %s", MigDiffLine.__tablename__)
+
+
+class MigDiffLine(SQLModel, table=True):
+    """
+    Model that represents the diff view for comparing MIG versions.
+    This view uses mig_hierarchy_materialized structure and compares line_status_std,
+    line_status_specification, and line_name.
+
+    Query with all 4 filter parameters to compare two specific versions:
+
+        SELECT * FROM v_mig_diff
+        WHERE old_format_version = 'FV2410'
+          AND new_format_version = 'FV2504'
+          AND old_format = 'UTILTS'
+          AND new_format = 'UTILTS'
+        ORDER BY sort_path;
+
+    diff_status can be: 'added', 'deleted', 'modified', 'unchanged'
+    All value columns exist twice (old_ and new_) to show the values from both versions.
+    """
+
+    __tablename__ = "v_mig_diff"
+
+    # Composite primary key
+    id_path: str = Field(primary_key=True)
+    old_format_version: Optional[EdifactFormatVersion] = Field(primary_key=True, default=None)
+    new_format_version: Optional[EdifactFormatVersion] = Field(primary_key=True, default=None)
+    old_format: Optional[EdifactFormat] = Field(primary_key=True, default=None)
+    new_format: Optional[EdifactFormat] = Field(primary_key=True, default=None)
+
+    # Common fields
+    sort_path: str = Field()
+    path: str = Field()
+    line_type: Optional[str] = Field(default=None)
+
+    # Diff status: 'added', 'deleted', 'modified', 'unchanged'
+    diff_status: str = Field()
+
+    # Which columns changed (for modified rows only, NULL otherwise)
+    changed_columns: Optional[str] = Field(default=None)
+
+    # Old version columns
+    old_segmentgroup_id: Optional[str] = Field(default=None)
+    old_segment_id: Optional[str] = Field(default=None)
+    old_dataelement_id: Optional[str] = Field(default=None)
+    old_code_value: Optional[str] = Field(default=None)
+    old_line_status_std: Optional[str] = Field(default=None)
+    old_line_status_specification: Optional[str] = Field(default=None)
+    old_line_name: Optional[str] = Field(default=None)
+
+    # New version columns
+    new_segmentgroup_id: Optional[str] = Field(default=None)
+    new_segment_id: Optional[str] = Field(default=None)
+    new_dataelement_id: Optional[str] = Field(default=None)
+    new_code_value: Optional[str] = Field(default=None)
+    new_line_status_std: Optional[str] = Field(default=None)
+    new_line_status_specification: Optional[str] = Field(default=None)
+    new_line_name: Optional[str] = Field(default=None)
+
+
+__all__ = ["create_mig_diff_view", "MigDiffLine"]


The mig_diff_view module does not define or export a DiffStatus enum like the similar ahb_diff_view module does. For consistency with the ahb_diff_view pattern and better API design, a DiffStatus enum should be defined here as well. This would provide type safety and clearer semantics when working with diff_status values.

Copilot · 2026-01-07T17:04:44Z

unittests/test_mig_views.py

+def test_create_db_and_populate_with_mig_view() -> None:
+    """Test the convenience function to create and populate MIG database"""
+    mig_paths = [Path(__file__).parent / "example_files" / "UTILTS_MIG_1.1d_Konsultationsfassung_2024_04_02.xml"]
+    actual_sqlite_path = create_db_and_populate_with_mig_view(mig_files=mig_paths)
+    assert actual_sqlite_path.exists()


The test function name 'test_create_db_and_populate_with_mig_view' doesn't follow the pattern of checking returned value. While the test does verify that the database file exists, the actual database path variable is named 'actual_sqlite_path', suggesting an assertion pattern that is incomplete. Consider either renaming the variable to match convention or adding more comprehensive assertions about the returned value's validity.

unittests/test_mig_views.py

Change MIG diff view to match rows by human-readable 'path' column instead of structural 'id_path'. This provides more semantically meaningful comparisons since paths are based on element names. Changes: - Match on path instead of id_path in create_mig_diff_view.sql - Add path deduplication in materialize_mig_view.sql (append @sort_path for duplicates) - Document matching strategy and limitations in SQL and Python docstrings Limitations documented: - Renamed elements appear as added+deleted rather than modified - Elements with identical names may be incorrectly matched - For structural comparisons, use id_path directly 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

The IFTSTA snapshot compares FV2504 (2.0f) vs FV2510 (2.0g) and shows: - 69 added entries - 4 deleted entries - 3 modified entries (line_status_specification changed from 'C' to 'R') This complements the COMDIS (small) and PRICAT (large) snapshots by demonstrating that the path-based matching correctly identifies semantic modifications, not just added/deleted rows. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Documents the MIG comparison feature similar to AHB: - create_db_and_populate_with_mig_view() for flattening MIG hierarchy - MigHierarchyMaterialized for querying flattened MIG data - create_mig_diff_view() for comparing MIG versions - v_mig_diff view with diff_status (added/deleted/modified/unchanged) - Explains path-based matching strategy and its limitations 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

hf-kklein and others added 7 commits January 7, 2026 13:12

docs: add MIG SQLModels documentation to README

6979c80

Update SQL Models section to include MIG support with code example. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

style: fix formatting and line length issues

d306310

Apply black/isort formatting and fix line-too-long pylint error. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

chore: add .claude/ to .gitignore

1a68e91

🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Base automatically changed from feat/mig-sqlmodels to main January 7, 2026 14:25

hf-kklein and others added 3 commits January 7, 2026 15:27

Merge origin/main into feat/mig-hierarchy-and-diff-view

fc0bc0d

Resolve conflicts after squash merge of feat/mig-sqlmodels into main. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

hf-kklein marked this pull request as ready for review January 7, 2026 16:59

hf-kklein requested a review from Copilot January 7, 2026 16:59

Copilot started reviewing on behalf of hf-kklein January 7, 2026 16:59 View session

Copilot AI reviewed Jan 7, 2026

View reviewed changes

hf-kklein and others added 6 commits January 7, 2026 18:31

chore: add segmente, hierarchie to spellcheck ignore list

17d7bee

🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Merge branch 'main' into feat/mig-hierarchy-and-diff-view

ba7231f

fmt: format MIG SQL files with JetBrains SQL formatter

4006917

🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

hf-kklein requested a review from Copilot January 7, 2026 19:06

Copilot started reviewing on behalf of hf-kklein January 7, 2026 19:07 View session

Copilot AI reviewed Jan 7, 2026

View reviewed changes

hf-kklein and others added 2 commits January 9, 2026 09:10

Update unittests/test_mig_views.py

3f6932c

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update unittests/test_mig_views.py

2497e3f

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

hf-kklein self-assigned this Jan 9, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: add MIG hierarchy materialized view and diff view #237

feat: add MIG hierarchy materialized view and diff view #237

Uh oh!

hf-kklein commented Jan 7, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI Jan 7, 2026

Uh oh!

Copilot AI Jan 7, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

feat: add MIG hierarchy materialized view and diff view #237

Are you sure you want to change the base?

feat: add MIG hierarchy materialized view and diff view #237

Uh oh!

Conversation

hf-kklein commented Jan 7, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Key Changes

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants