Fixes 21329: exclude temporal table period columns from autoClassification sampling

[edg956]

Summary

• Fixes list index out of range error in autoClassification ingestion for AzureSQL tables with Temporal Tables enabled (issue autoClassification ingestion fails due to AzureSQL Temporal Table fields #21329)
• SQL Server temporal tables add GENERATED ALWAYS AS ROW START/END period columns (ValidFrom/ValidTo) that break the autoClassification sampler in two ways: TABLESAMPLE is not supported on temporal tables (SQL Server error 13541), and pyodbc cannot reliably handle period column types
• Fix 1 (mssql/utils.py): filters generated_always_type IN (1, 2) in get_columns so temporal period columns are never stored in entity metadata
• Fix 2 (azuresql/sampler.py): adds _get_temporal_column_names() querying sys.columns to detect and exclude temporal period columns in fetch_sample_data, as a defense-in-depth guard

Test plan

• Unit test added to test_azuresql_sampling.py: verifies fetch_sample_data filters out columns returned by _get_temporal_column_names (mocked)
• PII processor test case added (test_cases/azuresql_temporal_table.py): table entity has 5 columns including ValidFrom/ValidTo, sample data has only the 3 regular columns — verifies the processor handles this correctly and detects PII on email
• Integration test added (test_azuresql_temporal_table.py): skipped unless AZURE_SQL_HOST, AZURE_SQL_DATABASE, AZURE_SQL_USERNAME, AZURE_SQL_PASSWORD env vars are set; creates a real temporal table and asserts temporal columns are absent from sample data

Closes #21329

🤖 Generated with Claude Code

---

Summary by Gitar

• Dependency management:
  • Added spacy to ingestion/setup.py to pin compatibility and resolve issues with spacy<3.8.
• Database ingestion/sampling fixes:
  • Modified mssql/utils.py to filter out temporal period columns using generated_always_type.
  • Added _get_temporal_column_names to AzureSQLSampler to identify and exclude period columns from fetch_sample_data.
• Testing:
  • Added integration tests for AzureSQL temporal tables covering metadata and classification workflows.
  • Added unit tests to verify proper column exclusion and handling of empty column sets in AzureSQLSampler.
  • Added PII processor test case for temporal table scenarios.

_{This will update automatically on new commits.}

[github-actions]

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

[github-actions]

🟡 Playwright Results — all passed (15 flaky)

✅ 4013 passed · ❌ 0 failed · 🟡 15 flaky · ⏭️ 86 skipped

Shard	Passed	Failed	Flaky	Skipped
✅ Shard 1	299	0	0	4
🟡 Shard 2	751	0	4	8
🟡 Shard 3	756	0	3	7
🟡 Shard 4	788	0	2	18
🟡 Shard 5	686	0	1	41
🟡 Shard 6	733	0	5	8

🟡 15 flaky test(s) (passed on retry)

• Features/ActivityAPI.spec.ts › Activity event shows the actor who made the change (shard 2, 1 retry)
• Features/BulkEditEntity.spec.ts › Glossary (shard 2, 1 retry)
• Features/DataQuality/BundleSuiteBulkOperations.spec.ts › Create new Bundle Suite with bulk selected test cases (shard 2, 1 retry)
• Features/Glossary/GlossaryWorkflow.spec.ts › should start term as Draft when glossary has reviewers (shard 2, 2 retries)
• Features/RTL.spec.ts › Verify Following widget functionality (shard 3, 1 retry)
• Flow/ObservabilityAlerts.spec.ts › Alert operations for a user with and without permissions (shard 3, 2 retries)
• Flow/PersonaDeletionUserProfile.spec.ts › User profile loads correctly before and after persona deletion (shard 3, 1 retry)
• Pages/Domains.spec.ts › Domain Rbac (shard 4, 1 retry)
• Pages/Entity.spec.ts › Tag Add, Update and Remove for child entities (shard 4, 1 retry)
• Pages/Entity.spec.ts › Announcement create, edit & delete (shard 5, 1 retry)
• Pages/GlossaryImportExport.spec.ts › Glossary CSV import preserves typed relations (shard 6, 1 retry)
• Pages/Lineage/DataAssetLineage.spec.ts › Column lineage for mlModel -> mlModel (shard 6, 1 retry)
• Pages/Lineage/LineageFilters.spec.ts › Verify lineage schema filter selection (shard 6, 1 retry)
• Pages/Lineage/LineageRightPanel.spec.ts › Verify custom properties tab IS visible for supported type: searchIndex (shard 6, 1 retry)
• Pages/UserDetails.spec.ts › Admin user can edit teams from the user profile (shard 6, 1 retry)

📦 Download artifacts

How to debug locally

# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace

[gitar-bot]

Code Review ✅ Approved 2 resolved / 2 findings

Excludes temporal table period columns from autoClassification sampling to resolve SQL Server compatibility issues and pins spacy versioning. Addresses bypass logic for empty column lists and redundant database queries.

✅ 2 resolved

✅ Bug: Filtered columns bypass when all are excluded (empty list falsy)

📄 ingestion/src/metadata/sampler/sqlalchemy/azuresql/sampler.py:87
In fetch_sample_data, if columns is a non-empty list but every column is filtered out (all temporal or all in NOT_COMPUTE_PYODBC), sqa_columns ends up as []. Because empty list is falsy in Python, sqa_columns or columns evaluates to columns — the original unfiltered list — completely defeating the filter. This means the temporal/unsupported columns will still be passed to super().fetch_sample_data(), likely causing the same pyodbc error this PR aims to fix.

The fix should use an explicit None check rather than relying on truthiness.

✅ Performance: Unnecessary DB query when columns is None

📄 ingestion/src/metadata/sampler/sqlalchemy/azuresql/sampler.py:78
_get_temporal_column_names() executes a database round-trip (opening a new session) on every call to fetch_sample_data, even when columns is None. When columns is None, the temporal column names are never used since the filtering loop is skipped entirely. Moving the call inside the if columns: block would avoid the unnecessary query.

Options

Display: compact → Showing less information.

Comment with these commands to change:

Compact

gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Gitar}

[sonarqubecloud]

Quality Gate passed for 'open-metadata-ingestion'

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
60.9% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud