fix: normalize usageSummary.date format and add explicit ES mapping#26693
fix: normalize usageSummary.date format and add explicit ES mapping#26693kojikokojiko wants to merge 2 commits intoopen-metadata:mainfrom
Conversation
|
Hi there 👋 Thanks for your contribution! The OpenMetadata team will review the PR shortly! Once it has been labeled as Let us know if you need any help! |
🟡 Playwright Results — all passed (15 flaky)✅ 3417 passed · ❌ 0 failed · 🟡 15 flaky · ⏭️ 208 skipped
🟡 15 flaky test(s) (passed on retry)
How to debug locally# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip # view trace |
ad9b3d6 to
502c253
Compare
63e9ca3 to
f07a57e
Compare
|
@harshach |
… parse errors MySQL JDBC may return DATE columns as 'yyyy-MM-dd HH:mm:ss' depending on driver configuration, causing Elasticsearch to fail parsing usageSummary.date when the field is dynamically mapped as strict_date_optional_time. - Use r.getDate() instead of r.getString() in UsageDetailsMapper and UsageDetailsWithIdMapper (CollectionDAO) and UsageRepository to ensure the date is always normalized to 'yyyy-MM-dd' format via java.sql.Date.toString() - Add explicit 'date' field mapping to all 36 ES/OS index mapping files with format 'strict_date_optional_time||yyyy-MM-dd HH:mm:ss||epoch_millis' to handle both formats defensively Fixes open-metadata#26678 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Covers the fix for usageSummary.date parse errors in Elasticsearch. Tests verify that getDate().toString() always returns yyyy-MM-dd format for both non-null and null usageDate values, across all three mappers: - CollectionDAO.UsageDAO.UsageDetailsMapper - CollectionDAO.UsageDAO.UsageDetailsWithIdMapper - UsageRepository.UsageDetailsMapper Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
f07a57e to
a666955
Compare
Code Review ✅ ApprovedNormalizes usageSummary.date format and adds explicit Elasticsearch mapping to prevent parse errors. No issues found. OptionsAuto-apply is off → Gitar will not commit updates to this branch. Comment with these commands to change:
Was this helpful? React with 👍 / 👎 | Gitar |
|



Problem
Elasticsearch/OpenSearch throws parse errors when reindexing entities with
usageSummary:Root cause (two issues):
usageSummary.datewas not defined in ES index mappings, so ES used dynamic mapping and inferred the type asdatewith default format (strict_date_optional_time). This format only accepts ISO 8601 (T-separated), not space-separated datetime strings.MySQL JDBC's
ResultSet.getString()returnsDATEcolumns as"yyyy-MM-dd HH:mm:ss"(with time component), which doesn't match the dynamically-inferred ES format, causing parse errors.Fix
1. Java: normalize date output (root cause)
Changed
r.getString("usageDate")tor.getDate("usageDate").toString()in bothUsageDetailsMapperandUsageDetailsWithIdMapper.java.sql.Date.toString()always returns"yyyy-MM-dd"regardless of JDBC driver behavior.Affected files:
CollectionDAO.java(2 mappers)UsageRepository.java(1 mapper)2. ES mapping: add explicit
usageSummary.datefield (defensive fix)Added explicit
datefield definition tousageSummary.propertiesin all index mapping JSON files across all languages (en,jp,zh,ru) for all entities that haveusageSummary(table, dashboard, mlmodel, database, databaseSchema, container, and others).This ensures that even if legacy data with
"yyyy-MM-dd HH:mm:ss"format exists, reindexing will not fail.Why both fixes are needed
Testing
"2026-03-19"(new format) indexes successfully ✅"2026-03-19 00:00:00"(legacy format) indexes successfully ✅ (was failing before)Fixes #26678
🤖 Generated with Claude Code