Fix: Skip alias-type fields in doc-level monitor query index mapping#2158
Open
thecodingshrimp wants to merge 3 commits into
Open
Conversation
DocLevelMonitorQueries.upsertQueryIndex failed with HTTP 500 whenever a source index contained fields with type:alias. leafNodeProcessor rewrote the alias path value by appending _<indexName>_<monitorId>, producing a synthesized path that did not exist in the percolate query index. - Skip alias fields in leafNodeProcessor (early return with empty props) - Skip empty-props entries in the properties.forEach accumulation loop - Upgrade log.debug to log.warn on first PUT _mapping failure - Fast-fail on unrecoverable mapping errors before the retry path - Defensive deep copy of sourceAsMap before traverseMappingsAndUpdate
… list recursion Three gaps identified in PR review, now addressed: 1. deepCopyMap list-of-maps: toMutableList() was shallow — map elements inside lists were shared between original and copy. Fixed by recursing into map elements; test verifies mutation isolation. 2. allFlattenPaths filter simulation: traverseMappingsAndUpdate adds all leaves to flattenPaths unconditionally before calling the processor, so alias paths land in allFlattenPaths even when the alias-skipping processor returns empty props. New test documents the required post-traversal filter that must also be applied when building allFlattenPaths for doIndexAllQueries. 3. getAllConflictingFields alias passthrough: identity processor does not skip alias-type fields, causing false conflicts when two indices share an alias field name with different path values. New test documents this known gap by simulating the conflict-detection logic directly via traverseMappingsAndUpdate with an identity processor.
…e in doc-level monitors H1 (indexDocLevelQueries): filter flattenPaths entries with type=="alias" before populating allFlattenPaths so alias field names never flow into doIndexAllQueries query rewrites, which would produce ts_alias__myindex_monitorid references to non-existent query-index fields. H2 (getAllConflictingFields): add type=="alias" guard in the flattenPaths forEach loop and an alias-skipping leaf processor so alias fields with differing path values across indices are not flagged as conflicting, preventing incorrect field renaming downstream. Also updates the two formerly-gap tests to assert the fixed behavior using the correct filtering predicate (type!="alias" rather than isEmpty(), because flattenPaths stores original nodeProps captured before the processor runs).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix: Skip alias-type fields in doc-level monitor query index mapping
Related issue
Fixes: #2157
Summary
DocLevelMonitorQueries.upsertQueryIndexfails with HTTP 500 whenever a source index contains fields with"type": "alias". This PR adds a two-line guard inleafNodeProcessorthat skips alias fields during mapping traversal, preventing invalidPUT _mappingcalls to the percolate query index.The fix is minimal and surgical: it does not change how any other field type is handled.
Root cause
File:
alerting/src/main/kotlin/org/opensearch/alerting/util/DocLevelMonitorQueries.ktFunction:
leafNodeProcessorWhen
upsertQueryIndexbuilds the query index mappings it callstraverseMappingsAndUpdate, which recursively walks every leaf field of every source index and delegates toleafNodeProcessor. For each leaf,leafNodeProcessorrewrites field names and the aliaspathvalue by appending_<indexName>_<monitorId>to produce a per-monitor suffix. Fortype: aliasfields this produces a synthesized path that does not exist as a concrete field in the query index. OpenSearch then rejects thePUT _mappingcall:The percolate query index (
.opensearch-alerting-queries-*) storesquery_stringqueries, not documents. Alias resolution is not used during percolate matching. Alias fields copied to the query index therefore serve no semantic purpose — they are index-local contracts that are only meaningful in the context of the index where they are defined.The plugin is not idempotent: alias fields left in a source index by a prior monitor lifecycle are re-discovered on every subsequent
upsertQueryIndexcall and trigger the same failure. ThedeleteQueryIndexInEveryRun=trueretry path does not escape this because the same source mappings produce the same invalid synthesized path on retry.Changes
alerting/src/main/kotlin/org/opensearch/alerting/util/DocLevelMonitorQueries.ktleafNodeProcessor— early return for alias fields.Add a guard before the existing
val newProps = props.toMutableMap()line. Whenprops["type"] == "alias", return a triple with an emptymutableMapOf()as the properties value. This signals the accumulation loop that the field should be excluded fromupdatedProperties.traverseMappingsAndUpdate/properties.forEachaccumulation loop — skip empty property maps.After calling
leafNodeProcessor, check whether the returned properties map is empty. If it is, do not write an entry toupdatedProperties. This prevents a zero-property alias stub from being submitted in thePUT _mappingbody.These two changes together ensure that alias fields never appear in the
PUT _mappingrequest body sent to the query index.No changes to any public API, monitor data model, or query execution path. The query index schema for all non-alias field types is unaffected.
Testing notes
Unit test — alias field excluded from query index mappings.
Create a
MappingMetadatafixture that includes atype: aliasfield alongside a concrete keyword field. Assert that aftertraverseMappingsAndUpdatethe returned map contains the concrete field and does not contain the alias field.Integration test — monitor creation succeeds on index with alias fields.
type: aliasfield and one concrete field.ACTIVE..opensearch-alerting-queries-*) mapping does not contain any field with"type": "alias".Regression test — idempotency across delete/re-create cycle.
PUT _mapping500 error in logs.Existing test suite.
Run the full
DocLevelMonitorQueriestest class and the doc-level monitor integration tests. No existing test should regress: the guard only affects fields with"type": "alias", which the existing tests do not exercise.Out of scope
The SA plugin (
opensearch-project/security-analytics) writestype: aliasfields to backing indices on detector creation and does not remove them on detector deletion. That cleanup is tracked separately. This PR makes the alerting plugin resilient to alias fields regardless of their origin, which addresses the immediate breakage independently of when the SA cleanup lands.