Skip to content

Fix: Skip alias-type fields in doc-level monitor query index mapping#2158

Open
thecodingshrimp wants to merge 3 commits into
opensearch-project:mainfrom
thecodingshrimp:fix/doc-level-monitor-alias-field-collision
Open

Fix: Skip alias-type fields in doc-level monitor query index mapping#2158
thecodingshrimp wants to merge 3 commits into
opensearch-project:mainfrom
thecodingshrimp:fix/doc-level-monitor-alias-field-collision

Conversation

@thecodingshrimp
Copy link
Copy Markdown

@thecodingshrimp thecodingshrimp commented May 23, 2026

Fix: Skip alias-type fields in doc-level monitor query index mapping

Related issue

Fixes: #2157


Summary

DocLevelMonitorQueries.upsertQueryIndex fails with HTTP 500 whenever a source index contains fields with "type": "alias". This PR adds a two-line guard in leafNodeProcessor that skips alias fields during mapping traversal, preventing invalid PUT _mapping calls to the percolate query index.

The fix is minimal and surgical: it does not change how any other field type is handled.


Root cause

File: alerting/src/main/kotlin/org/opensearch/alerting/util/DocLevelMonitorQueries.kt
Function: leafNodeProcessor

When upsertQueryIndex builds the query index mappings it calls traverseMappingsAndUpdate, which recursively walks every leaf field of every source index and delegates to leafNodeProcessor. For each leaf, leafNodeProcessor rewrites field names and the alias path value by appending _<indexName>_<monitorId> to produce a per-monitor suffix. For type: alias fields this produces a synthesized path that does not exist as a concrete field in the query index. OpenSearch then rejects the PUT _mapping call:

Invalid [path] value [<originalPath>_<backingIndexName>_<monitorId>] for field alias [...]:
an alias must refer to an existing field in the mappings.

The percolate query index (.opensearch-alerting-queries-*) stores query_string queries, not documents. Alias resolution is not used during percolate matching. Alias fields copied to the query index therefore serve no semantic purpose — they are index-local contracts that are only meaningful in the context of the index where they are defined.

The plugin is not idempotent: alias fields left in a source index by a prior monitor lifecycle are re-discovered on every subsequent upsertQueryIndex call and trigger the same failure. The deleteQueryIndexInEveryRun=true retry path does not escape this because the same source mappings produce the same invalid synthesized path on retry.


Changes

alerting/src/main/kotlin/org/opensearch/alerting/util/DocLevelMonitorQueries.kt

  1. leafNodeProcessor — early return for alias fields.
    Add a guard before the existing val newProps = props.toMutableMap() line. When props["type"] == "alias", return a triple with an empty mutableMapOf() as the properties value. This signals the accumulation loop that the field should be excluded from updatedProperties.

    // Guard added at entry of leafNodeProcessor:
    if (props["type"] == "alias") {
        return Triple(fieldName, fieldName, mutableMapOf())
    }
  2. traverseMappingsAndUpdate / properties.forEach accumulation loop — skip empty property maps.
    After calling leafNodeProcessor, check whether the returned properties map is empty. If it is, do not write an entry to updatedProperties. This prevents a zero-property alias stub from being submitted in the PUT _mapping body.

    // In the forEach loop that builds updatedProperties:
    val (newKey, _, newProps) = leafNodeProcessor(...)
    if (newProps.isNotEmpty()) {
        updatedProperties[newKey] = newProps
    }

These two changes together ensure that alias fields never appear in the PUT _mapping request body sent to the query index.

No changes to any public API, monitor data model, or query execution path. The query index schema for all non-alias field types is unaffected.


Testing notes

  1. Unit test — alias field excluded from query index mappings.
    Create a MappingMetadata fixture that includes a type: alias field alongside a concrete keyword field. Assert that after traverseMappingsAndUpdate the returned map contains the concrete field and does not contain the alias field.

  2. Integration test — monitor creation succeeds on index with alias fields.

    • Create an index with at least one type: alias field and one concrete field.
    • Create a doc-level monitor targeting that index.
    • Assert monitor creation returns HTTP 200 and monitor status is ACTIVE.
    • Verify the query index (.opensearch-alerting-queries-*) mapping does not contain any field with "type": "alias".
  3. Regression test — idempotency across delete/re-create cycle.

    • Create a doc-level monitor, delete it, and re-create it against the same index.
    • Assert the second creation succeeds (HTTP 200) with no PUT _mapping 500 error in logs.
  4. Existing test suite.
    Run the full DocLevelMonitorQueries test class and the doc-level monitor integration tests. No existing test should regress: the guard only affects fields with "type": "alias", which the existing tests do not exercise.


Out of scope

The SA plugin (opensearch-project/security-analytics) writes type: alias fields to backing indices on detector creation and does not remove them on detector deletion. That cleanup is tracked separately. This PR makes the alerting plugin resilient to alias fields regardless of their origin, which addresses the immediate breakage independently of when the SA cleanup lands.

DocLevelMonitorQueries.upsertQueryIndex failed with HTTP 500 whenever a
source index contained fields with type:alias. leafNodeProcessor rewrote
the alias path value by appending _<indexName>_<monitorId>, producing a
synthesized path that did not exist in the percolate query index.

- Skip alias fields in leafNodeProcessor (early return with empty props)
- Skip empty-props entries in the properties.forEach accumulation loop
- Upgrade log.debug to log.warn on first PUT _mapping failure
- Fast-fail on unrecoverable mapping errors before the retry path
- Defensive deep copy of sourceAsMap before traverseMappingsAndUpdate
… list recursion

Three gaps identified in PR review, now addressed:

1. deepCopyMap list-of-maps: toMutableList() was shallow — map elements inside
   lists were shared between original and copy. Fixed by recursing into map
   elements; test verifies mutation isolation.

2. allFlattenPaths filter simulation: traverseMappingsAndUpdate adds all leaves
   to flattenPaths unconditionally before calling the processor, so alias paths
   land in allFlattenPaths even when the alias-skipping processor returns empty
   props. New test documents the required post-traversal filter that must also
   be applied when building allFlattenPaths for doIndexAllQueries.

3. getAllConflictingFields alias passthrough: identity processor does not skip
   alias-type fields, causing false conflicts when two indices share an alias
   field name with different path values. New test documents this known gap by
   simulating the conflict-detection logic directly via traverseMappingsAndUpdate
   with an identity processor.
…e in doc-level monitors

H1 (indexDocLevelQueries): filter flattenPaths entries with type=="alias" before
populating allFlattenPaths so alias field names never flow into doIndexAllQueries
query rewrites, which would produce ts_alias__myindex_monitorid references to
non-existent query-index fields.

H2 (getAllConflictingFields): add type=="alias" guard in the flattenPaths forEach
loop and an alias-skipping leaf processor so alias fields with differing path
values across indices are not flagged as conflicting, preventing incorrect field
renaming downstream.

Also updates the two formerly-gap tests to assert the fixed behavior using the
correct filtering predicate (type!="alias" rather than isEmpty(), because
flattenPaths stores original nodeProps captured before the processor runs).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Doc-level monitor upsertQueryIndex fails with HTTP 500 when source index contains alias-type fields

1 participant