fix(apache): add custom.date as fallback source for ocsf.time#22986
Open
jbfeldman-dd wants to merge 1 commit intomasterfrom
Open
fix(apache): add custom.date as fallback source for ocsf.time#22986jbfeldman-dd wants to merge 1 commit intomasterfrom
jbfeldman-dd wants to merge 1 commit intomasterfrom
Conversation
When the Datadog Agent pre-parses Apache access logs, it stores the timestamp as custom.date (epoch ms) rather than date_access. The OCSF sub-pipeline's time remapper only looked at date_access, so ocsf.time was missing for these events, causing ~2,100 attribute_required_missing validation errors per day. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
|
This PR does not modify any files shipped with the agent. To help streamline the release process, please consider adding the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
date_access→ocsf.time, but when the Datadog Agent pre-parses Apache access logs it stores the timestamp ascustom.dateinstead ofdate_accessdate_accessin these pre-parsed events, leavingocsf.timeunsetattribute_required_missingvalidation errors per day (timemissing) in the OCSF validation jobChange: Add
custom.dateas a second source on the existingocsf.timeschema-remapper. Thedate_accesssource remains first so Agent-side-parsed events that do have it are unaffected.Test plan
custom.date) haveocsf.timepopulated after the pipeline runsattribute_required_missingerror count drops for sourceapacheinevent-jobs-security-monitoringdate_access) continue to pass🤖 Generated with Claude Code