Skip to content

[cloudflare_logpush] Ingest Pipeline Improvments#19163

Closed
brijesh-elastic wants to merge 9 commits into
mainfrom
feature/cloudflare_logpush-enhancement
Closed

[cloudflare_logpush] Ingest Pipeline Improvments#19163
brijesh-elastic wants to merge 9 commits into
mainfrom
feature/cloudflare_logpush-enhancement

Conversation

@brijesh-elastic
Copy link
Copy Markdown
Contributor

@brijesh-elastic brijesh-elastic commented May 22, 2026

Proposed commit message

cloudflare_logpush: Ingest Pipeline Improvments

- Update ECS to 9.3.0 and add new fields across multiple data streams
- Replace `rename` processors with typed `convert` processors for fields whose `fields.yml` mapping
  is `ip`, `long`, `double`, or `boolean`. This validates incoming values against the declared mapping
  and prevents off-type values from being silently indexed.
- Replace grok processors with dissect for simple delimiter-based pattern.
- Add `tag` key to each processor in the ingest pipelines.
- Standardize the error message format and use the latest null-removal script.
- Improve ECS mapping across multiple data streams.
- Consolidate timestamp handling to a single script across all data streams.
- Normalize severity handling in CASB data stream.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

  • Clone integrations repo.
  • Install elastic package locally.
  • Start elastic stack using elastic-package.
  • Move to integrations/packages/cloudflare_logpush directory.
  • Run the following command to run tests.

elastic-package test -v

Related issues

@brijesh-elastic
cloudflare_logpush: update ECS to 9.3.0 and add new fields across mul…
94b22b3 
…tiple data streams (#18685)

Enhancements:
- Bump ECS version to `git@v9.3.0` and `format_version` to `3.3.2`.
- Replace deprecated `agent.yml` with `beats.yml` and use `external: ecs` in `base-fields.yml` across
  all 21 data streams.
- Add new fields with corresponding ingest pipeline processors:
  - `device_posture` (RegistrationID)
  - `firewall_event` (FraudUserID)
  - `gateway_dns` (12 fields including InternalDNS*, QueryApplication*, RequestContext*)
  - `gateway_http` (AppControlInfo, ApplicationStatuses, RedirectTargetURI, RegistrationID)
  - `gateway_network` (RegistrationID)
  - `http_request` (11 fields including Fraud*, WebAssets*, WorkerScriptName)
  - `network_analytics` (DNSQueryName, DNSQueryType, PFPCustomTag)
  - `network_session` (InitialOriginIP, RegistrationID, ResolvedFQDN, SNI)
  - `workers_trace` (CPUTimeMs, WallTimeMs)
@brijesh-elastic
cloudflare_logpush: enhance ingestion logic with typed conversions an…
590f9c5 
…d use of dissect processors (#18952)

- Replace `rename` with typed `convert` processors for `ip`, `long`, and `boolean` fields.
  This validates values against `fields.yml` mappings and prevents off-type indexing.
- Replace `grok` with `dissect` for simple delimiter-based patterns.
@brijesh-elastic
cloudflare_logpush: add a tag key to each processor (#18953)
6665cf5 
Run `elastic-package modify -m pipeline-tag` to add a tag key to each processor.
@brijesh-elastic
cloudflare_logpush: standardize the error message format and use the …
a322f7d 
…latest null-removal script (#19003)

- Standardize the `error.message` format and implement the latest null-removal script;
  additionally, add `on_failure` handlers to all script processors.
- Remove `ignore_failure: true` from JSON processors and append processors for `related.*` fields.
- Utilize custom fields instead of standard ECS fields within the `related.*` append processors
  to improve maintainability and consistency.
- Apply `ignore_empty_value: true` and `ignore_missing: true` across processors, and
  refine `if` conditions to ensure the ingest pipeline is fully null-safe.
@brijesh-elastic
cloudflare_logpush: Improve ECS mapping across multiple data streams (#…
77eec5c 
…19043)
@brijesh-elastic
cloudflare_logpush: simplify timestamp normalization across all data …
66491d2 
…streams (#19085)

Factor the string-to-number conversion out of each data stream's
Painless timestamp script into a preceding convert processor. This
leaves the script responsible only for magnitude adjustment
(nanoseconds to milliseconds or seconds to milliseconds) and lets
it assume its input is already numeric.

The previous script performed type checking, string parsing, and
magnitude adjustment in a single try/catch block that silently
swallowed all exceptions. With the conversion handled by a separate
processor, the script's on_failure handler now surfaces errors
properly via error.message.
@brijesh-elastic
cloudflare_logpush: normalize severity handling in CASB data stream (#…
3e72507 
…19117)
@brijesh-elastic brijesh-elastic self-assigned this May 22, 2026
@brijesh-elastic brijesh-elastic added documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. enhancement New feature or request Integration:cloudflare_logpush Cloudflare Logpush Category: Integration quality Category: Quality used for SI planning Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] Team:SDE-Crest Crest developers on the Security Integrations team [elastic/sit-crest-contractors] labels May 22, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 22, 2026

TL;DR

Buildkite failed before any test/build steps because the PR head could not be auto-merged with main in the checkout hook. Rebase (or merge) main into feature/cloudflare_logpush-enhancement, resolve the listed conflicts, and push again.

Remediation

  • Update the branch with latest main and resolve merge conflicts in:
    • packages/cloudflare_logpush/changelog.yml
    • packages/cloudflare_logpush/data_stream/network_analytics/_dev/test/pipeline/test-pipeline-network-analytics.log
    • packages/cloudflare_logpush/data_stream/network_analytics/_dev/test/pipeline/test-pipeline-network-analytics.log-expected.json
    • packages/cloudflare_logpush/data_stream/network_analytics/elasticsearch/ingest_pipeline/default.yml
    • packages/cloudflare_logpush/data_stream/network_analytics/sample_event.json
    • packages/cloudflare_logpush/docs/README.md
    • packages/cloudflare_logpush/manifest.yml
  • After resolving and pushing, rerun Buildkite for PR [cloudflare_logpush] Ingest Pipeline Improvments #19163.
Investigation details

Root Cause

The failing step is :pipeline::arrow_up: Upload Pipeline: .buildkite/pipeline.yml, but the actual failure happens in the repository post-checkout hook during automatic merge setup for pr_merge_19163. The merge exits with conflicts, so pipeline upload never proceeds.

Evidence

Auto-merging packages/cloudflare_logpush/changelog.yml
CONFLICT (content): Merge conflict in packages/cloudflare_logpush/changelog.yml
...
Auto-merging packages/cloudflare_logpush/manifest.yml
CONFLICT (content): Merge conflict in packages/cloudflare_logpush/manifest.yml
Automatic merge failed; fix conflicts and then commit the result.
Merge failed: 1
Error: running "repository post-checkout" shell hook: exited with status 1

Verification

  • Not run: test/build jobs are blocked because checkout/merge preparation fails first.

Follow-up

Once conflicts are resolved, CI should advance past pipeline upload and run the normal package checks.

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

@elasticmachine
Copy link
Copy Markdown

elasticmachine commented May 25, 2026

💔 Build Failed

Failed CI Steps

History

cc @brijesh-elastic

@brijesh-elastic brijesh-elastic mentioned this pull request May 28, 2026
Closed
@brijesh-elastic
Copy link
Copy Markdown
Contributor Author

brijesh-elastic commented May 28, 2026

Whenever we push large changes to main, a single massive PR is difficult to review and maintain. To manage this, we use a long-lived feature branch approach, merging multiple smaller PRs into the feature branch before opening one final PR to main.

However, when the feature branch conflicts with main, branch protection rules prevent us from pushing conflict resolutions directly to it. Furthermore, creating a separate PR just to resolve those conflicts introduces a lot of issues.

For now, our workaround is to create an unprotected branch from the feature branch, resolve the conflicts with main there, and then raise a PR directly to main.

cc: @kcreddy

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@brijesh-elastic brijesh-elastic

Labels

Category: Integration quality Category: Quality used for SI planning documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. enhancement New feature or request Integration:cloudflare_logpush Cloudflare Logpush Team:SDE-Crest Crest developers on the Security Integrations team [elastic/sit-crest-contractors] Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@brijesh-elastic @elasticmachine