Greylilac09/fix mem tracking by GreyLilac09 · Pull Request #4 · GreyLilac09/vector

GreyLilac09 · 2025-10-22T17:52:54Z

Summary

Vector configuration

How did you test this PR?

Change Type

Bug fix
New feature
Non-functional (chore, refactoring, docs)
Performance

Is this a breaking change?

Yes
No

Does this PR include user facing changes?

Yes. Please add a changelog fragment based on our guidelines.
No. A maintainer will apply the no-changelog label to this PR.

References

Notes

Please read our Vector contributor resources.
Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
Some CI checks run only after we manually approve them.
- We recommend adding a pre-push hook, please see this template.
- Alternatively, we recommend running the following locally before pushing to the remote branch:
  - make fmt
  - make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
  - make test
After a review is requested, please avoid force pushes to help us review incrementally.
- Feel free to push as many commits as you want. They will be squashed into one before merging.
- For example, you can run git merge origin master and git push.
If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
run make build-licenses to regenerate the license inventory and commit the changes (if any). More details here.

* naively copy e2e otel-logs to otel-metrics * try to fix * Remove vector_default.yaml custom mapping * Test all metric types * Update vrl to main after fix was merged * Add additional attributes and verify them in tests * Format + remove Summary support * simplify assert_metric_attributes logic * Revert Cargo.toml changes * Add ci e2e trigger * Make test more concise * reuse common Dockerfiles * Use common functions in opentelemetry-logs * Hard code vector_otlp.yaml * Fix CI trigger * Add service to integration.yml matrix * Add and use generic assert_service_name_with * ranem data_type -> test_type --------- Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com>

…ectordotdev#24112)

…v#24115) chore(ci): parallelize e2e tests

* Use X logo instead of twitter * Properly use flex in social-buttons.html and add offset for icons * Use text-lg * Remove unused twitter url

* [StepSecurity] ci: Harden GitHub Actions Signed-off-by: StepSecurity Bot <bot@stepsecurity.io> * Add step-security-bot to cla allowlist * Fix permissions is MMQ --------- Signed-off-by: StepSecurity Bot <bot@stepsecurity.io> Co-authored-by: Thomas <thomas.schneider@datadoghq.com>

[StepSecurity] ci: Harden GitHub Actions Signed-off-by: StepSecurity Bot <bot@stepsecurity.io>

…tdev#24539)

…ectordotdev#24457) Bumps [preact](https://github.com/preactjs/preact) from 10.28.0 to 10.28.2. - [Release notes](https://github.com/preactjs/preact/releases) - [Commits](preactjs/preact@10.28.0...10.28.2) --- updated-dependencies: - dependency-name: preact dependency-version: 10.28.2 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…ectordotdev#24529) Bumps [lodash](https://github.com/lodash/lodash) from 4.17.21 to 4.17.23. - [Release notes](https://github.com/lodash/lodash/releases) - [Commits](lodash/lodash@4.17.21...4.17.23) --- updated-dependencies: - dependency-name: lodash dependency-version: 4.17.23 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix(ci): respect DISABLE_MOLD in setup action * Move DISABLE_MOLD flag

…time (vectordotdev#24546) Revert "enhancement(observability): Add metrics to measure total event processing time (vectordotdev#24481)" This reverts commit f5d0c56.

… component_received_events_total metric (vectordotdev#24537) * fix(opentelemetry source): count individual items in OTLP batches for component_received_events_total metric * Format * chore(vdev): apply vdev rust check fixes * Make count_otlp_items common * Simplify count_from_events

* chore(releasing): prepare v0.53.0 release (vectordotdev#24554) * fix(opentelemetry source): count individual items in OTLP batches for component_received_events_total metric (vectordotdev#24537) * fix(opentelemetry source): count individual items in OTLP batches for component_received_events_total metric * Format * chore(vdev): apply vdev rust check fixes * Make count_otlp_items common * Simplify count_from_events * chore(releasing): Pinned VRL version to 0.30.0 * chore(releasing): Generated release CUE file * chore(releasing): Updated website/cue/reference/administration/interfaces/kubectl.cue vector version to 0.53.0 * chore(releasing): Updated distribution/install.sh vector version to 0.53.0 * chore(releasing): Add 0.53.0 to versions.cue * chore(releasing): Created release md file * Add description w/ breaking changes * Add missing doris.md file * Changes to release description * Add last commit * Apply suggestions from code review Co-authored-by: Bryce Eadie <bryce.eadie@datadoghq.com> * Apply suggestions from code review --------- Co-authored-by: Bryce Eadie <bryce.eadie@datadoghq.com> * cargo vdev build manifests * Bump Vector + use git VRL --------- Co-authored-by: Bryce Eadie <bryce.eadie@datadoghq.com>

chore(buffers): Add custom instrumentation hook (unused)

* docs(external): match cla link to gh workflow * Shorten link text --------- Co-authored-by: Thomas <thomasqueirozb@gmail.com>

…dotdev#24556)

…ectordotdev#24394) * fix(log_to_metric transform): Use correct keys for histogram and summary metric values * Add changelog fragment

…ordotdev#24255) * Update Azure crates and migrate to the new SDK * add a reference comment for empty headers * Update Cargo.toml Co-authored-by: Thomas <thomasqueirozb@gmail.com> * Fix failing tests * Add optional = true to azure_core * Add dep:azure_core to sinks-azure_blob * more changes suggested in the PR * Fix shared_key_policy * Bump reqwest to 0.12 and consolidate reqwest versions * remove azure_identity crate, use defaults for other azure * Separate reqwest v0.11 and v0.12 again * Allow CDLA-Permissive-2.0 * update dates in tests to 2026 so they pass --------- Co-authored-by: Thomas <thomasqueirozb@gmail.com> Co-authored-by: Thomas <thomas.schneider@datadoghq.com>

…4520) 1. Avoid cloning the string generated by the `group_by` template when we can pass in ownership. 2. Defer generating the string value of the `key_field` until it is needed, which can be expensive if the underlying value is not a UTF-8 string.

…#24555) Avoid calling `Utc::now()` in a loop and instead use the same timestamp for all the events.

…4575)

* chore(ci): bump actions/cache from 5.0.1 to 5.0.3 Bumps [actions/cache](https://github.com/actions/cache) from 5.0.1 to 5.0.3. - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](actions/cache@9255dc7...cdf6c1f) --- updated-dependencies: - dependency-name: actions/cache dependency-version: 5.0.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Bump cache action in other workflows --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Thomas <thomas.schneider@datadoghq.com>

) Bumps [actions/checkout](https://github.com/actions/checkout) from 6.0.1 to 6.0.2. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@8e8c483...de0fac2) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: 6.0.2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

… provider config (vectordotdev#24341) * enhancement(http provider): allow environment interpolation from http provider config * use to_string_lossy * make fmt --------- Co-authored-by: Thomas <thomas.schneider@datadoghq.com>

…dev#24580) Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.31.9 to 4.32.0. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@5d4e8d1...b20883b) --- updated-dependencies: - dependency-name: github/codeql-action dependency-version: 4.32.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…#24582) Bumps [docker/login-action](https://github.com/docker/login-action) from 3.6.0 to 3.7.0. - [Release notes](https://github.com/docker/login-action/releases) - [Commits](docker/login-action@5e57cd1...c94ce9f) --- updated-dependencies: - dependency-name: docker/login-action dependency-version: 3.7.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…ctordotdev#24409) * refactor: split arrow encoder into smaller files * enhancement(clickhouse sink): add support for complex types * refactor: use idiomatic Arrow types instead of Arc * refactor: unify primitives, decimal, temporal + complex logic * refactor: use append_option where possible * refactor: use iterator method for building record batch * chore: remove duplicated code in arrow tests * chore: update docs * chore: add changelog fragment * chore: add support for named tuples * chore: improve handling for to_arrow * refactor: use serde_arrow for Arrow encoding * refactor: use nom for clickhouse type parsing * refactor: replace to_arrow() with TryFrom impl * refactor: use iterator chain instead of loops * chore: avoid fallible unwrap when fetching schema * refactor: implement FromStr for ClickHouseType * refactor: use String directly * refactor: use try_collect for schema parsing * refactor: simplify schema parsing with TryFrom for Field * refactor: use form_urlencoded for query string building * refactor: return to single file format/arrow.rs * refactor: defer to serde_arrow error handling * chore: add recursion warning for try_from Co-authored-by: Thomas <thomasqueirozb@gmail.com> * chore: add validation for DataType::Map * refactor: eliminate unreachable!() in ClickHouseType conversion * chore: clippy * chore: update license * Update licenses --------- Co-authored-by: Thomas <thomasqueirozb@gmail.com> Co-authored-by: Thomas <thomas.schneider@datadoghq.com>

… 1 directory (vectordotdev#24500) chore(deps): bump clap in the clap group across 1 directory Bumps the clap group with 1 update in the / directory: [clap](https://github.com/clap-rs/clap). Updates `clap` from 4.5.53 to 4.5.54 - [Release notes](https://github.com/clap-rs/clap/releases) - [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md) - [Commits](clap-rs/clap@clap_complete-v4.5.53...clap_complete-v4.5.54) --- updated-dependencies: - dependency-name: clap dependency-version: 4.5.54 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: clap ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump the tokio group with 5 updates Bumps the tokio group with 5 updates: | Package | From | To | | --- | --- | --- | | [tokio-stream](https://github.com/tokio-rs/tokio) | `0.1.17` | `0.1.18` | | [tokio-postgres](https://github.com/rust-postgres/rust-postgres) | `0.7.13` | `0.7.15` | | [tokio-tungstenite](https://github.com/snapview/tokio-tungstenite) | `0.20.1` | `0.21.0` | | [tokio-test](https://github.com/tokio-rs/tokio) | `0.4.4` | `0.4.5` | | [tokio](https://github.com/tokio-rs/tokio) | `1.47.1` | `1.49.0` | Updates `tokio-stream` from 0.1.17 to 0.1.18 - [Release notes](https://github.com/tokio-rs/tokio/releases) - [Commits](tokio-rs/tokio@tokio-stream-0.1.17...tokio-stream-0.1.18) Updates `tokio-postgres` from 0.7.13 to 0.7.15 - [Release notes](https://github.com/rust-postgres/rust-postgres/releases) - [Commits](rust-postgres/rust-postgres@tokio-postgres-v0.7.13...tokio-postgres-v0.7.15) Updates `tokio-tungstenite` from 0.20.1 to 0.21.0 - [Changelog](https://github.com/snapview/tokio-tungstenite/blob/master/CHANGELOG.md) - [Commits](snapview/tokio-tungstenite@v0.20.1...v0.21.0) Updates `tokio-test` from 0.4.4 to 0.4.5 - [Release notes](https://github.com/tokio-rs/tokio/releases) - [Commits](tokio-rs/tokio@tokio-test-0.4.4...tokio-test-0.4.5) Updates `tokio` from 1.47.1 to 1.49.0 - [Release notes](https://github.com/tokio-rs/tokio/releases) - [Commits](tokio-rs/tokio@tokio-1.47.1...tokio-1.49.0) --- updated-dependencies: - dependency-name: tokio-stream dependency-version: 0.1.18 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: tokio - dependency-name: tokio-postgres dependency-version: 0.7.15 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: tokio - dependency-name: tokio-tungstenite dependency-version: 0.21.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: tokio - dependency-name: tokio-test dependency-version: 0.4.5 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: tokio - dependency-name: tokio dependency-version: 1.49.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: tokio ... Signed-off-by: dependabot[bot] <support@github.com> * Update licenses * Make tokio-stream workspace * Make tokio-tungstenite a workspace dep * Use tokio-stream workspace everywhere * Update cargo.lock * Make tokio-test a workspace dep * Revert tokio-tungstenite to 0.20.1 --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Thomas <thomas.schneider@datadoghq.com>

GreyLilac09 added 3 commits October 21, 2025 10:58

fix

d6b9ddc

fix memory tracking final alloc

448d59e

fix

e37428c

github-actions bot added domain: sinks domain: core labels Oct 22, 2025

GreyLilac09 added 9 commits October 22, 2025 14:21

change to trace level

8861ced

get lru

dd60a34

fix log lvl

51bd83e

fix

f9e948a

remove some logs

e55b610

remove some logs

6f61c49

fix clone

41b983f

wip

df4eaee

add metrics

b0eb047

github-actions bot added the domain: transforms label Oct 30, 2025

GreyLilac09 and others added 7 commits October 30, 2025 10:43

use crate

ea26332

fix metrics

8577fe7

lint

c6b8b96

fix(dev): multicast_and_unicast_udp_message no longer hangs on macOS (v…

ae8ad71

…ectordotdev#24112)

fix eviction count

16f99f6

wip

9c2c93b

github-actions bot added the domain: external docs label Oct 30, 2025

GreyLilac09 and others added 7 commits October 30, 2025 12:36

fix

376d616

make const

62efd61

Merge branch 'master' into greylilac09/fix-mem-tracking

b3ae03d

chore(ci): parallelize e2e tests (ci-integration-review) (vectordotde…

d8abed5

…v#24115) chore(ci): parallelize e2e tests

fix(dev): fix journald tests for local macOS (vectordotdev#24114)

e486428

fix(dev): aws-kinesis-firehose tests (vectordotdev#24117)

42f7106

wip

1c16e26

thomasqueirozb and others added 30 commits January 22, 2026 20:42

feat(website): Use X logo instead of twitter (vectordotdev#24534)

4084133

* Use X logo instead of twitter * Properly use flex in social-buttons.html and add offset for icons * Use text-lg * Remove unused twitter url

chore(ci): Pin actions to full commit sha (vectordotdev#24538)

22bd2ae

[StepSecurity] ci: Harden GitHub Actions Signed-off-by: StepSecurity Bot <bot@stepsecurity.io>

fix(external docs): csv enrichment guide incorrect severity (vectordo…

9cf8e20

…tdev#24539)

fix(ci): respect DISABLE_MOLD in setup action (vectordotdev#24548)

f5632b6

* fix(ci): respect DISABLE_MOLD in setup action * Move DISABLE_MOLD flag

revert(observability): Add metrics to measure total event processing …

6f6b7ab

…time (vectordotdev#24546) Revert "enhancement(observability): Add metrics to measure total event processing time (vectordotdev#24481)" This reverts commit f5d0c56.

chore(buffers): Add custom instrumentation hook (vectordotdev#24558)

7594b8a

chore(buffers): Add custom instrumentation hook (unused)

docs(internal): match cla link to gh workflow (vectordotdev#24565)

0af6553

* docs(external): match cla link to gh workflow * Shorten link text --------- Co-authored-by: Thomas <thomasqueirozb@gmail.com>

chore(observability): Refactor EWMA + Gauge into a new struct (vector…

22e92a6

…dotdev#24556)

fix(log_to_metric transform): Use correct keys for histogram/summary (v…

2da8b24

…ectordotdev#24394) * fix(log_to_metric transform): Use correct keys for histogram and summary metric values * Add changelog fragment

Merge branch 'master' into greylilac09/fix-mem-tracking

5e63065

enhancement(websocket source): Micro-optimize send loop (vectordotdev…

25c2819

…#24555) Avoid calling `Utc::now()` in a loop and instead use the same timestamp for all the events.

chore(releasing): minor release template improvements (vectordotdev#2…

9422128

…4575)

chore(ci): Remove orphaned audit.yml (vectordotdev#24584)

cb701eb

Merge branch 'master' into greylilac09/fix-mem-tracking

1a4e17a

fix cargo error

2c1992a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Greylilac09/fix mem tracking#4

Greylilac09/fix mem tracking#4
GreyLilac09 wants to merge 235 commits intomasterfrom
greylilac09/fix-mem-tracking

GreyLilac09 commented Oct 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Conversation

GreyLilac09 commented Oct 22, 2025

Summary

Vector configuration

How did you test this PR?

Change Type

Is this a breaking change?

Does this PR include user facing changes?

References

Notes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants