Skip to content

Add propertiesSupplier support to EmfMetricLoggingPublisher#6735

Open
humanzz wants to merge 7 commits intoaws:masterfrom
humanzz:emf-props-supplier
Open

Add propertiesSupplier support to EmfMetricLoggingPublisher#6735
humanzz wants to merge 7 commits intoaws:masterfrom
humanzz:emf-props-supplier

Conversation

@humanzz
Copy link

@humanzz humanzz commented Feb 13, 2026

Closes #6595

Enable EmfMetricLoggingPublisher to accept an optional Supplier<Map<String, String>> for enriching EMF records with custom properties at publish time.

Motivation and Context

Users of EmfMetricLoggingPublisher have no way to add custom metadata (e.g. Lambda request IDs, trace IDs) to EMF records. The aws-embedded-metrics-java library supports this via putProperty(), but the SDK's built-in publisher does not. This change adds a propertiesSupplier to the builder, evaluated on each publish() call, allowing dynamic enrichment of EMF output with searchable properties in CloudWatch Logs Insights. As a user of Lambda Powertools, powertools add default properties to logs/metrics, to allow correlating them to Lambda Invocation's request id. The main property I want to use this is request id (Lambda invocation), so I can correlate the metric records to specific invocations similar to what one gets for free from Powertools.

Modifications

  • Add propertiesSupplier field and builder method to EmfMetricLoggingPublisher.Builder
  • Add propertiesSupplier field, accessor, and builder setter to EmfMetricConfiguration, defaulting to empty map when null
  • Add resolveProperties() to MetricEmfConverter which invokes the supplier from config once per convert call, handling null returns and exceptions gracefully
  • Add writeCustomProperties() to MetricEmfConverter which writes custom properties to the EMF JSON, silently skipping any keys that collide with _aws, dimension names, or metric names

Testing

Added 7 tests to MetricEmfConverterTest:

  • Custom properties appear in EMF output
  • No supplier configured produces identical output to current behavior
  • Empty properties map produces no extra keys
  • Property key matching _aws is silently skipped — _aws metadata object is preserved
  • Property key matching a dimension name is silently skipped — dimension retains real value
  • Property key matching a metric name is silently skipped — metric retains real value
  • Batched EMF records (220 metrics, 3 batches) all contain custom properties

Added 3 tests to EmfMetricLoggingPublisherTest:

  • Supplier throwing exception logs warning and publishes without custom properties
  • Supplier returning null publishes without custom properties
  • Stateful supplier returns different maps on successive publish calls

All 26 tests pass. mvn clean install -pl :emf-metric-logging-publisher succeeds (compile, checkstyle, tests, javadoc, API compatibility).

Screenshots (if appropriate)

N/A

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)

Checklist

  • I have read the CONTRIBUTING document
  • Local run of mvn install succeeds
  • My code follows the code style of this project
  • My change requires a change to the Javadoc documentation
  • I have updated the Javadoc documentation accordingly
  • I have added tests to cover my changes
  • All new and existing tests passed
  • I have added a changelog entry. Adding a new entry must be accomplished by running the scripts/new-change script and following the instructions. Commit the new file created by the script in .changes/next-release with your changes.
  • My change is to implement 1.11 parity feature and I have updated LaunchChangelog

License

  • I confirm that this pull request can be released under the Apache 2 license

Enable `EmfMetricLoggingPublisher` to accept an optional `Supplier<Map<String, String>>` for enriching EMF records with custom properties at publish time

- Add `propertiesSupplier` field and builder method to `EmfMetricLoggingPublisher.Builder`
- Add `propertiesSupplier` field, accessor, and builder setter to `EmfMetricConfiguration`, defaulting to empty map when null
- Add `resolveProperties()` to `MetricEmfConverter` which invokes the supplier from config once per convert call, handling null returns and exceptions gracefully
- Add `writeCustomProperties()` to `MetricEmfConverter` which writes properties first in the EMF JSON so `_aws`, dimensions, and metrics overwrite any key collisions

Closes aws#6595
@humanzz humanzz requested a review from a team as a code owner February 13, 2026 19:13
* or {@code null} to disable custom properties
* @return this builder
*/
public Builder propertiesSupplier(Supplier<Map<String, String>> propertiesSupplier) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm what is the advantage of taking a Supplier as opposed to a Map directly in this case?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because the properties are not necessarily static and known at publisher initialization time. An example is if I want to include a request id to correlate to other log lines/metrics within the same overall request

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah that makes sense but it seems like it would be hard in many scenarios to correctly determine that data, especially in a multi tenant or highly concurrent environment given that it doesn't have take any input.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm taking inspiration but simplifying what's happening in aws-embedded-metrics-java for injecting ambient context into EMF output.

In that library, the Environment interface defines a configureContext(MetricsContext) method that each environment implementation uses to enrich EMF records with runtime properties. For example, LambdaEnvironment.configureContext() injects executionEnvironment, functionVersion, logStreamId, and traceId by reading environment variables — notably, it also takes no per-request input, it just reads from ambient state. These properties end up as top-level keys in the EMF JSON.

The Supplier<Map<String, String>> is the same concept, just simplified. Instead of requiring users to implement a full Environment interface, we provide a lightweight Supplier callback that does the same thing — provide ambient context at publish time. The supplier implementer would have the responsibility to define their own strategy for dealing with multi-tenancy or concurrency, if required — though in Lambda environments, where this publisher makes the most sense, each invocation runs in its own execution context so ambient state is naturally request-scoped.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, another thing I wanna call out — I went with Map<String, String> rather than Map<String, Object>. The EMF spec allows any valid JSON type for root node members, and the aws-embedded-metrics-java library uses Map<String, Object> for its putProperty. I chose String to keep the implementation simple — the primary use case is contextual metadata like request IDs and trace IDs, and String covers that well. Using Object would require type-dispatching in the serializer since JsonWriter has separate overloads for String, long, double, boolean, etc., plus handling for unsupported types. If needed, we can always widen to Object later without breaking backward compatibility (narrowing would be a breaking change). Happy to change if you'd prefer Object from the start.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the detailed reply!

  1. I did take at the interface you liked previously, and I agree on the Map<String, String> over Map<String, Object> as it does greatly simplify the interface while addressing what we foresee as the majority use case. And as you said, it's not a one-way door.
  2. I agree that in a Lambda environment, the Supplier is more than sufficient; I was imagining other scenarios where the interface might benefit from information in order to compute the correct map values to enter. One thing I could think of is if a user may want to emit some high cardinality information like ServiceEndpoint as a property, which would be difficult to plumb through with this interface. I think even in this case though this isn't a one way door since we could overload it with a Function or something similar in the future. Let me take this discussion back to the rest of the team to get their input and get back to you.

int customIndex = emfLog.indexOf("\"OperationName\":\"should-be-overwritten\"");
int realIndex = emfLog.indexOf("\"OperationName\":\"GetObject\"");
assertThat(customIndex).isGreaterThanOrEqualTo(0);
assertThat(realIndex).isGreaterThan(customIndex);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like we rely on explicit ordering within the EMF object to ensure that the correct dimension is picked up by CloudWatch, which seems brittle. Can we instead address this upstream so that the writer doesn't write custom properties that have names that collide?

Copy link
Author

@humanzz humanzz Mar 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're absolutely right, relying on write order was brittle and actually produced duplicate keys in the JSON output. That was a miss on my end. I'll push a fix that filters colliding keys upstream in writeCustomProperties before writing, so reserved keys (_aws, dimension names, metric names) are simply skipped. The collision tests will also be updated accordingly.

@humanzz humanzz requested a review from dagnir March 12, 2026 01:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Enable setting properties to enrich the emf records emitted by EmfMetricLoggingPublisher

2 participants