Skip to content

fix: Add system property substitution to declarative config #8073

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
MikeGoldsmith wants to merge 3 commits into
base: main
Choose a base branch
from
+200 −21
Open

fix: Add system property substitution to declarative config #8073

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d8fe841
Add system property substitution to declarative config
MikeGoldsmith Feb 12, 2026
cc12372
clean up class ref
MikeGoldsmith Feb 13, 2026
b352442
fix import after applying pr feedback
MikeGoldsmith Feb 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 50 additions & 19 deletions ...in/java/io/opentelemetry/sdk/extension/incubator/fileconfig/DeclarativeConfiguration.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,8 +57,9 @@
public final class DeclarativeConfiguration {

private static final Logger logger = Logger.getLogger(DeclarativeConfiguration.class.getName());
// Matches ${VAR_NAME}, ${env:VAR_NAME}, or ${sys:property.name} with optional :-default
private static final Pattern ENV_VARIABLE_REFERENCE =
Pattern.compile("\\$\\{([a-zA-Z_][a-zA-Z0-9_]*)(:-([^\n}]*))?}");
Pattern.compile("\\$\\{(?:(env|sys):)?([a-zA-Z_][a-zA-Z0-9_.]*)(?::-([^\\n}]*))?}");
Copy link
Member

@jack-berg jack-berg Feb 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technically this is non compliant with the spec. The syntax describes "Optionally followed by env:" with no affordance for alternative prefixes.

I would have preferred none of this prefix business at all, but env: allows for compatibility with the collector's syntax, which had a lot of momentum and was stabilizing.

The collector also supports alternative prefixes to substitute from other places besides env vars (things like other files, network locations), AND allows you to do things like substitute entire objects or array (where as declarative config explicitly limits substitution to primitive types). So collector substitution is a superset of declarative config substitution. The motivation behind this was simplicity: since declarative config is going to be implemented 10+ times, we want the implementation burden to be lower.

The question we need to answer in the java is whether system property support is important enough for us to either:

  • Not complying with the spec
  • Try to update the spec to reflect our use cases

Personally, I don't think spec non-compliance is that big of a deal here. But I also want to be use case motivated. @MikeGoldsmith (or anyone else) have you heard this requested from users?

Copy link
Member

@trask trask Feb 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, I think it's good to support both env vars and system properties in the Java ecosystem

Copy link
Member Author

@MikeGoldsmith MikeGoldsmith Feb 13, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've not heard anything directly that using system properties like this is a hard requirement but I definitely see the benefit for those who prefer them over env vars.

I've opened a spec issue and draft PR to add support for per-language prefixes.

Copy link
Member

@jack-berg jack-berg Feb 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even if we don't add support for the sys prefix, we should add support for env for better interop with collector and other languages.

private static final ComponentLoader DEFAULT_COMPONENT_LOADER =
ComponentLoader.forClassLoader(DeclarativeConfigProperties.class.getClassLoader());

Expand Down Expand Up @@ -140,31 +141,37 @@ private static ExtendedOpenTelemetrySdk create(
/**
* Parse the {@code configuration} YAML and return the {@link OpenTelemetryConfigurationModel}.
*
* <p>During parsing, environment variable substitution is performed as defined in the <a
* <p>During parsing, environment variable and system property substitution is performed as
* defined in the <a
* href="https://opentelemetry.io/docs/specs/otel/configuration/data-model/#environment-variable-substitution">
* OpenTelemetry Configuration Data Model specification</a>.
*
* @throws DeclarativeConfigException if unable to parse
*/
public static OpenTelemetryConfigurationModel parse(InputStream configuration) {
try {
return parse(configuration, System.getenv());
return parse(configuration, System.getenv(), System.getProperties());
} catch (RuntimeException e) {
throw new DeclarativeConfigException("Unable to parse configuration input stream", e);
}
}

// Visible for testing
static OpenTelemetryConfigurationModel parse(
InputStream configuration, Map<String, String> environmentVariables) {
Object yamlObj = loadYaml(configuration, environmentVariables);
InputStream configuration,
Map<String, String> environmentVariables,
Map<Object, Object> systemProperties) {
Object yamlObj = loadYaml(configuration, environmentVariables, systemProperties);
return MAPPER.convertValue(yamlObj, OpenTelemetryConfigurationModel.class);
}

// Visible for testing
static Object loadYaml(InputStream inputStream, Map<String, String> environmentVariables) {
static Object loadYaml(
InputStream inputStream,
Map<String, String> environmentVariables,
Map<Object, Object> systemProperties) {
LoadSettings settings = LoadSettings.builder().setSchema(new CoreSchema()).build();
Load yaml = new EnvLoad(settings, environmentVariables);
Load yaml = new EnvLoad(settings, environmentVariables, systemProperties);
return yaml.loadFromInputStream(inputStream);
}

Expand All @@ -185,7 +192,7 @@ public static DeclarativeConfigProperties toConfigProperties(Object model) {
* @return a generic {@link DeclarativeConfigProperties} representation of the model
*/
public static DeclarativeConfigProperties toConfigProperties(InputStream configuration) {
Object yamlObj = loadYaml(configuration, System.getenv());
Object yamlObj = loadYaml(configuration, System.getenv(), System.getProperties());
return toConfigProperties(yamlObj, DEFAULT_COMPONENT_LOADER);
}

Expand Down Expand Up @@ -256,11 +263,16 @@ private static final class EnvLoad extends Load {

private final LoadSettings settings;
private final Map<String, String> environmentVariables;
private final Map<Object, Object> systemProperties;

private EnvLoad(LoadSettings settings, Map<String, String> environmentVariables) {
private EnvLoad(
LoadSettings settings,
Map<String, String> environmentVariables,
Map<Object, Object> systemProperties) {
super(settings);
this.settings = settings;
this.environmentVariables = environmentVariables;
this.systemProperties = systemProperties;
}

@Override
Expand All @@ -271,46 +283,54 @@ public Object loadFromInputStream(InputStream yamlStream) {
settings,
new ParserImpl(
settings, new StreamReader(settings, new YamlUnicodeReader(yamlStream))),
environmentVariables));
environmentVariables,
systemProperties));
}
}

/**
* A YAML Composer that performs environment variable substitution according to the <a
* A YAML Composer that performs environment variable and system property substitution according
* to the <a
* href="https://opentelemetry.io/docs/specs/otel/configuration/data-model/#environment-variable-substitution">
* OpenTelemetry Configuration Data Model specification</a>.
*
* <p>This composer supports:
*
* <ul>
* <li>Environment variable references: {@code ${ENV_VAR}} or {@code ${env:ENV_VAR}}
* <li>System property references: {@code ${sys:property.name}}
* <li>Default values: {@code ${ENV_VAR:-default_value}}
* <li>Escape sequences: {@code $$} is replaced with a single {@code $}
* </ul>
*
* <p>Environment variable substitution only applies to scalar values. Mapping keys are not
* candidates for substitution. Referenced environment variables that are undefined, null, or
* empty are replaced with empty values unless a default value is provided.
* <p>Substitution only applies to scalar values. Mapping keys are not candidates for
* substitution. Referenced variables that are undefined, null, or empty are replaced with empty
* values unless a default value is provided.
*
* <p>The {@code $} character serves as an escape sequence where {@code $$} in the input is
* translated to a single {@code $} in the output. This prevents environment variable substitution
* for the escaped content.
* translated to a single {@code $} in the output. This prevents substitution for the escaped
* content.
*/
private static final class EnvComposer extends Composer {

private final Load load;
private final Map<String, String> environmentVariables;
private final Map<Object, Object> systemProperties;
private final ScalarResolver scalarResolver;

private static final String ESCAPE_SEQUENCE = "$$";
private static final int ESCAPE_SEQUENCE_LENGTH = ESCAPE_SEQUENCE.length();
private static final char ESCAPE_SEQUENCE_REPLACEMENT = '$';

private EnvComposer(
LoadSettings settings, ParserImpl parser, Map<String, String> environmentVariables) {
LoadSettings settings,
ParserImpl parser,
Map<String, String> environmentVariables,
Map<Object, Object> systemProperties) {
super(settings, parser);
this.load = new Load(settings);
this.environmentVariables = environmentVariables;
this.systemProperties = systemProperties;
this.scalarResolver = settings.getSchema().getScalarResolver();
}

Expand Down Expand Up @@ -397,12 +417,23 @@ private StringBuilder envVarSubstitution(
int offset = 0;
do {
MatchResult matchResult = matcher.toMatchResult();
String envVarKey = matcher.group(1);
String prefix = matcher.group(1); // "env", "sys", or null
String key = matcher.group(2); // variable/property name
String defaultValue = matcher.group(3);
if (defaultValue == null) {
defaultValue = "";
}
String replacement = environmentVariables.getOrDefault(envVarKey, defaultValue);

String replacement;
if ("sys".equals(prefix)) {
// System property substitution
Object sysProp = systemProperties.get(key);
replacement = sysProp != null ? sysProp.toString() : defaultValue;
} else {
// Environment variable substitution (default or explicit "env:" prefix)
replacement = environmentVariables.getOrDefault(key, defaultValue);
}

newVal.append(val, offset, matchResult.start()).append(replacement);
offset = matchResult.end();
} while (matcher.find());
Expand Down
152 changes: 150 additions & 2 deletions ...o/opentelemetry/sdk/extension/incubator/fileconfig/DeclarativeConfigurationParseTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,7 @@
import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
import java.util.stream.Stream;
import javax.annotation.Nullable;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
Expand Down Expand Up @@ -999,6 +1000,7 @@ void coreSchemaValues(String rawYaml, Object expectedYamlResult) {
Object yaml =
DeclarativeConfiguration.loadYaml(
new ByteArrayInputStream(rawYaml.getBytes(StandardCharsets.UTF_8)),
Collections.emptyMap(),
Collections.emptyMap());
assertThat(yaml).isEqualTo(expectedYamlResult);
}
Expand Down Expand Up @@ -1029,7 +1031,8 @@ void envSubstituteAndLoadYaml(String rawYaml, Object expectedYamlResult) {
Object yaml =
DeclarativeConfiguration.loadYaml(
new ByteArrayInputStream(rawYaml.getBytes(StandardCharsets.UTF_8)),
environmentVariables);
environmentVariables,
Collections.emptyMap());
assertThat(yaml).isEqualTo(expectedYamlResult);
}

Expand Down Expand Up @@ -1131,6 +1134,64 @@ private static Map<String, Object> mapOf(Map.Entry<String, ?>... entries) {
return result;
}

@ParameterizedTest
@MethodSource("sysPropertySubstitutionArgs")
void sysPropertySubstituteAndLoadYaml(String rawYaml, Object expectedYamlResult) {
Map<Object, Object> systemProperties = new HashMap<>();
systemProperties.put("foo.bar", "BAR");
systemProperties.put("str.1", "value1");
systemProperties.put("str.2", "value2");
systemProperties.put("value.with.escape", "value$$");
systemProperties.put("empty.str", "");
systemProperties.put("bool.prop", "true");
systemProperties.put("int.prop", "1");
systemProperties.put("float.prop", "1.1");
systemProperties.put("hex.prop", "0xdeadbeef");

Object yaml =
DeclarativeConfiguration.loadYaml(
new ByteArrayInputStream(rawYaml.getBytes(StandardCharsets.UTF_8)),
Collections.emptyMap(),
systemProperties);
assertThat(yaml).isEqualTo(expectedYamlResult);
}

@SuppressWarnings("unchecked")
private static Stream<Arguments> sysPropertySubstitutionArgs() {
return java.util.stream.Stream.of(
// Simple cases with sys: prefix
Arguments.of("key1: ${sys:str.1}\n", mapOf(entry("key1", "value1"))),
Arguments.of("key1: ${sys:bool.prop}\n", mapOf(entry("key1", true))),
Arguments.of("key1: ${sys:int.prop}\n", mapOf(entry("key1", 1))),
Arguments.of("key1: ${sys:float.prop}\n", mapOf(entry("key1", 1.1))),
Arguments.of("key1: ${sys:hex.prop}\n", mapOf(entry("key1", 3735928559L))),
// Default values
Arguments.of("key1: ${sys:not.set:-value1}\n", mapOf(entry("key1", "value1"))),
Arguments.of("key1: ${sys:not.set:-true}\n", mapOf(entry("key1", true))),
Arguments.of("key1: ${sys:not.set:-1}\n", mapOf(entry("key1", 1))),
// Multiple property references
Arguments.of("key1: ${sys:str.1}${sys:str.2}\n", mapOf(entry("key1", "value1value2"))),
Arguments.of("key1: ${sys:str.1} ${sys:str.2}\n", mapOf(entry("key1", "value1 value2"))),
Arguments.of(
"key1: ${sys:str.1} ${sys:not.set:-default} ${sys:str.2}\n",
mapOf(entry("key1", "value1 default value2"))),
// Undefined / empty system property
Arguments.of("key1: ${sys:empty.str}\n", mapOf(entry("key1", null))),
Arguments.of("key1: ${sys:str.3}\n", mapOf(entry("key1", null))),
Arguments.of("key1: ${sys:str.1} ${sys:str.3}\n", mapOf(entry("key1", "value1 "))),
// Quoted system properties
Arguments.of("key1: \"${sys:hex.prop}\"\n", mapOf(entry("key1", "0xdeadbeef"))),
Arguments.of("key1: \"${sys:str.1}\"\n", mapOf(entry("key1", "value1"))),
Arguments.of("key1: '${sys:str.1}'\n", mapOf(entry("key1", "value1"))),
// Escaped
Arguments.of("key1: ${sys:foo.bar}\n", mapOf(entry("key1", "BAR"))),
Arguments.of("key1: $${sys:foo.bar}\n", mapOf(entry("key1", "${sys:foo.bar}"))),
Arguments.of("key1: $$${sys:foo.bar}\n", mapOf(entry("key1", "$BAR"))),
Arguments.of("key1: $$$${sys:foo.bar}\n", mapOf(entry("key1", "$${sys:foo.bar}"))),
// Mixed env and sys
Arguments.of("key1: ${sys:value.with.escape}\n", mapOf(entry("key1", "value$$"))));
}

@Test
void read_WithEnvironmentVariables() {
String yaml =
Expand All @@ -1149,7 +1210,9 @@ void read_WithEnvironmentVariables() {
envVars.put("OTEL_EXPORTER_OTLP_ENDPOINT", "http://collector:4317");
OpenTelemetryConfigurationModel model =
DeclarativeConfiguration.parse(
new ByteArrayInputStream(yaml.getBytes(StandardCharsets.UTF_8)), envVars);
new ByteArrayInputStream(yaml.getBytes(StandardCharsets.UTF_8)),
envVars,
Collections.emptyMap());
assertThat(model)
.isEqualTo(
new OpenTelemetryConfigurationModel()
Expand All @@ -1175,4 +1238,89 @@ void read_WithEnvironmentVariables() {
.withOtlpHttp(
new OtlpHttpExporterModel())))))));
}

@Test
void read_WithSystemProperties() {
String yaml =
"file_format: \"1.0-rc.1\"\n"
+ "tracer_provider:\n"
+ " processors:\n"
+ " - batch:\n"
+ " exporter:\n"
+ " otlp_http:\n"
+ " endpoint: ${sys:otel.exporter.otlp.endpoint}\n"
+ " - batch:\n"
+ " exporter:\n"
+ " otlp_http:\n"
+ " endpoint: ${sys:unset.sys.prop}\n";
Map<Object, Object> sysProps = new HashMap<>();
sysProps.put("otel.exporter.otlp.endpoint", "http://collector:4318");
OpenTelemetryConfigurationModel model =
DeclarativeConfiguration.parse(
new ByteArrayInputStream(yaml.getBytes(StandardCharsets.UTF_8)),
Collections.emptyMap(),
sysProps);
assertThat(model)
.isEqualTo(
new OpenTelemetryConfigurationModel()
.withFileFormat("1.0-rc.1")
.withTracerProvider(
new TracerProviderModel()
.withProcessors(
Arrays.asList(
new SpanProcessorModel()
.withBatch(
new BatchSpanProcessorModel()
.withExporter(
new SpanExporterModel()
.withOtlpHttp(
new OtlpHttpExporterModel()
.withEndpoint(
"http://collector:4318")))),
new SpanProcessorModel()
.withBatch(
new BatchSpanProcessorModel()
.withExporter(
new SpanExporterModel()
.withOtlpHttp(
new OtlpHttpExporterModel())))))));
}

@Test
void read_WithMixedEnvVarsAndSystemProperties() {
String yaml =
"file_format: \"1.0-rc.1\"\n"
+ "resource:\n"
+ " attributes:\n"
+ " - name: service.name\n"
+ " value: ${SERVICE_NAME}\n"
+ " - name: service.version\n"
+ " value: ${sys:app.version}\n"
+ " - name: deployment.environment\n"
+ " value: ${env:DEPLOYMENT_ENV:-production}\n";
Map<String, String> envVars = new HashMap<>();
envVars.put("SERVICE_NAME", "my-service");
Map<Object, Object> sysProps = new HashMap<>();
sysProps.put("app.version", "1.2.3");
OpenTelemetryConfigurationModel model =
DeclarativeConfiguration.parse(
new ByteArrayInputStream(yaml.getBytes(StandardCharsets.UTF_8)), envVars, sysProps);
assertThat(model)
.isEqualTo(
new OpenTelemetryConfigurationModel()
.withFileFormat("1.0-rc.1")
.withResource(
new ResourceModel()
.withAttributes(
Arrays.asList(
new AttributeNameValueModel()
.withName("service.name")
.withValue("my-service"),
new AttributeNameValueModel()
.withName("service.version")
.withValue("1.2.3"),
new AttributeNameValueModel()
.withName("deployment.environment")
.withValue("production")))));
}
}
Loading