ZOOKEEPER-5024: Allow to set TLS version and ciphers for PrometheusMetricsProvider by PDavid · Pull Request #2360 · apache/zookeeper

PDavid · 2026-03-11T14:48:27Z

Since PrometheusMetricsProvider only receives those configs which are prefixed with metricsProvider, introduced two new properties:

metricsProvider.ssl.enabledProtocols
metricsProvider.ssl.ciphersuites

Also documented metricsProvider properties which were missing in zookeeperAdmin.md

…tricsProvider Since PrometheusMetricsProvider only receives those configs which are prefixed with `metricsProvider`, introduced two new properties: - metricsProvider.ssl.enabledProtocols - metricsProvider.ssl.ciphersuites

…etricsProvider.ssl.enabledProtocols properties

PDavid · 2026-03-11T15:19:06Z

Testing

Tested this locally as follows:

Created keystore:

keytool -genkeypair -alias zkAdmin -keyalg RSA -keysize 2048 \
  -dname "CN=your.server.com" -validity 365 \
  -keystore keystore.jks -storepass password -keypass password

Created truststore:

# Export the cert
keytool -export -alias zkAdmin -file zkAdmin.crt \
  -keystore keystore.jks -storepass password

# Import into truststore
keytool -import -alias zkAdmin -file zkAdmin.crt \
  -keystore truststore.jks -storepass password -noprompt

Added these to zoo.cfg:

...
ssl.enabledProtocols=TLSv1.2,TLSv1.3
ssl.ciphersuites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
ssl.keyStore.type=jks
ssl.keyStore.location=keystore.jks
ssl.keyStore.password=password
ssl.trustStore.type=jks
ssl.trustStore.location=truststore.jks
ssl.trustStore.password=password
ssl.quorum.enabledProtocols=TLSv1.2,TLSv1.3
ssl.quorum.ciphersuites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
ssl.quorum.keyStore.type=jks
ssl.quorum.keyStore.location=keystore.jks
ssl.quorum.keyStore.password=password
ssl.quorum.trustStore.type=jks
ssl.quorum.trustStore.location=truststore.jks
ssl.quorum.trustStore.password=password
ssl.clientAuth=none
metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider
#metricsProvider.httpPort=7000
metricsProvider.httpsPort=7000
metricsProvider.ssl.keyStore.location=keystore.jks
metricsProvider.ssl.keyStore.password=password
metricsProvider.ssl.trustStore.location=keystore.jks
metricsProvider.ssl.trustStore.password=password
metricsProvider.ssl.need.client.auth=false
metricsProvider.ssl.enabledProtocols=TLSv1.2,TLSv1.3
metricsProvider.ssl.ciphersuites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

Started ZooKeeper:

mvn clean install -DskipTests && bin/zkServer.sh start

ZooKeeper log:

...
2026-03-11 15:46:36,926 [myid:] - INFO  [main:o.a.z.s.q.QuorumPeerConfig@167] - Reading configuration from: /home/david/projects/zookeeper/bin/../conf/zoo.cfg
2026-03-11 15:46:36,932 [myid:] - INFO  [main:o.a.z.s.q.QuorumPeerConfig@434] - clientPortAddress is 0.0.0.0:2181
2026-03-11 15:46:36,933 [myid:] - INFO  [main:o.a.z.s.q.QuorumPeerConfig@438] - secureClientPort is not set
2026-03-11 15:46:36,933 [myid:] - INFO  [main:o.a.z.s.q.QuorumPeerConfig@454] - observerMasterPort is not set
2026-03-11 15:46:36,933 [myid:] - INFO  [main:o.a.z.s.q.QuorumPeerConfig@471] - metricsProvider.className is org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider
2026-03-11 15:46:36,935 [myid:] - INFO  [main:o.a.z.s.DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2026-03-11 15:46:36,936 [myid:] - INFO  [main:o.a.z.s.DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2026-03-11 15:46:36,936 [myid:] - INFO  [main:o.a.z.s.DatadirCleanupManager@101] - Purge task is not scheduled.
2026-03-11 15:46:36,936 [myid:] - WARN  [main:o.a.z.s.q.QuorumPeerMain@139] - Either no config or no quorum defined in config, running in standalone mode
2026-03-11 15:46:36,939 [myid:] - INFO  [main:o.a.z.j.ManagedUtil@46] - Log4j 1.2 jmx support not found; jmx disabled.
2026-03-11 15:46:36,939 [myid:] - INFO  [main:o.a.z.s.q.QuorumPeerConfig@167] - Reading configuration from: /home/david/projects/zookeeper/bin/../conf/zoo.cfg
2026-03-11 15:46:36,940 [myid:] - INFO  [main:o.a.z.s.q.QuorumPeerConfig@434] - clientPortAddress is 0.0.0.0:2181
2026-03-11 15:46:36,940 [myid:] - INFO  [main:o.a.z.s.q.QuorumPeerConfig@438] - secureClientPort is not set
2026-03-11 15:46:36,940 [myid:] - INFO  [main:o.a.z.s.q.QuorumPeerConfig@454] - observerMasterPort is not set
2026-03-11 15:46:36,940 [myid:] - INFO  [main:o.a.z.s.q.QuorumPeerConfig@471] - metricsProvider.className is org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider
2026-03-11 15:46:36,940 [myid:] - INFO  [main:o.a.z.s.ZooKeeperServerMain@122] - Starting server
2026-03-11 15:46:36,946 [myid:] - INFO  [main:o.a.z.m.p.PrometheusMetricsProvider@122] - Initializing Prometheus metrics with Jetty, configuration: {ssl.enabledProtocols=TLSv1.2,TLSv1.3, ssl.trustStore.password=password, httpsPort=7000, ssl.keyStore.location=keystore.jks, ssl.keyStore.password=password, ssl.need.client.auth=false, ssl.ciphersuites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, ssl.trustStore.location=keystore.jks}
2026-03-11 15:46:36,978 [myid:] - INFO  [main:o.a.z.m.p.PrometheusMetricsProvider@161] - Starting Prometheus Jetty server...
2026-03-11 15:46:36,983 [myid:] - INFO  [main:o.e.j.u.l.Log@170] - Logging initialized @436ms to org.eclipse.jetty.util.log.Slf4jLog
2026-03-11 15:46:36,997 [myid:] - INFO  [main:o.a.z.m.p.PrometheusMetricsProvider@245] - Setting enabled protocols: 'TLSv1.2,TLSv1.3'
2026-03-11 15:46:36,997 [myid:] - INFO  [main:o.a.z.m.p.PrometheusMetricsProvider@251] - Setting enabled cipherSuites: 'TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384'
2026-03-11 15:46:37,036 [myid:] - INFO  [main:o.e.j.s.Server@375] - jetty-9.4.58.v20250814; built: 2025-08-14T02:28:49.637Z; git: 8f1440587e9e4ae7db3d74cf205643f3d707148d; jvm 1.8.0_422-b05
2026-03-11 15:46:37,065 [myid:] - INFO  [main:o.e.j.s.h.ContextHandler@921] - Started o.e.j.s.ServletContextHandler@1165b38{/,null,AVAILABLE}
2026-03-11 15:46:37,163 [myid:] - INFO  [main:o.e.j.u.s.SslContextFactory@358] - x509=X509@4c39bec8(zkadmin,h=[your.server.com],a=[],w=[]) for Server@1f59a598[provider=null,keyStore=file:///home/david/projects/zookeeper/keystore.jks,trustStore=file:///home/david/projects/zookeeper/keystore.jks]
2026-03-11 15:46:37,211 [myid:] - INFO  [main:o.e.j.s.AbstractConnector@333] - Started ServerConnector@16ec5519{SSL, (ssl, http/1.1)}{0.0.0.0:7000}
2026-03-11 15:46:37,212 [myid:] - INFO  [main:o.e.j.s.Server@415] - Started @665ms
2026-03-11 15:46:37,212 [myid:] - INFO  [main:o.a.z.m.p.PrometheusMetricsProvider@199] - Prometheus metrics provider with Jetty started. HTTP port: disabled, HTTPS port: 7000
2026-03-11 15:46:37,218 [myid:] - INFO  [main:o.a.z.s.ServerMetrics@64] - ServerMetrics initialized with provider org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider@68e965f5
2026-03-11 15:46:37,225 [myid:] - INFO  [main:o.a.z.s.a.DigestAuthenticationProvider@51] - ACL digest algorithm is: SHA1
2026-03-11 15:46:37,226 [myid:] - INFO  [main:o.a.z.s.a.DigestAuthenticationProvider@65] - zookeeper.DigestAuthenticationProvider.enabled = true
2026-03-11 15:46:37,227 [myid:] - INFO  [main:o.a.z.s.p.FileTxnSnapLog@124] - zookeeper.snapshot.trust.empty : false
2026-03-11 15:46:37,232 [myid:] - INFO  [main:o.a.z.ZookeeperBanner@42] - 
2026-03-11 15:46:37,232 [myid:] - INFO  [main:o.a.z.ZookeeperBanner@42] -   ______                  _                                          
2026-03-11 15:46:37,232 [myid:] - INFO  [main:o.a.z.ZookeeperBanner@42] -  |___  /                 | |                                         
2026-03-11 15:46:37,232 [myid:] - INFO  [main:o.a.z.ZookeeperBanner@42] -     / /    ___     ___   | | __   ___    ___   _ __     ___   _ __   
2026-03-11 15:46:37,232 [myid:] - INFO  [main:o.a.z.ZookeeperBanner@42] -    / /    / _ \   / _ \  | |/ /  / _ \  / _ \ | '_ \   / _ \ | '__|
2026-03-11 15:46:37,232 [myid:] - INFO  [main:o.a.z.ZookeeperBanner@42] -   / /__  | (_) | | (_) | |   <  |  __/ |  __/ | |_) | |  __/ | |    
2026-03-11 15:46:37,232 [myid:] - INFO  [main:o.a.z.ZookeeperBanner@42] -  /_____|  \___/   \___/  |_|\_\  \___|  \___| | .__/   \___| |_|
2026-03-11 15:46:37,232 [myid:] - INFO  [main:o.a.z.ZookeeperBanner@42] -                                               | |                     
2026-03-11 15:46:37,232 [myid:] - INFO  [main:o.a.z.ZookeeperBanner@42] -                                               |_|                     
2026-03-11 15:46:37,232 [myid:] - INFO  [main:o.a.z.ZookeeperBanner@42] - 
2026-03-11 15:46:37,233 [myid:] - INFO  [main:o.a.z.Environment@98] - Server environment:zookeeper.version=3.10.0-SNAPSHOT-c084c2537ec60ce47131bb61181b9833bd6630e6-dirty, built on 2026-03-11 14:43 UTC

Call PrometheusMetricsProvider:

curl -k -v https://localhost:7000/metrics 
* Host localhost:7000 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:7000...
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / secp256r1 / RSASSA-PSS
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: CN=your.server.com
*  start date: Mar 10 12:58:28 2026 GMT
*  expire date: Mar 10 12:58:28 2027 GMT
*  issuer: CN=your.server.com
*  SSL certificate verify result: self-signed certificate (18), continuing anyway.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* Connected to localhost (::1) port 7000
* using HTTP/1.x
> GET /metrics HTTP/1.1
> Host: localhost:7000
> User-Agent: curl/8.15.0
> Accept: */*
> 
* Request completely sent off
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
< HTTP/1.1 200 OK
< Date: Wed, 11 Mar 2026 15:15:54 GMT
< Content-Type: text/plain; version=0.0.4; charset=utf-8
< Content-Length: 38630
< Server: Jetty(9.4.58.v20250814)
< 
# HELP add_dead_watcher_stall_time_total add_dead_watcher_stall_time counter
# TYPE add_dead_watcher_stall_time_total counter
add_dead_watcher_stall_time_total 0.0
# HELP approximate_data_size approximate_data_size
# TYPE approximate_data_size gauge
approximate_data_size 44.0
# HELP auth_failed_count auth_failed_count
# TYPE auth_failed_count gauge
auth_failed_count 0.0
# HELP avg_latency avg_latency
# TYPE avg_latency gauge
avg_latency 0.0
# HELP bytes_received_count_total bytes_received_count counter
# TYPE bytes_received_count_total counter
bytes_received_count_total 0.0
# HELP close_session_prep_time close_session_prep_time summary
# TYPE close_session_prep_time summary
close_session_prep_time{quantile="0.5"} NaN
close_session_prep_time{quantile="0.95"} NaN
close_session_prep_time{quantile="0.99"} NaN
close_session_prep_time_count 0
close_session_prep_time_sum 0.0
# HELP cnxn_closed_without_zk_server_running_total cnxn_closed_without_zk_server_running counter
# TYPE cnxn_closed_without_zk_server_running_total counter
cnxn_closed_without_zk_server_running_total 0.0

...

# HELP write_final_proc_time_ms write_final_proc_time_ms summary
# TYPE write_final_proc_time_ms summary
write_final_proc_time_ms{quantile="0.5"} NaN
write_final_proc_time_ms{quantile="0.95"} NaN
write_final_proc_time_ms{quantile="0.99"} NaN
write_final_proc_time_ms_count 0
write_final_proc_time_ms_sum 0.0
# HELP znode_count znode_count
# TYPE znode_count gauge
znode_count 5.0
* Connection #0 to host localhost left intact

…properties

…ties

zookeeper-docs/src/main/resources/markdown/zookeeperAdmin.md

...metrics/src/main/java/org/apache/zookeeper/metrics/prometheus/PrometheusMetricsProvider.java

The logging of setting the enabled protocols and ciphers in PrometheusMetricsProvider is now changed to DEBUG.

PDavid · 2026-03-12T11:09:31Z

There was a test failure in the C client module:

[exec] /home/runner/work/zookeeper/zookeeper/zookeeper-client/zookeeper-client-c/tests/TestReconfig.cc:199: Assertion: equality assertion failed [Expected: 1, Actual  : 0]

I think it is unrelated.

anmolnar

Patch lgtm. The C client unit test failure is probably unrelated. Have you checked it?

PDavid · 2026-03-24T09:10:40Z

Patch lgtm. The C client unit test failure is probably unrelated. Have you checked it?

OK, I now looked into this a bit more.

This C client unit test failed in the PR build:

[exec] /home/runner/work/zookeeper/zookeeper/zookeeper-client/zookeeper-client-c/tests/TestReconfig.cc:199: Assertion: equality assertion failed [Expected: 1, Actual  : 0]

I now run the C unit tests locally on this branch:

$ mvn -Pfull-build verify -pl zookeeper-client/zookeeper-client-c

...

     [exec] /home/david/projects/upstream/zookeeper/zookeeper-client/zookeeper-client-c/tests/TestReadOnlyClient.cc:118: Assertion: equality assertion failed [Expected: 0, Actual  : -15]
     [exec] /home/david/projects/upstream/zookeeper/zookeeper-client/zookeeper-client-c/tests/TestClient.cc:902: Assertion: equality assertion failed [Expected: 0, Actual  : -15]
     [exec] /home/david/projects/upstream/zookeeper/zookeeper-client/zookeeper-client-c/tests/TestClient.cc:913: Assertion: equality assertion failed [Expected: 0, Actual  : -15]
     [exec] Failures !!!
     [exec] Run: 94   Failure total: 3   Failures: 3   Errors: 0
     [exec] FAIL: zktest-mt
     [exec] ==========================================
     [exec] 1 of 2 tests failed
     [exec] Please report to user@zookeeper.apache.org
     [exec] ==========================================
     [exec] make[1]: Leaving directory '/home/david/projects/upstream/zookeeper/zookeeper-client/zookeeper-client-c/target/c'
     [exec] make[1]: *** [Makefile:1903: check-TESTS] Error 1
     [exec] make: *** [Makefile:2168: check-am] Error 2
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  07:10 min
[INFO] Finished at: 2026-03-24T09:32:55+01:00
[INFO] ------------------------------------------------------------------------

TestReconfig.cc did not fail for me locally, these 3 other tests above failed though. I tried to execute the tests again 2 times, I got the same result. These 3 other tests are also failing for me on master.

…figs to document since when the config is used by ZK.

…sProvider.ssl.enabledProtocols properties to guide

PDavid added 2 commits March 11, 2026 15:46

ZOOKEEPER-5024: Documented new metricsProvider.ssl.ciphersuites and m…

6b0ca98

…etricsProvider.ssl.enabledProtocols properties

PDavid marked this pull request as ready for review March 11, 2026 15:44

PDavid added 2 commits March 11, 2026 16:46

ZOOKEEPER-5024: Documented missing metricsProvider.httpsPort and ssl …

e0c1d9e

…properties

ZOOKEEPER-5024: Documented missing metricsProvider client auth proper…

146b527

…ties

PDavid commented Mar 11, 2026

View reviewed changes

zookeeper-docs/src/main/resources/markdown/zookeeperAdmin.md Show resolved Hide resolved

ZOOKEEPER-5024: Fix markdown indentation

9a7303c

PDavid commented Mar 11, 2026

View reviewed changes

...metrics/src/main/java/org/apache/zookeeper/metrics/prometheus/PrometheusMetricsProvider.java Outdated Show resolved Hide resolved

meszibalu approved these changes Mar 12, 2026

View reviewed changes

ZOOKEEPER-5024: Changed logging to DEBUG

c8ff1a5

The logging of setting the enabled protocols and ciphers in PrometheusMetricsProvider is now changed to DEBUG.

anmolnar approved these changes Mar 23, 2026

View reviewed changes

PDavid added 2 commits March 25, 2026 13:17

ZOOKEEPER-5024: Added ZK version number to secure metricsProvider con…

8c26464

…figs to document since when the config is used by ZK.

ZOOKEEPER-5024: Added new metricsProvider.ssl.ciphersuites and metric…

7e679bf

…sProvider.ssl.enabledProtocols properties to guide

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ZOOKEEPER-5024: Allow to set TLS version and ciphers for PrometheusMetricsProvider#2360

ZOOKEEPER-5024: Allow to set TLS version and ciphers for PrometheusMetricsProvider#2360
PDavid wants to merge 8 commits intoapache:masterfrom
PDavid:ZOOKEEPER-5024-prometheus-https-ciphers-protocols

PDavid commented Mar 11, 2026 •

edited

Loading

Uh oh!

PDavid commented Mar 11, 2026

Uh oh!

Uh oh!

Uh oh!

PDavid commented Mar 12, 2026

Uh oh!

anmolnar left a comment

Uh oh!

PDavid commented Mar 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

PDavid commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

PDavid commented Mar 11, 2026

Testing

Uh oh!

Uh oh!

Uh oh!

PDavid commented Mar 12, 2026

Uh oh!

anmolnar left a comment

Choose a reason for hiding this comment

Uh oh!

PDavid commented Mar 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

PDavid commented Mar 11, 2026 •

edited

Loading