Support running embedded-zk in "ensemble" mode #2391

gerlowskija · 2024-04-05T21:57:45Z

Description

Prior to this commit, Solr only supported running embedded-ZK in "standalone" mode (which cannot take part in any larger ZK ensemble or quorum). But there are usecases that would benefit from being able to do this, both on the development and testing side, and event for adventurous users who might want the benefits of a small multi-node SolrCloud cluster without the headache of also deploying ZK.

Solution

This commit augments our embedded-ZK code to support running embedded-ZK
in "quorum" or ensemble mode. Multiple Solr nodes can now all have
their embedded-ZK's join a multi-node quorum upon startup. Other than
Solr and ZK sharing a process, the embedded- ZK ensemble behaves
identically to one formed of independent processes: nodes can join or
leave the cluster, etc.

Embedded-ensemble-ZK is enabled any time the zkQuorumRun system
property is present, along with an explicitly specified ZK host string.
On startup, Solr will identify which host in the zk-conn-string it
should be (based on admittedly hacky heuristics), and then spins up a
'ZooKeeperServerEmbedded' instance in-process to join the ensemble. e.g.

export LH="localhost"

# Run each below in a separate terminal, one for each host in the zk-conn string
bin/solr start -p 8983 -z $LH:9983,$LH:9984,$LH:9985 -DzkQuorumRun
bin/solr start -p 8984 -z $LH:9983,$LH:9984,$LH:9985 -DzkQuorumRun
bin/solr start -p 8985 -z $LH:9983,$LH:9984,$LH:9985 -DzkQuorumRun

Some notes:
- this doesn't (yet) work with ZK's dynamic-ensemble feature, so all
ZK nodes must be specified in a static ZK conn string provided at
startup
- this appears to run best when the security-manager is disabled.
- the interface in particular for how we expose this is pretty rough, and there's a lot of room for improvement.

Tests

None, yet.

Checklist

Please review the following and check all that apply:

I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
I have created a Jira issue and added the issue ID to my pull request title.
I have given Solr maintainers access to contribute to my PR branch. (optional but recommended)
I have developed this patch against the main branch.
I have run ./gradlew check.
I have added tests for my changes.
I have added documentation for the Reference Guide

…found" with an out-dated ClusterState (apache#2363)" This reverts commit 5c399dd.

This commit augments our embedded-ZK code to support running embedded-ZK in "quorum" or ensemble mode. Multiple Solr nodes can now all have their embedded-ZK's join a multi-node quorum upon startup. Other than Solr and ZK sharing a process, the embedded- ZK ensemble behaves identically to one formed of independent processes: nodes can join or leave the cluster, etc. Embedded-ensemble-ZK is enabled any time the `zkQuorumRun` system property is present, along with an explicitly specified ZK host string. On startup, Solr will identify which host in the zk-conn-string it should be (based on admittedly hacky heuristics), and then spins up a 'ZooKeeperServerEmbedded' instance in-process to join the ensemble. e.g. ``` export LH="localhost" bin/solr start -p 8983 -z $LH:9983,$LH:9984,$LH:9985 -DzkQuorumRun bin/solr start -p 8984 -z $LH:9983,$LH:9984,$LH:9985 -DzkQuorumRun bin/solr start -p 8985 -z $LH:9983,$LH:9984,$LH:9985 -DzkQuorumRun ``` Some notes: - this doesn't (yet) work with ZK's dynamic-ensemble feature, so all ZK nodes must be specified in a static ZK conn string provided at startup - this appears to run best when the security-manager is disabled.

gerlowskija · 2024-04-05T22:02:58Z

FYI: https://cwiki.apache.org/confluence/display/SOLR/SIP-14+Embedded+Zookeeper for context on others' interest in moving this direction.

janhoy · 2024-04-29T20:24:21Z

Great work. I think we should rip out the existing embedded ZK, which is a hack.

Do you want to proceed in a PR or perhaps in a central feature branch?

…ion not found" with an out-dated ClusterState (apache#2363)"" This reverts commit 1fb376b.

gerlowskija · 2025-09-15T12:31:14Z

Spent a few minutes pulling in the latest 'main'. Hoping to give it a bit more cleanup in the next few days if I can, but I've been hoping that for more than a year at this point (😬 ), so if anyone else is interested please feel free to move this forward!

epugh · 2025-10-14T01:13:55Z

@gerlowskija I took a stab at getting in the use of the solr roles to determine if we hsould run embedded zk quorum mode, and it worked! Here is my start script, where each runs with -f so you need three terminals. Omit the -f and you don't need the seperate terminals.

mkdir -p ./three_nodes/node1/solr
mkdir -p ./three_nodes/node2/solr
mkdir -p ./three_nodes/node3/solr

SOLR_SECURITY_MANAGER_ENABLED=false bin/solr start -f -p 8983 --solr-home "./three_nodes/node1/solr" -z localhost:9983,localhost:9984,localhost:9985 -Dsolr.node.roles=data:on,overseer:allowed,zookeeper_quorum:on
SOLR_SECURITY_MANAGER_ENABLED=false bin/solr start -f -p 8984 --solr-home "./three_nodes/node2/solr" -z localhost:9983,localhost:9984,localhost:9985 -Dsolr.node.roles=data:on,overseer:allowed,zookeeper_quorum:on
SOLR_SECURITY_MANAGER_ENABLED=false bin/solr start -f -p 8985 --solr-home "./three_nodes/node3/solr" -z localhost:9983,localhost:9984,localhost:9985 -Dsolr.node.roles=data:on,overseer:allowed,zookeeper_quorum:on

# Conflicts: # solr/core/src/java/org/apache/solr/core/ZkContainer.java # solr/packaging/build.gradle

janhoy · 2025-10-15T23:49:37Z

Merged in main branch
Properly clean up the ZK server resource in close()

solr/core/src/java/org/apache/solr/core/ZkContainer.java

epugh · 2025-10-16T00:19:38Z

FYI, the .bats test worked a few times for me and then quit working ;-(

Uses a base class SolrCloudWithEmbeddedZkQuorumTestCase New MiniSolrCloudCluster constructor that spins up a quorum cluster

janhoy · 2025-10-16T08:13:19Z

With the help of Claude Code I added a JUnit test TestEmbeddedZkQuorum, that actually spins up a quorum cluster with zookeeper_quorum:on node roles (after securing a port so we know beforehand what the zkHost string will be).

epugh

Three committers on the same PR! Woot!

epugh · 2025-10-16T10:02:26Z

solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java

+    this.jettyConfig = Objects.requireNonNull(jettyConfig);
+    this.solrXml = solrXml == null ? DEFAULT_CLOUD_SOLR_XML : solrXml;
+    this.trackJettyMetrics = trackJettyMetrics;
+    this.externalZkServer = true; // No ZkTestServer in quorum mode


something about this being true seems odd... Should it be false since there isn't a externalZkServer?

Yea, what is even an "external" zk in the context of our test framework?

epugh · 2025-10-16T10:04:07Z

solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java

+    }
+
+    // Set system properties for embedded ZK quorum mode
+    System.setProperty("solr.zookeeper.server.enabled", "true");


I am a bit curious if we need this system property AND zookeeper_quorum:on? I wonder if the existence of zookeeper_quorum:on is enough? if this system property is what turns on zookeeper, I wish maybe it was solr.mode=clustered or something to specificy that it's the clustered (cloud) mode...

Sounds like node role should be enough, it should turn things on, perhaps set that property automagically?

If you dont have the node role, then you need it right? OR!!!????? here is an idea... What if you are starting a single solr node and want zk... The bin/solr start of SOlr 10, and was previoulsly bin/solr start -c in 9x... What if, instead, if you do NOT supply a -Dsolr.node.roles=data:on,overseer:allowed,zookeeper_quorum:on or a -z property then we do that FOR you in the start up script. I believe that we ALREADY have logic that if you don't provide a solr.node.roles then you get data:on,overseer:allowed by default. See https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/core/NodeRoles.java#L32

Would that simplify things both internally and from a user perspective?

You either:

Start Solr old school with bin/solr start --user-managed

Or you start single node with embedded zk with bin/solr start

Or you start three nodes with embedded zk quorum with bin/solr start -z blahblah:blahblah:blahblah -Dsolr.node.roles=data:on,overseer:allowed,zookeeper_quorum:on

Or you start N nodes and connect to an external ZK with bin/solr start -z external_zk:2181.

I wonder if -z should be reserved for connecting to an external ZK quorum, and we invent a new option for telling SolrCloud which solr nodes will run the zk node role? That would make it clearer to read, and we make ZK an implementation detail by not refering to it as ZK, but as quorum-node, e.g.

bin/solr start --quorum-nodes=solr-node-1:8983,solr-node-2:8983,solr-node-3:8983

That would mean we want Solr to manage our ZK, and translate into the correct solr.zk.host setting for each host, no matter if the host you are starting is one of the quorum nodes or not. And if the node you are starting is a quorum node, it will detect that it's own host name matches a quorum host, and add -Dsolr.node.roles... during startup, and the ZK bootstrap code will configure its internal ZK config using the --quorum / -Dsolr.quorum.nodes values.

epugh · 2025-10-16T10:05:02Z

solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java

+      if (!zkClient.exists("/solr", true)) {
+        zkClient.makePath("/solr", false, true);
+      }
+      if (!zkClient.exists("/solr/initialized", true)) {


is this a concept in solr? this path? New to me!

Could be some hallucination going on

epugh · 2025-10-16T10:06:24Z

solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java

-              formatZkServer);
+      MiniSolrCloudCluster cluster;
+
+      if (useEmbeddedZkQuorum) {


this big if statement seems challenging... Could a builder pattern make it easier? Thir are a lot of params and overlap with subtle differences ;-).

Hmm, this code is inside the builder :) So the builder chose to instantiate the MSCC using one of two constructors.

solr/test-framework/src/java/org/apache/solr/cloud/SolrCloudWithEmbeddedZkQuorumTestCase.java

epugh · 2025-10-16T10:09:42Z

solr/core/src/test/org/apache/solr/cloud/TestEmbeddedZkQuorum.java

+
+  @Test
+  public void testBasicQuorumFunctionality() throws Exception {
+    log.info("Starting testBasicQuorumFunctionality");


I wonder if claude tools could go back and revamp old unit tests to be clearer?

epugh · 2025-10-16T10:12:03Z

solr/core/src/test/org/apache/solr/cloud/TestEmbeddedZkQuorum.java

This test is verifying the happy path! I think that we need a test that demonstrates that when you LOSE a node from a three node cluster, we still function, just degraded. I don't know if we have a test that verifies this behavior elsewhere (a stress test??) that could be reused with embedded zk?

There used to be such a test in the first phase, but it kept failing, so it was removed 😂🤣

Good point though. With this foundation it should be possible to keep adding more exceptional tests. Hey, perahps even randomize running with and without qorum. But currently we requre >=3 nodes. We could evolve the quorum to use 1 zk node if <3 and use 3 zk nodes for >=3. Then if you kill a solr node, it should all work even if that node also has zk.

But I think there were some issues with stopping a JettySolrRunner in quorum mode and then restarting it...

epugh · 2025-10-16T10:16:34Z

solr/core/src/java/org/apache/solr/core/ZkContainer.java

          "Cannot start Solr in cloud mode - no cloud config provided");
+    }
+
+    if (config == null) {


How does this even get hit? Sometimes I get confused by all the null checks! If you are running in standalone, wouldn't this class never be run???!

Who knows? Tons of tests failed due to NPE, and the existing code on main has this check, which was removed in this PR...

…ing flag to not do the saem thing twice Our code is just single threaded at this point...

epugh · 2025-11-03T12:41:01Z

I asked Claude to summarize the last seven commits and he said:

Recent work on the spike-zk-quorum branch has focused on refining Solr's embedded ZooKeeper quorum mode implementation through targeted code improvements. The codebase has been streamlined by removing unnecessary variables and conditional logic in the SolrZkServer class, particularly eliminating redundant checks around the zkServerEnabled flag and renaming methods like startZKSE to more descriptive alternatives. Initialization has been optimized by removing the /solr/initialized ZooKeeper node, which was only needed for multi-threading protection that's unnecessary in the current implementation. The ZooKeeper initialization logic in ZkContainer has been restructured with clearer nesting and better separation between standalone and quorum modes, making the codebase more maintainable. These changes collectively enhance the embedded ZooKeeper quorum functionality while simplifying the code structure, resulting in a more robust and easier-to-understand implementation that properly manages ZooKeeper server resources throughout Solr's lifecycle.

I think my summary is: I discovered that we have TWO ways of starting ZK, not including our independent ZkTestServer that runs a ZkTestServer.ZKServerMain during tests.

We now have:

For testing only, we have org.apache.solr.cloud.ZkTestServer.ZKServerMain that wraps org.apache.zookeeper.server.ZooKeeperServer.
org.apache.solr.cloud.SolrZkServer that wraps either a org.apache.zookeeper.server.ZooKeeperServer OR if there is more then one, then does a org.apache.zookeeper.server.quorum.QuorumPeerMain
org.apache.solr.core.ZkContainer that starts up org.apache.zookeeper.server.embedded.ZooKeeperServerEmbedded

I am now wondering could we convert SolrZkServer to using a ZooKeeperServerEmbedded instead of a ZooKeeperServer?

Secondly, now seeing the use of QuorumPeerMain, which currently kicks in if there are multiple ZooKeeper servers defined (presumably only in an external cluster set up), I wonder if we need to rename roles: zookeeper_quorum to zookeeper. If you have zookeeper then you fire up ZooKeeperServerEmbedded. And if not, then you default to QuorumPeerMain.

Thirdly, why is ZkContainer in o.a.s.core and not o.a.s.cloud?

I did take a stab at the SolrZkServer using the ZooKeeperServerEmbedded, updating the start/stop methods, but failed ;-(. Going to need to put this down for a bit.

janhoy · 2025-11-03T13:03:01Z

I did take a stab at the SolrZkServer using the ZooKeeperServerEmbedded, updating the start/stop methods, but failed ;-(. Going to need to put this down for a bit.

Thanks for chipping away on this. At some point we'll have something that can be released in 10.1 or 10.2 as an experimental feature for folks to try out, without dynamic reconfig. Then we can tackle the dynamic stuff...

epugh · 2026-01-28T12:40:41Z

Thanks for taking a look at this @janhoy I think I've stalled a bit on this... It feels "so close' yet not sure how to get it over the hump.

gerlowskija added 2 commits April 5, 2024 15:02

Revert "SOLR-17153: CloudSolrClient should not throw "Collection not …

1fb376b

…found" with an out-dated ClusterState (apache#2363)" This reverts commit 5c399dd.

github-actions bot added jetty-server client:solrj cat:api labels Apr 5, 2024

gerlowskija added 2 commits September 14, 2025 15:46

Revert "Revert "SOLR-17153: CloudSolrClient should not throw "Collect…

459ecf6

…ion not found" with an out-dated ClusterState (apache#2363)"" This reverts commit 1fb376b.

Merge branch 'main' into spike-zk-quorum

de59401

gerlowskija force-pushed the spike-zk-quorum branch from 88c8da4 to de59401 Compare September 14, 2025 22:49

github-actions bot removed jetty-server client:solrj cat:api labels Sep 14, 2025

epugh mentioned this pull request Sep 16, 2025

SOLR-17492: Introduce recommendations of WAYS of running Solr from small to massive #2783

Open

epugh added 2 commits October 13, 2025 19:24

Merge remote-tracking branch 'upstream/main' into pr/2391

ff0a9ca

Take advantage of Solr node roles to determine when to start embedded zk

cd6ecc5

Strip out old log4j workaround not needed, look at ide warnings.

9e6ef68

github-actions bot added the cat:cloud label Oct 14, 2025

check in some work to be removed

479e85f

github-actions bot added the tool:build label Oct 14, 2025

janhoy added 2 commits October 16, 2025 01:28

Merge branch 'refs/heads/main' into spike-zk-quorum

0601701

# Conflicts: # solr/core/src/java/org/apache/solr/core/ZkContainer.java # solr/packaging/build.gradle

Properly clean up ZK server resources

27ea8e9

epugh reviewed Oct 16, 2025

View reviewed changes

solr/core/src/java/org/apache/solr/core/ZkContainer.java Show resolved Hide resolved

epugh reviewed Oct 16, 2025

View reviewed changes

solr/core/src/java/org/apache/solr/core/ZkContainer.java Show resolved Hide resolved

janhoy added 2 commits October 16, 2025 09:18

Fix precommit in ZkContainer

6bd62ec

New test TestEmbeddedZkQuorum

8ee628d

Uses a base class SolrCloudWithEmbeddedZkQuorumTestCase New MiniSolrCloudCluster constructor that spins up a quorum cluster

github-actions bot added test-framework tests labels Oct 16, 2025

Handle standalone case in ZkContainer.initZookeeper

d8bac96

epugh reviewed Oct 16, 2025

View reviewed changes

epugh added 10 commits October 16, 2025 08:25

Spent too much time on this, backing it out.

c5ee205

Merge remote-tracking branch 'upstream/main' into spike-zk-quorum

991ba1e

add change log

47e928b

Redo explanation to be clearer

7cd745d

update variable name

42d5213

remove unneed variable and if statement, and add a reminder

f8d4f8a

remove the /solr/initialized zk node, it appears to be a multi thread…

bacb4af

…ing flag to not do the saem thing twice Our code is just single threaded at this point...

Remove intermediate test class and simplify cluster set up

5199b84

Better nesting of zkServerEnabled check and if in quorum mode...

3f785da

zkEnabled does actually do anything!

367d37b

janhoy added 2 commits January 28, 2026 10:15

Merge branch 'main' into spike-zk-quorum

898897a

Update code to work with latest main

ac9bf9e

Support running embedded-zk in "ensemble" mode #2391

Are you sure you want to change the base?

Support running embedded-zk in "ensemble" mode #2391

Uh oh!

Conversation

gerlowskija commented Apr 5, 2024

Description

Solution

Tests

Checklist

Uh oh!

gerlowskija commented Apr 5, 2024

Uh oh!

janhoy commented Apr 29, 2024

Uh oh!

gerlowskija commented Sep 15, 2025

Uh oh!

epugh commented Oct 14, 2025

Uh oh!

janhoy commented Oct 15, 2025

Uh oh!

Uh oh!

Uh oh!

epugh commented Oct 16, 2025

Uh oh!

janhoy commented Oct 16, 2025

Uh oh!

epugh left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

janhoy Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

epugh commented Nov 3, 2025

Uh oh!

janhoy commented Nov 3, 2025

Uh oh!

epugh commented Jan 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

janhoy Nov 3, 2025 •

edited

Loading