SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role by gerlowskija · Pull Request #2391 · apache/solr

gerlowskija · 2024-04-05T21:57:45Z

Description

Prior to this commit, Solr only supported running embedded-ZK in "standalone" mode (which cannot take part in any larger ZK ensemble or quorum). But there are usecases that would benefit from being able to do this, both on the development and testing side, and even for adventurous users who might want the benefits of a small multi-node SolrCloud cluster without the headache of also deploying ZK.

See SIP-14, SOLR-15636 for context.

⚠️ Experimental / Alpha feature — this feature is intentionally rough and should not be used in production. The API and configuration format may change without notice.

Solution

This commit augments the embedded-ZK code to support running in "quorum" or ensemble mode. Multiple Solr nodes can now all have their embedded ZK join a multi-node quorum upon startup. Other than Solr and ZK sharing a process, the embedded-ZK ensemble behaves identically to one formed of independent processes.

Embedded-ensemble-ZK is enabled by assigning the new zookeeper_quorum node role (on/off) via the solr.node.roles system property, along with an explicit ZK host string containing all quorum members. Solr identifies which host in the ZK connection string it should be (based on host/port matching) and starts a ZooKeeperServerEmbedded instance in-process to join the ensemble. For example:

export LH="localhost"

# Run each below in a separate terminal, one for each host in the zk-conn string
bin/solr start -p 8983 -z $LH:9983,$LH:9984,$LH:9985 -Dsolr.node.roles=data:on,overseer:allowed,zookeeper_quorum:on
bin/solr start -p 8984 -z $LH:9983,$LH:9984,$LH:9985 -Dsolr.node.roles=data:on,overseer:allowed,zookeeper_quorum:on
bin/solr start -p 8985 -z $LH:9983,$LH:9984,$LH:9985 -Dsolr.node.roles=data:on,overseer:allowed,zookeeper_quorum:on

Note: ZK client port is solr_port + 1000, quorum port is zk_client_port + 1, election port is zk_client_port + 2.

Known limitations (Phase 1 / experimental):

Does not work with ZK's dynamic-ensemble feature — all ZK nodes must be specified in a static connection string at startup.
Quorum loss and recovery is not yet reliable (see testQuorumLossAndRecovery marked @AwaitsFix).
Requires that hostnames in the ZK connection string exactly match Solr's configured host (solr.host).
Runs best with the security manager disabled.
The configuration interface is rough and subject to change.

Tests

TestEmbeddedZkQuorum — basic quorum startup, collection creation, and node join/leave scenarios

Checklist

Please review the following and check all that apply:

I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
I have created a Jira issue and added the issue ID to my pull request title.
I have given Solr maintainers access to contribute to my PR branch. (optional but recommended)
I have developed this patch against the main branch.
I have run ./gradlew check.
I have added tests for my changes.
I have added documentation for the Reference Guide

…found" with an out-dated ClusterState (apache#2363)" This reverts commit 5c399dd.

This commit augments our embedded-ZK code to support running embedded-ZK in "quorum" or ensemble mode. Multiple Solr nodes can now all have their embedded-ZK's join a multi-node quorum upon startup. Other than Solr and ZK sharing a process, the embedded- ZK ensemble behaves identically to one formed of independent processes: nodes can join or leave the cluster, etc. Embedded-ensemble-ZK is enabled any time the `zkQuorumRun` system property is present, along with an explicitly specified ZK host string. On startup, Solr will identify which host in the zk-conn-string it should be (based on admittedly hacky heuristics), and then spins up a 'ZooKeeperServerEmbedded' instance in-process to join the ensemble. e.g. ``` export LH="localhost" bin/solr start -p 8983 -z $LH:9983,$LH:9984,$LH:9985 -DzkQuorumRun bin/solr start -p 8984 -z $LH:9983,$LH:9984,$LH:9985 -DzkQuorumRun bin/solr start -p 8985 -z $LH:9983,$LH:9984,$LH:9985 -DzkQuorumRun ``` Some notes: - this doesn't (yet) work with ZK's dynamic-ensemble feature, so all ZK nodes must be specified in a static ZK conn string provided at startup - this appears to run best when the security-manager is disabled.

gerlowskija · 2024-04-05T22:02:58Z

FYI: https://cwiki.apache.org/confluence/display/SOLR/SIP-14+Embedded+Zookeeper for context on others' interest in moving this direction.

janhoy · 2024-04-29T20:24:21Z

Great work. I think we should rip out the existing embedded ZK, which is a hack.

Do you want to proceed in a PR or perhaps in a central feature branch?

…ion not found" with an out-dated ClusterState (apache#2363)"" This reverts commit 1fb376b.

gerlowskija · 2025-09-15T12:31:14Z

Spent a few minutes pulling in the latest 'main'. Hoping to give it a bit more cleanup in the next few days if I can, but I've been hoping that for more than a year at this point (😬 ), so if anyone else is interested please feel free to move this forward!

epugh · 2025-10-14T01:13:55Z

@gerlowskija I took a stab at getting in the use of the solr roles to determine if we hsould run embedded zk quorum mode, and it worked! Here is my start script, where each runs with -f so you need three terminals. Omit the -f and you don't need the seperate terminals.

mkdir -p ./three_nodes/node1/solr
mkdir -p ./three_nodes/node2/solr
mkdir -p ./three_nodes/node3/solr

SOLR_SECURITY_MANAGER_ENABLED=false bin/solr start -f -p 8983 --solr-home "./three_nodes/node1/solr" -z localhost:9983,localhost:9984,localhost:9985 -Dsolr.node.roles=data:on,overseer:allowed,zookeeper_quorum:on
SOLR_SECURITY_MANAGER_ENABLED=false bin/solr start -f -p 8984 --solr-home "./three_nodes/node2/solr" -z localhost:9983,localhost:9984,localhost:9985 -Dsolr.node.roles=data:on,overseer:allowed,zookeeper_quorum:on
SOLR_SECURITY_MANAGER_ENABLED=false bin/solr start -f -p 8985 --solr-home "./three_nodes/node3/solr" -z localhost:9983,localhost:9984,localhost:9985 -Dsolr.node.roles=data:on,overseer:allowed,zookeeper_quorum:on

# Conflicts: # solr/core/src/java/org/apache/solr/core/ZkContainer.java # solr/packaging/build.gradle

janhoy · 2025-10-15T23:49:37Z

Merged in main branch
Properly clean up the ZK server resource in close()

epugh · 2025-10-16T00:19:38Z

FYI, the .bats test worked a few times for me and then quit working ;-(

Uses a base class SolrCloudWithEmbeddedZkQuorumTestCase New MiniSolrCloudCluster constructor that spins up a quorum cluster

…down

…aging/build.gradle

…e (experimental)

janhoy · 2026-05-11T08:23:18Z

I addressed Copilot's review points, then ran another review with Claude and addressed some of those concerns.

Added ref-guide docs, marked as "experimental":

Release-notes:

Zookeeper config:

Node roles:

…ed ZK quorum

…ectServers()

… avoid disk roundtrip

Single-machine example with consecutive Solr ports 8983/8984/8985 caused ZK quorum/election port collisions (node N's quorum port = node N+1's client port). Fix by separating the multi-machine and single-machine examples, and using ports spaced ≥3 apart (8983, 8986, 8989) for the single-machine case.

janhoy · 2026-05-11T09:41:20Z

+
+bin/solr start -p 20000 $ZK $ROLES
+bin/solr start -p 30000 $ZK $ROLES
+bin/solr start -p 40000 $ZK $ROLES


Running three nodes like this causes all three embedded zookeepers to use exact same server/solr/zoo_home/data folder, which is not good of course.

I fixed it it commit ccf6ec3 - appending -<port> to the zoo_home folder name.

so, we can't do 8983, 8984,8985? That is definitly a conceptual change for all of us. I'm used to zk being 1000 more that my port...

janhoy · 2026-05-11T09:52:35Z

+
+|ZK leader election
+|`zk_client_port + 2`
+|9985


I modified the default ports used for quorum peer and leader election ports. They were something like client-port minus 3000 and minus 4000. Now they are +1 and +2. I feel we spread our port numbers too wide. But then after the change we got another problem that we could not start three test solr nodes on 8983, 8984, 8985, since the first one would occupy port 9983, 9984, 9985 for zookeeper.

I am willing to revert the +1, +2 to something else, but -3000 and -4000 seemed a bit arbitrary, why not +2000 and +3000 since client-port is +1000?

Yeah, after commenting elsewhere, I like your +2000 and +3000 idea.

I’d prefer if Solr occupied a more narrow port space, say within 100 ports total, so you could start Solr on 8000, 8100, 8200 and each instance would have ports close. Instead of node 1 using 8000, 9000, 10000, 11000, while node 2 using 8100, 9100, 10100, 11100….

And come to think of it, each and every of these ports should probably be configurable individually even if they default to some port offset...

janhoy · 2026-05-13T12:52:31Z

+        final Path zkDataHome =
+            Path.of(
+                EnvUtils.getProperty(
+                    "solr.zookeeper.server.datadir", solrHome.resolve("zoo_data").toString()));


Do we normally keep constants like this inline or do we have somewhere we try to keep all sysprop-names for e.g. zookeeper together in one place?

gerlowskija added 2 commits April 5, 2024 15:02

Revert "SOLR-17153: CloudSolrClient should not throw "Collection not …

1fb376b

…found" with an out-dated ClusterState (apache#2363)" This reverts commit 5c399dd.

github-actions Bot added jetty-server client:solrj cat:api labels Apr 5, 2024

gerlowskija added 2 commits September 14, 2025 15:46

Revert "Revert "SOLR-17153: CloudSolrClient should not throw "Collect…

459ecf6

…ion not found" with an out-dated ClusterState (apache#2363)"" This reverts commit 1fb376b.

Merge branch 'main' into spike-zk-quorum

de59401

gerlowskija force-pushed the spike-zk-quorum branch from 88c8da4 to de59401 Compare September 14, 2025 22:49

github-actions Bot removed jetty-server client:solrj cat:api labels Sep 14, 2025

epugh mentioned this pull request Sep 16, 2025

SOLR-17492: Introduce recommendations of WAYS of running Solr from small to massive #2783

Open

epugh added 2 commits October 13, 2025 19:24

Merge remote-tracking branch 'upstream/main' into pr/2391

ff0a9ca

Take advantage of Solr node roles to determine when to start embedded zk

cd6ecc5

Strip out old log4j workaround not needed, look at ide warnings.

9e6ef68

github-actions Bot added the cat:cloud label Oct 14, 2025

check in some work to be removed

479e85f

github-actions Bot added the tool:build label Oct 14, 2025

janhoy added 2 commits October 16, 2025 01:28

Merge branch 'refs/heads/main' into spike-zk-quorum

0601701

# Conflicts: # solr/core/src/java/org/apache/solr/core/ZkContainer.java # solr/packaging/build.gradle

Properly clean up ZK server resources

27ea8e9

epugh reviewed Oct 16, 2025

View reviewed changes

Comment thread solr/core/src/java/org/apache/solr/core/ZkContainer.java

epugh reviewed Oct 16, 2025

View reviewed changes

Comment thread solr/core/src/java/org/apache/solr/core/ZkContainer.java

janhoy added 2 commits October 16, 2025 09:18

Fix precommit in ZkContainer

6bd62ec

New test TestEmbeddedZkQuorum

8ee628d

Uses a base class SolrCloudWithEmbeddedZkQuorumTestCase New MiniSolrCloudCluster constructor that spins up a quorum cluster

janhoy reviewed May 11, 2026

View reviewed changes

Comment thread solr/core/src/java/org/apache/solr/core/ZkContainer.java Outdated

janhoy reviewed May 11, 2026

View reviewed changes

Comment thread solr/core/src/java/org/apache/solr/core/ZkContainer.java Outdated

janhoy reviewed May 11, 2026

View reviewed changes

Comment thread solr/packaging/build.gradle Outdated

janhoy added 5 commits May 11, 2026 10:06

SOLR-18094: Make zkServerEmbedded field volatile

63dc879

SOLR-18094: Wait for ZK quorum threads to stop after embedded ZK shut…

8b25153

…down

SOLR-18094: Validate zkHost entry format when starting quorum mode

5adaaef

SOLR-18094: Revert unrelated changes from libs.versions.toml and pack…

442e3c9

…aging/build.gradle

SOLR-18094: Add ref-guide documentation for zookeeper_quorum node rol…

4228fa8

…e (experimental)

github-actions Bot added documentation Improvements or additions to documentation and removed dependencies Dependency upgrades tool:build labels May 11, 2026

janhoy added 7 commits May 11, 2026 10:27

Add major-changes doc header

0cc2d38

SOLR-18094: Add null check for zkThread in SolrZkServer.stop()

d954146

SOLR-18094: zookeeper_quorum role alone is sufficient to start embedd…

47251b9

…ed ZK quorum

SOLR-18094: Log 'Closed embedded ZK' after thread-drain wait, not before

c573590

SOLR-18094: Add TODO comment explaining dead code in SolrZkServer.inj…

9db9a2c

…ectServers()

SOLR-18094: Pass Properties directly to startZooKeeperServerEmbedded,…

829ec2f

… avoid disk roundtrip

SOLR-18094: Add zookeeper_quorum to node-roles API example output

1e1f7f2

janhoy reviewed May 11, 2026

View reviewed changes

Comment thread solr/core/src/java/org/apache/solr/core/ZkContainer.java Outdated

janhoy reviewed May 11, 2026

View reviewed changes

Comment thread solr/core/src/java/org/apache/solr/core/ZkContainer.java Outdated

janhoy reviewed May 11, 2026

View reviewed changes

Comment thread solr/core/src/java/org/apache/solr/core/ZkContainer.java Outdated

janhoy reviewed May 11, 2026

View reviewed changes

Avoid zoo_home collision on same host

ccf6ec3

janhoy reviewed May 11, 2026

View reviewed changes

epugh added 2 commits May 12, 2026 16:01

Eliminate SolrZkServer in favour of using ZooKeeperServerEmbedded

f281185

Much nicer to read.

fe81c20

janhoy reviewed May 13, 2026

View reviewed changes

Better solution for zoo_home configuration in the example

f51ee69

Conversation

gerlowskija commented Apr 5, 2024 • edited by janhoy Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Solution

Tests

Checklist

Uh oh!

gerlowskija commented Apr 5, 2024

Uh oh!

janhoy commented Apr 29, 2024

Uh oh!

gerlowskija commented Sep 15, 2025

Uh oh!

epugh commented Oct 14, 2025

Uh oh!

janhoy commented Oct 15, 2025

Uh oh!

Uh oh!

Uh oh!

epugh commented Oct 16, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

janhoy commented May 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gerlowskija commented Apr 5, 2024 •

edited by janhoy

Loading

janhoy commented May 11, 2026 •

edited

Loading