HIVE-29391: Enable Independent Scaling of HMS REST Catalog from HMS by difin · Pull Request #6270 · apache/hive

difin · 2026-01-14T17:19:32Z

What changes were proposed in this pull request?

Adds a standalone HMS REST Catalog Server that can scale independently from HMS.
Currently, HMS REST Catalog Server is tied to HMS and can only be started together with HMS in a single instance.
This PR introduces a standalone REST Catalog server that:

Runs independently of HMS in a separate JVM/process
Connects to external HMS instances via Thrift
Supports horizontal scaling through Kubernetes load balancing
Provides a health check endpoint for Kubernetes readiness/liveness probes
Reuses the same REST Catalog servlet code used in embedded HMS mode

Architecture:
Client → Kubernetes Load Balancer/API Gateway → Standalone REST Catalog Server → HMS

Why are the changes needed?

Allows independent scaling of the HMS REST Catalog Server from HMS, enabling:

Horizontal Scaling: Scale REST Catalog instances based on REST API load, independent of HMS load
Resource Optimization: Allocate resources separately for REST API serving vs. metadata operations
Deployment Flexibility: Deploy REST Catalog servers separately from HMS
Cloud-Native Deployment: Leverage Kubernetes for load balancing, health checks, and auto-scaling

Does this PR introduce any user-facing change?

Yes. Adds a new standalone server mode:

New Class: StandaloneRESTCatalogServer - can be run as a standalone application
Backward Compatible: Embedded HMS REST Catalog mode remains unchanged and continues to work as before
Port Conflict Detection: Enhanced error messages when port conflicts occur between embedded and standalone modes

How was this patch tested?

New integration tests.

okumin · 2026-01-14T23:29:13Z

The intention sounds great! I have one challenge: Do we really need active-passive with ZooKeeper? RESTful API should always be able to make use of a load balancer, whose configuration is typically easier than ZK.

difin · 2026-01-15T00:25:56Z

@okumin Active-passive mode is not necessary for scaling, but active-active seems to what is needed. I used Zookeeper for consistency and code reuse, because it is already used in several places in Hive.

deniskuzZ · 2026-01-15T00:32:09Z

@okumin Active-passive mode is not necessary for scaling, but active-active seems to what is needed. I used Zookeeper for consistency and code reuse, because it is already used in several places in Hive.

@difin FYI there is an ongoing work to decommission Zookeeper.
cc @abstractdog

btw, why do we need coordinator here? i would envision the following flow:
Client → Load Balancer / API Gateway → HMS REST instance

abstractdog · 2026-01-15T06:48:38Z

g work to decommission Zookeeper.

@okumin Active-passive mode is not necessary for scaling, but active-active seems to what is needed. I used Zookeeper for consistency and code reuse, because it is already used in several places in Hive.

@difin FYI there is an ongoing work to decommission Zookeeper. cc @abstractdog

btw, why do we need coordinator here? i would envision the following flow: Client → Load Balancer / API Gateway → HMS REST instance

while we're looking for native kubernetes alternatives for things we're currently doing with ZK, ZK is still a valid choice, as getting rid of it in the whole hive codebase would be too much in one go, especially because it's battle-tested, so reusing ZkRegistryBase is fine now (not to mention that hive still runs in old-school clusters with ZK nowadays in many places I guess)

okumin · 2026-01-15T08:34:35Z

Client → Load Balancer / API Gateway → HMS REST instance

I also think this is more than enough in most cases.

difin · 2026-01-19T22:19:09Z

Thanks everyone! I am changing the implementation to support this flow, without Zookeeper:

Client → Load Balancer / API Gateway → HMS REST instance

okumin

+1

okumin · 2026-02-08T01:44:27Z

...etastore/metastore-rest-catalog/src/main/java/org/apache/iceberg/rest/HMSCatalogFactory.java

    if (configExtWarehouse != null) {
      properties.put("external-warehouse", configExtWarehouse);
+      // HiveCatalog reads this property directly from Configuration, not from properties map
+      configuration.set("hive.metastore.warehouse.external.dir", configExtWarehouse);


You might not have an answer. I'm just curious what makes the drift between Iceberg's HiveCatalog and our HiveCatalog.

hive/iceberg/iceberg-catalog/src/main/java/org/apache/iceberg/hive/HiveCatalog.java

Lines 750 to 751 in 7b54334

database.setLocationUri(new Path(getExternalWarehouseLocation(), namespace.level(0)).toString() + ".db");

database.setManagedLocationUri(databaseLocation(namespace.level(0)));

https://github.com/apache/iceberg/blob/91902a87970ff8841e99e6fb3f27f583f7645a6b/hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java#L772

okumin · 2026-02-08T01:58:47Z

...st-catalog/src/main/java/org/apache/iceberg/rest/standalone/StandaloneRESTCatalogServer.java

+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0


Please use asf.header at this point. I locally have a follow-up patch of HIVE-29245, but it's not complete.
https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6270&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true

...st-catalog/src/main/java/org/apache/iceberg/rest/standalone/StandaloneRESTCatalogServer.java

deniskuzZ · 2026-02-11T19:51:54Z

...st-catalog/src/main/java/org/apache/iceberg/rest/standalone/StandaloneRESTCatalogServer.java

+ * 
+ * <p>Multiple instances can run behind a Kubernetes Service for load balancing.
+ */
+public class StandaloneRESTCatalogServer {


would be nice to use spring-boot or something, but that's not a blocker

...etastore/metastore-rest-catalog/src/main/java/org/apache/iceberg/rest/HMSCatalogFactory.java

deniskuzZ

LGTM, some minor things

sonarqubecloud · 2026-02-15T03:05:12Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

asf-ci-hive added tests pending tests failed and removed tests pending labels Jan 14, 2026

difin force-pushed the hms_rest_catalog_server_scaling branch from f846f08 to d02fe25 Compare January 14, 2026 18:10

asf-ci-hive added tests pending and removed tests failed labels Jan 14, 2026

asf-ci-hive added tests unstable and removed tests pending labels Jan 15, 2026

difin force-pushed the hms_rest_catalog_server_scaling branch from fc55364 to 139244e Compare January 22, 2026 22:43

asf-ci-hive added tests pending tests failed and removed tests unstable tests pending labels Jan 22, 2026

difin force-pushed the hms_rest_catalog_server_scaling branch from 139244e to de28c69 Compare January 24, 2026 01:54

asf-ci-hive added tests pending tests passed and removed tests failed tests pending labels Jan 24, 2026

difin marked this pull request as ready for review January 24, 2026 04:34

difin requested a review from deniskuzZ January 26, 2026 15:20

difin force-pushed the hms_rest_catalog_server_scaling branch from de28c69 to b20f5e7 Compare January 26, 2026 18:25

asf-ci-hive added tests pending and removed tests passed tests pending labels Jan 26, 2026

asf-ci-hive added tests unstable and removed tests pending labels Jan 27, 2026

HIVE-29391: Enable Independent Scaling of HMS REST Catalog from HMS

d00bf81

difin force-pushed the hms_rest_catalog_server_scaling branch from 7afe72b to d00bf81 Compare January 28, 2026 15:11

asf-ci-hive added tests pending tests passed and removed tests unstable tests pending labels Jan 28, 2026

okumin approved these changes Feb 8, 2026

View reviewed changes

deniskuzZ reviewed Feb 11, 2026

View reviewed changes

...st-catalog/src/main/java/org/apache/iceberg/rest/standalone/StandaloneRESTCatalogServer.java Show resolved Hide resolved

deniskuzZ reviewed Feb 11, 2026

View reviewed changes

...etastore/metastore-rest-catalog/src/main/java/org/apache/iceberg/rest/HMSCatalogFactory.java Show resolved Hide resolved

deniskuzZ approved these changes Feb 11, 2026

View reviewed changes

asf-ci-hive added tests pending and removed tests passed labels Feb 13, 2026

difin force-pushed the hms_rest_catalog_server_scaling branch from a233585 to a3ce8ce Compare February 13, 2026 19:36

asf-ci-hive added tests failed tests pending tests unstable and removed tests pending tests failed labels Feb 13, 2026

code review comments - Feb 13.

4c81548

difin force-pushed the hms_rest_catalog_server_scaling branch from a3ce8ce to 4c81548 Compare February 15, 2026 01:43

asf-ci-hive added tests pending and removed tests unstable labels Feb 15, 2026

asf-ci-hive added tests passed and removed tests pending labels Feb 15, 2026

deniskuzZ merged commit 399200a into apache:master Feb 15, 2026
2 checks passed

	database.setLocationUri(new Path(getExternalWarehouseLocation(), namespace.level(0)).toString() + ".db");
	database.setManagedLocationUri(databaseLocation(namespace.level(0)));

Conversation

difin commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

okumin commented Jan 14, 2026

Uh oh!

difin commented Jan 15, 2026

Uh oh!

deniskuzZ commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

abstractdog commented Jan 15, 2026

Uh oh!

okumin commented Jan 15, 2026

Uh oh!

difin commented Jan 19, 2026

Uh oh!

okumin left a comment

Choose a reason for hiding this comment

Uh oh!

okumin Feb 8, 2026

Choose a reason for hiding this comment

Uh oh!

okumin Feb 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

deniskuzZ Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

deniskuzZ left a comment

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud bot commented Feb 15, 2026

Quality Gate passed

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

difin commented Jan 14, 2026 •

edited

Loading

deniskuzZ commented Jan 15, 2026 •

edited

Loading