[feature] Support custom HTTP headers for REST Iceberg catalog via header.* properties #61388
Description
Feature Request: Support header.* Properties for REST Iceberg Catalog
Summary
Apache Doris should support passing custom HTTP headers to the Iceberg REST catalog client via header.<Header-Name> catalog properties. This is part of the standard Iceberg REST catalog specification and is required to integrate with catalogs such as GCP BigLake Metastore that mandate custom headers for authentication and routing.
Motivation and Use Case
GCP BigLake Metastore
Google Cloud BigLake Metastore exposes an Iceberg REST catalog endpoint but requires callers to include a x-goog-user-project header on every request to identify the billing project. Without this header, requests are rejected.
The standard way to supply this in the Iceberg ecosystem is via the header.x-goog-user-project catalog property, which the Iceberg REST client translates into an HTTP header on all outgoing requests.
Example of the expected Doris CREATE CATALOG syntax once this feature is supported:
CREATE CATALOG biglake_catalog
PROPERTIES (
'type' = 'iceberg',
'iceberg.catalog.type' = 'rest',
'iceberg.rest.uri' = 'https://biglake.googleapis.com/iceberg/v1/restcatalog',
'warehouse' = 'gs://<warehouse-name>',
'iceberg.rest.header.x-goog-user-project' = '<BILLING_PROJECT>'
);Without header.* support, it is impossible to use Doris with BigLake Metastore (and any other REST catalog that relies on custom request headers for auth/routing).
Standard Iceberg Behaviour
The header.* property prefix is a first-class convention in the Iceberg REST catalog client (org.apache.iceberg.rest.RESTCatalog). Any property whose key starts with header. is stripped of the prefix and injected as an HTTP header on every request the client makes.
This behaviour is documented and used across the Iceberg ecosystem:
- Apache Iceberg Java / Spark –
spark.sql.catalog.my_catalog.header.X-Catalog-Name=default - PyIceberg –
header.x-goog-user-project,header.X-Iceberg-Access-Delegation, etc. (docs) - Iceberg Rust client – recently added the same feature
Root Cause
Doris appears to pass only an explicitly enumerated set of properties to the underlying RESTCatalog instance rather than forwarding all catalog properties. As a result, header.* keys are silently dropped before reaching the Iceberg client, even though the client itself is capable of processing them.
The fix is to ensure that all catalog properties (or at minimum those with a header. prefix) are forwarded to the restProperties map that is handed to the Iceberg RESTCatalog.
Expected Behaviour
Any header.<Name> = <value> property specified in CREATE CATALOG ... PROPERTIES (...) should be forwarded to the Iceberg REST catalog client and sent as the HTTP header <Name>: <value> on all requests to the REST catalog endpoint.
Alternatives Considered
- Network-layer proxy – injecting headers via a sidecar/reverse-proxy is possible but fragile, adds operational complexity, and should not be required for a standard protocol feature.
- OAuth2 / SigV4 – these are separate authentication mechanisms and do not cover arbitrary header requirements such as billing-project routing.
Environment
- Apache Doris version: 3.x / 4.x (latest)
- Feature affects: Iceberg REST catalog integration (
iceberg.catalog.type = 'rest')
References
- Apache Iceberg REST catalog spec – header properties
- PyIceberg configuration docs
- GCP BigLake Metastore Iceberg REST API
- Related Doris issue: [Bug] S3 properties not work wher create iceberg catalog #23671 (property pass-through for REST catalog)