Skip to content

Commit 3216019

Browse files
committed
sea-napi-binding: address review — sql-kernel rename, loader class + DI seam, packaging, tests
Rebuilds native/sea against the merged kernel main (binding renamed @databricks/sea-native -> @databricks/sql-kernel via kernel #82) and addresses the /full-review findings on PR #380: - C1: commit the napi-rs router (native/sea/index.js, un-ignored) + a prepack assertion so the publish tarball can never ship without it. Companion kernel fix (databricks-sql-kernel#93) corrects the base package name so the router require paths resolve. - C2: drop the @sea-native tsconfig path alias; use a relative import so no unresolvable specifier leaks into the emitted .d.ts. - C3/C11: error hints no longer name a 404-ing package; dlopen hint now includes the underlying dlerror string + concrete remediation. - C4: document M0 = linux-x64-gnu-only scope + npm scope-lock note. - C5: SeaNativeBinding = typeof import('../../native/sea') (no drift). - C6: SeaNativeLoader is now a class with an injectable load seam; getSeaNative/tryGetSeaNative are thin process-global wrappers. - C7: re-exports renamed Sea* to avoid colliding with Thrift types. - C8: version.test.ts fails loud on the linux-x64 CI runner + shape checks; new loader.test.ts covers the hint branches, Node gate, shape check, and caching via the injected seam. - C9: Node-version guard fails closed (NaN or < floor). - C13: e2e smoke uses the shared tests/e2e/utils/config.ts creds. - C10/C12 are resolved upstream in merged kernel main / deferred to M1. index.d.ts regenerated from merged main: ExecuteOptions dropped (catalog/schema/sessionConf are session-level on openSession), close() awaits DeleteSession, schema() is sync, sessionId/statementId getters added. Verified: 11 unit tests; e2e SELECT 1 against a live warehouse, direct and through mitmproxy (SEA REST: POST /sql/sessions -> POST /sql/statements -> DELETE /sql/sessions/<id>). Co-authored-by: Isaac
1 parent 548a14b commit 3216019

12 files changed

Lines changed: 913 additions & 178 deletions

File tree

.gitattributes

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -32,8 +32,9 @@ Dockerfile* text
3232
.gitattributes export-ignore
3333
.gitignore export-ignore
3434

35-
# napi-rs auto-generates this file from the kernel's `napi-binding/napi/`
36-
# crate; regenerated by `npm run build:native`. Tell git/GitHub it's
37-
# machine-generated so it collapses in diffs and is excluded from
35+
# napi-rs auto-generates these files from the kernel's `napi-binding/napi/`
36+
# crate; regenerated by `npm run build:native`. Tell git/GitHub they're
37+
# machine-generated so they collapse in diffs and are excluded from
3838
# blame and language stats.
3939
native/sea/index.d.ts linguist-generated=true
40+
native/sea/index.js linguist-generated=true

.gitignore

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,10 @@ dist
1212
lib/version.ts
1313

1414
# SEA native binding — copied/generated from kernel workspace by `npm run build:native`.
15-
# The committed contract is `native/sea/index.d.ts` (TypeScript declarations).
16-
# Everything else under native/sea/ is a build artifact and must not be committed.
17-
native/sea/index.js
15+
# The committed contract is `native/sea/index.d.ts` (TypeScript declarations) and
16+
# `native/sea/index.js` (the napi-rs platform router — small, stable, and required in
17+
# the publish tarball so a missing build step can't ship a tarball that can't load).
18+
# The `.node` binaries are large per-platform artifacts and must NOT be committed;
19+
# in production they arrive via the `@databricks/sql-kernel-<triple>` optional deps.
1820
native/sea/index.node
1921
native/sea/index.*.node

.npmignore

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
!thrift/**/*
55

66
# SEA napi-rs router shim + TypeScript declarations. The router (index.js)
7-
# selects the per-platform `.node` artifact from `@databricks/sea-native-*`
7+
# selects the per-platform `.node` artifact from `@databricks/sql-kernel-*`
88
# optionalDependencies (populated when the kernel CI publishes them);
99
# the .d.ts is the consumer-facing type contract.
1010
!native/sea/index.js

lib/sea/SeaNativeLoader.ts

Lines changed: 149 additions & 57 deletions
Original file line numberDiff line numberDiff line change
@@ -17,31 +17,42 @@
1717
*
1818
* Mirrors the load-failure-tolerant pattern of `lib/utils/lz4.ts`: the
1919
* `.node` artifact ships via per-platform optional dependencies
20-
* (`@databricks/sea-native-<triple>`), so its absence must not crash
20+
* (`@databricks/sql-kernel-<triple>`), so its absence must not crash
2121
* a Thrift-only consumer of the driver. Callers that actually need
22-
* SEA invoke `getSeaNative()`, which throws a structured error if
23-
* the binding could not be loaded.
22+
* SEA construct a {@link SeaNativeLoader} (or use the process-global
23+
* {@link getSeaNative}) which throws a structured error if the binding
24+
* could not be loaded.
25+
*
26+
* M0 publishes a single triple (`linux-x64-gnu`); see
27+
* `native/sea/README.md` for the supported-platform policy.
2428
*/
2529

2630
import type {
2731
Connection as NativeConnection,
2832
Statement as NativeStatement,
29-
ConnectionOptions,
30-
ExecuteOptions,
31-
ArrowBatch,
32-
ArrowSchema,
33-
} from '@sea-native';
34-
35-
export type { ConnectionOptions, ExecuteOptions, ArrowBatch, ArrowSchema };
36-
export type Connection = NativeConnection;
37-
export type Statement = NativeStatement;
38-
39-
export interface SeaNativeBinding {
40-
version(): string;
41-
openSession(options: ConnectionOptions): Promise<NativeConnection>;
42-
Connection: typeof NativeConnection;
43-
Statement: typeof NativeStatement;
44-
}
33+
ConnectionOptions as NativeConnectionOptions,
34+
ArrowBatch as NativeArrowBatch,
35+
ArrowSchema as NativeArrowSchema,
36+
} from '../../native/sea';
37+
38+
// SEA-prefixed re-exports. The kernel-generated `.d.ts` keeps the
39+
// napi-rs default names (`ConnectionOptions`, `ArrowBatch`, …); we
40+
// disambiguate on the TS-wrapper side so these never collide with the
41+
// Thrift-side `ConnectionOptions` (lib/contracts/IDBSQLClient.ts) or
42+
// `ArrowBatch` (lib/result/utils.ts) when imported elsewhere.
43+
export type SeaConnectionOptions = NativeConnectionOptions;
44+
export type SeaArrowBatch = NativeArrowBatch;
45+
export type SeaArrowSchema = NativeArrowSchema;
46+
export type SeaConnection = NativeConnection;
47+
export type SeaStatement = NativeStatement;
48+
49+
/**
50+
* The full native binding surface, derived from the generated module
51+
* so it can never drift from the `.d.ts` contract: when the kernel
52+
* adds or renames a free function / class, this type follows
53+
* automatically and `defaultRequire`'s cast stays correct.
54+
*/
55+
export type SeaNativeBinding = typeof import('../../native/sea');
4556

4657
const MIN_NODE_MAJOR = 18;
4758

@@ -50,68 +61,149 @@ function detectNodeMajor(): number {
5061
return parseInt(process.version.slice(1), 10);
5162
}
5263

64+
function platformLabel(): string {
65+
return `${process.platform}-${process.arch}`;
66+
}
67+
5368
function loadFailureHint(err: NodeJS.ErrnoException): string {
54-
const platform = `${process.platform}-${process.arch}`;
55-
const installHint = `Install the matching optional dependency (e.g. @databricks/sea-native-${platform}).`;
69+
const platform = platformLabel();
70+
// Do not name a concrete package: the published name uses the napi-rs
71+
// triple (e.g. `-linux-x64-gnu` / `-linux-x64-musl` / `-win32-x64-msvc`),
72+
// not the bare `${platform}` shown here, so a literal example would
73+
// 404. Point at the README's supported-triple list instead.
74+
const installHint =
75+
'Install the matching @databricks/sql-kernel-* optional dependency for your platform ' +
76+
'(see native/sea/README.md for the supported triples; M0 ships linux-x64-gnu only).';
5677
if (err.code === 'MODULE_NOT_FOUND') {
5778
return `SEA native binding not installed for platform ${platform} on Node ${process.version}. ${installHint}`;
5879
}
5980
if (err.code === 'ERR_DLOPEN_FAILED') {
60-
return `SEA native binding present but failed to dlopen on platform ${platform} / Node ${process.version} — likely a libc or Node ABI mismatch. The binding requires Node >=${MIN_NODE_MAJOR}.`;
81+
// Surface the underlying dlerror string (e.g. `GLIBC_2.32 not found`)
82+
// plus concrete remediation — without it the cause is invisible.
83+
return (
84+
`SEA native binding present but failed to dlopen on platform ${platform} / Node ${process.version}: ` +
85+
`${err.message}. Common causes: glibc/musl mismatch (e.g. Alpine Linux — install the -musl variant), ` +
86+
`Node ABI mismatch (try \`rm -rf node_modules && npm install\`), or CPU-architecture mismatch. ` +
87+
`The binding requires Node >=${MIN_NODE_MAJOR}.`
88+
);
6189
}
6290
return `SEA native binding failed to load on platform ${platform} / Node ${process.version}: ${err.message}`;
6391
}
6492

65-
let cached: SeaNativeBinding | null | undefined;
66-
let cachedError: Error | undefined;
93+
/**
94+
* Default loader: resolves `native/sea/index.js` (the napi-rs router),
95+
* which selects the per-platform `.node`. `.js` is omitted so eslint's
96+
* `import/extensions` rule accepts the call.
97+
*/
98+
function defaultRequire(): SeaNativeBinding {
99+
// eslint-disable-next-line @typescript-eslint/no-var-requires, global-require
100+
return require('../../native/sea') as SeaNativeBinding;
101+
}
67102

68-
function tryLoad(): SeaNativeBinding | undefined {
69-
const nodeMajor = detectNodeMajor();
70-
if (Number.isFinite(nodeMajor) && nodeMajor < MIN_NODE_MAJOR) {
71-
cachedError = new Error(
72-
`SEA native binding requires Node >=${MIN_NODE_MAJOR}; running Node ${process.version}. Continue using the Thrift backend on this runtime.`,
103+
/**
104+
* Verify the loaded module exposes the surface the driver depends on.
105+
* Catches kernel-side renames at load time rather than letting them
106+
* surface as `undefined is not a function` deep in a call path.
107+
*/
108+
function assertBindingShape(binding: SeaNativeBinding): void {
109+
const missing: string[] = [];
110+
if (typeof binding.version !== 'function') missing.push('version');
111+
if (typeof binding.openSession !== 'function') missing.push('openSession');
112+
if (typeof binding.Connection !== 'function') missing.push('Connection');
113+
if (typeof binding.Statement !== 'function') missing.push('Statement');
114+
if (missing.length > 0) {
115+
throw new Error(
116+
`SEA native binding loaded but is missing expected export(s): ${missing.join(', ')}. ` +
117+
`The kernel-generated binding and the JS loader are out of sync.`,
73118
);
74-
return undefined;
75119
}
120+
}
121+
122+
/**
123+
* Loads and caches the SEA native binding. Exposed as a class with an
124+
* injectable `load` seam so consumers (e.g. `SeaBackend`) can be unit
125+
* tested with a stub binding instead of requiring a real `.node` on the
126+
* test machine. Most production code uses the process-global default
127+
* via {@link getSeaNative} / {@link tryGetSeaNative}.
128+
*/
129+
export class SeaNativeLoader {
130+
private cached: SeaNativeBinding | null | undefined;
131+
132+
private cachedError: Error | undefined;
133+
134+
constructor(private readonly load: () => SeaNativeBinding = defaultRequire) {}
76135

77-
try {
78-
// The require path resolves to `native/sea/index.js` (the napi-rs
79-
// router). `.js` is omitted so eslint's `import/extensions` rule
80-
// accepts the call.
81-
// eslint-disable-next-line @typescript-eslint/no-var-requires, global-require
82-
return require('../../native/sea') as SeaNativeBinding;
83-
} catch (err) {
84-
if (err instanceof Error && 'code' in err) {
85-
cachedError = new Error(loadFailureHint(err as NodeJS.ErrnoException));
136+
private tryLoad(): SeaNativeBinding | undefined {
137+
const nodeMajor = detectNodeMajor();
138+
// Fail closed: if we cannot determine the Node major (NaN) or it is
139+
// below the floor, refuse the load and fall back to Thrift.
140+
if (!Number.isFinite(nodeMajor) || nodeMajor < MIN_NODE_MAJOR) {
141+
this.cachedError = new Error(
142+
`SEA native binding requires Node >=${MIN_NODE_MAJOR}; running Node ${process.version}. ` +
143+
`Continue using the Thrift backend on this runtime.`,
144+
);
145+
return undefined;
146+
}
147+
148+
try {
149+
const binding = this.load();
150+
assertBindingShape(binding);
151+
return binding;
152+
} catch (err) {
153+
if (err instanceof Error && 'code' in err) {
154+
this.cachedError = new Error(loadFailureHint(err as NodeJS.ErrnoException));
155+
} else if (err instanceof Error) {
156+
// Shape-check failure or any other Error — preserve its message.
157+
this.cachedError = err;
158+
} else {
159+
this.cachedError = new Error(`SEA native binding failed to load with non-standard error: ${String(err)}`);
160+
}
86161
return undefined;
87162
}
88-
cachedError = new Error(`SEA native binding failed to load with non-standard error: ${String(err)}`);
89-
return undefined;
163+
}
164+
165+
/**
166+
* Returns the loaded native binding. Throws a structured error if the
167+
* binding is unavailable on this platform / Node version.
168+
*/
169+
get(): SeaNativeBinding {
170+
if (this.cached === undefined) {
171+
this.cached = this.tryLoad() ?? null;
172+
}
173+
if (this.cached === null) {
174+
throw this.cachedError ?? new Error('SEA native binding unavailable');
175+
}
176+
return this.cached;
177+
}
178+
179+
/**
180+
* Returns the loaded binding or `undefined` if it could not be
181+
* loaded. Use this for capability-detection at startup; use
182+
* {@link get} at the point where SEA is actually required.
183+
*/
184+
tryGet(): SeaNativeBinding | undefined {
185+
if (this.cached === undefined) {
186+
this.cached = this.tryLoad() ?? null;
187+
}
188+
return this.cached ?? undefined;
90189
}
91190
}
92191

192+
// Process-global default instance + thin convenience wrappers.
193+
const defaultLoader = new SeaNativeLoader();
194+
93195
/**
94-
* Returns the loaded native binding. Throws a structured error if
95-
* the binding is unavailable on this platform / Node version.
196+
* Returns the loaded native binding from the process-global loader.
197+
* Throws a structured error if the binding is unavailable.
96198
*/
97199
export function getSeaNative(): SeaNativeBinding {
98-
if (cached === undefined) {
99-
cached = tryLoad() ?? null;
100-
}
101-
if (cached === null) {
102-
throw cachedError ?? new Error('SEA native binding unavailable');
103-
}
104-
return cached;
200+
return defaultLoader.get();
105201
}
106202

107203
/**
108-
* Returns the loaded binding or `undefined` if it could not be
109-
* loaded. Use this for capability-detection at startup; use
110-
* `getSeaNative()` at the point where SEA is actually required.
204+
* Returns the loaded binding from the process-global loader, or
205+
* `undefined` if it could not be loaded.
111206
*/
112207
export function tryGetSeaNative(): SeaNativeBinding | undefined {
113-
if (cached === undefined) {
114-
cached = tryLoad() ?? null;
115-
}
116-
return cached ?? undefined;
208+
return defaultLoader.tryGet();
117209
}

native/sea/README.md

Lines changed: 35 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,15 @@
11
# `native/sea/` — consumer-side directory for the Rust napi binding
22

33
**The Rust binding source lives in the kernel repo** at
4-
`databricks-sql-kernel/napi-binding/napi/`. Building it requires a
5-
local checkout of that repo — see "Build for local dev" below.
4+
`databricks-sql-kernel/napi/`. Building it requires a local checkout
5+
of that repo — see "Build for local dev" below. The published npm
6+
package is `@databricks/sql-kernel-<triple>`.
67

78
## Workspace topology
89

910
The napi crate is a **standalone Cargo workspace** (`[workspace]
10-
members = ["."]` in `napi-binding/napi/Cargo.toml`), **not** a
11-
sibling of `pyo3/` in the kernel root workspace.
11+
members = ["."]` in `napi/Cargo.toml`), **not** a sibling of `pyo3/`
12+
in the kernel root workspace.
1213

1314
The reason is Cargo feature unification. pyo3 builds the kernel with
1415
the default `tls-native` feature (system OpenSSL via `native-tls`).
@@ -32,31 +33,55 @@ and reintroduce the same clash. Standalone-workspace is the fix.
3233
- `index.js` — napi-rs's per-platform router shim. Gitignored;
3334
populated by `npm run build:native` for local dev. In published
3435
tarballs it ships alongside the `.d.ts` and `require()`s the
35-
right `@databricks/sea-native-<triple>` optional dependency.
36+
right `@databricks/sql-kernel-<triple>` optional dependency.
3637
- `index.*.node` — the actual native binary, one per platform.
3738
Gitignored. In production these live in the per-triple optional
38-
dependencies (`@databricks/sea-native-linux-x64-gnu`, etc.); for
39+
dependencies (`@databricks/sql-kernel-linux-x64-gnu`, etc.); for
3940
local dev `npm run build:native` copies one into this directory.
4041

4142
## Build for local dev
4243

4344
```bash
4445
# From the nodejs repo root:
45-
export DATABRICKS_SQL_KERNEL_REPO=/path/to/your/databricks-sql-kernel/napi-binding
46+
export DATABRICKS_SQL_KERNEL_REPO=/path/to/your/databricks-sql-kernel
4647
npm run build:native # release build (default)
4748
BUILD_PROFILE= npm run build:native # debug build (empty BUILD_PROFILE drops --release)
4849
```
4950

50-
`DATABRICKS_SQL_KERNEL_REPO` is required when your kernel checkout
51-
isn't at `../../databricks-sql-kernel/napi-binding` relative to the
51+
`DATABRICKS_SQL_KERNEL_REPO` points at the kernel repo root (the
52+
directory containing `napi/`) and is required when your kernel
53+
checkout isn't at `../../databricks-sql-kernel` relative to the
5254
nodejs repo.
5355

5456
## Production load path
5557

5658
At release time the kernel's CI publishes
57-
`@databricks/sea-native-<triple>` npm packages — one per supported
59+
`@databricks/sql-kernel-<triple>` npm packages — one per supported
5860
platform — each containing a single `.node` binary. The nodejs
5961
driver lists them as `optionalDependencies`; npm installs only the
6062
one matching the consumer's `process.platform` / `process.arch`.
6163
`native/sea/index.js` (the napi-rs router) then `require()`s the
6264
installed package at load time.
65+
66+
## Supported platforms (M0)
67+
68+
M0 publishes a **single** triple: **`linux-x64-gnu`** (package
69+
`@databricks/sql-kernel-linux-x64-gnu`). It is the only entry in the
70+
driver's `optionalDependencies`.
71+
72+
On every other platform (macOS, Windows, linux-arm64, linux-x64-musl
73+
/ Alpine, …) the SEA binding is simply absent: `SeaNativeLoader`
74+
returns `undefined` from `tryGet()` / throws a structured
75+
`MODULE_NOT_FOUND` hint from `get()`, and the driver continues to use
76+
the Thrift backend exclusively. This is expected, not a regression —
77+
additional triples are added to `optionalDependencies` as the kernel
78+
CI starts publishing them in later milestones.
79+
80+
## Supply-chain note
81+
82+
The unpublished triple names (`@databricks/sql-kernel-darwin-arm64`,
83+
`…-win32-x64-msvc`, etc.) referenced by the router are **not**
84+
squat-able: `@databricks` is a Databricks-owned npm scope, and npm
85+
only allows org members to publish under a scope it owns. A third
86+
party therefore cannot register `@databricks/sql-kernel-*` and have
87+
the router autoload it. No placeholder packages are required.

0 commit comments

Comments
 (0)