Skip to content

Commit 5f1728a

Browse files
samikshya-dbclaude
andauthored
[4/7] Telemetry Event Emission and Aggregation (#327)
* Add telemetry infrastructure: CircuitBreaker and FeatureFlagCache This is part 2 of 7 in the telemetry implementation stack. Components: - CircuitBreaker: Per-host endpoint protection with state management - FeatureFlagCache: Per-host feature flag caching with reference counting - CircuitBreakerRegistry: Manages circuit breakers per host Circuit Breaker: - States: CLOSED (normal), OPEN (failing), HALF_OPEN (testing recovery) - Default: 5 failures trigger OPEN, 60s timeout, 2 successes to CLOSE - Per-host isolation prevents cascade failures - All state transitions logged at debug level Feature Flag Cache: - Per-host caching with 15-minute TTL - Reference counting for connection lifecycle management - Automatic cache expiration and refetch - Context removed when refCount reaches zero Testing: - 32 comprehensive unit tests for CircuitBreaker - 29 comprehensive unit tests for FeatureFlagCache - 100% function coverage, >80% line/branch coverage - CircuitBreakerStub for testing other components Dependencies: - Builds on [1/7] Types and Exception Classifier * Add authentication support for REST API calls Implements getAuthHeaders() method for authenticated REST API requests: - Added getAuthHeaders() to IClientContext interface - Implemented in DBSQLClient using authProvider.authenticate() - Updated FeatureFlagCache to fetch from connector-service API with auth - Added driver version support for version-specific feature flags - Replaced placeholder implementation with actual REST API calls Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Fix feature flag and telemetry export endpoints - Change feature flag endpoint to use NODEJS client type - Fix telemetry endpoints to /telemetry-ext and /telemetry-unauth - Update payload to match proto with system_configuration - Add shared buildUrl utility for protocol handling * Match JDBC telemetry payload format - Change payload structure to match JDBC: uploadTime, items, protoLogs - protoLogs contains JSON-stringified TelemetryFrontendLog objects - Remove workspace_id (JDBC doesn't populate it) - Remove debug logs added during testing * Fix lint errors - Fix import order in FeatureFlagCache - Replace require() with import for driverVersion - Fix variable shadowing - Disable prefer-default-export for urlUtils * Add telemetry infrastructure: CircuitBreaker and FeatureFlagCache This is part 2 of 7 in the telemetry implementation stack. Components: - CircuitBreaker: Per-host endpoint protection with state management - FeatureFlagCache: Per-host feature flag caching with reference counting - CircuitBreakerRegistry: Manages circuit breakers per host Circuit Breaker: - States: CLOSED (normal), OPEN (failing), HALF_OPEN (testing recovery) - Default: 5 failures trigger OPEN, 60s timeout, 2 successes to CLOSE - Per-host isolation prevents cascade failures - All state transitions logged at debug level Feature Flag Cache: - Per-host caching with 15-minute TTL - Reference counting for connection lifecycle management - Automatic cache expiration and refetch - Context removed when refCount reaches zero Testing: - 32 comprehensive unit tests for CircuitBreaker - 29 comprehensive unit tests for FeatureFlagCache - 100% function coverage, >80% line/branch coverage - CircuitBreakerStub for testing other components Dependencies: - Builds on [1/7] Types and Exception Classifier * Add telemetry client management: TelemetryClient and Provider This is part 3 of 7 in the telemetry implementation stack. Components: - TelemetryClient: HTTP client for telemetry export per host - TelemetryClientProvider: Manages per-host client lifecycle with reference counting TelemetryClient: - Placeholder HTTP client for telemetry export - Per-host isolation for connection pooling - Lifecycle management (open/close) - Ready for future HTTP implementation TelemetryClientProvider: - Reference counting tracks connections per host - Automatically creates clients on first connection - Closes and removes clients when refCount reaches zero - Thread-safe per-host management Design Pattern: - Follows JDBC driver pattern for resource management - One client per host, shared across connections - Efficient resource utilization - Clean lifecycle management Testing: - 31 comprehensive unit tests for TelemetryClient - 31 comprehensive unit tests for TelemetryClientProvider - 100% function coverage, >80% line/branch coverage - Tests verify reference counting and lifecycle Dependencies: - Builds on [1/7] Types and [2/7] Infrastructure * Add authentication support for REST API calls Implements getAuthHeaders() method for authenticated REST API requests: - Added getAuthHeaders() to IClientContext interface - Implemented in DBSQLClient using authProvider.authenticate() - Updated FeatureFlagCache to fetch from connector-service API with auth - Added driver version support for version-specific feature flags - Replaced placeholder implementation with actual REST API calls Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Fix feature flag and telemetry export endpoints - Change feature flag endpoint to use NODEJS client type - Fix telemetry endpoints to /telemetry-ext and /telemetry-unauth - Update payload to match proto with system_configuration - Add shared buildUrl utility for protocol handling * Match JDBC telemetry payload format - Change payload structure to match JDBC: uploadTime, items, protoLogs - protoLogs contains JSON-stringified TelemetryFrontendLog objects - Remove workspace_id (JDBC doesn't populate it) - Remove debug logs added during testing * Fix lint errors - Fix import order in FeatureFlagCache - Replace require() with import for driverVersion - Fix variable shadowing - Disable prefer-default-export for urlUtils * Add telemetry infrastructure: CircuitBreaker and FeatureFlagCache This is part 2 of 7 in the telemetry implementation stack. Components: - CircuitBreaker: Per-host endpoint protection with state management - FeatureFlagCache: Per-host feature flag caching with reference counting - CircuitBreakerRegistry: Manages circuit breakers per host Circuit Breaker: - States: CLOSED (normal), OPEN (failing), HALF_OPEN (testing recovery) - Default: 5 failures trigger OPEN, 60s timeout, 2 successes to CLOSE - Per-host isolation prevents cascade failures - All state transitions logged at debug level Feature Flag Cache: - Per-host caching with 15-minute TTL - Reference counting for connection lifecycle management - Automatic cache expiration and refetch - Context removed when refCount reaches zero Testing: - 32 comprehensive unit tests for CircuitBreaker - 29 comprehensive unit tests for FeatureFlagCache - 100% function coverage, >80% line/branch coverage - CircuitBreakerStub for testing other components Dependencies: - Builds on [1/7] Types and Exception Classifier * Add telemetry client management: TelemetryClient and Provider This is part 3 of 7 in the telemetry implementation stack. Components: - TelemetryClient: HTTP client for telemetry export per host - TelemetryClientProvider: Manages per-host client lifecycle with reference counting TelemetryClient: - Placeholder HTTP client for telemetry export - Per-host isolation for connection pooling - Lifecycle management (open/close) - Ready for future HTTP implementation TelemetryClientProvider: - Reference counting tracks connections per host - Automatically creates clients on first connection - Closes and removes clients when refCount reaches zero - Thread-safe per-host management Design Pattern: - Follows JDBC driver pattern for resource management - One client per host, shared across connections - Efficient resource utilization - Clean lifecycle management Testing: - 31 comprehensive unit tests for TelemetryClient - 31 comprehensive unit tests for TelemetryClientProvider - 100% function coverage, >80% line/branch coverage - Tests verify reference counting and lifecycle Dependencies: - Builds on [1/7] Types and [2/7] Infrastructure * Add telemetry event emission and aggregation This is part 4 of 7 in the telemetry implementation stack. Components: - TelemetryEventEmitter: Event-based telemetry emission using Node.js EventEmitter - MetricsAggregator: Per-statement aggregation with batch processing TelemetryEventEmitter: - Event-driven architecture using Node.js EventEmitter - Type-safe event emission methods - Respects telemetryEnabled configuration flag - All exceptions swallowed and logged at debug level - Zero impact when disabled Event Types: - connection.open: On successful connection - statement.start: On statement execution - statement.complete: On statement finish - cloudfetch.chunk: On chunk download - error: On exception with terminal classification MetricsAggregator: - Per-statement aggregation by statement_id - Connection events emitted immediately (no aggregation) - Statement events buffered until completeStatement() called - Terminal exceptions flushed immediately - Retryable exceptions buffered until statement complete - Batch size (default 100) triggers flush - Periodic timer (default 5s) triggers flush Batching Strategy: - Optimizes export efficiency - Reduces HTTP overhead - Smart flushing based on error criticality - Memory efficient with bounded buffers Testing: - 31 comprehensive unit tests for TelemetryEventEmitter - 32 comprehensive unit tests for MetricsAggregator - 100% function coverage, >90% line/branch coverage - Tests verify exception swallowing - Tests verify debug-only logging Dependencies: - Builds on [1/7] Types, [2/7] Infrastructure, [3/7] Client Management * Add authentication support for REST API calls Implements getAuthHeaders() method for authenticated REST API requests: - Added getAuthHeaders() to IClientContext interface - Implemented in DBSQLClient using authProvider.authenticate() - Updated FeatureFlagCache to fetch from connector-service API with auth - Added driver version support for version-specific feature flags - Replaced placeholder implementation with actual REST API calls Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Fix feature flag and telemetry export endpoints - Change feature flag endpoint to use NODEJS client type - Fix telemetry endpoints to /telemetry-ext and /telemetry-unauth - Update payload to match proto with system_configuration - Add shared buildUrl utility for protocol handling Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Match JDBC telemetry payload format - Change payload structure to match JDBC: uploadTime, items, protoLogs - protoLogs contains JSON-stringified TelemetryFrontendLog objects - Remove workspace_id (JDBC doesn't populate it) - Remove debug logs added during testing * Fix lint errors - Fix import order in FeatureFlagCache - Replace require() with import for driverVersion - Fix variable shadowing - Disable prefer-default-export for urlUtils * Add telemetry infrastructure: CircuitBreaker and FeatureFlagCache This is part 2 of 7 in the telemetry implementation stack. Components: - CircuitBreaker: Per-host endpoint protection with state management - FeatureFlagCache: Per-host feature flag caching with reference counting - CircuitBreakerRegistry: Manages circuit breakers per host Circuit Breaker: - States: CLOSED (normal), OPEN (failing), HALF_OPEN (testing recovery) - Default: 5 failures trigger OPEN, 60s timeout, 2 successes to CLOSE - Per-host isolation prevents cascade failures - All state transitions logged at debug level Feature Flag Cache: - Per-host caching with 15-minute TTL - Reference counting for connection lifecycle management - Automatic cache expiration and refetch - Context removed when refCount reaches zero Testing: - 32 comprehensive unit tests for CircuitBreaker - 29 comprehensive unit tests for FeatureFlagCache - 100% function coverage, >80% line/branch coverage - CircuitBreakerStub for testing other components Dependencies: - Builds on [1/7] Types and Exception Classifier * Add telemetry client management: TelemetryClient and Provider This is part 3 of 7 in the telemetry implementation stack. Components: - TelemetryClient: HTTP client for telemetry export per host - TelemetryClientProvider: Manages per-host client lifecycle with reference counting TelemetryClient: - Placeholder HTTP client for telemetry export - Per-host isolation for connection pooling - Lifecycle management (open/close) - Ready for future HTTP implementation TelemetryClientProvider: - Reference counting tracks connections per host - Automatically creates clients on first connection - Closes and removes clients when refCount reaches zero - Thread-safe per-host management Design Pattern: - Follows JDBC driver pattern for resource management - One client per host, shared across connections - Efficient resource utilization - Clean lifecycle management Testing: - 31 comprehensive unit tests for TelemetryClient - 31 comprehensive unit tests for TelemetryClientProvider - 100% function coverage, >80% line/branch coverage - Tests verify reference counting and lifecycle Dependencies: - Builds on [1/7] Types and [2/7] Infrastructure * Add telemetry event emission and aggregation This is part 4 of 7 in the telemetry implementation stack. Components: - TelemetryEventEmitter: Event-based telemetry emission using Node.js EventEmitter - MetricsAggregator: Per-statement aggregation with batch processing TelemetryEventEmitter: - Event-driven architecture using Node.js EventEmitter - Type-safe event emission methods - Respects telemetryEnabled configuration flag - All exceptions swallowed and logged at debug level - Zero impact when disabled Event Types: - connection.open: On successful connection - statement.start: On statement execution - statement.complete: On statement finish - cloudfetch.chunk: On chunk download - error: On exception with terminal classification MetricsAggregator: - Per-statement aggregation by statement_id - Connection events emitted immediately (no aggregation) - Statement events buffered until completeStatement() called - Terminal exceptions flushed immediately - Retryable exceptions buffered until statement complete - Batch size (default 100) triggers flush - Periodic timer (default 5s) triggers flush Batching Strategy: - Optimizes export efficiency - Reduces HTTP overhead - Smart flushing based on error criticality - Memory efficient with bounded buffers Testing: - 31 comprehensive unit tests for TelemetryEventEmitter - 32 comprehensive unit tests for MetricsAggregator - 100% function coverage, >90% line/branch coverage - Tests verify exception swallowing - Tests verify debug-only logging Dependencies: - Builds on [1/7] Types, [2/7] Infrastructure, [3/7] Client Management * Add telemetry export: DatabricksTelemetryExporter This is part 5 of 7 in the telemetry implementation stack. Components: - DatabricksTelemetryExporter: HTTP export with retry logic and circuit breaker - TelemetryExporterStub: Test stub for integration tests DatabricksTelemetryExporter: - Exports telemetry metrics to Databricks via HTTP POST - Two endpoints: authenticated (/api/2.0/sql/telemetry-ext) and unauthenticated (/api/2.0/sql/telemetry-unauth) - Integrates with CircuitBreaker for per-host endpoint protection - Retry logic with exponential backoff and jitter - Exception classification (terminal vs retryable) Export Flow: 1. Check circuit breaker state (skip if OPEN) 2. Execute with circuit breaker protection 3. Retry on retryable errors with backoff 4. Circuit breaker tracks success/failure 5. All exceptions swallowed and logged at debug level Retry Strategy: - Max retries: 3 (default, configurable) - Exponential backoff: 100ms * 2^attempt - Jitter: Random 0-100ms to prevent thundering herd - Terminal errors: No retry (401, 403, 404, 400) - Retryable errors: Retry with backoff (429, 500, 502, 503, 504) Circuit Breaker Integration: - Success → Record success with circuit breaker - Failure → Record failure with circuit breaker - Circuit OPEN → Skip export, log at debug - Automatic recovery via HALF_OPEN state Critical Requirements: - All exceptions swallowed (NEVER throws) - All logging at LogLevel.debug ONLY - No console logging - Driver continues when telemetry fails Testing: - 24 comprehensive unit tests - 96% statement coverage, 84% branch coverage - Tests verify exception swallowing - Tests verify retry logic - Tests verify circuit breaker integration - TelemetryExporterStub for integration tests Dependencies: - Builds on all previous layers [1/7] through [4/7] * Add authentication support for REST API calls Implements getAuthHeaders() method for authenticated REST API requests: - Added getAuthHeaders() to IClientContext interface - Implemented in DBSQLClient using authProvider.authenticate() - Updated FeatureFlagCache to fetch from connector-service API with auth - Added driver version support for version-specific feature flags - Replaced placeholder implementation with actual REST API calls Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Update DatabricksTelemetryExporter to use authenticated export - Use getAuthHeaders() method for authenticated endpoint requests - Remove TODO comments about missing authentication - Add auth headers when telemetryAuthenticatedExport is true Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Fix feature flag endpoint and telemetry export - Use NODEJS client type instead of OSS_NODEJS for feature flags - Use /telemetry-ext and /telemetry-unauth (not /api/2.0/sql/...) - Update payload to match proto: system_configuration with snake_case - Add URL utility to handle protocol correctly Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Match JDBC telemetry payload format - Change payload structure to match JDBC: uploadTime, items, protoLogs - protoLogs contains JSON-stringified TelemetryFrontendLog objects - Remove workspace_id (JDBC doesn't populate it) - Remove debug logs added during testing * Fix lint errors - Fix import order in FeatureFlagCache - Replace require() with import for driverVersion - Fix variable shadowing - Disable prefer-default-export for urlUtils * Add telemetry infrastructure: CircuitBreaker and FeatureFlagCache This is part 2 of 7 in the telemetry implementation stack. Components: - CircuitBreaker: Per-host endpoint protection with state management - FeatureFlagCache: Per-host feature flag caching with reference counting - CircuitBreakerRegistry: Manages circuit breakers per host Circuit Breaker: - States: CLOSED (normal), OPEN (failing), HALF_OPEN (testing recovery) - Default: 5 failures trigger OPEN, 60s timeout, 2 successes to CLOSE - Per-host isolation prevents cascade failures - All state transitions logged at debug level Feature Flag Cache: - Per-host caching with 15-minute TTL - Reference counting for connection lifecycle management - Automatic cache expiration and refetch - Context removed when refCount reaches zero Testing: - 32 comprehensive unit tests for CircuitBreaker - 29 comprehensive unit tests for FeatureFlagCache - 100% function coverage, >80% line/branch coverage - CircuitBreakerStub for testing other components Dependencies: - Builds on [1/7] Types and Exception Classifier * Add telemetry client management: TelemetryClient and Provider This is part 3 of 7 in the telemetry implementation stack. Components: - TelemetryClient: HTTP client for telemetry export per host - TelemetryClientProvider: Manages per-host client lifecycle with reference counting TelemetryClient: - Placeholder HTTP client for telemetry export - Per-host isolation for connection pooling - Lifecycle management (open/close) - Ready for future HTTP implementation TelemetryClientProvider: - Reference counting tracks connections per host - Automatically creates clients on first connection - Closes and removes clients when refCount reaches zero - Thread-safe per-host management Design Pattern: - Follows JDBC driver pattern for resource management - One client per host, shared across connections - Efficient resource utilization - Clean lifecycle management Testing: - 31 comprehensive unit tests for TelemetryClient - 31 comprehensive unit tests for TelemetryClientProvider - 100% function coverage, >80% line/branch coverage - Tests verify reference counting and lifecycle Dependencies: - Builds on [1/7] Types and [2/7] Infrastructure * Add telemetry event emission and aggregation This is part 4 of 7 in the telemetry implementation stack. Components: - TelemetryEventEmitter: Event-based telemetry emission using Node.js EventEmitter - MetricsAggregator: Per-statement aggregation with batch processing TelemetryEventEmitter: - Event-driven architecture using Node.js EventEmitter - Type-safe event emission methods - Respects telemetryEnabled configuration flag - All exceptions swallowed and logged at debug level - Zero impact when disabled Event Types: - connection.open: On successful connection - statement.start: On statement execution - statement.complete: On statement finish - cloudfetch.chunk: On chunk download - error: On exception with terminal classification MetricsAggregator: - Per-statement aggregation by statement_id - Connection events emitted immediately (no aggregation) - Statement events buffered until completeStatement() called - Terminal exceptions flushed immediately - Retryable exceptions buffered until statement complete - Batch size (default 100) triggers flush - Periodic timer (default 5s) triggers flush Batching Strategy: - Optimizes export efficiency - Reduces HTTP overhead - Smart flushing based on error criticality - Memory efficient with bounded buffers Testing: - 31 comprehensive unit tests for TelemetryEventEmitter - 32 comprehensive unit tests for MetricsAggregator - 100% function coverage, >90% line/branch coverage - Tests verify exception swallowing - Tests verify debug-only logging Dependencies: - Builds on [1/7] Types, [2/7] Infrastructure, [3/7] Client Management * Add telemetry export: DatabricksTelemetryExporter This is part 5 of 7 in the telemetry implementation stack. Components: - DatabricksTelemetryExporter: HTTP export with retry logic and circuit breaker - TelemetryExporterStub: Test stub for integration tests DatabricksTelemetryExporter: - Exports telemetry metrics to Databricks via HTTP POST - Two endpoints: authenticated (/api/2.0/sql/telemetry-ext) and unauthenticated (/api/2.0/sql/telemetry-unauth) - Integrates with CircuitBreaker for per-host endpoint protection - Retry logic with exponential backoff and jitter - Exception classification (terminal vs retryable) Export Flow: 1. Check circuit breaker state (skip if OPEN) 2. Execute with circuit breaker protection 3. Retry on retryable errors with backoff 4. Circuit breaker tracks success/failure 5. All exceptions swallowed and logged at debug level Retry Strategy: - Max retries: 3 (default, configurable) - Exponential backoff: 100ms * 2^attempt - Jitter: Random 0-100ms to prevent thundering herd - Terminal errors: No retry (401, 403, 404, 400) - Retryable errors: Retry with backoff (429, 500, 502, 503, 504) Circuit Breaker Integration: - Success → Record success with circuit breaker - Failure → Record failure with circuit breaker - Circuit OPEN → Skip export, log at debug - Automatic recovery via HALF_OPEN state Critical Requirements: - All exceptions swallowed (NEVER throws) - All logging at LogLevel.debug ONLY - No console logging - Driver continues when telemetry fails Testing: - 24 comprehensive unit tests - 96% statement coverage, 84% branch coverage - Tests verify exception swallowing - Tests verify retry logic - Tests verify circuit breaker integration - TelemetryExporterStub for integration tests Dependencies: - Builds on all previous layers [1/7] through [4/7] * Add telemetry integration into driver components This is part 6 of 7 in the telemetry implementation stack. Integration Points: - DBSQLClient: Telemetry lifecycle management and configuration - DBSQLOperation: Statement event emissions - DBSQLSession: Session ID propagation - CloudFetchResultHandler: Chunk download events - IDBSQLClient: ConnectionOptions override support DBSQLClient Integration: - initializeTelemetry(): Initialize all telemetry components - Feature flag check via FeatureFlagCache - Create TelemetryClientProvider, EventEmitter, MetricsAggregator, Exporter - Wire event listeners between emitter and aggregator - Cleanup on close(): Flush metrics, release clients, release feature flag context - Override support via ConnectionOptions.telemetryEnabled Event Emission Points: - connection.open: On successful openSession() with driver config - statement.start: In DBSQLOperation constructor - statement.complete: In DBSQLOperation.close() - cloudfetch.chunk: In CloudFetchResultHandler.downloadLink() - error: In DBSQLOperation.emitErrorEvent() with terminal classification Session ID Propagation: - DBSQLSession passes sessionId to DBSQLOperation constructor - All events include sessionId for correlation - Statement events include both sessionId and statementId Error Handling: - All telemetry code wrapped in try-catch - All exceptions logged at LogLevel.debug ONLY - Driver NEVER throws due to telemetry failures - Zero impact on driver operations Configuration Override: - ConnectionOptions.telemetryEnabled overrides config - Per-connection control for testing - Respects feature flag when override not specified Testing: - Integration test suite: 11 comprehensive E2E tests - Tests verify full telemetry flow: connection → statement → export - Tests verify feature flag behavior - Tests verify driver works when telemetry fails - Tests verify no exceptions propagate Dependencies: - Builds on all previous layers [1/7] through [5/7] - Completes the telemetry data flow pipeline * Add authentication support for REST API calls Implements getAuthHeaders() method for authenticated REST API requests: - Added getAuthHeaders() to IClientContext interface - Implemented in DBSQLClient using authProvider.authenticate() - Updated FeatureFlagCache to fetch from connector-service API with auth - Added driver version support for version-specific feature flags - Replaced placeholder implementation with actual REST API calls Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Update DatabricksTelemetryExporter to use authenticated export - Use getAuthHeaders() method for authenticated endpoint requests - Remove TODO comments about missing authentication - Add auth headers when telemetryAuthenticatedExport is true Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Fix telemetry event listeners and add config options - Fix event listener names: Remove 'telemetry.' prefix - Add support for telemetryBatchSize and telemetryAuthenticatedExport config options - Update telemetry files with fixed endpoints and proto format Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Match JDBC telemetry payload format - Change payload structure to match JDBC: uploadTime, items, protoLogs - protoLogs contains JSON-stringified TelemetryFrontendLog objects - Remove workspace_id (JDBC doesn't populate it) - Remove debug logs added during testing * Fix lint errors - Fix import order in FeatureFlagCache - Replace require() with import for driverVersion - Fix variable shadowing - Disable prefer-default-export for urlUtils * Add missing getAuthHeaders method to ClientContextStub Fix TypeScript compilation error by implementing getAuthHeaders method required by IClientContext interface. * Add missing getAuthHeaders method to ClientContextStub Fix TypeScript compilation error by implementing getAuthHeaders method required by IClientContext interface. * Add missing getAuthHeaders method to ClientContextStub Fix TypeScript compilation error by implementing getAuthHeaders method required by IClientContext interface. * Add missing getAuthHeaders method to ClientContextStub Fix TypeScript compilation error by implementing getAuthHeaders method required by IClientContext interface. * Add missing getAuthHeaders method to ClientContextStub Fix TypeScript compilation error by implementing getAuthHeaders method required by IClientContext interface. * Fix prettier formatting * Fix prettier formatting * Fix prettier formatting * Fix prettier formatting * Fix prettier formatting * Use nodejs-sql-driver as driver name in telemetry Changed from '@databricks/sql' to 'nodejs-sql-driver' to match JDBC driver naming convention. Added DRIVER_NAME constant to types.ts. * Add DRIVER_NAME constant for nodejs-sql-driver * Add DRIVER_NAME constant for nodejs-sql-driver * Add DRIVER_NAME constant for nodejs-sql-driver * Add DRIVER_NAME constant for nodejs-sql-driver * Add missing telemetry fields to match JDBC Added osArch, runtimeVendor, localeName, charSetEncoding, and processName fields to DriverConfiguration to match JDBC implementation. * Add missing telemetry fields to match JDBC Added osArch, runtimeVendor, localeName, charSetEncoding, and processName fields to DriverConfiguration to match JDBC implementation. * Populate all telemetry system configuration fields Added helper methods to populate osArch, runtimeVendor, localeName, charSetEncoding, and processName to match JDBC implementation: - osArch: from os.arch() - runtimeVendor: 'Node.js Foundation' - localeName: from LANG env var (format: en_US) - charSetEncoding: UTF-8 (Node.js default) - processName: from process.title or script name * Add missing telemetry fields to match JDBC Added osArch, runtimeVendor, localeName, charSetEncoding, and processName fields to DriverConfiguration to match JDBC implementation. * Add missing telemetry fields to match JDBC Added osArch, runtimeVendor, localeName, charSetEncoding, and processName fields to DriverConfiguration to match JDBC implementation. * Add missing telemetry fields to match JDBC Added osArch, runtimeVendor, localeName, charSetEncoding, and processName fields to DriverConfiguration to match JDBC implementation. * Fix telemetry aggregator cleanup in client close Changed from flush() to close() to ensure: - Periodic flush timer is stopped - Incomplete statements are finalized - Final flush is performed Previously, only flush() was called which left the timer running and didn't complete remaining statements. * Fix TypeScript compilation: add missing fields to system_configuration interface * Fix TypeScript compilation: add missing fields to system_configuration interface * Fix TypeScript compilation: add missing fields to system_configuration interface * Fix TypeScript compilation: add missing fields to system_configuration interface * Add telemetry testing and documentation This is part 7 of 7 in the telemetry implementation stack - FINAL LAYER. Documentation: - README.md: Add telemetry overview section - docs/TELEMETRY.md: Comprehensive telemetry documentation - spec/telemetry-design.md: Detailed design document - spec/telemetry-sprint-plan.md: Implementation plan - spec/telemetry-test-completion-summary.md: Test coverage report README.md Updates: - Added telemetry overview section - Configuration examples with all 7 options - Privacy-first design highlights - Link to detailed TELEMETRY.md TELEMETRY.md - Complete User Guide: - Overview and benefits - Privacy-first design (what is/isn't collected) - Configuration guide with examples - Event types with JSON schemas - Feature control (server-side flag + client override) - Architecture overview - Troubleshooting guide - Privacy & compliance (GDPR, CCPA, SOC 2) - Performance impact analysis - FAQ (12 common questions) Design Document (telemetry-design.md): - Complete system architecture - Component specifications - Data flow diagrams - Error handling requirements - Testing strategy - Implementation phases Test Coverage Summary: - 226 telemetry tests passing - 97.76% line coverage - 90.59% branch coverage - 100% function coverage - Critical requirements verified Test Breakdown by Component: - ExceptionClassifier: 51 tests (100% coverage) - CircuitBreaker: 32 tests (100% functions) - FeatureFlagCache: 29 tests (100% functions) - TelemetryEventEmitter: 31 tests (100% functions) - TelemetryClient: 31 tests (100% functions) - TelemetryClientProvider: 31 tests (100% functions) - MetricsAggregator: 32 tests (94% lines, 82% branches) - DatabricksTelemetryExporter: 24 tests (96% statements) - Integration: 11 E2E tests Critical Test Verification: ✅ All exceptions swallowed (no propagation) ✅ Debug-only logging (no warn/error) ✅ No console logging ✅ Driver works when telemetry fails ✅ Reference counting correct ✅ Circuit breaker behavior correct This completes the 7-layer telemetry implementation stack! Signed-off-by: samikshya-chand_data <samikshya.chand@databricks.com> * Add authentication support for REST API calls in telemetry Implement proper authentication for feature flag fetching and telemetry export by adding getAuthHeaders() method to IClientContext. - **IClientContext**: Add getAuthHeaders() method to expose auth headers - **DBSQLClient**: Implement getAuthHeaders() using authProvider.authenticate() - Returns empty object gracefully if no auth provider available - **FeatureFlagCache**: Implement actual server API call - Endpoint: GET /api/2.0/connector-service/feature-flags/OSS_NODEJS/{version} - Uses context.getAuthHeaders() for authentication - Parses JSON response with flags array - Updates cache duration from server-provided ttl_seconds - Looks for: databricks.partnerplatform.clientConfigsFeatureFlags.enableTelemetryForNodeJs - All exceptions swallowed with debug logging only - **DatabricksTelemetryExporter**: Add authentication to authenticated endpoint - Uses context.getAuthHeaders() when authenticatedExport=true - Properly authenticates POST to /api/2.0/sql/telemetry-ext - Removes TODO comments about missing authentication Follows same pattern as JDBC driver: - Endpoint: /api/2.0/connector-service/feature-flags/OSS_JDBC/{version} (JDBC) - Endpoint: /api/2.0/connector-service/feature-flags/OSS_NODEJS/{version} (Node.js) - Auth headers from connection's authenticate() method - Response format: { flags: [{ name, value }], ttl_seconds } - Build: ✅ Successful - E2E: ✅ Verified with real credentials - Feature flag fetch now fully functional - Telemetry export now properly authenticated Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> Signed-off-by: samikshya-chand_data <samikshya.chand@databricks.com> * Fix telemetry and feature flag implementation - Fix event listener names: use 'connection.open' not 'telemetry.connection.open' - Fix feature flag endpoint: use NODEJS client type instead of OSS_NODEJS - Fix telemetry endpoints: use /telemetry-ext and /telemetry-unauth (not /api/2.0/sql/...) - Update telemetry payload to match proto: use system_configuration with snake_case fields - Add URL utility to handle hosts with or without protocol - Add telemetryBatchSize and telemetryAuthenticatedExport config options - Remove debug statements and temporary feature flag override Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> Signed-off-by: samikshya-chand_data <samikshya.chand@databricks.com> * Fix prettier formatting Signed-off-by: samikshya-chand_data <samikshya.chand@databricks.com> * Update telemetry design doc with system config and protoLogs format Added detailed documentation for: - System configuration fields (osArch, runtimeVendor, localeName, charSetEncoding, processName) with JDBC equivalents - protoLogs payload format matching JDBC TelemetryRequest structure - Complete log object structure with all field descriptions - Example JSON payloads showing actual format sent to server Clarified that: - Each log is JSON-stringified before adding to protoLogs array - Connection events include full system_configuration - Statement events include operation_latency_ms and sql_operation - The items field is required but always empty Signed-off-by: samikshya-chand_data <samikshya.chand@databricks.com> * Document telemetry export lifecycle and timing Added comprehensive section 6.5 explaining exactly when telemetry exports occur: - Statement close: Aggregates metrics, exports only if batch full - Connection close: ALWAYS exports all pending metrics via aggregator.close() - Process exit: NO automatic export unless close() was called - Batch size/timer: Automatic background exports Included: - Code examples showing actual implementation - Summary table comparing all lifecycle events - Best practices for ensuring telemetry export (SIGINT/SIGTERM handlers) - Key differences from JDBC (JVM shutdown hooks vs manual close) Clarified that aggregator.close() does three things: 1. Stops the periodic flush timer 2. Completes any remaining incomplete statements 3. Performs final flush to export all buffered metrics Signed-off-by: samikshya-chand_data <samikshya.chand@databricks.com> * Add connection open latency tracking and enable telemetry by default Changes: - Track and export connection open latency (session creation time) - Enable telemetry by default (was false), gated by feature flag - Update design doc to document connection latency Implementation: - DBSQLClient.openSession(): Track start time and calculate latency - TelemetryEventEmitter: Accept latencyMs in connection event - MetricsAggregator: Include latency in connection metrics - DatabricksTelemetryExporter: Export operation_latency_ms for connections Config changes: - telemetryEnabled: true by default (in DBSQLClient and types.ts) - Feature flag check still gates initialization for safe rollout Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> Signed-off-by: samikshya-chand_data <samikshya.chand@databricks.com> * Populate sql_operation, statement_id, and auth_type in telemetry Fixes: - sql_operation now properly populated by fetching metadata before statement close - statement_id always populated from operation handle GUID - auth_type now included in driver_connection_params Changes: - DBSQLOperation: Fetch metadata before emitting statement.complete to ensure resultFormat is available for sql_operation field - DBSQLClient: Track authType from connection options and include in driver configuration - DatabricksTelemetryExporter: Export auth_type in driver_connection_params - types.ts: Add authType to DriverConfiguration interface - Design doc: Document auth_type, resultFormat population, and connection params Implementation details: - emitStatementComplete() is now async to await metadata fetch - Auth type defaults to 'access-token' if not specified - Result format fetched even if not explicitly requested by user - Handles metadata fetch failures gracefully (continues without resultFormat) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> Signed-off-by: samikshya-chand_data <samikshya.chand@databricks.com> * Map auth type to telemetry auth enum - Convert 'access-token' (or undefined) to 'pat' - Convert 'databricks-oauth' to 'external-browser' (U2M) or 'oauth-m2m' (M2M) - Distinguish M2M from U2M by checking for oauthClientSecret - Keep 'custom' as 'custom' Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Add SqlExecutionEvent fields to telemetry - Add statement_type field from operationType - Add is_compressed field from compression tracking - Export both fields in sql_operation for statement metrics - Fields populated from CloudFetch chunk events Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Filter out NIL UUID from statement ID in telemetry - Exclude '00000000-0000-0000-0000-000000000000' from sql_statement_id - Only include valid statement IDs in telemetry logs Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Only populate sql_operation fields when present - statement_type only included if operationType is set - is_compressed only included if compressed value is set - execution_result only included if resultFormat is set - sql_operation object only created if any field is present Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Map Thrift operation type to proto Operation.Type enum - Convert TOperationType (Thrift) to proto Operation.Type names - EXECUTE_STATEMENT remains EXECUTE_STATEMENT - GET_TYPE_INFO -> LIST_TYPE_INFO - GET_CATALOGS -> LIST_CATALOGS - GET_SCHEMAS -> LIST_SCHEMAS - GET_TABLES -> LIST_TABLES - GET_TABLE_TYPES -> LIST_TABLE_TYPES - GET_COLUMNS -> LIST_COLUMNS - GET_FUNCTIONS -> LIST_FUNCTIONS - UNKNOWN -> TYPE_UNSPECIFIED Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Move auth_type to top level per proto definition - auth_type is field 5 at OssSqlDriverTelemetryLog level, not nested - Remove driver_connection_params (not populated in Node.js driver) - Export auth_type directly in sql_driver_log for connection events Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Map result format to proto ExecutionResult.Format enum - ARROW_BASED_SET -> INLINE_ARROW - COLUMN_BASED_SET -> COLUMNAR_INLINE - ROW_BASED_SET -> INLINE_JSON - URL_BASED_SET -> EXTERNAL_LINKS Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Refactor telemetry type mappers to separate file - Create lib/telemetry/telemetryTypeMappers.ts - Move mapOperationTypeToTelemetryType (renamed from mapOperationTypeToProto) - Move mapResultFormatToTelemetryType (renamed from mapResultFormatToProto) - Keep all telemetry-specific mapping functions in one place Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Add driver_connection_params with available fields - http_path: API endpoint path - socket_timeout: Connection timeout in milliseconds - enable_arrow: Whether Arrow format is enabled - enable_direct_results: Whether direct results are enabled - enable_metric_view_metadata: Whether metric view metadata is enabled - Only populate fields that are present Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Document proto field coverage in design doc - Add section 14 detailing implemented and missing proto fields - List all fields from OssSqlDriverTelemetryLog that are implemented - Document which fields are not implemented and why - Explain that missing fields require additional instrumentation Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Include system_configuration, driver_connection_params, and auth_type in all telemetry logs - Cache driver config in MetricsAggregator when connection event is processed - Include cached driver config in all statement and error metrics - Export system_configuration, driver_connection_params, and auth_type for every log - Each telemetry log is now self-contained with full context This ensures every telemetry event (connection, statement, error) includes the driver configuration context, making logs independently analyzable. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Add connection close telemetry event Implement CONNECTION_CLOSE telemetry event to track session lifecycle: - Add CONNECTION_CLOSE event type to TelemetryEventType enum - Add emitConnectionClose() method to TelemetryEventEmitter - Add processConnectionCloseEvent() handler in MetricsAggregator - Track session open time in DBSQLSession and emit close event with latency - Remove unused TOperationType import from DBSQLOperation This provides complete session telemetry: connection open, statement execution, and connection close with latencies for each operation. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Fix unit tests for connection close telemetry Update test files to match new telemetry interface changes: - Add latencyMs parameter to all emitConnectionOpen() test calls - Add missing DriverConfiguration fields in test mocks (osArch, runtimeVendor, localeName, charSetEncoding, authType, processName) This fixes TypeScript compilation errors introduced by the connection close telemetry implementation. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Add connection.close event listener to telemetry wire-up Fix missing event listener for CONNECTION_CLOSE events in DBSQLClient telemetry initialization. Without this listener, connection close events were being emitted but not routed to the aggregator for processing. Now all 3 telemetry events are properly exported: - CONNECTION_OPEN (connection latency) - STATEMENT_COMPLETE (execution latency) - CONNECTION_CLOSE (session duration) Verified with e2e test showing 3 successful telemetry exports. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Make telemetry logging silent by default Remove verbose telemetry logs to minimize noise in customer logs. Only log essential startup/shutdown messages and errors: Kept (LogLevel.debug): - "Telemetry: enabled" - on successful initialization - "Telemetry: disabled" - when feature flag disables it - "Telemetry: closed" - on graceful shutdown - Error messages only when failures occur Removed: - Individual metric flushing logs - Export operation logs ("Exporting N metrics") - Success confirmations ("Successfully exported") - Client lifecycle logs (creation, ref counting) - All intermediate operational logs Updated spec/telemetry-design.md to document the silent logging policy. Telemetry still functions correctly - exports happen silently in the background without cluttering customer logs. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Ensure statement_type always populated in telemetry Fix issue where statement_type was null in telemetry payloads. Changes: - mapOperationTypeToTelemetryType() now always returns a string, defaulting to 'TYPE_UNSPECIFIED' when operationType is undefined - statement_type always included in sql_operation telemetry log This ensures that even if the Thrift operationHandle doesn't have operationType set, the telemetry will include 'TYPE_UNSPECIFIED' instead of null. Root cause: operationHandle.operationType from Thrift response can be undefined, resulting in null statement_type in telemetry logs. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Add operation types to connection metrics Connection metrics now include operation type in sql_operation: - CREATE_SESSION for connection open events - DELETE_SESSION for connection close events This matches the proto Operation.Type enum which includes session-level operations in addition to statement-level operations. Before: sql_operation: null After: sql_operation: { statement_type: "CREATE_SESSION" // or "DELETE_SESSION" } Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Fix telemetry proto field mapping Correct issue where Operation.Type values were incorrectly placed in statement_type field. Per proto definition: - statement_type expects Statement.Type (QUERY, SQL, UPDATE, METADATA, VOLUME) - operation_type goes in operation_detail.operation_type and uses Operation.Type Changes: - Connection metrics: Set sql_operation.operation_detail.operation_type to CREATE_SESSION or DELETE_SESSION - Statement metrics: Set both statement_type (QUERY or METADATA based on operation) and operation_detail.operation_type (EXECUTE_STATEMENT, etc.) - Added mapOperationToStatementType() to convert Operation.Type to Statement.Type This ensures telemetry payloads match the OssSqlDriverTelemetryLog proto structure correctly. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Add operation_detail field to telemetry interface and enhance test - Added operation_detail field to DatabricksTelemetryLog interface - Enhanced telemetry-local.test.ts to capture and display actual payloads - Verified all three telemetry events (CONNECTION_OPEN, STATEMENT_COMPLETE, CONNECTION_CLOSE) - Confirmed statement_type and operation_detail.operation_type are properly populated Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Add error scenario test for telemetry validation - Added test for invalid query execution (TABLE_OR_VIEW_NOT_FOUND) - Confirms SQL execution errors are handled as failed statements - Verified telemetry payloads still correctly formatted during errors - Note: Driver-level errors (connection/timeout) would need emitErrorEvent wiring Test output shows correct behavior: - CONNECTION_OPEN with CREATE_SESSION - STATEMENT_COMPLETE with QUERY + EXECUTE_STATEMENT (even on error) - CONNECTION_CLOSE with DELETE_SESSION Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Fix telemetry PR review comments from #325 Three fixes addressing review feedback: 1. Fix documentation typo (sreekanth-db comment) - DatabricksTelemetryExporter.ts:94 - Changed "TelemetryFrontendLog" to "DatabricksTelemetryLog" 2. Add proxy support (jadewang-db comment) - DatabricksTelemetryExporter.ts:exportInternal() - Get HTTP agent from connection provider - Pass agent to fetch for proxy support - Follows same pattern as CloudFetchResultHandler and DBSQLSession - Supports http/https/socks proxies with authentication 3. Fix flush timer to prevent rate limiting (sreekanth-db comment) - MetricsAggregator.ts:flush() - Reset timer after manual flushes (batch size, terminal errors) - Ensures consistent 30s spacing between exports - Prevents rapid successive flushes (e.g., batch at 25s, timer at 30s) All changes follow existing driver patterns and maintain backward compatibility. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Fix telemetry PR review comments from #325 Three fixes addressing review feedback: 1. Fix documentation typo (sreekanth-db comment) - DatabricksTelemetryExporter.ts:94 - Changed "TelemetryFrontendLog" to "DatabricksTelemetryLog" 2. Add proxy support (jadewang-db comment) - DatabricksTelemetryExporter.ts:exportInternal() - Get HTTP agent from connection provider - Pass agent to fetch for proxy support - Follows same pattern as CloudFetchResultHandler and DBSQLSession - Supports http/https/socks proxies with authentication 3. Fix flush timer to prevent rate limiting (sreekanth-db comment) - MetricsAggregator.ts:flush() - Reset timer after manual flushes (batch size, terminal errors) - Ensures consistent 30s spacing between exports - Prevents rapid successive flushes (e.g., batch at 25s, timer at 30s) All changes follow existing driver patterns and maintain backward compatibility. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Add proxy support to feature flag fetching Feature flag fetching was also missing proxy support like telemetry exporter was. Applied the same fix: - Get HTTP agent from connection provider - Pass agent to fetch call for proxy support - Follows same pattern as CloudFetchResultHandler and DBSQLSession - Supports http/https/socks proxies with authentication This completes proxy support for all HTTP operations in the telemetry system (both telemetry export and feature flag fetching). Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Add proxy support to feature flag fetching Feature flag fetching was also missing proxy support like telemetry exporter was. Applied the same fix: - Get HTTP agent from connection provider - Pass agent to fetch call for proxy support - Follows same pattern as CloudFetchResultHandler and DBSQLSession - Supports http/https/socks proxies with authentication This completes proxy support for all HTTP operations in the telemetry system (both telemetry export and feature flag fetching). Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Make feature flag cache extensible for multiple flags Refactored FeatureFlagCache to support querying any feature flag, not just the telemetry flag: **Changes:** - Store all flags from server in Map<string, string> - Add generic isFeatureEnabled(host, flagName) method - Keep isTelemetryEnabled() as convenience method - fetchFeatureFlags() now stores all flags for future use **Benefits:** - Extensible to any safe feature flag - No code changes needed to add new flags - Single fetch stores all flags from response - Backward compatible (isTelemetryEnabled still works) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Add circuit breaker protection to feature flag fetching Feature flags now use the same circuit breaker protection as telemetry for resilience against endpoint failures. **Changes:** - FeatureFlagCache now accepts optional CircuitBreakerRegistry - Feature flag fetches wrapped in circuit breaker execution - Shared circuit breaker registry between feature flags and telemetry - Per-host circuit breaker isolation maintained - Falls back to cached values when circuit is OPEN **Benefits:** - Protects against repeated failures to feature flag endpoint - Fails fast when endpoint is down (circuit OPEN) - Auto-recovery after timeout (60s default) - Same resilience patterns as telemetry export **Configuration:** - Failure threshold: 5 consecutive failures - Timeout: 60 seconds - Per-host isolation (failures on one host don't affect others) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Fix telemetry unit tests for extensible feature flags and proto payload format - Update FeatureFlagCache tests to use new extensible flags Map - Fix DatabricksTelemetryExporter tests to use protoLogs format - Verify telemetry endpoints use correct paths (/telemetry-ext, /telemetry-unauth) - 213 passing, 13 logging assertion tests need investigation * Merge main into telemetry-3-client-management; prefer main versions for infrastructure Takes main's versions of CircuitBreaker, FeatureFlagCache, DatabricksTelemetryExporter, MetricsAggregator, TelemetryEventEmitter, and types — bringing in SSRF hardening, overflow protection, async flush/close, errorStack redaction, and IAuthentication-based auth headers. Removes IClientContext.getAuthHeaders() in favour of the direct authProvider injection pattern from main. Co-authored-by: samikshya-chand_data * fix: prettier formatting and add coverage/ to .prettierignore Co-authored-by: samikshya-chand_data * fix: delete host entry before awaiting close to prevent stale-client race If getOrCreateClient ran after refCount hit 0 but before clients.delete(host), it would receive a closing TelemetryClient. Deleting synchronously first means any concurrent caller gets a fresh instance instead. Co-authored-by: samikshya-chand_data * refactor: simplify TelemetryClient.close() and TelemetryClientProvider - close() made synchronous (no I/O, no reason for async); uses finally to guarantee this.closed=true even when logger throws - TelemetryClientProvider.releaseClient() made synchronous for the same reason - Misleading ConcurrentHashMap reference removed from docstring - getRefCount/getActiveClients marked @internal (test-only surface) - Update tests to match: rejects→throws stubs, remove await on sync calls Co-authored-by: samikshya-chand_data * chore: move telemetry MD docs to follow-up PR Remove docs/TELEMETRY.md, spec/telemetry-design.md, spec/telemetry-sprint-plan.md, spec/telemetry-test-completion-summary.md and revert README.md — these ~4.9k lines of markdown are being split into a stacked docs-only PR on top of this branch to keep the [4/7] diff focused on code. Co-authored-by: Isaac Signed-off-by: samikshya-chand_data <samikshya.chand@databricks.com> * chore: drop tests/e2e/telemetry-local.test.ts Local-only e2e harness that duplicates what tests/e2e/telemetry/telemetry-integration.test.ts covers in CI. Co-authored-by: Isaac Signed-off-by: samikshya-chand_data <samikshya.chand@databricks.com> * chore: remove unused CircuitBreaker import in FeatureFlagCache Co-authored-by: Isaac * chore: apply prettier formatting to telemetry files Fixes the lint CI job which runs prettier --check before eslint. Co-authored-by: Isaac Signed-off-by: samikshya-chand_data <samikshya.chand@databricks.com> * feat: emit per-chunk timing telemetry on the FetchResults path Aggregate initial/slowest/sum chunk latencies in MetricsAggregator, emit them in the proto chunk_details block, and time each FetchResults RPC in RowSetProvider so the inline-Arrow path populates chunk telemetry alongside CloudFetch (mirrors Go's chunkTimingAccumulator). Also fix MetricsAggregator clobbering accumulated chunkCount and bytesDownloaded to 0 on STATEMENT_COMPLETE when the event omitted those fields — this hid chunk_details from every path. Co-authored-by: Isaac * fix(telemetry): address review feedback on aggregator and close() - Snapshot driverConfig on each statement at first event so a later CONNECTION_OPEN can't retroactively rewrite the config reported by in-flight statements (and their buffered errors). - Attach a defensive .catch() to the fire-and-forget exporter.export() call so any future regression that leaks a rejection logs at debug rather than surfacing as an unhandled promise rejection. - Document the unref()'d flush timer on DBSQLClient.close(): callers must await close() on shutdown to drain buffered telemetry; otherwise metrics between flush ticks are lost. Co-authored-by: Isaac * fix(telemetry): iter-2 review fixes — rebase regressions, type-safe wiring, error telemetry Rebase the regressed exporter/aggregator/feature-flag-cache on main's hardened versions and re-apply only the genuinely new functionality (CONNECTION_CLOSE event, chunk-timing aggregation) on top. Closes the critical findings from the multi-reviewer audit: - SSRF guard, redactSensitive, sanitizeProcessName, hasAuthorization, auth-missing warn-once — all restored via main's telemetryUtils. - MetricsAggregator memory bounds (maxPendingMetrics with error-preferring drop, maxErrorsPerStatement, statementTtlMs eviction) restored. - FeatureFlagCache in-flight fetch dedup and TTL clamp [60s, 3600s] restored; lib/telemetry/urlUtils.ts deleted. - close() now properly awaits aggregator drain — fixes the close()/flush race that PR #362 already fixed once. - Driver version reads lib/version.ts via buildUserAgentString instead of hardcoded '1.0.0'; uuidv4() restored in place of Math.random(). - TelemetryTerminalError re-exported from lib/index.ts. Type-safe wiring: - Added optional getTelemetryEmitter() / getTelemetryAggregator() to IClientContext; removed all 7 `(this.context as any)` casts. - Six copy-pasted event listeners in DBSQLClient.initializeTelemetry collapsed into one `Object.values(TelemetryEventType)` loop — closes the listener-name mismatch that silently dropped error events. - mapAuthType now covers all 6 authType values instead of defaulting everything to 'pat'. TelemetryClient now owns the host-scoped resources: - TelemetryClientProvider is a process-wide singleton (getInstance()). - TelemetryClient owns DatabricksTelemetryExporter, MetricsAggregator, CircuitBreakerRegistry, and FeatureFlagCache for its host. Implements IClientContext itself so the owned components have a stable context that survives any single DBSQLClient's close. - DBSQLClient instances on the same host share the breaker counters, feature-flag cache, exporter, and HTTP batches. Fixes the per-instance breaker-fragmentation noted in iter-2 architecture review. - Each DBSQLClient still holds its own TelemetryEventEmitter (respects per-client telemetryEnabled); emitters bridge into the shared aggregator. - Exporter falls back to context.getAuthProvider() when no explicit auth provider is passed, so the shared exporter resolves auth through the TelemetryClient's FIFO of registered DBSQLClients. Error telemetry wired across operation entry points: - Re-added emitErrorEvent(error) on DBSQLOperation; uses ExceptionClassifier.isTerminal() to classify. - fetchChunk, cancel, close, getMetadata wrap their bodies in try/catch that calls emitErrorEvent before re-throwing. Verified end-to-end against a real Azure Databricks workspace: failed query produces STATEMENT_COMPLETE + ERROR (with redacted stack) on the wire. - Removed the await getMetadata() call from emitStatementComplete — eliminates the extra Thrift RPC on every close (F19) AND prevents spurious error telemetry from getMetadata's wrapper firing during close-cleanup of an already-failed operation. Other: - Iterating Map.keys() while mutating made safe via snapshot in close(). - STATEMENT_COMPLETE no longer zeroes accumulated chunk metrics when the emit doesn't supply them (matches sibling-field guards). - Tests for the rebased modules restored from main; provider tests updated for the singleton API; deleted unused TelemetryExporterStub. 484 unit tests passing. Diff vs main: ~+2110/-383, down from the original PR's +3640/-1173. Co-authored-by: Isaac * chore(telemetry): satisfy CI lint and prettier - Apply prettier to TelemetryClientProvider.test.ts (sed-edits in the prior commit didn't preserve formatting). - Silence eslint `no-await-in-loop` on the auth-context fall-through in TelemetryClient.getConnectionProvider — sequential by intent. - Drop the empty public constructor on TelemetryClientProvider; leave a comment explaining the singleton + test-friendly construction contract. Co-authored-by: Isaac * chore(lint): disable func-names in test files Mocha tests need `function () {}` so they can use `this.timeout()` / `this.skip()` — arrow functions don't bind `this` to the test context. The `func-names` rule was firing on every test in the suite (including pre-existing tests in `protocol_versions.test.ts`); moving the rule to the test-file override block silences those warnings. Co-authored-by: Isaac * fix(telemetry): address review findings — wiring, lifecycle, redaction, knobs Iter-3 review fixes addressing 17 distinct findings from the multi-agent review. Telemetry is now functionally correct and operationally safe. Critical - F1: TelemetryClient ctor wires getOrCreateContext on FeatureFlagCache. isTelemetryEnabled was previously short-circuiting to false in production because no caller registered the host — every customer silently emitted zero events. - F2: integration test asserts the documented default (true), not the prior off-by-default. Test was contradicting production code. - F3: IClientContext.getAuthProvider now optional; consumers use ?.() so external implementers don't break on upgrade. High / privacy - F4: explicit DATABRICKS_TELEMETRY_DISABLED parser (1/true/yes/on, case insensitive). Avoids the footgun where DATABRICKS_TELEMETRY_DISABLED=false also disabled telemetry. Documented in CHANGELOG and TSDoc. - F12: TelemetryClient.registerContext warns on telemetry-config and userAgentEntry divergence so multi-tenant misconfig is visible. - F9: connect()-on-reconnect releases prior refcount; close() clears the emitter ref so post-close events can't smuggle into a closed aggregator. - M-1: redactSensitive strips /home/<user>/, /Users/<user>/, and C:\Users\<user>\ patterns from stack traces. - M-3: FeatureFlagCache.getAuthHeaders falls through to the context's auth provider — feature-flag GET is no longer unconditionally unauth. Operational - F7: MetricsAggregator.close races the final flush against a configurable telemetryCloseTimeoutMs (default 2s) so a flapping endpoint can't hang process.exit(0). - F8: flushInFlight serializer prevents concurrent fire-and-forget flushes from starving the user's HTTP socket pool. Drain pattern in close() awaits any in-flight flush, then issues a fresh one to capture close-time metrics that would otherwise be stranded. - F16: maxStatementMetrics cap (default 5000) with oldest-first eviction. Buffered errors emitted as standalone metrics first so the first-failure signal survives. - DBSQLSession.close() emits connection.close even when closeSession fails so failed-close rates are visible in dashboards. Maintainability - F10/F17: single withErrorTelemetry helper covers fetchChunk, cancel, close, finished, hasMoreRows, getSchema, getMetadata. safeEmit helper consolidates seven copy-pasted "get emitter, emit, swallow at debug" blocks across DBSQLOperation, DBSQLClient, DBSQLSession, CloudFetchResultHandler, RowSetProvider. Also fixes the inconsistency where DBSQLSession.close() lacked the swallow wrapper that the other six sites had. API surface - F13: ConnectionOptions exposes nine telemetry knobs (was three) with TSDoc. Adds telemetryFlushIntervalMs, telemetryMaxRetries, telemetryCircuitBreakerThreshold, telemetryCircuitBreakerTimeout, telemetryCloseTimeoutMs, telemetryMaxStatementMetrics. Tests - ClientContextStub gains telemetryEmitter / telemetryAggregator hooks so unit tests can assert on emit calls instead of silently no-op'ing. - 18 new unit tests covering F1 refcount, F12 divergence warn, async-close idempotency, error-telemetry wrappers (cancel, close, getMetadata, closed-op finished/getSchema/hasMoreRows), multi-context FIFO, and a new tests/unit/result/RowSetProvider.test.ts file (RowSetProvider had no test file at all). 783 unit tests pass; live e2e against adb-27363120558779.19.azuredatabricks.net validates the full pipeline. Co-authored-by: Isaac * chore(lint): apply prettier to MetricsAggregator Co-authored-by: Isaac * fix(telemetry): address review highs — refcount leak, FIFO fallthrough, public types, README, test gaps - DBSQLClient.initializeTelemetry: release the per-host refcount on the catch path so a thr…
1 parent 6a4a7c4 commit 5f1728a

30 files changed

Lines changed: 3920 additions & 547 deletions

.eslintrc

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,8 @@
2929
"files": ["*.test.js", "*.test.ts"],
3030
"rules": {
3131
"no-unused-expressions": "off",
32-
"@typescript-eslint/no-unused-expressions": "off"
32+
"@typescript-eslint/no-unused-expressions": "off",
33+
"func-names": "off"
3334
}
3435
}
3536
]

CHANGELOG.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,13 @@
1212
- Add metric view metadata support (databricks/databricks-sql-nodejs#312 by @shivam2680)
1313
- Fix: Avoid calling require('lz4') if it's really not required (databricks/databricks-sql-nodejs#316 by @ikkala)
1414
- Add telemetry foundation (off by default) (databricks/databricks-sql-nodejs#324 by @samikshya-db)
15+
- Telemetry event emission and per-host aggregation (databricks/databricks-sql-nodejs#327 by @samikshya-db).
16+
**Default change:** `telemetryEnabled` now defaults to `true` (gated by a remote feature flag).
17+
To opt out programmatically, pass `telemetryEnabled: false` to `connect()`.
18+
To disable globally without code changes, set the environment variable
19+
`DATABRICKS_TELEMETRY_DISABLED` to one of `1`, `true`, `yes`, or `on`
20+
(case-insensitive). Other values (empty, `0`, `false`, etc.) are ignored
21+
— the runtime config takes precedence.
1522

1623
## 1.12.0
1724

README.md

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,91 @@ client
5151
});
5252
```
5353

54+
## Telemetry
55+
56+
Starting with version 1.13, the driver collects telemetry — connection,
57+
statement, and CloudFetch chunk metrics, plus error events with redacted
58+
stack traces — to help Databricks improve driver performance and
59+
reliability. **Telemetry is enabled by default and gated by a server-side
60+
feature flag**: events are emitted only when the workspace's feature flag
61+
is on. No SQL text, parameter values, or row data are ever included.
62+
63+
### What's collected
64+
65+
- Connection lifecycle (`CREATE_SESSION`, `DELETE_SESSION`) with latency.
66+
- Statement lifecycle (`STATEMENT_START`, `STATEMENT_COMPLETE`) with
67+
execution latency, operation type, and result format.
68+
- CloudFetch chunk timings and byte counts.
69+
- Error events with redacted stack traces (Bearer/JWT tokens, OAuth
70+
secrets, home-directory paths, and Databricks PATs are stripped before
71+
emission).
72+
73+
See `TelemetryEvent` and `TelemetryMetric` in the package exports for the
74+
exact payload shapes.
75+
76+
### Multi-tenant SaaS deployments — read this before enabling telemetry
77+
78+
The telemetry layer shares one per-host `TelemetryClient` across every
79+
`DBSQLClient` connected to the same Databricks workspace host. The
80+
authenticated export path uses the **first-registered** client's auth
81+
provider, User-Agent, and `telemetryAuthenticatedExport` value — these
82+
fields are snapshotted at the host singleton and are **not** per-tenant.
83+
84+
If you are operating a SaaS layer that fronts multiple tenants against the
85+
same Databricks workspace host with a shared driver process, telemetry from
86+
tenant B's queries can be POSTed under tenant A's auth headers, with
87+
tenant A's `userAgentEntry`. A tenant B that explicitly set
88+
`telemetryAuthenticatedExport: false` will still ride tenant A's
89+
authenticated pipeline.
90+
91+
> **Recommendation for multi-tenant deployments**: set
92+
> `telemetryEnabled: false` on all `DBSQLClient` instances, or partition
93+
> by Databricks workspace host so each tenant owns its own
94+
> `TelemetryClient`. Subsequent registrants with diverging auth/UA values
95+
> emit a warn-level log so the leak is at least visible.
96+
97+
### Opting out
98+
99+
Three independent ways to disable telemetry, in order of precedence:
100+
101+
1. **Environment variable** — set `DATABRICKS_TELEMETRY_DISABLED` to one
102+
of `1`, `true`, `yes`, or `on` (case-insensitive). Other values
103+
(empty, `0`, `false`, `off`, `no`) are ignored, leaving the runtime
104+
config in charge.
105+
2. **Programmatic** — pass `telemetryEnabled: false` to `connect()`:
106+
```javascript
107+
await client.connect({
108+
host,
109+
path,
110+
token,
111+
telemetryEnabled: false,
112+
});
113+
```
114+
3. **Server-side** — Databricks-managed feature flag; if disabled for
115+
your workspace, the driver does not emit telemetry regardless of
116+
client config.
117+
118+
### Tuning
119+
120+
If you keep telemetry on, the following knobs are available on
121+
`ConnectionOptions` (see JSDoc on `IDBSQLClient.ts` for defaults and
122+
units):
123+
124+
- `telemetryAuthenticatedExport` — set to `false` to ship reduced
125+
payloads (no statement/session correlation IDs, generic User-Agent)
126+
via the unauthenticated endpoint.
127+
- `telemetryBatchSize`, `telemetryFlushIntervalMs`, `telemetryMaxRetries`
128+
— batching and retry tuning.
129+
- `telemetryCircuitBreakerThreshold`, `telemetryCircuitBreakerTimeout`
130+
circuit-breaker tuning for the export endpoint.
131+
- `telemetryCloseTimeoutMs` — bound on `await client.close()` waiting for
132+
the final flush.
133+
134+
> **Note for short-lived processes**: always `await client.close()`
135+
> before `process.exit(0)` so the final batch is flushed. Without an
136+
> explicit close, the periodic flush timer is `unref()`'d to avoid
137+
> holding the event loop open, so any unflushed events are dropped.
138+
54139
## Run Tests
55140

56141
### Unit tests

0 commit comments

Comments
 (0)