feat(grpc): Add integration support for GRPC#3721
Conversation
Adds a new `sentry_grpc` package with `SentryGrpcInterceptor`, a gRPC `ClientInterceptor` that provides: - Distributed tracing via `sentry-trace` / `baggage` header injection - Child span creation with `grpc.client` operation per unary call - Breadcrumbs for outgoing RPC calls (method, status, duration) - Optional exception capture for non-OK gRPC status codes - Span-first lifecycle compatibility - Support for grpc >=4.0.4 <6.0.0 Co-Authored-By: Claude <noreply@anthropic.com>
Each example is a Flutter app with three buttons: - Good Request: GET rsa4096.badssl.com (valid TLS) - Bad Request: GET expired.badssl.com (expired cert, captured by Sentry) - gRPC Request: grpcb.in:9001 GRPCBin/Empty via SentryGrpcInterceptor grpc4 resolves grpc 4.x; grpc5 resolves grpc 5.x. Both produce identical behaviour to demonstrate version-agnostic compatibility. Co-Authored-By: Claude <noreply@anthropic.com>
DummyUnary sends DummyMessage { f_string: "hello from sentry_grpc" }
and displays the echoed f_string from the response. The message is
hand-encoded via proto wire format (field 1, wire type 2) without
needing a protobuf codegen step.
RandomError triggers a random gRPC error from grpcb.in, which is
automatically captured by SentryGrpcInterceptor and reported to Sentry.
Co-Authored-By: Claude <noreply@anthropic.com>
- Add captureRequestHeaders option to attach outgoing gRPC metadata to spans, with PII filtering for sensitive headers - Attach gRPC error details (ErrorInfo, BadRequest, RetryInfo, DebugInfo, PreconditionFailure, ResourceInfo, QuotaFailure) to spans on failed calls - Remove captureBody (not captured by other SDK integrations) - Improve example apps: typed DummyMessage, WithHeaders button with custom metadata, SentryFlutter init in grpc4 Co-Authored-By: Claude <noreply@anthropic.com>
Semver Impact of This PR🟡 Minor (new features) 📋 Changelog PreviewThis is how your changes will appear in the changelog. Features
FixesDart
Flutter
EnhancementsFlutter
DependenciesDeps
Internal Changes
🤖 This preview updates automatically when you update the PR. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #3721 +/- ##
==========================================
- Coverage 86.96% 86.85% -0.11%
==========================================
Files 336 336
Lines 11982 11984 +2
==========================================
- Hits 10420 10409 -11
- Misses 1562 1575 +13
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
lucas-zimerman
changed the title
Android Performance metrics 🚀
|
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| 20faa47 | 400.26 ms | 392.90 ms | -7.37 ms |
| 6ad8fc4 | 489.92 ms | 484.96 ms | -4.96 ms |
| e8f0fc4 | 348.40 ms | 353.34 ms | 4.94 ms |
| 57eccdf | 437.96 ms | 437.94 ms | -0.02 ms |
| 2e4277e | 346.60 ms | 358.29 ms | 11.70 ms |
| e90cf93 | 374.35 ms | 368.40 ms | -5.95 ms |
| a920b0f | 391.75 ms | 378.98 ms | -12.77 ms |
| 55b7a00 | 418.43 ms | 398.07 ms | -20.35 ms |
| af96ef2 | 375.84 ms | 382.02 ms | 6.18 ms |
| 1adf565 | 389.67 ms | 395.96 ms | 6.29 ms |
App size
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| 20faa47 | 13.93 MiB | 15.18 MiB | 1.25 MiB |
| 6ad8fc4 | 13.93 MiB | 15.06 MiB | 1.13 MiB |
| e8f0fc4 | 14.55 MiB | 15.87 MiB | 1.31 MiB |
| 57eccdf | 14.30 MiB | 15.49 MiB | 1.19 MiB |
| 2e4277e | 14.55 MiB | 15.87 MiB | 1.31 MiB |
| e90cf93 | 14.31 MiB | 15.56 MiB | 1.25 MiB |
| a920b0f | 14.09 MiB | 15.28 MiB | 1.19 MiB |
| 55b7a00 | 14.30 MiB | 15.49 MiB | 1.19 MiB |
| af96ef2 | 13.93 MiB | 15.18 MiB | 1.25 MiB |
| 1adf565 | 14.09 MiB | 15.28 MiB | 1.19 MiB |
Previous results on branch: lz/grpc
Startup times
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| 247fbe6 | 352.18 ms | 336.59 ms | -15.59 ms |
| a4d676d | 354.69 ms | 354.87 ms | 0.18 ms |
| 166b609 | 358.77 ms | 357.23 ms | -1.54 ms |
App size
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| 247fbe6 | 14.55 MiB | 15.87 MiB | 1.31 MiB |
| a4d676d | 14.55 MiB | 15.87 MiB | 1.31 MiB |
| 166b609 | 14.55 MiB | 15.87 MiB | 1.31 MiB |
iOS Performance metrics 🚀
|
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| 6ba4675 | 1223.12 ms | 1238.17 ms | 15.04 ms |
| b3b2719 | 1257.07 ms | 1264.57 ms | 7.50 ms |
| e1ab497 | 1260.92 ms | 1260.22 ms | -0.69 ms |
| 944b773 | 1252.82 ms | 1254.08 ms | 1.27 ms |
| 765aa8b | 1259.09 ms | 1269.90 ms | 10.82 ms |
| 3fe9e80 | 1238.04 ms | 1245.22 ms | 7.18 ms |
| bfabaf2 | 1251.72 ms | 1253.38 ms | 1.67 ms |
| 9d516f3 | 1256.04 ms | 1262.70 ms | 6.66 ms |
| 8825ed8 | 1243.35 ms | 1240.39 ms | -2.96 ms |
| 40c8f93 | 1234.27 ms | 1261.98 ms | 27.71 ms |
App size
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| 6ba4675 | 7.86 MiB | 9.44 MiB | 1.58 MiB |
| b3b2719 | 5.83 MiB | 6.28 MiB | 462.27 KiB |
| e1ab497 | 5.53 MiB | 6.01 MiB | 487.96 KiB |
| 944b773 | 5.53 MiB | 6.00 MiB | 479.98 KiB |
| 765aa8b | 7.86 MiB | 9.44 MiB | 1.58 MiB |
| 3fe9e80 | 5.83 MiB | 6.28 MiB | 461.20 KiB |
| bfabaf2 | 5.53 MiB | 6.01 MiB | 487.95 KiB |
| 9d516f3 | 5.73 MiB | 6.18 MiB | 463.54 KiB |
| 8825ed8 | 5.53 MiB | 5.97 MiB | 453.71 KiB |
| 40c8f93 | 5.53 MiB | 6.00 MiB | 479.94 KiB |
Previous results on branch: lz/grpc
Startup times
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| 247fbe6 | 1248.98 ms | 1251.13 ms | 2.15 ms |
| 166b609 | 1263.55 ms | 1261.00 ms | -2.55 ms |
| a4d676d | 1256.67 ms | 1256.33 ms | -0.34 ms |
App size
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| 247fbe6 | 5.83 MiB | 6.28 MiB | 460.84 KiB |
| 166b609 | 5.83 MiB | 6.28 MiB | 462.27 KiB |
| a4d676d | 5.83 MiB | 6.28 MiB | 462.27 KiB |
![sentry-warden[bot]](https://avatars.githubusercontent.com/in/2755838?s=60&v=4){kind=small}
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Remove Flutter dependency, platform folders (linux/android/web), and empty widget tests. Examples now run with `dart run lib/main.dart`. Co-Authored-By: Claude <noreply@anthropic.com>
Flutter tool file, not needed for building or running. Co-Authored-By: Claude <noreply@anthropic.com>
buenaflor
left a comment
buenaflor
left a comment
There was a problem hiding this comment.
Did a quick PR review for now (didn't look into the details yet)
| class MockHub with NoSuchMethodProvider implements Hub { | ||
| List<CaptureExceptionCall> captureExceptionCalls = []; | ||
| List<AddBreadcrumbCall> addBreadcrumbCalls = []; | ||
| int closeCalls = 0; | ||
| bool _isEnabled = true; | ||
|
|
||
| @override | ||
| Scope get scope => Scope(_options); | ||
|
|
||
| final _options = defaultTestOptions(); | ||
|
|
||
| @override | ||
| // ignore: invalid_use_of_internal_member | ||
| SentryOptions get options => _options; | ||
|
|
||
| void reset() { | ||
| captureExceptionCalls = []; | ||
| addBreadcrumbCalls = []; | ||
| closeCalls = 0; | ||
| _isEnabled = true; | ||
| } |
There was a problem hiding this comment.
I think we already have a MockHub in the dart package, maybe it makes sense to move that mockhub from the dart package to the _sentry_testing package so we can use it easily across packages
There was a problem hiding this comment.
Is it ok if I do this task on a follow-up PR?
| span.setData('grpc.error_info.reason', detail.reason); | ||
| span.setData('grpc.error_info.domain', detail.domain); | ||
| if (detail.metadata.isNotEmpty) { | ||
| span.setData('grpc.error_info.metadata', detail.metadata.toString()); | ||
| } | ||
| } else if (detail is rpc.BadRequest) { | ||
| span.setData( | ||
| 'grpc.bad_request.field_violations', | ||
| detail.fieldViolations | ||
| .map((v) => '${v.field_1}: ${v.description}') | ||
| .join('; '), | ||
| ); | ||
| } else if (detail is rpc.RetryInfo) { | ||
| span.setData( | ||
| 'grpc.retry_info.retry_delay', | ||
| '${detail.retryDelay.seconds}s', | ||
| ); | ||
| } else if (detail is rpc.DebugInfo) { | ||
| span.setData('grpc.debug_info.detail', detail.detail); | ||
| } else if (detail is rpc.PreconditionFailure) { | ||
| span.setData( | ||
| 'grpc.precondition_failure.violations', | ||
| detail.violations | ||
| .map((v) => '${v.type}: ${v.subject} - ${v.description}') | ||
| .join('; '), | ||
| ); | ||
| } else if (detail is rpc.ResourceInfo) { | ||
| span.setData('grpc.resource_info.type', detail.resourceType); |
There was a problem hiding this comment.
Since we add a bunch of new attributes double check if OTEL has any equivalent attribute: https://opentelemetry.io/docs/specs/semconv/rpc/
if not, we should open a PR adding new ones in https://getsentry.github.io/sentry-conventions/
Let's double check also the python and go instrumentation to see how they structure their attributes and span ops
also let's add them to packages/dart/constants.dart in SemanticAttributesConstants
- Remove manual CHANGELOG entry (auto-generated from PR title) - Add gRPC error detail attribute constants to SemanticAttributesConstants - Use constants in interceptor instead of raw strings - Wrap fire-and-forget response.then() with unawaited() - Capture StackTrace in onError and pass to captureException - Gate ErrorInfo.metadata behind sendDefaultPii - Check tracePropagationTargets before injecting trace headers - Fix noSuchMethod to throw UnsupportedError instead of silently dropping - Add tests for _attachErrorDetails (ErrorInfo, BadRequest, RetryInfo, DebugInfo, ResourceInfo) including PII guard for metadata - Add tests for tracePropagationTargets header injection control Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
| } else if (detail is rpc.DebugInfo) { | ||
| span.setData( | ||
| SemanticAttributesConstants.grpcDebugInfoDetail, | ||
| detail.detail, | ||
| ); |
There was a problem hiding this comment.
DebugInfo.detail attached to span without sendDefaultPii guard
Unlike ErrorInfo.metadata (which is gated on sendDefaultPii), DebugInfo.detail is always added to the span — this field typically contains server-side stack traces that could expose sensitive internals and should also require sendDefaultPii to be enabled.
Evidence
ErrorInfo.metadatais only attached when_hub.options.sendDefaultPiiis true (line 270).DebugInfo.detailat lines 288–292 has no such guard; it is always written to the span viaspan.setData.- The gRPC
DebugInfoproto is explicitly designed to carry server stack traces (stack_entries) and free-form detail text, both potentially sensitive. - The PreconditionFailure, ResourceInfo, and QuotaFailure blocks also lack any PII gate, but those fields (type/subject/description) are lower risk than raw stack trace text.
Suggested fix: Wrap the DebugInfo setData call in a sendDefaultPii check, consistent with the ErrorInfo.metadata guard.
| } else if (detail is rpc.DebugInfo) { | |
| span.setData( | |
| SemanticAttributesConstants.grpcDebugInfoDetail, | |
| detail.detail, | |
| ); | |
| if (_hub.options.sendDefaultPii) { | |
| span.setData( | |
| SemanticAttributesConstants.grpcDebugInfoDetail, | |
| detail.detail, | |
| ); | |
| } |
Also found at 1 additional location
packages/grpc/lib/src/sentry_grpc_interceptor.dart:119
Identified by Warden code-review, find-bugs · J93-NEA
| CallOptions _buildModifiedOptions( | ||
| CallOptions options, |
There was a problem hiding this comment.
Non-standard credential headers bypass PII filter and are sent to Sentry by default
The _sensitiveHeaders allowlist only covers 4 headers (authorization, cookie, set-cookie, proxy-authorization). Common credential headers such as x-api-key, x-auth-token, x-access-token, and grpc-authorization pass through unconditionally when captureRequestHeaders is true (the default) and sendDefaultPii is false, leaking credentials into Sentry spans.
Evidence
_sensitiveHeadersis defined at lines 41-46 with exactly 4 entries:authorization,cookie,set-cookie,proxy-authorization._attachRequestDataat line 186 skips a key only when!sendPii && _sensitiveHeaders.contains(normalizedKey)— every other key proceeds tospan.setDataat line 187.captureRequestHeadersdefaults totrue(constructor line 69), so this runs for all users unless explicitly disabled.sendDefaultPiidefaults tofalseinSentryOptions, meaning no PII opt-in is needed to trigger the leak.- Headers like
x-api-keyare widely used for gRPC service authentication and would be sent to Sentry without any user awareness.
Identified by Warden find-bugs · 25T-W9K
Correct attribute keys per the OTEL gRPC semconv spec: - rpc.system.name instead of rpc.system - rpc.response.status_code (string) instead of rpc.grpc.status_code (int) - rpc.request.metadata.<key> instead of http.request.header.* - rpc.method now contains full service/method path (no rpc.service) Co-Authored-By: Claude <noreply@anthropic.com>
|
|
||
| /// Prefix for RPC request metadata. Append the lowercase metadata key. | ||
| /// | ||
| /// See https://opentelemetry.io/docs/specs/semconv/rpc/grpc/ |
There was a problem hiding this comment.
rpcRequestMetadataPrefix uses 'rpc.request.metadata.' instead of OTel-specified 'rpc.grpc.request.metadata.'
The OTel gRPC spec (linked in the doc comment) scopes request metadata under rpc.grpc.request.metadata.<key>; the current prefix rpc.request.metadata. is non-standard and breaks backend interoperability.
Evidence
- Doc comment on line 433 links to the OTel gRPC semconv.
- OTel gRPC spec defines metadata attributes as
rpc.grpc.request.metadata.<key>. - Constant value is
'rpc.request.metadata.'(line 435), missing thegrpc.infix. _attachRequestDatain the interceptor constructs keys as${rpcRequestMetadataPrefix}$normalizedKey, so all captured metadata headers will have the wrong OTel key prefix.
Also found at 1 additional location
packages/dart/lib/src/constants.dart:421
Identified by Warden code-review · B83-A56
| /// The RPC system identifier. For gRPC, always `'grpc'`. | ||
| /// | ||
| /// See https://opentelemetry.io/docs/specs/semconv/rpc/grpc/ | ||
| static const rpcSystemName = 'rpc.system.name'; |
There was a problem hiding this comment.
rpcSystemName string value diverges from OTel spec ('rpc.system.name' vs 'rpc.system')
The constant value is 'rpc.system.name' but the OpenTelemetry RPC semantic convention (cited in the doc comment) defines this attribute as rpc.system. Spans emitted with this key will not be recognized by OTel-compatible backends (Grafana, Jaeger, etc.) that filter or group by rpc.system.
Evidence
- OTel RPC semconv (https://opentelemetry.io/docs/specs/semconv/rpc/grpc/) defines the attribute as
rpc.system, notrpc.system.name. - The doc comment directly links to that spec page yet the string value
'rpc.system.name'is incorrect. _attachRpcAttributesinsentry_grpc_interceptor.dartcallsspan.setData(SemanticAttributesConstants.rpcSystemName, 'grpc'), so spans in production will carry the non-standard key.- Tests assert against
SemanticAttributesConstants.rpcSystemName(line 115 in test file), so they pass but validate the wrong key.
Suggested fix: Change the string value to the correct OTel attribute name.
| static const rpcSystemName = 'rpc.system.name'; | |
| static const rpcSystemName = 'rpc.system'; |
Identified by Warden find-bugs · U3Z-TQL
There was a problem hiding this comment.
rpc.name doesn't exist
| /// The string representation of the gRPC status code (e.g., `'OK'`, `'NOT_FOUND'`). | ||
| /// | ||
| /// See https://opentelemetry.io/docs/specs/semconv/rpc/grpc/ | ||
| static const rpcResponseStatusCode = 'rpc.response.status_code'; |
There was a problem hiding this comment.
rpcResponseStatusCode string value diverges from OTel spec ('rpc.response.status_code' vs 'rpc.grpc.status_code')
The constant value is 'rpc.response.status_code' but the OTel gRPC semantic convention (cited in the doc comment) defines the gRPC status code attribute as rpc.grpc.status_code. This will emit a non-standard key that OTel tooling won't map to gRPC status metrics or dashboards.
Evidence
- OTel RPC/gRPC semconv specifies
rpc.grpc.status_codefor the gRPC status code;rpc.response.status_codeis not defined in that spec. - The doc comment says 'See https://opentelemetry.io/docs/specs/semconv/rpc/grpc/' but the string contradicts it.
interceptUnaryinsentry_grpc_interceptor.dartcallsspan.setData(SemanticAttributesConstants.rpcResponseStatusCode, 'OK')and similar on the error path (lines 113 and 133), propagating the wrong key on every span.- Tests at lines 134 and 259 validate against the constant, so they pass but verify the non-standard key.
Suggested fix: Change the string value to the correct OTel attribute name.
| static const rpcResponseStatusCode = 'rpc.response.status_code'; | |
| static const rpcResponseStatusCode = 'rpc.grpc.status_code'; |
Identified by Warden find-bugs · GCT-9GL
There was a problem hiding this comment.
rpc.grpc.status_code is not there, as odd as it sounds, they adopted to use a general key name from rpc instead of adding grpc to the key name
2 participants
📜 Summary
captureRequestHeadersoption toSentryGrpcInterceptorto attach outgoing gRPC metadata (request headers) to spans, with automatic PII filtering for sensitive headers (authorization,cookie,set-cookie,proxy-authorization)ErrorInfo,BadRequest,RetryInfo,DebugInfo,PreconditionFailure,ResourceInfo,QuotaFailure) to spans on failed callscaptureBody— no other Sentry SDK gRPC integration captures the request bodyDummyMessageclass (replaces rawList<int>hand-encoding),WithHeadersbutton demonstrating custom gRPC metadata,SentryFlutterinit in grpc4 example💡 Motivation and Context
Brings the Dart gRPC interceptor closer to parity with the Python and JS integrations for request header propagation, while adding richer error detail capture that those SDKs don't have.
💚 How did you test it?
grpcb.inhttp.request.header.*data when metadata is set📝 Checklist
sendDefaultPiiis enabled🔮 Next steps