Add otel metrics for seidb #114

yzang2019 · 2025-10-01T15:38:36Z

Describe your changes and provide context

This PR add couple of new metrics using otel for seidb

Testing performed to validate your change

Example:

* main: Integrate UCI versioning and release (#113) Rocksdb update interface (#112)

sc/memiavl/db.go

 	db.mtx.Lock()
 	defer db.mtx.Unlock()
-
+	startTime := time.Now()


sc/memiavl/db.go

masih

One blocker about measuring latency in relation to errors. Left a bunch of suggestions.

masih · 2025-10-06T15:35:40Z

sc/memiavl/db.go

-
+	startTime := time.Now()
+	defer func() {
+		metrics.SeiDBMetrics.RestartLatency.Record(context.Background(), time.Since(startTime).Seconds())


Tag by "status" in case there is an error. Otherwise the times measured could be missleading.

Ditto for all such defer statements for functions that may also return error.

Make sense, will add

masih · 2025-10-06T15:42:40Z

sc/memiavl/db.go

+	startTime := time.Now()
+	defer func() {
+
+		metrics.SeiDBMetrics.CommitLatency.Record(context.Background(), time.Since(startTime).Milliseconds())


declare context once and reuse it?

sc/memiavl/db.go

masih · 2025-10-06T15:47:05Z

sc/memiavl/mem_node.go

+	left Node,
+	right Node) *MemNode {
+	TotalNumOfMemNode.Add(1)
+	TotalMemNodeSize.Add(int64(120 + len(key)))


Move these increments to when the branch node is being added to its parent struct? Instantiation alone does not mean increase in size, right?

Ditto re newLeafNode.

Each time we call newBranchNode or newLeafNode, it's always gonna mean the size would be increase, so I feel this is the best place to add size and count increase

Actually ignore my previous comment, let me check again, the setRecursive function might be a bit confusing

Ok, just double checked, there are only 2 places where new memNodes are created:

setRecursive (

sei-db/sc/memiavl/node.go

Line 37 in cc4d74f

func setRecursive(node Node, key, value []byte, version, cowVersion uint32) (*MemNode, bool) {

)

mutate (

sei-db/sc/memiavl/persisted_node.go

Line 147 in cc4d74f

func (node PersistedNode) Mutate(version, _ uint32) *MemNode {

)

So both places are creating memnodes and making it as part of the tree

I could either choose to adding the bump logic to all these places or I can choose to just add the bump size logic in initialization, which is less code duplicates. Feel like that latter is easier?

codecov · 2025-10-06T18:03:01Z

Codecov Report

❌ Patch coverage is 94.59459% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 40.00%. Comparing base (f369e78) to head (0b567e7).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
common/metrics/metrics.go	0.00%	4 Missing ⚠️

❌ Your project status has failed because the head coverage (40.00%) is below the target coverage (60.00%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #114      +/-   ##
==========================================
+ Coverage   39.80%   40.00%   +0.19%     
==========================================
  Files          58       59       +1     
  Lines        7477     7510      +33     
==========================================
+ Hits         2976     3004      +28     
- Misses       4156     4161       +5     
  Partials      345      345

Files with missing lines	Coverage Δ
sc/memiavl/db.go	`61.81% <100.00%> (+1.37%)`	⬆️
sc/memiavl/mem_node.go	`93.37% <100.00%> (+1.06%)`	⬆️
sc/memiavl/multitree.go	`71.60% <100.00%> (+0.08%)`	⬆️
sc/memiavl/node.go	`71.66% <100.00%> (-1.78%)`	⬇️
sc/memiavl/persisted_node.go	`91.12% <100.00%> (-0.61%)`	⬇️
common/metrics/metrics.go	`0.00% <0.00%> (ø)`

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

sc/memiavl/db.go

 		fileLock FileLock
 	)
-
+	startTime := time.Now()


sc/memiavl/db.go

 	db.mtx.Lock()
 	defer db.mtx.Unlock()
-
+	startTime := time.Now()


sc/memiavl/db.go

 	go func() {
 		defer close(ch)
-
+		startTime := time.Now()


blindchaser · 2025-10-06T21:46:20Z

sc/memiavl/mem_node.go


 func newLeafNode(key, value []byte, version uint32) *MemNode {
+	TotalNumOfMemNode.Add(1)
+	TotalMemNodeSize.Add(int64(120 + len(key) + len(value)))


can we add comments about why it's 120?

Good point, added

Alignment & offsets (64-bit)

height = 1

padding after height = 3 (to align uint32)

version = 4 → offset now 8

size = 8 → offset 16

key = 24 → offset 40

value = 24 → offset 64

left = 16 → offset 80

right = 16 → offset 96

hash = 24 → offset 120

Total struct size = 120 bytes (already a multiple of 8, so no trailing padding).

blindchaser · 2025-10-06T21:55:16Z

common/metrics/metrics.go

+		SnapshotCreationLatency metric.Float64Histogram
+		CommitLatency           metric.Int64Histogram
+		ApplyChangesetLatency   metric.Int64Histogram
+		MemNodeTotalSize        metric.Int64Gauge


from chatGPT: MemNodeSizeObs metric.Int64ObservableGauge type is good for “now” value , same for MemNodeCount, for your consideration

Yeah, that's what we want, please check the graph here:

sc/memiavl/mem_node.go

 	"io"
-	"math"
+	"sync/atomic"
+	"unsafe"


masih

Couple of gosec issues to fix, otherwise LGTM 👍

* main: Revert "Merge pull request #105 from sei-protocol/PebbleSurfaceErrors" (#118)

yzang2019 added 8 commits October 1, 2025 08:38

Add otel metrics for seidb

530da86

Add commit latency metrics

488613e

Remove server

636e74a

Fix metrics

0b0a66e

Add metrics for seidb

cc4d74f

Merge branch 'main' into yzang/add-metrics

8049bb1

* main: Integrate UCI versioning and release (#113) Rocksdb update interface (#112)

fix go mod

534de71

Fix go version

5252396

github-advanced-security bot found potential problems Oct 6, 2025

View reviewed changes

Bump lint version

fb465e5

yzang2019 requested a review from blindchaser October 6, 2025 15:46

masih reviewed Oct 6, 2025

View reviewed changes

yzang2019 added 16 commits October 6, 2025 08:48

Bump lint version

ce1e035

Bump lint version

d888816

Bump lint version

d261396

Fix golint

b610ab8

Fix golint

06019a0

Fix golint

6fa9a20

Fix golint

e3f6e79

Fix golint

30fce5f

Fix golint

450451f

Fix golint

c547152

Fix golint

b20ee39

Fix golint

a41d132

Fix golint

c5a0004

Fix golint

ddc32cc

Fix golint

44bd4f8

Fix golint

c5de273

Kbhat1 approved these changes Oct 6, 2025

View reviewed changes

Add latency metric with success

1395e55

github-advanced-security bot found potential problems Oct 6, 2025

View reviewed changes

yzang2019 requested a review from masih October 6, 2025 20:52

blindchaser reviewed Oct 6, 2025

View reviewed changes

Add comment

7ace1a5

yzang2019 requested a review from blindchaser October 7, 2025 05:10

blindchaser approved these changes Oct 7, 2025

View reviewed changes

yzang2019 added 2 commits October 7, 2025 22:31

Merge latest

838e4f2

Update MemNode metrics from parent

8cfe551

github-advanced-security bot found potential problems Oct 8, 2025

View reviewed changes

sc/memiavl/mem_node.go

"io"

"math"

"sync/atomic"

"unsafe"

Check notice

Code scanning / CodeQL

Sensitive package import Note

Certain system packages contain functions which may be a possible source of non-determinism

Add one more metric

925e75b

masih approved these changes Oct 8, 2025

View reviewed changes

yzang2019 added 2 commits October 8, 2025 07:10

Fix lint

9f4ce5c

Merge branch 'main' into yzang/add-metrics

0b567e7

* main: Revert "Merge pull request #105 from sei-protocol/PebbleSurfaceErrors" (#118)

yzang2019 enabled auto-merge (squash) October 8, 2025 14:29

yzang2019 merged commit c3ee918 into main Oct 8, 2025
5 checks passed

yzang2019 deleted the yzang/add-metrics branch October 8, 2025 14:30

Add otel metrics for seidb #114

Add otel metrics for seidb #114

Uh oh!

Conversation

yzang2019 commented Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe your changes and provide context

Testing performed to validate your change

Uh oh!

Uh oh!

Check warning

Uh oh!

Uh oh!

masih left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yzang2019 Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yzang2019 Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Check warning

Check warning

Check warning

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Check notice

masih left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

yzang2019 commented Oct 1, 2025 •

edited

Loading

yzang2019 Oct 6, 2025 •

edited

Loading

yzang2019 Oct 6, 2025 •

edited

Loading

codecov bot commented Oct 6, 2025 •

edited

Loading