flatkv cache by cody-littley · Pull Request #3027 · sei-protocol/sei-chain

cody-littley · 2026-03-05T21:45:18Z

Describe your changes and provide context

Add a caching layer to FlatKV, more than doubling performance in cryptosim benchmarks.

Testing performed to validate your change

Unit tests, ran benchmark over several days.

github-actions · 2026-03-05T21:46:17Z

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
`✅ passed`	`✅ passed`	`✅ passed`	`✅ passed`	Mar 27, 2026, 1:05 PM

sei-db/state_db/sc/flatkv/flatcache/cache_impl.go

sei-db/db_engine/pebbledb/pebblecache/cache_impl.go

sei-db/db_engine/pebbledb/pebblecache/read_scheduler.go

codecov · 2026-03-06T19:02:20Z

Codecov Report

❌ Patch coverage is 74.03846% with 108 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.74%. Comparing base (9abe71e) to head (33378ce).
⚠️ Report is 8 commits behind head on main.

Files with missing lines	Patch %	Lines
sei-db/state_db/sc/flatkv/store_write.go	78.47%	18 Missing and 13 partials ⚠️
sei-db/db_engine/pebbledb/db.go	50.00%	18 Missing and 5 partials ⚠️
sei-db/state_db/sc/flatkv/config.go	65.62%	11 Missing and 11 partials ⚠️
sei-db/state_db/sc/flatkv/store.go	75.75%	8 Missing and 8 partials ⚠️
sei-db/db_engine/dbcache/cache_config.go	45.45%	6 Missing ⚠️
sei-db/db_engine/pebbledb/pebbledb_config.go	60.00%	2 Missing and 2 partials ⚠️
sei-db/common/threading/fixed_pool.go	0.00%	1 Missing and 1 partial ⚠️
sei-db/db_engine/pebbledb/batch.go	60.00%	1 Missing and 1 partial ⚠️
sei-db/state_db/sc/composite/store.go	60.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3027      +/-   ##
==========================================
+ Coverage   58.62%   58.74%   +0.12%     
==========================================
  Files        2099     2104       +5     
  Lines      173514   175077    +1563     
==========================================
+ Hits       101716   102852    +1136     
- Misses      62750    63139     +389     
- Partials     9048     9086      +38

Flag	Coverage Δ
sei-chain-pr	`73.22% <74.03%> (?)`
sei-db	`70.41% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
sei-db/common/metrics/phase_timer.go	`74.46% <100.00%> (+0.55%)`	⬆️
sei-db/config/sc_config.go	`100.00% <100.00%> (ø)`
sei-db/db_engine/dbcache/cache.go	`77.77% <100.00%> (+66.66%)`	⬆️
sei-db/db_engine/dbcache/cache_impl.go	`95.65% <100.00%> (ø)`
sei-db/db_engine/pebbledb/pebble_metrics.go	`69.09% <100.00%> (ø)`
sei-db/db_engine/pebbledb/pebbledb_test_config.go	`100.00% <100.00%> (ø)`
sei-db/db_engine/types/types.go	`100.00% <ø> (ø)`
sei-db/state_db/sc/flatkv/flatkv_test_config.go	`100.00% <100.00%> (ø)`
sei-db/state_db/sc/flatkv/snapshot.go	`66.76% <100.00%> (+0.19%)`	⬆️
sei-db/state_db/sc/flatkv/store_lifecycle.go	`62.50% <100.00%> (+4.75%)`	⬆️
... and 9 more

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

sei-db/db_engine/dbcache/cache_impl.go

sei-db/db_engine/pebbledb/pebblecache/shard.go

sei-db/state_db/sc/flatkv/store_write.go

sei-db/common/utils/work_pool.go

sei-db/common/threading/fixed_pool.go

sei-db/common/threading/pool_impl.go

yzang2019 · 2026-03-20T21:47:51Z

sei-db/state_db/sc/composite/store.go

 	}

-	return store
+	return store, nil


Instead of changing the interface to return an error, wouldn't it be easier to just panic inside this constructor? Is there any benefit to return the error if we failed to create the store?

Converted back to a panic().

As a general pattern, I prefer to return errors when possible instead of using panic, except in cases where I'm just doing sanity/invariant testing. But this isn't something I feel strongly enough to argue about, so I don't mind switching this one back. 😜

yzang2019 · 2026-03-20T22:07:23Z

sei-db/config/sc_config.go


 	// FlatKVConfig is the configuration for the FlatKV (EVM) backend
-	FlatKVConfig flatkv.Config
+	FlatKVConfig *flatkv.Config


Why do we need to make this a pointer?

Reverted back to a direct reference.

No need to make this a pointer, except that I prefer pointer types for any struct that has more than one or two fields. It's unlikely that this is going to be a major performance factor in this case, so I don't mind using a direct reference if that's your preferred pattern.

yzang2019 · 2026-03-20T23:18:45Z

sei-cosmos/storev2/rootmulti/store.go

 	if err := scStore.CleanupCrashArtifacts(); err != nil {
 		panic(err)
 	}
+	if err := scStore.CleanupCrashArtifacts(); err != nil {


Duplicate CleanupCrashArtifacts call here, we should remove the one above?

Removed duplicate call. This was probably happened when I merged upstream into my branch.

yzang2019 · 2026-03-20T23:19:17Z

sei-db/state_db/sc/flatkv/store_write.go

+		return fmt.Errorf("failed to batch read old values: %w", err)
+	}
+
+	s.phaseTimer.SetPhase("apply_change_sets_prepare")


Duplicate code s.phaseTimer.SetPhase("apply_change_sets_prepare")

yzang2019 · 2026-03-20T23:32:44Z

sei-db/db_engine/pebbledb/pebbledb_config.go

+)
+
+// Configuration for the PebbleDB database.
+type PebbleDBConfig struct {


These configs are not specific to PebbleDB, instead, they are specific to flatkv, I think we should move them to flatkv instead of put them here, we could change the db engine in the future for flatkv, but these configs are universal to all db engines

Extracted configuration.

yzang2019 · 2026-03-20T23:36:05Z

sei-db/db_engine/pebbledb/pebbledb_config.go

+	// The number of shards in the key-value cache. Must be a power of two and greater than 0.
+	CacheShardCount uint64
+	// The size of pebbleDB's internal block cache, in bytes.
+	BlockCacheSize int


Would recommend remove this BlockCacheSize from the config, we can set a tuned default value for now, we can add it in the future if we are 100% sure we need a different specific value for each DB.
In any way, we are not going rely on the PebbleDB internal cache for read optimization, so would recommend maybe just set a default number

Removed in this PR and set to previous default, but I'd like to discuss adding it back in a future.

The block cache still serves an important role in database performance. It's used by pebble to cache LSM blocks, and we can't just disable it.

The reason why I added this config was because I thought the default value of 0.5gb was way too large for unit tests, which I generally prefer to use very little resources. Eventually I'd like to run unit tests in parallel, that will be tricky if each FlatKV instance utilizes 2.5gb memory.

Although I haven't done so yet, I'd actually like to enable configuration of ALL pebbleDB settings (although we can give them sane defaults). Right now it's tricky to experiment with pebble configuration because it's all hard coded.

Perhaps good for a quick in-person discussion.

yzang2019 · 2026-03-20T23:39:23Z

sei-db/db_engine/pebbledb/db.go

-	opts types.OpenOptions,
-	enableMetrics bool,
+	config *PebbleDBConfig,
+	comparer *pebble.Comparer,


Do we have to pass the comparer in? Under what scenario will we pass in a custom comparer?

Removed comparer parameter.

yzang2019 · 2026-03-20T23:44:11Z

sei-db/state_db/sc/flatkv/config.go

+		return fmt.Errorf("account db config is invalid: %w", c.AccountDBConfig.Validate())
+	}
+	if c.CodeDBConfig.Validate() != nil {
+		return fmt.Errorf("code db config is invalid: %w", c.CodeDBConfig.Validate())


Nit: We shouldn't need to call Validate() twice here

yzang2019 · 2026-03-20T23:46:45Z

sei-db/state_db/sc/flatkv/config.go

+		return fmt.Errorf("metadata db config is invalid: %w", c.MetadataDBConfig.Validate())
+	}
+
+	if c.ReaderThreadsPerCore < 0 {


If ReaderThreadsPerCore == 0, and ReaderConstantThreadCount == 0, it will create a pool with 0 workers, causing deadlock issue?

Good point. I've updated the pool constructor to more elegantly handle this case:

if workers <= 0 { workers = 1 }

yzang2019 · 2026-03-20T23:51:07Z

sei-db/db_engine/pebbledb/pebbledb_config.go

+	if c.DataDir == "" {
+		return fmt.Errorf("data dir is required")
+	}
+	if c.CacheSize > 0 && (c.CacheShardCount&(c.CacheShardCount-1)) != 0 {


Shall we also validate and make sure CacheShardCount > 0?

Good point. Validation added.

yzang2019 · 2026-03-20T23:58:13Z

One blocker comment: I think we should decouple cache from db_engine as much as possible, which allows FlatKV to switch different db_engine easily without reimplmenting the cache for each engine in the future

Cody Littley added 7 commits March 5, 2026 13:31

Created a cache for flatKV.

7c5e216

checkpoint

4a404ee

incremental progress

d36e825

address feedback

2ccbe62

more fixes

f412e85

bugfix

e310037

wire in cache

cf1071c

cody-littley self-assigned this Mar 5, 2026

cody-littley added the non-app-hash-breaking label Mar 5, 2026

github-advanced-security bot found potential problems Mar 5, 2026

View reviewed changes

sei-db/state_db/sc/flatkv/flatcache/cache_impl.go Fixed Show fixed Hide fixed

sei-db/db_engine/pebbledb/pebblecache/cache_impl.go Fixed Show fixed Hide fixed

sei-db/db_engine/pebbledb/pebblecache/read_scheduler.go Fixed Show fixed Hide fixed

Cody Littley added 9 commits March 6, 2026 09:21

Merge branch 'main' into cjl/flatkv-cache

11232ff

incremental improvements

a8c1c75

checkin

221d114

Moved where the cache sits

8eca079

bugfix

267feae

Batch update the cache

50b0be6

Add batch read to cache

2ca00d6

Add batch get to db interface

8f8534a

integrate batch reads

23c0277

github-advanced-security bot found potential problems Mar 6, 2026

View reviewed changes

Cody Littley added 2 commits March 6, 2026 13:18

wire in cache

02d3ca1

Introduce work pool, size caches differently

7ee1b08

github-advanced-security bot found potential problems Mar 6, 2026

View reviewed changes

sei-db/common/utils/work_pool.go Fixed Show fixed Hide fixed

sei-db/common/threading/fixed_pool.go Fixed Show fixed Hide fixed

sei-db/common/threading/pool_impl.go Fixed Show fixed Hide fixed

Cody Littley added 5 commits March 6, 2026 14:46

bugfix

20c70c3

Add unit constants

b714789

refactor threading utils

cc9d41d

cleanup

53b2bd8

Cleanup, fix race condition

c10e0cd

Cody Littley added 10 commits March 17, 2026 11:36

made suggested changes

003fcc9

made suggested change

a208a1b

made suggested changes

157a600

fix unit test

b41639f

fix unit test

fe31475

Merge branch 'cjl/cache-impl' into cjl/flatkv-cache

0702197

Merge branch 'main' into cjl/flatkv-cache

6d435f5

fixed merge bugs

64f8530

fix teardown race

d9c5fc1

Add logging metric, clean up log files before/after run

14593ec

cody-littley marked this pull request as ready for review March 20, 2026 15:23

Cody Littley added 3 commits March 20, 2026 10:35

fix unit test

2d88076

fix unit tests

ccad074

fix unit test

38ffd35

yzang2019 reviewed Mar 20, 2026

View reviewed changes

Cody Littley added 5 commits March 25, 2026 08:21

made suggested changes

f143d30

config changes

a18fd93

made suggested changes

34e711d

Merge branch 'main' into cjl/flatkv-cache

b596f89

bugfix

33378ce

Conversation

cody-littley commented Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe your changes and provide context

Testing performed to validate your change

Uh oh!

github-actions bot commented Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Mar 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yzang2019 Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yzang2019 commented Mar 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

cody-littley commented Mar 5, 2026 •

edited

Loading

github-actions bot commented Mar 5, 2026 •

edited

Loading

codecov bot commented Mar 6, 2026 •

edited

Loading

yzang2019 Mar 20, 2026 •

edited

Loading