Skip to content

[Bug] Major Performance degradation with 20GB dataset. Timeouts with 200GB dataset. #124

@RossWilliams

Description

@RossWilliams

Describe the bug

Update operations become unreasonably slow if they change a GSI due to lack of an index. Problematic SQL missing indexes:

DELETE FROM "_ddb_<gsi_table>" WHERE base_pk = $1 AND base_sk_s = $2

At just 20GB I'm seeing updates take from 700ms-11s . At 200GB a single update_item call fails with a timeout. I'm using gsi_propagation_delay_ms=0.

To Reproduce

  1. Set gsi_propagation_delay_ms=0.
  2. Create a table with 3 gsis
  3. Load 20GB of data using all GSIs for the load
  4. with concurrency of 5 start updating the existing data with update_item. The updates should update a gsi key.
  5. Observe latency and max throughput

Expected behavior

DynamoDB offers consistent performance for common production databases.

Actual behavior

Over 30 seconds running:

Requests per second achieved: 4.4
p50 update latency: 730ms
p99 update latency: 11,677ms

A small database shows unreasonable performance degradation, likely severely impacting user-visible latency. A reasonably sized database will cease to operate.

Environment

  • ExtendDB version: 140a1e5
  • Operating system: macOS 26
  • Rust version (if building from source): 1.94
  • Client SDK/driver and version: https
  • Deployment method (binary, container, source): source

Checklist

  • I have searched existing issues for duplicates
  • I have included the ExtendDB version and environment details
  • I can reproduce this with the latest release

Metadata

Metadata

Labels

No labels
No labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions