Skip to content

[Bug] Data Loss / Inconsistency on process exit #125

@RossWilliams

Description

@RossWilliams

Describe the bug

If I restart the process I can see missing data in the GSIs. It appears delayed writes to GSIs are not crash safe and I do not see any repair or recovery mechanism.

In addition, I see that GSI write delays can be over 10 minutes. Possibly longer but I did not actively monitor. This is an unacceptable data inconsistency compared to managed DynamoDB.

To Reproduce

  1. Create a table with 3 GSIs
  2. Load 20GB of data
  3. Put an item. Inspect table data and GSI data with queries on an interval to check delay for GSI propagation.
  4. Load another 20GB of data.
  5. Kill and restart the process. wait. GSIs never become consistent with table.

Expected behavior

AWS DynamoDB has back pressure from the GSI to the table when latency breaches a certain level to have a bounded latency. ExtendDB as a replacement must offer a reasonable level of bounded latency.

GSI permanent inconsistency on crash or restart should be treated as a small step below a data loss condition and should never happen.

Actual behavior

No back pressure for GSI catchup leads to unbounded propagation delays. Restarts result in permanent inconsistency and loss of data in the GSI without a repair or recovery mechanism.

Environment

  • ExtendDB version: 140a1e5
  • Operating system: macOS 26
  • Rust version (if building from source): 1.94
  • Client SDK/driver and version: http
  • Deployment method (binary, container, source): source

Additional context

Effectively lost data would need a user to scan and re-touch every item to get GSIs back to consistent.

Checklist

  • I have searched existing issues for duplicates
  • I have included the ExtendDB version and environment details
  • I can reproduce this with the latest release

Metadata

Metadata

Labels

No labels
No labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions