"file is not a database", followed by "cannot rollback - no transaction is active" in ShapeDb

[alco]

This is a report from a user.

• Update to 1.4.11
• Restart Electric Fargate Task
• This corrupted the entire EFS with the Got: {:error, "file is not a database"} error
• Then I restarted the entire task with a fresh EFS volume to reset the shape logs.

That resulted in the following log dump (truncated; full log dump is almost 3MB):

Excerpt from the full log dump

timestamp,message
1772483160228,"20:26:00.226 pid=<0.27821.0> [error] GenServer {:""Electric.ProcessRegistry:single_stack"", {Electric.ShapeCache.ShapeStatus.ShapeDb.Statistics, nil}} terminating"
1772483160228,** (MatchError) no match of right hand side value:
1772483160228,"    {:error, ""cannot rollback - no transaction is active""}"
1772483160228,    (electric 1.4.11) lib/electric/shape_cache/shape_status/shape_db/connection.ex:429: Electric.ShapeCache.ShapeStatus.ShapeDb.Connection.transaction/2
1772483160228,    (electric 1.4.11) lib/electric/shape_cache/shape_status/shape_db/connection.ex:280: anonymous fn/3 in Electric.ShapeCache.ShapeStatus.ShapeDb.Connection.checkout_write!/4
1772483160228,    (nimble_pool 1.1.0) lib/nimble_pool.ex:462: NimblePool.checkout!/4
1772483160228,    (electric 1.4.11) lib/electric/shape_cache/shape_status/shape_db/statistics.ex:60: Electric.ShapeCache.ShapeStatus.ShapeDb.Statistics.handle_continue/2
1772483160228,    (stdlib 7.1) gen_server.erl:2424: :gen_server.try_handle_continue/3
1772483160228,    (stdlib 7.1) gen_server.erl:2291: :gen_server.loop/4
1772483160228,    (stdlib 7.1) proc_lib.erl:333: :proc_lib.init_p_do_apply/3
1772483160228,"Process Label: {:shape_db_statistics, ""single_stack""}"
1772483160228,"Last message: {:continue, {:initialize_stats, false}}"

1772483162256,20:26:02.255 pid=<0.27025.0> [error] WriteBuffer batch write failed: ** (MatchError) no match of right hand side value:
1772483162256,"    {:error, ""cannot rollback - no transaction is active""}"
1772483162256,    (electric 1.4.11) lib/electric/shape_cache/shape_status/shape_db/connection.ex:429: Electric.ShapeCache.ShapeStatus.ShapeDb.Connection.transaction/2
1772483162256,    (electric 1.4.11) lib/electric/shape_cache/shape_status/shape_db/connection.ex:280: anonymous fn/3 in Electric.ShapeCache.ShapeStatus.ShapeDb.Connection.checkout_write!/4
1772483162256,    (nimble_pool 1.1.0) lib/nimble_pool.ex:462: NimblePool.checkout!/4
1772483162256,    (electric 1.4.11) lib/electric/shape_cache/shape_status/shape_db/write_buffer.ex:433: Electric.ShapeCache.ShapeStatus.ShapeDb.WriteBuffer.do_batch_write/2
1772483162256,    (electric 1.4.11) lib/electric/shape_cache/shape_status/shape_db/write_buffer.ex:379: anonymous fn/5 in Electric.ShapeCache.ShapeStatus.ShapeDb.WriteBuffer.flush_entries/2
1772483162256,    (opentelemetry 1.7.0) /builder/electric/deps/opentelemetry/src/otel_tracer_default.erl:47: :otel_tracer_default.with_span/5
1772483162256,    (electric 1.4.11) lib/electric/telemetry/open_telemetry.ex:87: anonymous fn/5 in Electric.Telemetry.OpenTelemetry.do_with_span/5
1772483162256,    (telemetry 1.3.0) /builder/electric/deps/telemetry/src/telemetry.erl:324: :telemetry.span/3
1772483162256,    (electric 1.4.11) lib/electric/shape_cache/shape_status/shape_db/write_buffer.ex:323: Electric.ShapeCache.ShapeStatus.ShapeDb.WriteBuffer.handle_info/2
1772483162256,    (stdlib 7.1) gen_server.erl:2434: :gen_server.try_handle_info/3
1772483162256,    (stdlib 7.1) gen_server.erl:2420: :gen_server.handle_msg/3
1772483162256,    (stdlib 7.1) proc_lib.erl:333: :proc_lib.init_p_do_apply/3

1772483164297,"20:26:04.284 pid=<0.27836.0> [error] GenServer {:""Electric.ProcessRegistry:single_stack"", {Electric.ShapeCache, nil}} terminating"
1772483164297,** (MatchError) no match of right hand side value:
1772483164297,"    {:error, ""cannot rollback - no transaction is active""}"
1772483164297,    (electric 1.4.11) lib/electric/shape_cache/shape_status/shape_db/connection.ex:429: Electric.ShapeCache.ShapeStatus.ShapeDb.Connection.transaction/2
1772483164297,    (electric 1.4.11) lib/electric/shape_cache/shape_status/shape_db/connection.ex:280: anonymous fn/3 in Electric.ShapeCache.ShapeStatus.ShapeDb.Connection.checkout_write!/4
1772483164297,    (nimble_pool 1.1.0) lib/nimble_pool.ex:462: NimblePool.checkout!/4
1772483164297,    (electric 1.4.11) lib/electric/shape_cache/shape_status/shape_db.ex:104: Electric.ShapeCache.ShapeStatus.ShapeDb.handle_for_shape_inner/3
1772483164297,    (opentelemetry 1.7.0) /builder/electric/deps/opentelemetry/src/otel_tracer_default.erl:47: :otel_tracer_default.with_span/5
1772483164297,    (electric 1.4.11) lib/electric/telemetry/open_telemetry.ex:87: anonymous fn/5 in Electric.Telemetry.OpenTelemetry.do_with_span/5
1772483164297,    (telemetry 1.3.0) /builder/electric/deps/telemetry/src/telemetry.erl:324: :telemetry.span/3
1772483164297,    (electric 1.4.11) lib/electric/shape_cache.ex:318: Electric.ShapeCache.maybe_create_shape/2
1772483164297,    (electric 1.4.11) lib/electric/shape_cache.ex:281: Electric.ShapeCache.handle_call/3
1772483164297,    (stdlib 7.1) gen_server.erl:2470: :gen_server.try_handle_call/4
1772483164297,    (stdlib 7.1) gen_server.erl:2499: :gen_server.handle_msg/3
1772483164297,    (stdlib 7.1) proc_lib.erl:333: :proc_lib.init_p_do_apply/3
1772483164297,"Process Label: {:shape_cache, ""single_stack""}"
1772483164297,"Last message (from #PID<0.27828.0>): {:create_or_wait_shape_handle, Shape.new!(..., where: "...", columns: [...]), %{:""$__otel_baggage_ctx_key"" => ...}}"

1772483164359,"20:26:04.346 pid=<0.27404.0> request_id=GJkgCyvBgxgM0fIAGdSx [error] ** (exit) exited in: GenServer.call({:via, Registry, {:""Electric.ProcessRegistry:single_stack"", {Electric.ShapeCache, nil}}}, {:create_or_wait_shape_handle, Shape.new!(...), %{:""$__otel_baggage_ctx_key"" => ...}}, 30000)"
1772483164359,    ** (EXIT) an exception was raised:
1772483164359,        ** (MatchError) no match of right hand side value:
1772483164359,"    {:error, ""cannot rollback - no transaction is active""}"
1772483164359,            (electric 1.4.11) lib/electric/shape_cache/shape_status/shape_db/connection.ex:429: Electric.ShapeCache.ShapeStatus.ShapeDb.Connection.transaction/2
1772483164359,            (electric 1.4.11) lib/electric/shape_cache/shape_status/shape_db/connection.ex:280: anonymous fn/3 in Electric.ShapeCache.ShapeStatus.ShapeDb.Connection.checkout_write!/4
1772483164359,            (nimble_pool 1.1.0) lib/nimble_pool.ex:462: NimblePool.checkout!/4
1772483164359,            (electric 1.4.11) lib/electric/shape_cache/shape_status/shape_db.ex:104: Electric.ShapeCache.ShapeStatus.ShapeDb.handle_for_shape_inner/3
1772483164359,            (opentelemetry 1.7.0) /builder/electric/deps/opentelemetry/src/otel_tracer_default.erl:47: :otel_tracer_default.with_span/5
1772483164359,            (electric 1.4.11) lib/electric/telemetry/open_telemetry.ex:87: anonymous fn/5 in Electric.Telemetry.OpenTelemetry.do_with_span/5
1772483164359,            (telemetry 1.3.0) /builder/electric/deps/telemetry/src/telemetry.erl:324: :telemetry.span/3
1772483164359,            (electric 1.4.11) lib/electric/shape_cache.ex:318: Electric.ShapeCache.maybe_create_shape/2
1772483164359,            (electric 1.4.11) lib/electric/shape_cache.ex:281: Electric.ShapeCache.handle_call/3
1772483164359,            (stdlib 7.1) gen_server.erl:2470: :gen_server.try_handle_call/4
1772483164359,            (stdlib 7.1) gen_server.erl:2499: :gen_server.handle_msg/3
1772483164359,            (stdlib 7.1) proc_lib.erl:333: :proc_lib.init_p_do_apply/3
1772483164359,    (elixir 1.19.1) lib/gen_server.ex:1142: GenServer.call/3
1772483164359,    (electric 1.4.11) lib/electric/shapes/api.ex:295: Electric.Shapes.Api.handle_shape_info/2
1772483164359,    (opentelemetry 1.7.0) /builder/electric/deps/opentelemetry/src/otel_tracer_default.erl:47: :otel_tracer_default.with_span/5
1772483164359,    (electric 1.4.11) lib/electric/telemetry/open_telemetry.ex:87: anonymous fn/5 in Electric.Telemetry.OpenTelemetry.do_with_span/5
1772483164359,    (telemetry 1.3.0) /builder/electric/deps/telemetry/src/telemetry.erl:324: :telemetry.span/3
1772483164359,    (electric 1.4.11) lib/electric/shapes/api.ex:180: Electric.Shapes.Api.validate/2
1772483164359,    (electric 1.4.11) lib/electric/plug/serve_shape_plug.ex:105: Electric.Plug.ServeShapePlug.validate_request/2
1772483164359,    (electric 1.4.11) lib/electric/plug/serve_shape_plug.ex:1: Electric.Plug.ServeShapePlug.plug_builder_call/2
1772483164359,    (electric 1.4.11) deps/plug/lib/plug/error_handler.ex:80: Electric.Plug.ServeShapePlug.call/2
1772483164359,    (electric 1.4.11) deps/plug/lib/plug/router.ex:246: anonymous fn/4 in Electric.Plug.Router.dispatch/2
1772483164359,    (telemetry 1.3.0) /builder/electric/deps/telemetry/src/telemetry.erl:324: :telemetry.span/3
1772483164359,    (electric 1.4.11) deps/plug/lib/plug/router.ex:242: Electric.Plug.Router.dispatch/2
1772483164359,    (electric 1.4.11) lib/electric/plug/router.ex:1: Electric.Plug.Router.plug_builder_call/2
1772483164359,    (electric 1.4.11) lib/electric/plug/router.ex:1: Electric.Plug.Router.call/2
1772483164359,    (bandit 1.8.0) lib/bandit/pipeline.ex:131: Bandit.Pipeline.call_plug!/2
1772483164359,    (bandit 1.8.0) lib/bandit/pipeline.ex:42: Bandit.Pipeline.run/5
1772483164359,    (bandit 1.8.0) lib/bandit/http1/handler.ex:13: Bandit.HTTP1.Handler.handle_data/3
1772483164359,    (bandit 1.8.0) lib/bandit/delegating_handler.ex:18: Bandit.DelegatingHandler.handle_data/3
1772483164359,    (bandit 1.8.0) lib/bandit/delegating_handler.ex:8: Bandit.DelegatingHandler.handle_info/2
1772483164359,    (stdlib 7.1) gen_server.erl:2434: :gen_server.try_handle_info/3
1772483164359,    (stdlib 7.1) gen_server.erl:2420: :gen_server.handle_msg/3
1772483164359,    (stdlib 7.1) proc_lib.erl:333: :proc_lib.init_p_do_apply/3

[alco]

@magnetised can we close this issue?