Skip to content

Add S3 backend core#338

Open
shaypal5 wants to merge 2 commits intomasterfrom
feature/s3-backend
Open

Add S3 backend core#338
shaypal5 wants to merge 2 commits intomasterfrom
feature/s3-backend

Conversation

@shaypal5
Copy link
Member

Summary

Closes #41.

Adds a new s3 backend to cachier, allowing function results to be persistently cached in AWS S3 (or any S3-compatible service: MinIO, localstack, etc.).

  • New backend: @cachier(backend="s3", s3_bucket="my-bucket") - same decorator API as all other cores
  • Flexible client config: pass a pre-built boto3 client (s3_client), a callable factory (s3_client_factory), or let the core auto-create one from s3_region / s3_endpoint_url / s3_config
  • Data model: one pickled CacheEntry per cache key, stored at <s3_prefix>/<func_str>/<cache_key>.pkl
  • Async: async cached functions are supported via asyncio.to_thread delegation (boto3 is sync-only)
  • No import-time warnings: boto3 absence is only signalled when the S3 core is actually instantiated

Files changed

File Change
src/cachier/cores/s3.py New _S3Core class implementing the full _BaseCore interface
src/cachier/_types.py Add S3Client type alias; extend Backend literal with "s3"
src/cachier/core.py Wire S3 decorator parameters; import and instantiate _S3Core
pyproject.toml Add s3 pytest marker; add [project.optional-dependencies] extras
tests/s3_tests/ 18 tests using moto[s3] (no real AWS account required)
tests/requirements_s3.txt boto3 + moto[s3] test deps
examples/s3_example.py Runnable demo

Test plan

  • pytest -m s3 - all 18 S3 tests pass (uses moto for offline mocking)
  • pytest -m "not (mongo or redis or sql or s3)" - 219 existing tests pass, 0 regressions
  • ruff check - no linting errors on changed files
  • mypy src/cachier/ - only pre-existing SQL overload error; boto3 stubs handled with # type: ignore[import-untyped]

Open questions (from issue)

The following points from the original issue are left for follow-up if desired:

  • Async s3_client_factory (currently sync-only, matching boto3's nature)
  • Server-side encryption options in the core API
  • Multipart uploads for very large cached values

🤖 Generated with Claude Code

Implements a new cachier backend backed by AWS S3 (or any S3-compatible
service such as MinIO or localstack).

Key changes:
- src/cachier/cores/s3.py: new _S3Core implementing _BaseCore; stores one
  pickled CacheEntry per key under <prefix>/<func_str>/<key>.pkl; supports
  direct boto3 client, callable factory, or auto-created client via region /
  endpoint_url / Config options; syncs async callers via thread delegation
  from _BaseCore defaults (boto3 is sync-only).
- src/cachier/_types.py: add S3Client type alias; extend Backend literal with
  "s3".
- src/cachier/core.py: wire s3_bucket, s3_prefix, s3_client, s3_client_factory,
  s3_region, s3_endpoint_url, s3_config decorator parameters; import and
  instantiate _S3Core.
- pyproject.toml: add "s3" pytest marker; add [project.optional-dependencies]
  with per-backend extras (mongo, redis, sql, s3, all).
- tests/s3_tests/: 18 tests covering basic caching, skip/overwrite, stale_after,
  next_time, allow_none, entry_size_limit, clear_cache, clear_being_calculated,
  delete_stale_entries, client factory, thread safety, and error handling.
  Uses moto[s3] for offline testing with no real AWS account needed.
- tests/requirements_s3.txt: boto3 + moto[s3] test deps.
- examples/s3_example.py: runnable demo for basic caching, stale_after,
  client factory, and cache management.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@codecov
Copy link

codecov bot commented Feb 17, 2026

Codecov Report

❌ Patch coverage is 18.22917% with 157 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.57%. Comparing base (85539dc) to head (2827e10).

Files with missing lines Patch % Lines
src/cachier/cores/s3.py 16.21% 155 Missing ⚠️
src/cachier/core.py 50.00% 1 Missing and 1 partial ⚠️
Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff             @@
##            master     #338      +/-   ##
===========================================
- Coverage   100.00%   90.57%   -9.43%     
===========================================
  Files           11       12       +1     
  Lines         1477     1666     +189     
  Branches       185      209      +24     
===========================================
+ Hits          1477     1509      +32     
- Misses           0      156     +156     
- Partials         0        1       +1     
Flag Coverage Δ
local 58.16% <18.22%> (-5.28%) ⬇️
mongodb 40.93% <18.22%> (-3.21%) ⬇️
postgres 43.93% <18.22%> (-3.46%) ⬇️
redis 47.41% <18.22%> (-3.91%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/cachier/_types.py 100.00% <100.00%> (ø)
src/cachier/core.py 99.38% <50.00%> (-0.62%) ⬇️
src/cachier/cores/s3.py 16.21% <16.21%> (ø)

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 85539dc...2827e10. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add an S3 backend

1 participant