rclone Modal backend

Out-of-tree rclone build with an extra backend named modal for Modal v2 Volumes. It lets rclone copy and sync between Modal Volumes, local filesystems, S3, and other rclone remotes.

This backend is experimental and is not part of upstream rclone.

Build

Requires Go 1.24 or newer.

make deps
make build
make install

make install writes ~/.local/bin/rclone by default. Override the binary name or prefix when needed:

make install BIN_NAME=rclone-modal
make install PREFIX=/opt/local

Authentication

The backend reads Modal credentials from MODAL_TOKEN_ID and MODAL_TOKEN_SECRET, or from .modal.toml. Token values are not rclone backend options.

Useful options and environment variables:

• profile: Modal profile from .modal.toml; falls back to MODAL_PROFILE, then the active profile.
• environment: Modal environment name; falls back to MODAL_ENVIRONMENT, then the selected profile environment.
• MODAL_CONFIG_PATH: alternate path to .modal.toml.
• MODAL_SERVER_URL: alternate Modal API URL.
• sidecar_metadata: preserve source mtimes through a sidecar file. Default: true.
• create_volumes: create missing Modal Volumes on write or mkdir. Default: false.

Paths

Use the backend directly:

rclone lsd :modal:
rclone lsf :modal:my-volume/path

Or configure a named remote:

rclone config create modal-prod modal
rclone config update modal-prod profile=work environment=main

Path format:

:modal:volume-name/path/inside/volume
modal-prod:volume-name/path/inside/volume

The first path segment is the Modal Volume name. The rest is the path inside that Volume.

Operations resolve Volumes from Modal's existing Volume list and do not create missing Volumes by default. This avoids creating empty Volumes from typos in paths such as :modal:prod-data.

To allow write commands to create missing Volumes, opt in explicitly:

rclone copy ./data :modal:new-volume/data --modal-create-volumes
rclone mkdir :modal:new-volume --modal-create-volumes

Examples

rclone lsd :modal:
rclone lsf :modal:my-volume/datasets/
rclone copy ./data :modal:my-volume/data
rclone sync ./data :modal:my-volume/data
rclone sync :modal:my-volume/data ./data
rclone sync s3remote:bucket/prefix :modal:my-volume/prefix

Sync Metadata

Modal's v2 Volume API exposes size and Modal's own mtime, but not a full-file hash, and uploads cannot set the destination mtime. This backend reports no hashes, so checksum mode cannot compare Modal files by a Modal-side hash.

With sidecar_metadata = true, uploads write .rclone-modal-metadata.json at the Volume root. The sidecar records the source mtime plus Modal's size and physical mtime for files written through this backend. Later listings return the source mtime only while the current Modal size and physical mtime still match the sidecar entry; otherwise they fall back to Modal's native mtime.

The sidecar is hidden from listings while sidecar metadata is enabled.

If the sidecar JSON is missing, unreadable, or corrupt, the backend ignores it and falls back to Modal's native mtime.

The sidecar is not safe against concurrent rclone processes writing to the same Volume; the last writer can overwrite another process's metadata updates.

If files are changed or deleted outside this backend, prune stale sidecar entries with:

rclone backend prune-metadata :modal:my-volume

Use a path remote such as :modal:my-volume/data to prune only metadata entries under that path.

Disable sidecar metadata to expose only native Modal metadata:

rclone sync ./data :modal:my-volume/data --modal-sidecar-metadata=false

With sidecar metadata disabled, repeated syncs may need --size-only or --ignore-times. --size-only can miss same-size content changes; --ignore-times recopies even unchanged files.

Limitations and Notes

The backend does not support server-side Move or Copy, setting destination modtimes with SetModTime, About, or Modal-side hashes.

For predictable sidecar metadata, use a single writer process per Volume. The sidecar is just a file in the Volume, so concurrent writers can overwrite each other's metadata updates.

Metadata updates are debounced briefly and flushed on shutdown, but each flush still rewrites the full sidecar. Large syncs with many small files can spend substantial time and API calls updating metadata.

Uploads spool the full source object to local temporary storage so missing-block retries can reread blocks. Uploading a 100 GB file needs roughly 100 GB of temporary disk space.

The backend uses Modal's direct v2 Volume file APIs from outside Modal containers. It does not call commit(), and there is no backend commit command or commit_interval option.

License

MIT. See COPYING.