[Feature Request] Add Support for S3-Compatible Object Storage Backend

[ZiggiZagga]

Description

Propose adding support for open source, S3-compatible object storage as a primary file storage mechanism in OpenAS2. This allows for a scalable, cloud-native backend for message processing and storage, improving flexibility and modernizing deployment strategies.

Rationale

• Cloud Native Flexibility: Supports deployments using Amazon S3, MinIO, Ceph, or other compatible systems.
• Scalable Architecture: Object storage backends scale horizontally and support large datasets natively.
• Stateless Deployments: Enables running OpenAS2 in stateless environments (e.g., Kubernetes).
• Open Source Alignment: Using systems like MinIO keeps the OpenAS2 ecosystem vendor-neutral.

Proposed Implementation

• Develop a new storage module (S3StorageModule) adhering to OpenAS2's existing file interface structure.
• Integrate the AWS SDK or an open source-compatible alternative.
• Configuration supporting:
  • Bucket name
  • Region/endpoint
  • Path prefix
  • Access credentials (environment variables)
• Implement both read and write capabilities, including:
  • Multipart upload/download
  • Error handling and logging

Optional Enhancements

• Local disk fallback mechanism
• Metadata tagging support

[uhurusurfa]

For immediate use of OpenAS2 in AWS (and providers who support the S3 protocol) you can already use S3Fuse:
#307

I will happily support a generic cloud implementation supporting the more common clouds that can be extendable to support other lesser known cloud providers.

This library does a great job and would support a PR that provides cloud support using it:
https://jclouds.apache.org/

The implementation would need to provide enhanced versions of the following:

1. The directory poller module to support cloud based file systems. (Server/src/main/java/org/openas2/processor/receiver/AS2DirectoryPollingModule.java)
2. The file storage module to support cloud based file systems. (Server/src/main/java/org/openas2/processor/storage/MessageFileModule.java and Server/src/main/java/org/openas2/processor/storage/BaseStorageModule.java)
3. Possibly the IO util module which is used for moving files around based on state changes (Server/src/main/java/org/openas2/util/IOUtil.java)

[ZiggiZagga]

Hi @uhurusurfa ,

I haven't heard about jcloud before, will definetly have a closer look at it.
I have seen a lot of Opensource Java Projects implementing S3 compability, I think maybe there is already a Java S3-library out there, that is widely supported. In any ways I would like to make a proposal on how our ideas can be pushed forward.

[ZiggiZagga]

🔹 Part 1: Cloud-Ready OpenAS2 via Modular Backend

🎯 Goal

Enable OpenAS2 to operate seamlessly in cloud-native environments by extending core storage components and implementing a pluggable, S3-compatible module.

✅ Tasks

• Implement new CloudStorageModule using Apache jclouds for multi-cloud support(This unlocks cloud-native capabilities for OpenAS2 in a clean, scalable, and extensible way). Notice: jclouds retired in June 2025 and the move to the Attic is In Progress.
• Enhance key modules to support cloud file operations:
  • MessageFileModule (message payload & metadata storage)
  • BaseStorageModule (base functionality for storage modules)
  • IOUtil (for cloud-compatible file movement and state transitions)
  • AS2DirectoryPollingModule (to poll from S3-compatible storage buckets)
  • (Stretch) Add event-driven ingestion via S3-compatible systems (e.g., SQS, Kafka, webhooks)
• Add modular configuration options:
  • Bucket name and path prefix
  • Cloud endpoint, region, and provider
  • Credentials via environment variables
• Include:
  • Logging and error handling
  • Multipart upload support
  • Optional fallback to local disk for hybrid scenarios
• Ensure complete backward compatibility (module is opt-in)

🔄 Stretch Goal: Event-Driven Ingestion Support

Many S3-compatible systems support event notification mechanisms (e.g. MinIO → Pulsar, S3 → Kafka). As an evolution beyond polling, OpenAS2 could support:

• Webhook or queue-based listeners triggered by cloud storage events
• Near real-time file detection and lower resource usage compared to polling
• Optional pluggable modules depending on infrastructure (e.g., Pulsar, [Kafka](apache kafka github))

This approach enhances performance, reduces latency, and aligns with modern event-driven architecture patterns.

---

🔹 Part 2: Shared S3 Abstraction for the Java OSS Ecosystem

🎯 Goal

Create a reusable, vendor-neutral Java library for S3-compatible storage to reduce duplication across open-source projects and promote sustainable cloud-native adoption.

✅ Tasks

• Survey existing solutions (e.g., Apache Iceberg, Nessie, Polaris).
• Design a common API for basic object storage operations (get, put, list, delete).
• Architect a plug-in system to support different S3-compatible backends.
• Use OpenAS2 as a real-world reference implementation.
• Reach out to contributors and communities for potential collaboration.

---

🌱 Stretch Goals for Shared S3 Abstraction

• Cross-Provider Conformance Tests – Ensure consistent behavior across S3 backends (AWS, MinIO, Ceph, etc.)
• Pluggable Encryption Support – Allow integration with envelope encryption or customer-managed key schemes.
• Async & Reactive APIs – Add support for non-blocking runtimes (e.g., WebFlux, Vert.x).
• Cloud Event Hooks – Provide scaffolding for SQS, Kafka, NATS, or webhook integrations.
• File Integrity & Deduplication – Built-in checksum/ETag comparison, duplicate detection logic.
• Extended Metadata & Indexing – Rich tagging or custom metadata for downstream analysis or lifecycle control.
• Data Format Connectors – Optional modules to work with data lake formats like Parquet, ORC, Avro.

---

🔄 Compatibility & Migration

• Cloud module will be optional and not affect default file-based behavior.
• Clear, versioned documentation to guide adoption.
• Smooth path for hybrid deployments and gradual migration.

---

🌟 Benefits

• Cloud-Native Compatibility – Enables deployment on Kubernetes, containers, cloud VMs.
• Scalability – Taps into cloud-native elasticity and durability.
• Vendor Neutrality – Works with MinIO, Ceph, AWS S3, and other compatible platforms.
• Community Impact – Encourages sustainable reuse across OSS Java projects.
• Developer Experience – Modular, testable components that promote maintainability and evolution.

[uhurusurfa]

I am on board with the above and happy to leave it to you entirely to produce a solution or collaborate at whatever level you feel comfortable with though my time is likely to be limited due to other commitlents.

The key will be to retain backwards compatiblity within reason to avoid having to develop upgrade scripts.

The code should wrap the JClouds stuff so that when there is a successor to JClouds it is relativley easy to switch to the newer library (there is much discussion about why JClouds is moving to the attic and fundamentally it needs a rewrite so at some point there is likely to be a more streamlined rewrite).

I will migrate the documentation from using ODF to RST (https://www.sphinx-doc.org/en/master/usage/restructuredtext/index.html) and will make it easier to maintain and provide online access to it.