Distributed AI Workload Management via Active Storage and dataClay

Overview

Offload AI workloads from edge devices to powerful cloud servers—without changing your code.

This repository demonstrates how to seamlessly distribute AI training across heterogeneous devices using active storage. Run heavy PyTorch models on resource-constrained edge devices by intelligently offloading computation to the cloud.

Note: This work is part of the ICOS EU Project 🇪🇺 and builds upon our previous research in edge computing optimization, MetaOS architecture, and adaptive ML for constrained environments. See Related Publications for more details.

Active storage architecture spanning edge-to-cloud continuum

---

🚀 Why This Matters

Reduce memory usage by 80%+ on edge devices while maintaining performance. Perfect for deploying AI on resource-constrained hardware like Raspberry Pi, Orange Pi, or embedded systems.

📊 Real Performance Gains

Dramatic memory reduction on client devices

Faster training with intelligent workload offloading

🎯 Key Benefits

• Minimal code changes - Keep your existing PyTorch/scikit-learn code
• Transparent offloading - dataClay handles distribution automatically
• Resource flexibility - Mix and match edge devices with cloud resources
• Real datasets - Includes CPU utilization data on Hugging Face

---

📦 Project Structure

🔧 Server-Side Training with dataClay

To train a model on the server using dataClay:

1. Open train_dataclay.py and set the correct server_ip.

2. Ensure Docker Compose is up and running:

docker-compose up -d

3. Define two experiment names:

   • client_experiment for storing client results.
   • experiment_name for storing server results.

4. Run the training notebook:

./AIoffload/0_server_experiment.ipynb

---

🧪 Baseline (Non-dataClay) Training

For standalone (non-dataClay) experiments:

1. Open the notebook:

./Baseline/1_client_experiment.ipynb

2. Set experiment_name to track and store metrics and results.

---

⚙️ Environment Setup Instructions

You'll need to configure both the server and client environments.

🖥️ Server Setup

1. Install Miniconda:

• For Linux:

  wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
  bash Miniconda3-latest-Linux-x86_64.sh

• For macOS:

  curl -O https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
  bash Miniconda3-latest-MacOSX-x86_64.sh

2. Initialize Conda:

source ~/miniconda3/bin/activate
conda init

Then restart your terminal.

3. Create and activate the server environment:

conda create -n server_env python=3.10.16 -y
conda activate server_env

4. Clone the repository and install dependencies:

git clone <repository_url>
cd <repository_directory>
pip install -r requirements.txt

Make sure torch is included (add a specific CUDA version if needed).

5. Start Docker + dataClay:

docker-compose down
docker-compose up -d

---

💻 Client Setup

1. Install Miniconda (same steps as server).

2. Create and activate the client environment:

conda create -n client_env python=3.10.16 -y
conda activate client_env

3. Clone the repository and install dependencies:

git clone <repository_url>
cd <repository_directory>
pip install -r requirements-client.txt

⚠️ Do not include torch in client requirements if unnecessary.

4. Install Jupyter Notebook:

conda install -c conda-forge notebook

5. Launch Jupyter:

jupyter-notebook

---

📄 License

This project is licensed under CC BY-NC-SA 4.0. You are free to share and adapt the material with proper attribution, for non-commercial purposes only, and under the same license.

---

📚 Citation

If you use this work in your research, please cite:

Peer-Reviewed Paper

@article{barcel2025offloading,
  title={Offloading Artificial Intelligence Workloads across the Computing Continuum by means of Active Storage Systems},
  author={Barcel{\'o}, Alex and Ord{\'o}{\~n}ez, Sebasti{\'a}n A Cajas and Samanta, Jaydeep and Su{\'a}rez-Cetrulo, Andr{\'e}s L and Ghosh, Romila and Carbajo, Ricardo Sim{\'o}n and Queralt, Anna},
  journal={Future Generation Computer Systems},
  pages={108271},
  year={2025},
  publisher={Elsevier},
  doi={10.1016/j.future.2025.108271},
  url={https://www.sciencedirect.com/science/article/pii/S0167739X25005655}
}

Additional Resources

• 📖 Official Blog Post: CeADAR Publication
• 📄 arXiv Preprint: arXiv:2512.02646
• 📋 CITATION.cff: See CITATION.cff for structured citation metadata

Related ICOS Publications

This work builds upon and extends our previous research in the ICOS project:

1. ICOS: An Intelligent MetaOS for the Continuum Garcia et al., MECC '25

2. Adaptive Machine Learning for Resource-Constrained Environments Cajas Ordóñez et al., Lecture Notes in Computer Science, 2025

3. Drift-MoE: A Mixture of Experts Approach to Handle Concept Drifts arXiv:2507.18464

4. Intelligent Edge Computing and Machine Learning: A Survey of Optimization and Applications Future Internet 2025, 17(9), 417

---

🏅 Funding

🇪🇺 This work has received funding from the European Union's HORIZON research and innovation programme under grant agreement No. 101070177 (ICOS Project).

---

🌟 Star us on GitHub if this helps your research! 🌟