Implement unified pipeline for orchestrating bulk and single-cell RNA-seq analyses

omicsdm-server/pipelines/snakemake/bulk_sc/.Rprofile

@@ -0,0 +1,14 @@
+source("renv/activate.R")
+options(renv.config.pak.enabled = TRUE)
+
+# Configure BiocManager to use Posit Package Manager
+options(BioC_mirror = "https://packagemanager.posit.co/bioconductor/2025-12-30")
+
+# Configure BiocManager to load its configuration from Package Manager
+options(BIOCONDUCTOR_CONFIG_FILE = "https://packagemanager.posit.co/bioconductor/2025-12-30/config.yaml")
+
+# Set the Bioconductor version to prevent defaulting to a newer version
+Sys.setenv("R_BIOC_VERSION" = "3.22")
+
+# Configure a CRAN snapshot compatible with Bioconductor 3.22
+options(repos = c(CRAN = "https://packagemanager.posit.co/cran/__linux__/jammy/2025-12-30"))

omicsdm-server/pipelines/snakemake/bulk_sc/.dockerignore

@@ -0,0 +1,16 @@
+.Rproj.user
+.snakemake
+log
+out
+renv
+.dockerignore
+.gitignore
+.Rhistory
+.Rprofile
+*.Rproj
+docker-compose.yaml
+Dockerfile
+README.md
+renv_bk
+Snakefile
+venv

omicsdm-server/pipelines/snakemake/bulk_sc/.gitignore

@@ -0,0 +1,8 @@
+.snakemake
+log
+out
+renv
+venv
+*.Rproj
+.Rproj.user
+.Rhistory

omicsdm-server/pipelines/snakemake/bulk_sc/.renvignore

@@ -0,0 +1 @@
+venv

omicsdm-server/pipelines/snakemake/bulk_sc/Dockerfile

@@ -0,0 +1,76 @@
+FROM rocker/r-ver:4.5.2
+
+# Install system dependencies for both R and Python packages
+RUN apt-get update && apt-get install -y \
+  curl \
+  jq \
+  python3 \
+  python3.12-venv \
+  python3-pip \
+  libxml2 \
+  pandoc \
+  git \
+  libfontconfig1-dev \
+  libfreetype6-dev \
+  libgit2-dev \
+  libicu-dev \ 
+  libglpk-dev \
+  libpng-dev \ 
+  libxml2-dev \
+  libfribidi-dev \ 
+  libharfbuzz-dev  \  
+  zlib1g-dev \ 
+  libbz2-dev \ 
+  liblzma-dev \ 
+  libzstd-dev \ 
+  libdeflate-dev \
+  libpcre2-dev \ 
+  libreadline-dev \ 
+  libcurl4-openssl-dev \
+  libglpk-dev \
+  libmagick++-dev \
+  libssl-dev \
+  unzip
+
+# Set BLAS and LAPACK to libblas
+RUN ARCH=$(uname -m) && \
+  update-alternatives --set "libblas.so.3-${ARCH}-linux-gnu" "/usr/lib/${ARCH}-linux-gnu/blas/libblas.so.3" && \
+  update-alternatives --set "liblapack.so.3-${ARCH}-linux-gnu" "/usr/lib/${ARCH}-linux-gnu/lapack/liblapack.so.3"
+
+# Install Snakemake and Python packages
+RUN python3 -m venv /home/venv
+RUN /home/venv/bin/pip install snakemake
+RUN /home/venv/bin/pip install pulp==2.7.0
+
+# Install awscli
+RUN curl https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip -o awscliv2.zip && \
+  unzip awscliv2.zip && \
+  ./aws/install && \
+  rm -rf awscliv2.zip aws
+
+# Install Python dependencies
+COPY requirements.txt /tmp/requirements.txt
+RUN /home/venv/bin/pip install --no-cache-dir -r /tmp/requirements.txt
+
+# Install R package manager (pak) and renv
+RUN R -e 'install.packages("pak",repos = sprintf("https://r-lib.github.io/p/pak/stable/%s/%s/%s",.Platform$pkgType,R.Version()$os,R.Version()$arch));'
+RUN R -e "pak::pkg_install('renv')"
+
+# Set up working directory
+WORKDIR /home
+
+# Copy project files
+COPY .renvignore .
+COPY renv.lock .
+
+# Install R dependencies
+ENV RENV_CONFIG_PAK_ENABLED=TRUE
+RUN R -e "renv::restore()"
+
+# Copy the rest of the project
+COPY . .
+
+# Create necessary directories
+RUN mkdir -p in out log
+
+CMD [ "sh" ]

omicsdm-server/pipelines/snakemake/bulk_sc/README.md

@@ -0,0 +1,179 @@
+# Combined Pipeline for Bulk and Single-cell RNA-seq Analysis
+
+## Goal
+
+One Docker container image that is able to run the entire pipeline, integrating both bulk RNA-seq and single-cell RNA-seq analyses.
+
+## Pipeline Overview
+
+### Bulk RNA-seq Pipeline
+1. **DESeq2** - Differential expression analysis
+2. **FGSEA** - Functional gene set enrichment analysis (uses DESeq2 output)
+
+### Single-cell RNA-seq Pipeline
+3. **sc_normalisation** - Single-cell data normalization
+
+### Gene Set Scoring Pipelines
+Uses significant gene sets from FGSEA as input for:
+4. **z_scoring** - Z-score based gene set scoring
+5. **gsva** - GSVA gene set scoring
+
+## Pipeline Dependencies
+
+<!-- insert a mermaid diagramm here -->
+
+- FGSEA depends on DESeq2 output
+- z_scoring and gsva both depend on:
+  - FGSEA significant gene sets output
+  - sc_normalisation normalized data
+
+## Directory Structure
+
+```
+all/
+├── Dockerfile              # Combined Docker image with R and Python
+├── docker-compose.yaml     # Docker Compose configuration
+├── Snakefile              # Main Snakemake workflow
+├── makefile               # Build and run commands
+├── requirements.txt       # Python dependencies
+├── renv.lock             # R dependencies
+├── config/               # Configuration files for each pipeline
+│   ├── deseq2.json
+│   ├── fgsea.json
+│   ├── sc_normalisation.json
+│   ├── z_scoring.json
+│   └── gsva.json
+├── src/                  # Source scripts (symlinked from individual pipelines)
+│   ├── deseq2/
+│   ├── fgsea/
+│   ├── sc_normalisation/
+│   ├── z_scoring/
+│   ├── gsva/
+│   ├── lib/
+│   └── report/
+└── rules/                # Snakemake rules
+    └── renv.smk
+```
+
+## Building the Docker Image
+
+```bash
+make build
+```
+
+Or manually:
+
+```bash
+docker build -t omicsdm-all-pipeline:latest .
+```
+
+## Running the Pipeline
+
+### Using Make
+
+```bash
+# Run the complete pipeline
+make run
+
+# Open a shell in the container
+make shell
+
+# Clean output directories
+make clean
+```
+
+### Using Docker Directly
+
+```bash
+docker run --rm \
+  -v $(PWD)/Snakefile:/home/Snakefile:ro \
+  -v $(PWD)/src:/home/src:ro \
+  -v $(PWD)/config:/home/config:ro \
+  -v $(PWD)/docker-out:/home/out \
+  --add-host minio.omicsdm.cnag.dev:172.16.10.112 \
+  -it omicsdm-all-pipeline:latest /home/venv/bin/snakemake -F
+```
+
+## Configuration
+
+Each pipeline component requires its own configuration file in the `config/` directory:
+
+- `deseq2.json` - DESeq2 analysis configuration
+- `fgsea.json` - FGSEA analysis configuration
+- `sc_normalisation.json` - Single-cell normalization configuration
+- `z_scoring.json` - Z-scoring configuration
+- `gsva.json` - GSVA configuration
+
+## Outputs
+
+The pipeline generates outputs in the following structure:
+
+```
+out/
+├── results/
+│   ├── deseq2/                # DESeq2 analysis results
+│   ├── fgsea/                 # FGSEA results including significant_genesets.gmt
+│   ├── sc_normalisation/      # Normalized single-cell data
+│   ├── z_scoring/             # Z-score results
+│   └── gsva/                  # GSVA results
+├── _main.html                 # Combined report
+└── _book/                     # Report book directory
+```
+
+## Key Features
+
+1. **Single Docker Image**: All dependencies (R, Python, Bioconductor packages, Python packages) in one container
+2. **Snakemake Orchestration**: Automatic dependency management and parallel execution
+3. **Modular Design**: Each pipeline component is independent but integrated
+4. **Reproducible**: Version-controlled dependencies via renv.lock and requirements.txt
+
+## Technical Details
+
+### Docker Image
+- Base: rocker/r-ver:4.4.0
+- R version: 4.4.0
+- Python: 3.10
+- Includes: Snakemake, awscli, R packages (DESeq2, fgsea, etc.), Python packages (scanpy, anndata, etc.)
+
+### Dependencies
+- **R packages**: Managed via renv (see renv.lock)
+- **Python packages**: Listed in requirements.txt
+- **System packages**: Defined in Dockerfile
+
+## Troubleshooting
+
+### Dry Run
+Test the pipeline structure without execution:
+
+```bash
+cd /path/to/all
+snakemake --dry-run -j1
+```
+
+### View DAG
+Visualize the pipeline workflow:
+
+```bash
+snakemake --dag | dot -Tpdf > dag.pdf
+```
+
+## Development
+
+### Adding New Pipeline Steps
+
+1. Create source scripts in appropriate subdirectory under `src/`
+2. Add rule to `Snakefile`
+3. Update configuration if needed
+4. Test with `snakemake --dry-run`
+
+### Updating Dependencies
+
+**R packages:**
+```bash
+R -e 'renv::snapshot()'
+```
+
+**Python packages:**
+```bash
+pip freeze > requirements.txt
+```