Distributed Job Queue System

A distributed job queue system built with Node.js and Redis featuring distributed workers, retries, dead letter queues (DLQ), fault-tolerant recovery, and a live monitoring dashboard.

This project simulates how production-grade background job processing systems work internally.

---

Features

• Distributed worker architecture
• Concurrent job processing
• Reliable Redis-backed queue
• Visibility timeout handling
• Automatic retries
• Dead Letter Queue (DLQ)
• Fault-tolerant watchdog recovery
• Live monitoring dashboard
• Metrics API
• Horizontal worker scaling
• Dockerized Redis setup

---

Dashboard

Live monitoring dashboard showing queue state, processing jobs, completed jobs, failures, and DLQ metrics.

---

Distributed Workers Processing Jobs

Multiple workers consuming jobs concurrently from the Redis queue.

---

Retry Mechanism

Workers automatically retry failed jobs until the retry limit is exceeded.

---

Dead Letter Queue (DLQ)

Jobs exceeding maximum retry attempts are moved into the Dead Letter Queue for inspection and recovery.

---

Dead Letter Queue Stored In Redis

Failed jobs persisted inside Redis after exceeding retry limits.

---

Metrics API

Metrics endpoint exposing internal queue state and processing statistics.

---

Architecture

Client ↓ Producer API ↓ Redis Queue ↓ Distributed Workers ↓ ACK / Retry System ↓ Dead Letter Queue (DLQ)

---

Tech Stack

• Node.js
• Express.js
• Redis
• Docker
• HTML/CSS/JavaScript

---

How It Works

1. Producer API receives jobs through HTTP requests.
2. Jobs are pushed into a Redis queue.
3. Multiple distributed workers consume jobs concurrently.
4. Workers move jobs into a processing queue using atomic Redis operations.
5. Successfully completed jobs are ACKed and removed.
6. Failed jobs are retried automatically.
7. Jobs exceeding retry limits are moved to the Dead Letter Queue (DLQ).
8. A watchdog continuously monitors stale jobs and recovers failed processing.
9. Metrics are exposed through a monitoring dashboard and API.

---

Setup

Clone Repository

git clone https://github.com/CHINMOYSHARMA-debug/distributed_job_queue.git
cd distributed_job_queue

Install Dependencies

npm install

Start Redis

docker compose up

Run Application

npm run dev

---

Metrics API

GET /metrics

Returns:

{
  "queued": 3,
  "processing": 2,
  "deadLetter": 1,
  "completed": 8,
  "failed": 2
}

Deployment

The producer API and monitoring dashboard are deployed publicly on render. Distributed workers and watchdog processes are demonstrated locally due to free-tier limitations on long-running background services.

---