Golang Backend with RabbitMQ, PostgreSQL, Redis & SSE

This project is a backend application demonstrating expertise in RabbitMQ, PostgreSQL, Redis, and Golang.
It includes CSV ingestion, message queuing, data storage, an API with filtering, encryption, validation, logging, SSE for real-time updates, and Dockerization.

---

🚀 Features

• 📂 CSV File Upload in UI → Can upload CSV file in the UI and it will be sent into the service .
• 📂 CSV Ingestion → Parses CSV files and sends data to RabbitMQ queue.
• 📨 Message Queue Processing → Consumes messages, validates them, and stores data in PostgreSQL and Redis.
• 🔒 Security
  • Encrypts email column in the database.
  • Prevents SQL Injection attacks with proper query handling.
  • Uses data validation for email and name fields.
• 📝 Logging & Error Handling → Centralized logging for troubleshooting and debugging.
• ⚡ Performance Optimizations
  • Uses Redis cache before querying PostgreSQL.
  • Optimized SQL queries to improve API response times.
• 📡 Server-Sent Events (SSE) → Streams real-time user updates to the frontend.
• 🐳 Docker & Docker Compose → Fully containerized application.

---

📌 Tech Stack

Component	Technology Used
Language	Golang
Queue Broker	RabbitMQ
Database	PostgreSQL
Cache	Redis
API Framework	net/http
Logging	log
Validation	regex, custom functions
Frontend	HTML, JavaScript (SSE)

---

⚙ Project Structure

golang-backend/
│── api/
│   ├── handlers/
│   │   ├── sse_handler.go    # SSE event streaming
│── cmd/
│   ├── api/
│   │   ├── main.go           # API server setup
│   ├── consumer/
│   │   ├── main.go           # Message queue consumer
│── config/
│   ├── config.go             # Parse .env and load config
│── internal/
│   ├── models/
│   │   ├── user.go           # User struct definition
│   ├── rabbitmq/
│   │   ├── consumer.go       # PostgreSQL operations
│   │   ├── producer.go          # Redis operations
│   ├── repository/
│   │   ├── postgres.go       # PostgreSQL operations
│   │   ├── redis.go          # Redis operations
│   ├── services/
│   │   ├── encryption.go     # Logging service
│   │   ├── logging.go        # Logging service
│   │   ├── validation.go     # Validation functions
│── web/
│   ├── index.html            # Frontend with SSE integration
│── .env                      # Environment config
│── data.csv                  # CSV data to produce
│── docker-compose.yml        # Docker container configuration
│── Dockerfile                # Golang application Dockerfile
│── README.md                 # Documentation
│── go.mod                    # Go dependency module
│── go.sum                    # Go Indirect dependencies

---

🛠 Setup & Installation

1️⃣ Set Up Environment Variables

Create a .env file in the project root:

DATABASE_URL=postgres://user:password@db:5432/golang_db?sslmode=disable
RABBITMQ_URL=amqp://guest:guest@rabbitmq:5672/
REDIS_URL=redis://redis:6379

2️⃣ Run Services Using Docker

docker-compose up --build

• PostgreSQL runs on port 5432
• RabbitMQ runs on port 5672
• Redis runs on port 6379

3️⃣ Start Consumer, Producer and API Manually

go run cmd/consumer/main.go
go run cmd/api/main.go

---

🔥 Usage

📌 1️⃣ Upload CSV File

Send a POST request to upload a CSV file.

curl -X POST -F "file=@data.csv" http://localhost:8080/api/upload

📌 2️⃣ Real-Time User Updates via SSE

• Open index.html in the browser.
• Users will appear live as they are added.
• Or test via CLI:

curl -N http://localhost:8080/sse

---

📡 API Endpoints

1️⃣ CSV Upload

POST /api/upload

Parameter	Type	Description
file	File	CSV file to upload

---

2️⃣ Real-time Updates (SSE)

GET /sse

Response:

data: [{"id":1, "first_name":"John", "last_name":"Doe", "email":"john@example.com"}]

---

🔐 Security Measures

✔ Email column encryption in PostgreSQL
✔ Data validation before database insertion
✔ SQL injection protection using parameterized queries
✔ Secure API endpoints with input sanitization

---

📊 Performance Optimizations

✔ Uses Redis cache before querying PostgreSQL
✔ Filters API results efficiently
✔ Batch processing for CSV ingestion
✔ Uses SSE instead of polling for real-time updates

---

🧪 Testing

To run unit tests:

go test ./...