🌍 Off-Road Semantic Segmentation using SegFormer-B3

📦 Submission Package

This project implements semantic segmentation for off-road environments using a fine-tuned SegFormer-B3 transformer model.

The model predicts a semantic class for every pixel in an image and identifies terrain elements such as vegetation, rocks, sky, logs, and background.

---

📁 Folder Structure

TeamName_Submission/
│── train_segformer.py
│── test_segformer.py
│── best_segformer_b3.pth
│── last_segformer_b3.pth
│── segformer_stats/
│── predictions_segformer/
│── README.md
│── Report.pdf

---

🚀 Project Overview

Off-road environments are challenging for autonomous systems because they contain:

• 🌿 Irregular terrain
• 🌳 Vegetation
• 🪨 Rocks / logs
• 🌘 Shadows
• 🚧 Natural obstacles
• 🛤️ Unstructured paths
• ☀️ Dynamic outdoor lighting

This project uses deep learning to understand such terrain through pixel-wise segmentation.

---

🧠 Model Used

SegFormer-B3

Pretrained base model:

nvidia/segformer-b3-finetuned-ade-512-512

Then fine-tuned on the custom off-road segmentation dataset.

---

🎯 Classes Predicted

ID	Class Name
0	Background
1	Trees 🌳
2	Lush Bushes 🌿
3	Dry Grass 🌾
4	Dry Bushes 🍂
5	Ground Clutter 🟫
6	Logs 🪵
7	Rocks 🪨
8	Landscape 🏔️
9	Sky ☁️

---

📄 Files Description

🏋️ train_segformer.py

Training script for fine-tuning SegFormer-B3 on the dataset.

Features:

• ✅ Loads pretrained SegFormer-B3
• ✅ Fine-tunes on custom 10 classes
• ✅ Tracks validation metrics
• ✅ Saves best checkpoint
• ✅ Saves final checkpoint
• ✅ Generates training graphs

---

🧪 test_segformer.py

Evaluation / inference script.

Outputs:

• 🎯 Predicted masks
• 🎨 Colored masks
• 🖼️ Comparison images
• 📊 Mean IoU
• 📈 Per-class metrics report

---

💾 best_segformer_b3.pth

Checkpoint with highest validation IoU during training.

---

💾 last_segformer_b3.pth

Final checkpoint from last epoch.

---

📊 segformer_stats/

Contains:

training_curves.png
iou_curves.png
dice_curves.png
evaluation_metrics.txt

---

🖼️ predictions_segformer/

Contains:

masks/
masks_color/
comparisons/
evaluation_metrics.txt
per_class_metrics.png

---

🗂️ Dataset Structure

Offroad_Segmentation_Training_Dataset/
├── train/
│   ├── Color_Images/
│   └── Segmentation/
│
└── val/
    ├── Color_Images/
    └── Segmentation/

Offroad_Segmentation_testImages/
├── Color_Images/
└── Segmentation/

---

⚙️ Installation

Install dependencies:

pip install torch torchvision transformers opencv-python matplotlib tqdm numpy pillow

---

🏋️‍♂️ How to Train

python train_segformer.py

Training will:

• 🔥 Fine-tune SegFormer-B3
• 💾 Save best model
• 💾 Save final model
• 📈 Save graphs and metrics

---

🧪 How to Test

python test_segformer.py

Testing will generate predictions and evaluation outputs.

---

📐 Evaluation Metrics

The project uses:

• 📊 Mean IoU (Intersection over Union)
• 🎯 Dice Score
• ✅ Pixel Accuracy
• 📈 Per-Class IoU

---

🌟 Why SegFormer?

SegFormer was selected because it provides:

• 🧠 Strong segmentation performance
• 🌍 Transformer-based global context understanding
• ⚡ Efficient architecture
• 🔁 Strong transfer learning ability
• ⚖️ Great balance of speed and accuracy

---

🌐 Use Cases

• 🚙 Autonomous off-road vehicles
• 🚜 Agricultural robotics
• 🛰️ Outdoor navigation systems
• 🆘 Search & rescue robots
• 🚁 Drone terrain understanding

---

🔮 Future Improvements

• ⚡ Real-time deployment
• 🎥 Video segmentation
• 🧩 More terrain classes
• 📱 Lightweight mobile inference
• 📡 Sensor fusion with LiDAR / GPS
• 🛣️ Path planning integration

---

📝 Summary

This project successfully adapts a pretrained SegFormer-B3 model for off-road terrain semantic segmentation.

It produces both quantitative metrics and visual prediction outputs suitable for research, deployment prototypes, and competition submissions.

---

👨‍💻 Author

Developed as a deep learning computer vision project for off-road scene understanding.