# 🛡️ IPWatchdog
ML-Powered Suspicious IP Detection & Real-Time Log Monitoring
Upload a log → ML detects anomalies → See exactly why each IP was flagged
IPWatchdog is an open-source security dashboard for web administrators and DevOps engineers. It ingests access logs from any web server, runs an ensemble ML model (Isolation Forest + KMeans) over per-IP behavioural features, and surfaces the most suspicious IP addresses — with plain-English explanations of why they were flagged.
Unlike simple threshold-based tools, IPWatchdog learns the shape of normal traffic for your server and flags statistical outliers — catching low-and-slow attacks that trip no single rule.
| 🚀 Quick Start | ⚙️ Configuration | 📄 Log Formats |
| 🧠 Detection Engine | 🌐 Threat Intel | 🛡️ Automated Actions |
| 📡 Real-Time Monitor | 🏗️ Architecture | 🤝 Contributing |
|
|
┌──────────────────────────────────────────────────────────────────────┐
│ IPWatchdog Pipeline │
│ │
│ ┌────────────┐ ┌────────────────┐ ┌───────────────────────┐ │
│ │ Log Upload │───▶│ core/parser │───▶│ core/detector │ │
│ │ (.log/.txt)│ │ Auto-detect: │ │ ┌─────────────────┐ │ │
│ └────────────┘ │ • Apache │ │ │ Feature Eng. │ │ │
│ │ • Nginx │ │ │ req_count │ │ │
│ ┌────────────┐ │ • Combined │ │ │ error_rate │ │ │
│ │ Whitelist/ │ │ • JSON │ │ │ method_entropy │ │ │
│ │ Blacklist │ └────────────────┘ │ │ path_diversity │ │ │
│ └─────┬──────┘ │ │ burst_score │ │ │
│ │ │ └────────┬────────┘ │ │
│ │ │ │ │ │
│ │ │ ┌────────▼────────┐ │ │
│ │ │ │ IsolationForest │ │ │
│ │ │ │ + KMeans │ │ │
│ │ │ │ anomaly_score │ │ │
│ │ │ └────────┬────────┘ │ │
│ │ └───────────┼───────────┘ │
│ │ │ │
│ │ ┌─────────────────────────────────▼─────────┐ │
│ │ │ core/explainer │ │
│ │ │ "High error rate (52%) – possible scan" │ │
│ │ │ "Traffic burst – 80% in a single hour" │ │
│ │ └─────────────────────────────────┬─────────┘ │
│ │ │ │
│ │ ┌─────────────────────────────────▼─────────┐ │
│ │ │ core/threat_intel │ │
│ │ │ GeoIP country/ISP • AbuseIPDB score │ │
│ │ │ Tor / proxy detection │ │
│ │ └─────────────────────────────────┬─────────┘ │
│ └──────────────────┐ │ │
│ ▼ ▼ │
│ ┌──────────────────────────────────────────┐ │
│ │ Dashboard (Flask) │ │
│ │ Stats • Table • Charts • Block Rules │ │
│ │ CSV Export • Live Monitor (SSE) │ │
│ └──────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────────┘
Browser ──POST /analyze──▶ app.py
│
┌─────────▼──────────┐
│ 1. Validate file │ (type, size, secure_filename)
│ 2. LogParser │ (auto-detect format)
│ 3. IPDetector │ (feature eng + ML)
│ 4. Explainer │ (why-flagged reasons)
│ 5. ThreatIntel │ (GeoIP + AbuseIPDB)
│ 6. ActionEngine │ (.htaccess rules)
└─────────┬──────────┘
│
◀─────────┘ JSON response
(summary, results[], top_ips[], htaccess_rules)
# 1. Clone the repository
git clone https://github.com/PRATHAM777P/IPWatchdog.git
cd IPWatchdog
# 2. Create a virtual environment (recommended)
python -m venv venv
source venv/bin/activate # Windows: venv\Scripts\activate
# 3. Install dependencies
pip install -r requirements.txt
# 4. Configure environment variables
cp .env.example .env
# Edit .env and set at minimum SECRET_KEY
# 5. Start the app
python app.py🌐 Open your browser at http://127.0.0.1:5000
192.168.x.x - - [10/Oct/2023:13:55:36 -0700] "GET /index.html HTTP/1.1" 200 2326
10.x.x.x - - [10/Oct/2023:13:55:37 -0700] "POST /login HTTP/1.1" 401 512
All configuration is done via environment variables. Copy .env.example → .env:
| Variable | Default | Description |
|---|---|---|
SECRET_KEY |
(random) | Required in production. Flask session key |
FLASK_DEBUG |
false |
Set true for dev only |
FLASK_HOST |
127.0.0.1 |
Bind address |
FLASK_PORT |
5000 |
Listen port |
MAX_UPLOAD_MB |
50 |
Maximum log file size |
MONITOR_ALLOWED_PATHS |
(empty) | Colon-separated paths the live monitor may tail |
ABUSEIPDB_API_KEY |
(empty) | AbuseIPDB API key (optional) |
GEOIP_ENABLED |
true |
Enable free ip-api.com GeoIP lookups |
KMEANS_CLUSTERS |
10 |
Number of KMeans clusters |
ISOLATION_FOREST_CONTAMINATION |
0.05 |
Expected fraction of anomalies (0–0.5) |
ALERT_EMAIL_ENABLED |
false |
Enable SMTP email alerts |
SMTP_HOST / SMTP_USER / SMTP_PASSWORD |
(empty) | SMTP credentials |
ALERT_RECIPIENT |
(empty) | Alert email address |
🔒 Security: Never commit your
.envfile. It is excluded by.gitignore.
| Format | Auto-detected | Example |
|---|---|---|
| Apache Common | ✅ | 1.2.3.4 - - [date] "GET / HTTP/1.1" 200 512 |
| Apache Combined | ✅ | Same + user-agent field |
| Nginx default | ✅ | Same structure as Apache Combined |
| JSON (one obj/line) | ✅ | {"ip":"1.2.3.4","time":"...","method":"GET",...} |
Custom formats can be added in
core/parser.pyby extending the_parse_*methods.
| Feature | Description |
|---|---|
request_count |
Total requests from this IP |
error_rate |
Fraction of 4xx / 5xx responses |
method_entropy |
Shannon entropy of HTTP method distribution |
path_diversity |
Unique paths / total requests |
avg_bytes |
Mean response size |
burst_score |
Peak-hour requests / total requests |
Raw features ──▶ StandardScaler ──▶ IsolationForest ──▶ anomaly_score ∈ [-1, +1]
──▶ KMeans ──▶ cluster label
- IsolationForest — continuous score per IP; lower = more anomalous
- KMeans — groups IPs into behavioural clusters for pattern overview
- Fallback — z-score heuristic used automatically if scikit-learn is absent
| Source | Cost | Key Required | Data Returned |
|---|---|---|---|
| ip-api.com | Free (45 req/min) | ❌ No | Country, ISP, proxy/hosting flag |
| AbuseIPDB | Free tier available | ✅ ABUSEIPDB_API_KEY |
Confidence score, report count, Tor flag |
All lookups fail silently — the pipeline always completes regardless of network availability.
From the Block Rules tab, IPWatchdog generates ready-to-use firewall rules for all flagged IPs:
# Apache .htaccess
Deny from 192.168.1.100
# Nginx ACL (nginx.conf)
geo $blocked_ip { default 0; 192.168.1.100 1; }
# iptables shell script
iptables -I INPUT -s 192.168.1.100 -j DROPRules can be copied or downloaded directly from the dashboard. Email alerting is available via SMTP — configure SMTP_* and ALERT_RECIPIENT in .env.
- Set
MONITOR_ALLOWED_PATHSin.envto the directories you want to allow tailing - Go to the Live Monitor tab and enter the full path to your log file
- New entries appear in real time with colour-coded anomaly scores
🔒 For security, only paths explicitly listed in
MONITOR_ALLOWED_PATHScan be tailed.
IPWatchdog/
├── 📄 app.py ← Flask app, routes, SSE endpoint
├── ⚙️ config.py ← All config from environment variables
├── 🧩 core/
│ ├── __init__.py
│ ├── parser.py ← Multi-format log parser
│ ├── detector.py ← IsolationForest + KMeans ensemble
│ ├── explainer.py ← Human-readable flag reasons
│ ├── threat_intel.py ← GeoIP + AbuseIPDB integration
│ └── actions.py ← Block rule generators + email alerts
├── 🎨 templates/
│ └── dashboard.html ← Bootstrap 5 SPA dashboard
├── 🖼️ assets/ ← GIFs and images for README
├── 📦 requirements.txt
├── 🔒 .env.example ← Config template (safe to commit)
├── 🚫 .gitignore ← Excludes .env, *.log, result.txt, *.csv
├── 📖 README.md
├── 🛡️ SECURITY.md
└── 🤖 .github/
└── workflows/
└── codeql.yml ← Automated CodeQL security scanning
Contributions are welcome! 🎉
- Fork the repository and create a feature branch
git checkout -b feat/your-awesome-feature
- Make your changes and add tests where applicable
- Ensure the code passes CodeQL checks (runs automatically on push)
- Open a pull request with a clear description of your changes
Please read SECURITY.md before submitting code that touches the detection engine or any network-facing functionality.
Key hardening points:
- 🔑 All secrets loaded from environment variables — never hardcoded
- 📁 File uploads validated by extension and size before processing
- 🚧 Live monitor only allows explicitly whitelisted paths
- 🧱 Security headers (CSP, X-Frame-Options, etc.) applied to every response
- 🤖 CodeQL scans run on every push and pull request
Please see SECURITY.md for our full vulnerability disclosure policy.
