Implement Dynamic SEO Sitemap Generation with Feature Flag Control

[bhavabhuthi]

Overview

Implement a comprehensive sitemap solution for the application that generates sitemaps on-demand (not at build time) with support for large datasets through a multi-sitemap architecture, controlled by feature flags.

Requirements

1. On-Demand Sitemap Generation

• Sitemaps should be generated dynamically when requested by users/search engines
• No sitemap generation during build process
• All sitemap data fetched from Django API endpoints

2. Multi-Sitemap Architecture

• Support for 10,000+ URLs through sitemap index pattern
• Automatic pagination with configurable items per sitemap (default: 1000 items)
• Separate sitemaps for different entity types:
  • Datasets: /sitemap-datasets-[page].xml
  • Usecases: /sitemap-usecases-[page].xml
  • Publishers: /sitemap-publishers-[page].xml
  • Sectors: /sitemap-sectors-[page].xml

3. Sitemap Index Structure

• Main index at /sitemap.xml that references all child sitemaps
• Consolidated static URLs included in main sitemap index
• Dynamic calculation of required sitemaps based on API response counts

4. Feature Flag Control

• Enable/disable sitemap functionality via environment variables
• Configurable parameters:
  • FEATURE_SITEMAPS: Enable/disable entire sitemap system
  • FEATURE_SITEMAP_ITEMS_PER_PAGE: Number of items per sitemap file
  • FEATURE_SITEMAP_CACHE_DURATION: Cache duration for sitemap index
  • FEATURE_SITEMAP_CHILD_CACHE_DURATION: Cache duration for child sitemaps

5. Caching Strategy Defaults

• Sitemap index: 1 hour cache (frequent updates possible)
• Child sitemaps: 6 hours cache (datasets don't change frequently)
• Proper HTTP cache headers for SEO optimization

6. Error Handling

• Graceful degradation when API calls fail
• Proper HTTP status codes (404 for disabled features, 500 for errors)
• Empty sitemap fallback on critical failures

7. Static URLs Integration

• Important static pages included in main sitemap:
  • Homepage (/)
  • About us
  • Main listing pages (/datasets, /usecases, /publishers, /sectors)
  • Configurable priority and change frequency

Technical Implementation Details

Route Handlers Required:

1. app/sitemap.xml/route.ts - Main sitemap index
2. app/sitemap-[entity]-[page].xml/route.ts - Dynamic entity sitemaps

Configuration Files:

1. Environment variable support for all configurations

API Integration:

• Fetch dataset counts from Django paginated endpoints
• Support for standard Django REST framework pagination format
• Error-resistant API calls with proper fallbacks

Acceptance Criteria:

• Sitemaps generated only on user request (no build-time generation)
• Feature flags control sitemap availability
• Handles 10,000+ URLs through multi-sitemap architecture
• Static URLs consolidated in main sitemap index
• Proper caching headers implemented
• Error handling for API failures
• Works in development with feature flag override
• Production-ready with appropriate defaults
• SEO-compliant XML sitemap format
• robots.txt references main sitemap index

Environment Variables to Support:

FEATURE_SITEMAPS=true
FEATURE_SITEMAP_ITEMS_PER_PAGE=1000
FEATURE_SITEMAP_CACHE_DURATION=3600
FEATURE_SITEMAP_CHILD_CACHE_DURATION=21600
DJANGO_API_URL=https://your-api.com
NEXT_PUBLIC_SITE_URL=https://yourdomain.com

Testing Notes:

• Test with feature flags disabled (should return 404)
• Test with feature flags enabled in development
• Verify multi-sitemap generation with large dataset counts
• Confirm proper XML formatting and SEO compliance