Pod Conflicts During Hasura Database Upgrades - Need to Separate Database into StatefulSet

[mosoriob]

Current Behavior

The current Kubernetes deployment configuration combines both Hasura and its PostgreSQL database in a single pod within a Deployment resource. This is causing conflicts during application upgrades and potential stability issues.

Expected Behavior

The PostgreSQL database should be deployed separately using a StatefulSet to ensure proper lifecycle management and data persistence.

Technical Details

Current Issues:

1. Database and application are tightly coupled in the same pod
2. Hasura's readiness probe is checking PostgreSQL on localhost, which is problematic in a multi-container setup
3. Database upgrades and maintenance are complicated by the current architecture
4. PVC management is not optimal for a database workload

Proposed Solution:

1. Move PostgreSQL to a separate StatefulSet
2. Update Hasura's configuration to connect to the database via a service
3. Implement proper readiness probes for both components
4. Use volumeClaimTemplates for database storage

Implementation Plan

1. Create new StatefulSet for PostgreSQL:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mint-hasura-db
spec:
  serviceName: mint-hasura-db
  # ... StatefulSet configuration

2. Create headless service for database connectivity:

apiVersion: v1
kind: Service
metadata:
  name: mint-hasura-db
spec:
  clusterIP: None
  # ... Service configuration

3. Update Hasura deployment to remove database container and update connection settings

Benefits

• Improved stability during upgrades
• Better database lifecycle management
• Proper persistence handling
• Easier maintenance and debugging
• Independent scaling capabilities

Migration Strategy

1. Backup existing database
2. Deploy new StatefulSet and Service
3. Migrate data to new database instance
4. Update Hasura configuration
5. Validate functionality
6. Remove old deployment

Additional Notes

• The current password and username secrets can be reused
• Need to update documentation after implementation
• Consider adding monitoring for the database component

Related Files

• apps/v1/hasura.yaml
• Any database initialization scripts
• Related configuration maps and secrets

/cc @Database-Team @platform-team