Observability & Enhanced Sync

Added

• Sentry Error Tracking Integration - Production error monitoring and performance tracking
  • Optional Sentry SDK integration for real-time error tracking and transaction monitoring
  • Fiber middleware for automatic panic recovery and error reporting
  • SENTRY_DSN environment variable configuration
  • Request context preservation with tags and custom data
  • Helper functions for manual error capture (sentry.CaptureException, sentry.CaptureMessage)
  • Transaction tracking with smart sampling (10% default)
  • Error classification and context enhancement (CaptureErrorWithContext)
  • 5-second timeout during graceful shutdown for pending event delivery
  • Custom Instrumentation Helpers for detailed performance monitoring
    • StartSpan() - Create child spans within transactions
    • GetTransactionFromContext() - Access active transactions from any context
    • GetSpanFromContext() - Access active spans from any context
    • CaptureExceptionWithContext() - Capture errors in background workers without Fiber context
  • Background Worker Transaction Tracking
    • worker.refresh_oauth_tokens - OAuth token refresh operations (every 5 minutes)
    • worker.refresh_game_sessions - Game session refresh checks (every 5 minutes)
    • worker.cleanup_expired_sessions - Session cleanup operations (daily at 2 AM)
    • worker.full_sync - Pterodactyl panel synchronization jobs
    • worker.discord_webhook - Discord webhook dispatch operations
• Hytale Game Session Enhancements - Improved session lifecycle management
  • Game session refresh interval changed to every 5 minutes
  • Sessions are refreshed when 5 minutes before expiry
  • Session token refresh returns new sessionToken and identityToken from Hytale
  • Database method to update session tokens after refresh (UpdateGameSessionTokens)
  • Pterodactyl API integration to push refreshed tokens to panel servers
  • New UpdateServerEnvironment method for updating environment variables via Pterodactyl client API
  • Sentry transaction tracking for all session refresh operations
• Hytale Audit Logs API - New endpoint for retrieving Hytale OAuth event logs
  • GET /api/v1/hytale/logs?account_id={id}&limit={limit} endpoint
  • Query parameter validation (max 1000 logs per request)
  • Returns audit logs with event types (TOKEN_CREATED, TOKEN_REFRESHED, SESSION_CREATED, etc.)
  • Supports filtering by account ID
• Comprehensive Unit Test Suite - Core packages tested for reliability
  • Config loading and environment variable parsing tests
  • Error handling and HTTP status mapping tests
  • JWKS cache and key management tests
  • OAuth client configuration tests
  • Rate limiter token bucket algorithm tests
  • Type structure and serialization tests
• CI/CD Workflow Improvements - Automated quality gates
  • Coverage reporting with 30% minimum threshold
  • PostgreSQL 15 and Redis 7 service containers for integration tests
  • Multi-workflow setup: test-build, format, docker, dependencies, coverage
  • HTML coverage reports uploaded to GitHub artifacts
  • Coverage comments on pull requests with metrics
• Dual API Integration: Full Pterodactyl Client API support for user-server relationships
  • New server_subusers table tracking user access to servers with permissions
  • GetClientServers() method to fetch servers via Client API
  • GetServerSubusers() method to fetch subusers for specific servers
  • syncServerSubusers() sync method integrating Client API data into database
  • Configurable subuser sync (enabled by default, 25 servers per batch)
  • Automatic owner marking in server_subusers table
  • Rate limiting with 2-second delays between batches
  • New config options: SYNC_SUBUSERS_ENABLED, SYNC_SUBUSERS_BATCH_SIZE
  • Client API types: ClientServer and ClientSubuser structs
• Hytale Game Server Log Persistence - Automatic backup of console logs to database
  • Background worker periodically fetches logs from Hytale API (every 5 minutes)
  • Persistent storage in PostgreSQL with sync state tracking to prevent duplicates
  • Automatic cleanup of logs older than 30 days (runs daily at 4 AM)
  • API endpoints for historical log retrieval:
    • GET /api/v1/hytale/server-logs - Retrieve paginated logs for a game server
    • GET /api/v1/hytale/server-logs/count - Get total log count for a server
  • Sync state tracking to monitor last successful sync and handle API errors gracefully
  • Sentry transaction tracking for log persistence operations

Changed

• Sentry SDK Configuration - SDK API compatibility improvements
  • TracesSampler now correctly returns float64 instead of sentry.Sampled
  • Event tags use event.Tags map directly instead of SetTag() method
  • Span operations properly initialized with WithOpName and WithTransactionSource options
  • Structured logging integration enabled (EnableLogs: true) for automatic log capture
• Workflow Reliability - Fixed workflow configuration issues
  • Updated deprecated artifact actions (v3 → v4)
  • Added continue-on-error flags to linting steps to prevent false failures
  • Fixed gofmt check to use gofmt -l instead of go fmt ./... (non-destructive)
  • Added PR-conditional GitHub script execution to prevent undefined context errors
  • PostgreSQL readiness check added to coverage workflow for test database setup
  • Full sync now includes server subusers as Step 7 (non-blocking, continues on failure)
  • Progress tracking updated to 85% for subuser sync step
  • Config struct extended with SyncSubusersEnabled and SyncSubusersBatchSize fields

Fixed

• Hytale OAuth Handler - Corrected return value handling
  • RefreshGameSession handler now properly captures 2 return values from oauth client (sessionResp, err)
  • New tokens are now persisted to database immediately after refresh
• Test Compatibility - Aligned tests with actual implementation
  • Fixed OAuth endpoint URL format to match https://oauth.accounts.{host} pattern
• Sentry Background Worker Panic - Fixed nil pointer dereference in background jobs
  • Added StartBackgroundTransaction() function for workers without HTTP context
  • Updated StartTransaction() to handle nil fiber.Ctx gracefully
  • Fixed 6 worker functions: RefreshOAuthTokens, RefreshGameSessions, CleanupExpiredSessions, CleanupOldLogs, HandleDiscordWebhook, HandleFullSync
  • All background jobs now use proper Sentry transaction tracking with task operation type
• Sync Progress Updates - Enhanced frontend polling visibility
  • Added lastMessage field to all sync progress updates for real-time status display
  • Standardized status values to RUNNING (was inconsistently in_progress)
  • Progress updates now include user-friendly messages like "Syncing locations...", "Syncing nodes...", etc.
  • Added lastUpdated timestamp to metadata for frontend polling
• Config Validation - Removed duplicate field declarations
  • Fixed duplicate SyncSubusersEnabled and SyncSubusersBatchSize fields in Config struct
  • Corrected rate limiter Allow() return types (int vs int64)
  • Fixed test expectations for rate limiter cleanup behavior
  • Removed tests for private/unexported functions
  • Aligned JWKS cache tests with actual NewJWKSCache(useStaging bool) signature
• Swagger Documentation Generation - Fixed type reference issues in OpenAPI comments
  • Corrected Swagger parameter annotations from backslash to dot notation (types.RefreshTokenRequest instead of types.\RefreshTokenRequest)
  • Added missing types. package qualification to unqualified type references in @Success and @Failure annotations
  • Fixed missing types. prefix on SuccessResponse instantiation in SelectProfile handler
  • All Hytale OAuth handler endpoints now generate valid Swagger documentation

Hytale Support

Added

Hytale OAuth Device Code Flow

• Device Code Authorization - Full OAuth 2.0 Device Authorization Grant implementation
  • /api/hytale/device-code - Request device code for GSP authentication
  • Automatic code expiration (30 min) and polling timeout (15 min)
  • User-friendly device code format with verification URL
  • Session binding for code→token exchange
• Token Polling - /api/hytale/token-poll endpoint with configurable backoff
  • Polls Hytale OAuth server for authorization completion
  • Automatic retry-after header handling
  • Clean response on session limit errors (403 Forbidden)
  • Account and profile ID extraction from token response
• Token Refresh - /api/hytale/refresh-token for extending session validity
  • Automatic refresh window (7 days before expiry)
  • Transaction-safe token replacement in database
  • Preserves account_id and profile_id across refreshes
• Token Validation - JWT signature verification with Ed25519 keys
  • JWKS public key caching with hourly automatic refresh
  • Cryptographic signature validation preventing replay attacks
  • Token expiry validation with 60-second clock skew tolerance
  • Audience (aud) claim verification for correct token type
  • Subject (sub) claim extraction for profile identification

Game Session Management

• Session Endpoints - Complete game session lifecycle management
  • POST /api/hytale/game-session - Create new game session with validation
  • PATCH /api/hytale/game-session - Refresh existing session (10 min window)
  • DELETE /api/hytale/game-session - Terminate session with Hytale cleanup
  • Query parameter validation (account_id, profile_id, session_id)
• Session Tracking - Database persistence for audit and compliance
  • Account and profile associations
  • Session state tracking (created, active, terminated)
  • IP address and user agent logging
  • Timestamp recording for session lifecycle

JWT Token and JWKS Management

• JWKSCache - Thread-safe public key caching system
  • Automatic hourly refresh from Hytale JWKS endpoint
  • Base64URL decoding of Ed25519 public keys
  • In-memory bucket cleanup for stale entries (30 min inactivity)
  • Concurrent read access via RWMutex protection
• TokenValidator - Complete JWT validation pipeline
  • Header/payload/signature parsing from JWT format
  • Ed25519 signature verification with extracted public keys
  • Claim validation (iat, exp, sub, aud)
  • Base64URL decoding with proper padding
  • Support for both identity and session token types
• Database Token Storage - Secure token persistence
  • Encrypted token fields using AES-256-GCM
  • Account and profile associations
  • Expiry timestamps for automatic cleanup
  • Staging environment support for testing

Error Handling & HTTP Mapping

• HytaleError Type - Standardized error response structure
  • Application error codes (INVALID_REQUEST, UNAUTHORIZED, SESSION_LIMIT, etc.)
  • User-facing error messages (no sensitive data leakage)
  • Internal messages for debugging logs
  • Session limit flag for special handling
• HTTP Status Mapping:
  • 400 Bad Request → INVALID_REQUEST, INVALID_DEVICE_CODE
  • 401 Unauthorized → UNAUTHORIZED, EXPIRED_TOKEN, INVALID_TOKEN
  • 403 Forbidden → FORBIDDEN, SESSION_LIMIT_EXCEEDED (with entitlement hint)
  • 404 Not Found → ENDPOINT_NOT_FOUND, SESSION_NOT_FOUND
  • 429 Too Many Requests → RATE_LIMITED
  • 500+ Server Errors → SERVICE_ERROR, INTEGRATION_ERROR
• Response Types - Consistent JSON error responses
  • DetailedErrorResponse with code, message, status fields
  • SessionLimitErrorResponse with entitlement hint
  • NotFoundErrorResponse for missing resources
  • RateLimitErrorResponse with X-RateLimit headers

Audit Logging for Compliance

• Hytale Audit Log Repository - Compliance-grade event tracking
  • 8 event types: TOKEN_CREATED, TOKEN_REFRESHED, TOKEN_DELETED, AUTH_FAILED, SESSION_CREATED, SESSION_REFRESHED, SESSION_DELETED, PROFILE_SELECTED
  • Account, profile, and IP tracking for forensics
  • User agent capture for device fingerprinting
  • Queryable audit trail via GetAuditLogs(account_id, limit)
• Database Schema - Indexed audit table with constraints
  • UUID primary key with auto-generation
  • Event type ENUM validation (8 valid types)
  • Composite indexes on account_id, event_type, created_at DESC
  • JSON details field for flexible event metadata
  • Foreign key relationship to accounts table
• Audit Events Logged:
  • OAuth token creation/refresh/deletion with timing
  • Game session lifecycle (created, refreshed, deleted)
  • Authentication failures with reason codes
  • Profile selection for session binding
  • All events include IP, user agent, timestamp

Rate Limiting (Token Bucket Algorithm)

• Distributed Rate Limiter - Per-endpoint configurable limits
  • Token bucket refill algorithm with float64 precision
  • IP-based limiting for unauthenticated endpoints (device code)
  • Account-based limiting for authenticated endpoints (token ops, sessions)
  • Thread-safe via sync.RWMutex protection
• Endpoint Configurations:
  • Device Code: 5 requests per 15 minutes (per IP)
  • Token Poll: 10 requests per 5 minutes (per account)
  • Token Refresh: 6 requests per 1 hour (per account)
  • Game Session: 20 requests per 1 hour (per account)
• Response Headers - Standard rate limit signaling
  • X-RateLimit-Limit: requests allowed in window
  • X-RateLimit-Remaining: tokens left for requester
  • X-RateLimit-Reset: Unix timestamp when bucket refills
• Rate Limit Errors - Proper 429 responses on limit exceed

Type Consolidation

• Internal Types Package - Organized request/response definitions
  • types/auth.go - Authentication request/response types
  • types/hytale_oauth.go - OAuth and session types
  • types/error_responses.go - Error response types with Swagger annotations
  • types/rate_limit.go - Rate limit error responses
  • types/token_validation.go - JWT validation request/responses
  • Eliminates scattered type definitions across handler files
  • Centralized Swagger documentation via struct tags
  • Reduced import chains and circular dependency risk

Staging Environment Support

• Configuration Branching - Per-environment OAuth endpoints
  • Production Hytale OAuth: https://oauth.hytale.com
  • Staging Hytale OAuth: https://oauth.staging.hytale.com
  • Environment variable HYTALE_ENVIRONMENT (production|staging)
  • Automatic token endpoint and JWKS URL selection
• Testing Support - Isolated staging credentials
  • Separate staging database schema (optional)
  • Staging OAuth tokens won't interfere with production
  • Safe testing of device code, token, session flows

Changed

• OAuth Endpoints Middleware - All 8 Hytale routes now protected with rate limiting
  • Device code endpoint: 5 per 15 min per IP
  • Token polling: 10 per 5 min per account
  • Token refresh: 6 per 1 hr per account
  • Game sessions: 20 per 1 hr per account
• Middleware Chain - Enhanced request processing
  • API Key → JWT Bearer → Rate Limit validation in sequence
  • Proper error responses at each stage
  • Rate limit headers on all responses
• Error Response Format - Standardized across all endpoints
  • Consistent HytaleError structure vs inconsistent previous responses
  • Proper HTTP status codes (400/401/403/404/429/500)
  • No sensitive data in error messages (cleaned up before responding)

Fixed

• Database Connection Pool - Corrected pgxpool.Pool API usage
  • Changed from invalid r.db.conn.ExecContext to r.db.Pool.Exec
  • Changed from invalid r.db.conn.QueryContext to r.db.Pool.Query
  • Applies to all audit logging operations (LogTokenCreated, LogTokenRefreshed, LogTokenDeleted, LogSessionCreated, LogSessionRefreshed, LogSessionDeleted, LogAuthFailure, GetAuditLogs, GetLatestAuditLog)
• Type Safety - AuditLogType enum to string casting for SQL parameters
  • Ensures proper parameterized query execution
  • Prevents SQL injection and type mismatch errors
• Code Duplication - Removed duplicate contains() helper functions
  • Consolidated to single definition in appropriate package
  • Uses standard strings.Contains() for consistency

Technical Details

• Language: Go 1.24
• HTTP Framework: Fiber v2.52.5
• Job Queue: Asynq v0.24.1 (Redis-backed)
• Database Driver: pgx v5.7.2 with pgxpool connection pooling
• Crypto: crypto/ed25519 for JWT signature verification
• JWT Handling: Manual parsing with header/payload/signature validation
• Caching: In-memory thread-safe map for JWKS public keys
• Scheduler: robfig/cron v3.0.1 for token/session refresh jobs
• Logging: zerolog v1.33.0 for structured audit logging
• Container Runtime: Docker with docker-compose

Security

• OAuth Security:
  • Device code authorization flow prevents credential leakage in logs
  • Server-side session state prevents CSRF attacks
  • Automatic device code expiration (30 min) limits brute force window
  • Polling timeout (15 min) prevents indefinite waits
• Token Security:
  • JWT signature verification with Ed25519 (collision-resistant, smaller keys)
  • Token expiry validation prevents indefinite access
  • Audience claim verification prevents token type confusion
  • Rate limiting (6/hr) prevents token enumeration attacks
  • 60-second clock skew tolerance handles time sync issues
• Error Handling:
  • Session limit errors (403) don't leak entitlements (users told "buy upgrade")
  • All errors use generic messages (no leaking OAuth server internals)
  • HTTP status codes match RFC 7231 standards
  • Rate limit responses (429) don't expose implementation details...