Production-Grade Observability, Logging, and Analytics Integration

[johnsmccain]

Summary

This PR implements comprehensive production-grade observability, structured logging, and expanded analytics integration coverage for the NeuroWealth backend.

Changes

📊 Prometheus Metrics Endpoint

New Files:

• src/utils/metrics.ts - Prometheus metrics registry with counters and histograms
• src/routes/metrics.ts - /metrics endpoint for Prometheus scraping

Metrics Implemented:

• Event Processing: events_processed_total, events_processing_duration_seconds, events_processing_rate_per_minute
• Failures: failures_total, failure_rate
• Dead Letter Queue: dlq_size, dlq_retry_total
• Cursor/Lag: cursor_lag_ledgers, last_processed_ledger
• Agent Loop: agent_loop_heartbeat_timestamp, agent_loop_status, agent_rebalance_checks_total, agent_rebalances_triggered_total, agent_snapshot_duration_seconds
• Database: db_operation_duration_seconds, db_connections_active
• HTTP: http_requests_total, http_request_duration_seconds
• Analytics API: analytics_requests_total, analytics_request_duration_seconds

🔧 Instrumentation

Modified Files:

• src/stellar/events.ts - Added metrics recording for event processing, cursor lag, DB operations
• src/stellar/dlq.ts - Added DLQ size tracking on add/retry operations
• src/agent/loop.ts - Added heartbeat, status, rebalance, and snapshot metrics
• src/index.ts - Added /metrics route to Express app

📝 Structured Logging

Modified File:

• src/utils/logger.ts - Enhanced with:
  • Rotating file transports (10MB max, 5 files per log type)
  • Configurable log levels via LOG_LEVEL environment variable
  • JSON output in production environments
  • Fail-safe log directory handling
  • Sensitive data redaction (passwords, tokens, API keys, encryption keys)
  • Optional cloud logging adapter support
  • Exported addCloudLoggingAdapter() function for cloud integrations

Replaced all console. usage with structured logger methods:*

• src/config/env.ts - console.log → logger.info, console.warn → logger.warn
• src/middleware/authenticate.ts - console.error → logger.error
• src/routes/whatsapp.ts - console.error → logger.error
• src/stellar/wallet.ts - console.log → logger.info

🚦 ESLint Rules

Modified File:

• eslint.config.mjs - Added 'no-console': ['error', { allow: ['warn', 'error'] }] to enforce no console usage in production code

📚 Documentation

New File:

• docs/OBSERVABILITY.md - Comprehensive observability guide including:
  • All Prometheus metrics documentation
  • Recommended alert thresholds (critical, warning, info)
  • Prometheus alert rules examples
  • Grafana dashboard recommendations with panel queries
  • Deployment recommendations for Prometheus
  • Cloud logging adapter examples (AWS CloudWatch, GCP, Datadog)
  • Monitoring best practices
  • Troubleshooting guide

🧪 Analytics Integration Tests

New File:

• tests/helpers/analyticsTestFactory.ts - Reusable test data factories:
  • createTestUser(), createTestPosition(), createTestYieldSnapshots()
  • createTestProtocolRates(), createTestSession()
  • setupAnalyticsTestData() - Complete test data setup
  • cleanupAnalyticsTestData() - Proper database cleanup
  • createGroupedProtocolTestData() - For testing grouped protocol structures
  • createValidationFailureTestData() - For validation failure scenarios

Modified File:

• tests/integration/api/analytics.test.ts - Expanded coverage:
  • Authentication behavior tests (expired session, inactive user, missing token)
  • Validation failure tests (invalid period values, invalid types)
  • Grouped protocol structure tests (multiple protocols/assets/networks)
  • Edge case tests (empty data, no positions, null TVL values)
  • Both success and failure paths for all endpoints

📦 Dependencies

Modified File:

• package.json - Added prom-client dependency for Prometheus metrics
• package-lock.json - Updated with new dependency

Testing

All existing tests pass with the new changes. The expanded analytics integration tests cover:

• ✅ Authentication scenarios (valid, expired, inactive, missing)
• ✅ Validation failures (invalid periods, invalid types)
• ✅ Grouped protocol structures (multiple protocols/assets/networks)
• ✅ Edge cases (empty data, no positions, null TVL)
• ✅ Success paths for all endpoints

Configuration

Environment Variables

# Logging Configuration
LOG_LEVEL=info                    # debug, info, warn, error
NODE_ENV=production               # Enables JSON logging

# Metrics are automatically exposed at /metrics
# No additional configuration needed

Cloud Logging (Optional)

Examples provided in docs/OBSERVABILITY.md for:

• AWS CloudWatch
• Google Cloud Logging
• Datadog

Deployment

1. The /metrics endpoint is automatically available at http://your-server:port/metrics
2. Configure Prometheus to scrape this endpoint
3. Import the alert rules from docs/OBSERVABILITY.md
4. Set up Grafana dashboards using the provided panel queries
5. Optionally configure cloud logging adapters

Breaking Changes

None. This is a pure addition with no breaking changes to existing functionality.

Checklist

• Prometheus metrics endpoint implemented
• Event listener instrumented with metrics
• Processing pipeline instrumented with metrics
• Retry system (DLQ) instrumented with metrics
• Agent loop instrumented with metrics
• All console.* usage replaced with structured logger
• Sensitive data redaction implemented
• Rotating file transports configured
• JSON output in production
• ESLint no-console rule added
• Alert thresholds documented
• Grafana dashboard recommendations documented
• Cloud logging adapters documented
• Analytics test factory created
• Analytics integration tests expanded
• All tests passing

Close #94
Close #95
Close #96
Close #97

[drips-wave]

@johnsmccain Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits