Feature Request: Implement High-Performance Structured Logging with Pino

[Avtrkrb]

🎯 Overview

The Nanocoder project currently uses basic console.log() statements throughout the codebase, which provides limited debugging capabilities and no structured logging for production environments. This proposal implements a comprehensive, high-performance logging system using Pino (5-10x faster than Winston) while maintaining backward compatibility.

📊 Current State Analysis

🔍 Key Findings:

• 20+ files using basic console logging across the codebase
• No structured logging framework (Winston, Pino, Bunyan)
• No log persistence - logs lost on application restart
• No log levels - everything outputs at same priority
• Limited debugging - no context preservation across application lifecycle
• No performance metrics - no timing or execution tracking
• No log rotation - unlimited log file growth
• No security compliance - potential exposure of sensitive data

🏗️ Proposed Architecture

Core Implementation Strategy

• Primary Framework: Pino (ultra-high performance, 5-10x faster than Winston)
• Fallback Option: Winston for complex enterprise scenarios if needed
• Gradual Migration: Backward compatibility with feature flags
• Security Compliance: Built-in PII redaction and security logging standards

Directory Structure

source/utils/logging/
├── index.ts              # Main logging interface & facade
├── pino-logger.ts         # Pino implementation (primary)
├── config.ts               # Environment-based configuration
├── formatters.ts           # Structured output formatting (JSON, pretty, production)
├── transports.ts            # Environment-specific outputs
├── redaction.ts            # PII & sensitive data redaction
├── correlation.ts          # Request/response correlation IDs
├── performance.ts          # Timing & metrics collection
└── types.ts                # TypeScript definitions

📋 Implementation Plan

Phase 1: Core Infrastructure (Week 1)

• Dependencies: Add pino@^9.x, pino-pretty@^11.x, pino-roll@^5.x
• Configuration: Environment-based logging with proper directory structure
• Pino Implementation: Type-safe logger with async non-blocking operations
• Formatters: JSON structured output with pretty development formatting
• Redaction: Automatic PII filtering (API keys, tokens, personal data)
• Correlation: Request IDs for tracking AI interactions and MCP operations

Phase 2: Integration (Week 1-2)

• Critical Path Migration: Replace console.log in:
  • source/ai-sdk-client.ts (AI interactions - 376+ lines)
  • source/mcp/mcp-client.ts (MCP operations)
  • source/app.tsx (main application entry)
  • source/utils/error-formatter.ts (error handling)
  • source/utils/message-queue.tsx (UI messaging)
• Enhanced Error Context: Stack traces, error metadata, structured objects
• Performance Hooks: Request timing, memory usage, operation metrics
• Backward Compatibility: Graceful fallback to console.log during transition

Phase 3: Advanced Features (Week 2-3)

• Log Rotation: Time-based retention (7-30 days) with size limits (100MB)
• Hot Reloading: Runtime configuration changes without restart
• Aggregation: Log buffering and batch operations
• Query Interface: Built-in log search and filtering capabilities
• Monitoring Integration: Health checks, log volume alerts, error thresholds

Phase 4: Testing & Documentation (Week 3-4)

• Comprehensive Testing: Unit, integration, and performance benchmarks
• Migration Documentation: Step-by-step transition guide
• API Documentation: Logger interface and usage examples
• Performance Validation: Benchmarks against current console.log approach
• Production Readiness: Load testing, failure scenarios, recovery testing

🎛️ Configuration Strategy

Environment-Specific Logging

Development Environment:

const devConfig = {
  level: 'debug',
  transport: {
    target: 'pino-pretty',
    options: {
      colorize: true,
      translateTime: 'HH:MM:ss Z',
      ignore: 'pid,hostname'
    }
  },
  redact: ['apiKey', 'token', 'password'],
  correlation: true
};

Production Environment:

const prodConfig = {
  level: process.env.LOG_LEVEL || 'info',
  transport: {
    target: 'pino-roll',
    options: {
      frequency: 'daily',
      limit: { size: '100MB', count: 30, days: 7 },
      filename: `${logDirectory}/nanocoder-%Y-%m-%d.log`,
      compress: true
    }
  },
  redact: ['apiKey', 'token', 'password', 'email', 'userId'],
  correlation: true,
  serialize: true
};

Log Directory Strategy

• macOS: ~/Library/Preferences/nanocoder/logs/
• Linux: ~/.config/nanocoder/logs/
• Windows: %APPDATA%\nanocoder/logs\
• Fallback: Current working directory /logs

Log Hierarchy (RFC 5424 inspired)

FATAL (0): System crashes, unrecoverable errors
ERROR (1): Application errors, API failures  
WARN (2):  Potential issues, deprecated usage
INFO (3):  General information, user actions
HTTP (4): HTTP requests/responses (AI SDK)
DEBUG (5): Detailed debugging information
TRACE (6): Fine-grained execution tracing

🔧 Security & Compliance

Built-in Protection Features

• PII Redaction: Automatic detection and masking of sensitive data
• Stack Trace Sanitization: Remove internal paths and system details in production
• Access Control: Log-level permissions and audit trails
• Data Retention: Configurable retention policies for compliance
• Encryption: Optional log file encryption for sensitive environments

Security Logging Standards

• Access Events: User authentication, authorization failures
• Security Violations: Failed authentication attempts, blocked requests
• Data Access: Sensitive data access with user context
• System Events: Configuration changes, permission modifications

📈 Expected Benefits

Performance Improvements

• 5-10x faster logging with Pino vs current console.log approach
• <5% application overhead with async non-blocking operations
• Memory efficient structured logging vs string concatenation
• Reduced I/O blocking for high-throughput scenarios

Developer Experience

• Structured JSON logs for better IDE integration and debugging
• Consistent format across all modules and components
• Searchable logs with correlation IDs and metadata
• Enhanced error context with structured objects and stack traces
• Production-ready monitoring with proper log rotation and retention

Operational Benefits

• Centralized log management with configurable retention policies
• Time-based retention (7-30 days) for compliance and storage management
• Performance monitoring with request timing and resource usage
• Integration-ready for observability platforms and monitoring tools
• Debugging efficiency with preserved context and correlation

🔄 Migration Strategy

Backward Compatibility Approach

1. Feature Flags: Environment variables to enable structured logging per module
2. Facade Pattern: Logging interface that can switch between implementations
3. Gradual Replacement: Systematic replacement of console.log calls by module priority
4. Fallback Mechanisms: Automatic fallback to console.log if logger initialization fails
5. Parallel Operation: Both systems active during transition period

Rollout Priority Order

1. Critical Infrastructure (AI SDK, MCP clients, error handling)
2. Core Application (Main app, configuration, utilities)
3. UI Components (React messaging, user interactions)
4. Development Tools (CLI commands, build scripts)
5. Complete Migration (All console.log calls replaced)

Testing Strategy

• Unit Tests: Logger configuration, formatting, and redaction
• Integration Tests: End-to-end logging workflows
• Performance Benchmarks: Compare current vs new implementation
• Chaos Testing: Logger failure scenarios and recovery
• Production Simulation: Load testing with realistic log volumes

📊 Success Metrics & KPIs

Quantitative Targets

• Log Output Reduction: 40-60% reduction vs current verbose console output
• Debugging Time Improvement: 25-35% reduction in incident resolution time
• Performance Impact: <5% overhead vs current implementation
• Migration Success: 100% console.log replacement without breaking changes
• Test Coverage: >95% for new logging functionality

Qualitative Goals

• Improved Debugging Experience: Structured context preservation across application lifecycle
• Production Readiness: Compliance with enterprise logging standards
• Developer Satisfaction: Better tooling integration and log analysis capabilities
• Maintainability: Clear logging patterns and documentation

🛠️ Risk Assessment

Technical Risks (Low)

• Performance Regression: Minimal risk with Pino's high-performance design
• Memory Usage: Controlled through async operations and efficient serialization
• Breaking Changes: Mitigated through facade pattern and gradual migration

Migration Risks (Low-Medium)

• Developer Adoption: Addressed through comprehensive documentation and training
• Temporary Inconsistency: Managed through dual-logging during transition
• Configuration Complexity: Mitigated through environment-based defaults

Operational Risks (Low)

• Log Loss: Prevented through proper file rotation and backup strategies
• Performance Impact: Minimized through async non-blocking design
• Storage Management: Controlled through retention policies and monitoring

📦 Dependencies & Impact

New Dependencies

{
  "pino": "^9.0.0",
  "pino-pretty": "^11.0.0", 
  "pino-roll": "^5.0.0",
  "sonic-boom": "^4.0.0"
}

Bundle Size Impact

• Estimated Increase: ~150KB gzipped
• Runtime Memory: <2MB additional for structured logging
• Build Time: <10 seconds additional compilation

Compatibility

• Node.js: >=18.0.0 (already satisfied by existing requirements)
• TypeScript: Full compatibility with existing type definitions
• Existing Dependencies: No conflicts with current package ecosystem

🎯 Next Steps

Immediate Actions (Week 1)

1. Dependencies Setup: Add Pino ecosystem packages to package.json
2. Core Implementation: Base logger interface and Pino configuration
3. Directory Creation: Establish logging directory structure
4. Critical Migration: Replace console.log in AI SDK and MCP clients
5. Initial Testing: Validate structured output and performance

Short-term Goals (Weeks 2-4)

1. Complete Migration: Replace all console.log calls across codebase
2. Advanced Features: Implement correlation, performance metrics, log rotation
3. Documentation: Comprehensive usage guide and migration documentation
4. Testing: Full test suite with performance benchmarks
5. Production Readiness: Configuration validation and deployment preparation

Long-term Vision (Months 3-6)

1. Observability Integration: Connect with external monitoring platforms
2. Advanced Analytics: Log aggregation and querying capabilities
3. AI-Powered Insights: Automated log analysis and anomaly detection
4. Ecosystem Expansion: Additional transports and integrations as needed

---

This comprehensive logging implementation will transform the Nanocoder project from basic console output to a production-ready, high-performance structured logging solution while maintaining full backward compatibility and ensuring operational excellence.

📝 Implementation Questions

1. Priority: Should we proceed with Pino for maximum performance, or would you prefer Winston for more enterprise features?
2. Timeline: Is the proposed 4-6 week timeline acceptable for this implementation?
3. Compliance: Any specific security logging standards or compliance requirements we should consider?
4. Integration: Should we plan integration with any existing monitoring/observability tools?

---

Labels: enhancement, logging, performance, infrastructure, security

[Avtrkrb]

@will-lamerton I think having a more robust logging system in place will help us isolate root causes for issues like #118, #110 & others more easily. Runtime logs are captured & saved in the default NANOCODER_CONFIG_DIR directory in a file, users can then share these log files when reporting errors, this gives us with a stack trace to investigate.

Winston is another option. Pino claims to be high performance while Winston claims to have a lot of features. Let me know which logging tool you prefer...

[Avtrkrb]

📊 Winston vs. Pino Comparison

Feature	Winston	Pino	Recommendation
Performance	Moderate (1x baseline)	Ultra-Fast (5-10x)	Winston for production, Pino for performance-critical
Ecosystem	Extensive (15+ transports)	Good (plugin ecosystem)	Winston for enterprise features
TypeScript	Excellent (built-in types)	Excellent (built-in types)	Both provide full type safety
Features	Rich (child loggers, bulk ops)	Fast (minimal overhead)	Winston for comprehensive needs
Transport Options	15+ built-in options	Stream-based plugins	Winston for production flexibility
Exception Handling	Built-in (uncaught + rejections)	Manual implementation	Winston for reliability
Configuration	Flexible (multiple formatters)	Simple (performance-focused)	Winston for complex environments
Memory Usage	Higher (feature overhead)	Lower (minimal design)	Winston for development, Pino for production
File Rotation	Built-in (multiple strategies)	Via plugins	Winston for production rotation
Enterprise	Proven (12+ years)	Growing adoption	Winston for enterprise stability

---

📊 Implementation Decision Matrix

Choose Winston if:

• ✅ Enterprise features needed (child loggers, bulk operations, extensive transport options)
• ✅ Production stability prioritized over raw performance
• ✅ Complex environments with multiple transport requirements
• ✅ Existing monitoring tools integration required
• ✅ Regulatory compliance requiring specific log formats and retention

Choose Pino if:

• ⚡ Maximum performance is critical requirement
• ⚡ Microservices architecture with minimal overhead constraints
• ⚡ High-throughput scenarios with thousands of logs per second
• ⚡ Resource-constrained environments where memory usage is critical

Hybrid Approach (Recommended):

• 🏆 Primary: Winston for production, development flexibility, and enterprise features
• ⚡ Fallback: Pino for performance-critical paths and development debugging
• 🔄 Dynamic: Runtime framework switching based on environment and performance metrics
• 📈 Monitoring: Built-in performance monitoring to trigger automatic framework switches

[will-lamerton]

This looks really good! Giving it a read, I think a better logging system would be excellent.

We have a legacy logging system but it's poor. I think a command like "/logging" to change the verbosity would be awesome too so we can use logging whilst devving. What do you think?

[Avtrkrb]

We have a legacy logging system but it's poor. I think a command like "/logging" to change the verbosity would be awesome too so we can use logging whilst devving. What do you think?

This is a good idea, but I'd prefer the default file logging level to be DEBUG for the packaged & released binary. We could probably modify the in-app logging level with the proposed "/logging" command (maybe a shorter "/log" command ?), but always have a file log saved to the default NANOCODER_CONFIG_DIR with a default timestamp, json format, etc. My reasoning for this is to have a verbose log file that users can share in the github issues if they encounter errors, this will help us identify root causes of issues & investigate faster.

We could probably leverage the multiple transport feature that Winston ships for this...

I'm torn between Pino & Winston as the logging framework/library. I know the specifications talk about having a fallback, but that's just one of the approaches we could take. Ideally, we need to stick to a single logging framework/library to keep dependencies minimal.

Would like to know your preference of the library that we can go with.

[Avtrkrb]

@will-lamerton In continuation with our conversation from discord, I'll proceed with Pino as the preferred logging framework/library.

Additionally, at the start of every log capture I'm planning to capture the following details as well:

1. Provider
2. Model
3. MCP servers & tools connected
4. LSP servers & tools connected

This additional information will help in identifying root causes for any issues that users might encounter.