Add adaptive training features for improved GAN stability

[Rakshitha-Ireddi]

Authors

• Ireddi Rakshitha
• Yaswanth Devavarapu

This PR introduces research-level adaptive training features to CTGAN that improve training stability and convergence through dynamic parameter adjustment mechanisms.

Features Added

1. Adaptive Discriminator-Generator Step Balancing

• Dynamically adjusts `discriminator_steps` based on loss ratio between generator and discriminator
• Automatically balances training when one network becomes too strong
• Prevents mode collapse and improves convergence

2. Gradient Clipping and Monitoring

• Optional gradient clipping to prevent gradient explosion
• Tracks gradient norms for both generator and discriminator
• Improves training stability, especially for difficult datasets

3. Adaptive Learning Rate Scheduling

• Automatically reduces learning rate when loss plateaus
• Configurable patience period and reduction factor
• Applied independently to both generator and discriminator optimizers
• Helps escape local minima and improve final convergence

4. Early Stopping

• Stops training automatically when loss stops improving
• Configurable patience period
• Prevents overfitting and saves computational resources

5. Generator Eval Mode Fix

• Sets generator to eval mode during sampling (addresses issue Set generator to eval mode before sampling?  #309)
• Ensures consistent sampling behavior with BatchNorm layers
• Properly restores training mode after sampling

Implementation Details

• All features are backward compatible and opt-in (disabled by default)
• Comprehensive unit tests added for all new features
• Follows existing code patterns and Google-style docstrings
• No breaking changes to existing API

Usage Example

from ctgan import CTGAN

ctgan = CTGAN(
    epochs=100,
    adaptive_training=True,        # Enable adaptive step balancing
    gradient_clipping=1.0,          # Clip gradients to norm 1.0
    early_stopping=True,           # Enable early stopping
    early_stopping_patience=10,     # Wait 10 epochs
    adaptive_lr=True,               # Enable adaptive LR
    lr_patience=5,                  # Reduce LR after 5 epochs
    lr_factor=0.5                  # Reduce LR by 50%
)

ctgan.fit(data, discrete_columns=['col1', 'col2'])
synthetic_data = ctgan.sample(1000)

Research Impact

These features address common challenges in GAN training:

• Training instability: Gradient clipping and monitoring
• Mode collapse: Adaptive step balancing
• Slow convergence: Adaptive learning rate scheduling
• Overfitting: Early stopping mechanism
• Inconsistent sampling: Proper eval mode handling

Files Changed

• `ctgan/synthesizers/ctgan.py`: Core implementation of adaptive features
• `tests/unit/synthesizer/test_ctgan.py`: Comprehensive test coverage

Checklist

• Code follows existing patterns and style
• All tests pass
• Backward compatible (all features opt-in)
• Comprehensive documentation in docstrings
• No markdown files committed