Clean up CI workflows: rename Kokoro, remove dead framework checks#463
Clean up CI workflows: rename Kokoro, remove dead framework checks#463Alex-Wengg merged 1 commit intomainfrom
Conversation
- Rename tts-test.yml to kokoro-tts-test.yml and polish to match pocket-tts-test.yml (caching, PR comments, timeout, Swift 6.1, ffmpeg). Remove unused FLUIDAUDIO_ENABLE_TTS env var. - Delete framework-app-store-validation.yml and framework-validation.yml which target Sources/FluidAudio/Frameworks/** (path no longer exists, workflows never trigger).
|
Claude finished @Alex-Wengg's task —— View job PR Review Complete ✅Completed comprehensive review of PR #463: Clean up CI workflows: rename Kokoro, remove dead framework checks Todo List:
Review Summary:✅ Excellent cleanup with significant improvements:
No blocking issues identified. This is a well-executed cleanup that removes dead code and significantly improves the TTS testing workflow. |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review Summary
This PR makes a focused and well-reasoned cleanup of CI workflows. The changes are solid overall with good improvements to the TTS workflow.
✅ Excellent Improvements in kokoro-tts-test.yml:
- Comprehensive caching strategy (lines 25-34): Much better than original - includes model paths, Homebrew caches, and build artifacts
- Proper dependency management: Explicit Swift 6.1 setup and ffmpeg installation with fallback
- Enhanced error handling: Better smoke test logic with proper exit code capture and status reporting
- Professional PR automation: Excellent comment system with status table, file size reporting, and proper comment updating logic
- Security: Appropriate
pull-requests: writepermission andcontinue-on-error: truefor non-blocking PR comments - Timeout protection: 45-minute timeout prevents runaway jobs
✅ Justified Deletions:
The deletion of framework-app-store-validation.yml and framework-validation.yml is completely justified:
- Both workflows filter on
Sources/FluidAudio/Frameworks/**which does not exist in the current codebase framework-app-store-validation.ymlreferences a non-existentFrameworkLinkTeststest class- No other workflows or documentation reference these deleted files
- These workflows would never trigger, making them dead code
🔧 Minor Observations:
- The removal of
FLUIDAUDIO_ENABLE_TTS: "1"environment variable appears intentional and likely unused - The workflow naming is now consistent with other test workflows in the repository
- The new workflow follows modern GitHub Actions best practices
📋 Test Plan Verification:
- ✅ Verified that
Sources/FluidAudio/Frameworks/directory does not exist - ✅ Confirmed no other workflows reference the deleted files
- ✅ The Kokoro TTS workflow should run successfully on this PR
Overall Assessment: This is a high-quality cleanup that removes dead code and significantly improves the TTS testing workflow. No blocking issues identified.
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
PocketTTS Smoke Test ✅
Runtime: 0m29s Note: PocketTTS uses CoreML MLState (macOS 15) KV cache + Mimi streaming state. CI VM lacks physical GPU — audio quality may differ from Apple Silicon. |
Kokoro TTS Smoke Test ✅
Runtime: 0m47s Note: Kokoro TTS uses CoreML flow matching + Vocos vocoder. CI VM lacks physical ANE — performance may differ from Apple Silicon. |
Parakeet EOU Benchmark Results ✅Status: Benchmark passed Performance Metrics
Streaming Metrics
Test runtime: 1m8s • 03/29/2026, 01:38 AM EST RTFx = Real-Time Factor (higher is better) • Processing includes: Model inference, audio preprocessing, state management, and file I/O |
Qwen3-ASR int8 Smoke Test ✅
Performance Metrics
Runtime: 3m29s Note: CI VM lacks physical GPU — CoreML MLState (macOS 15) KV cache produces degraded results on virtualized runners. On Apple Silicon: ~1.3% WER / 2.5x RTFx. |
ASR Benchmark Results ✅Status: All benchmarks passed Parakeet v3 (multilingual)
Parakeet v2 (English-optimized)
Streaming (v3)
Streaming (v2)
Streaming tests use 5 files with 0.5s chunks to simulate real-time audio streaming 25 files per dataset • Test runtime: 6m58s • 03/29/2026, 01:44 AM EST RTFx = Real-Time Factor (higher is better) • Calculated as: Total audio duration ÷ Total processing time Expected RTFx Performance on Physical M1 Hardware:• M1 Mac: ~28x (clean), ~25x (other) Testing methodology follows HuggingFace Open ASR Leaderboard |
Sortformer High-Latency Benchmark ResultsES2004a Performance (30.4s latency config)
Sortformer High-Latency • ES2004a • Runtime: 3m 8s • 2026-03-29T05:45:16.186Z |
VAD Benchmark ResultsPerformance Comparison
Dataset Details
✅: Average F1-Score above 70% |
Offline VBx Pipeline ResultsSpeaker Diarization Performance (VBx Batch Mode)Optimal clustering with Hungarian algorithm for maximum accuracy
Offline VBx Pipeline Timing BreakdownTime spent in each stage of batch diarization
Speaker Diarization Research ComparisonOffline VBx achieves competitive accuracy with batch processing
Pipeline Details:
🎯 Offline VBx Test • AMI Corpus ES2004a • 1049.0s meeting audio • 220.2s processing • Test runtime: 3m 47s • 03/29/2026, 01:45 AM EST |
Speaker Diarization Benchmark ResultsSpeaker Diarization PerformanceEvaluating "who spoke when" detection accuracy
Diarization Pipeline Timing BreakdownTime spent in each stage of speaker diarization
Speaker Diarization Research ComparisonResearch baselines typically achieve 18-30% DER on standard datasets
Note: RTFx shown above is from GitHub Actions runner. On Apple Silicon with ANE:
🎯 Speaker Diarization Test • AMI Corpus ES2004a • 1049.0s meeting audio • 39.0s diarization time • Test runtime: 2m 17s • 03/29/2026, 01:51 AM EST |
1 participant
Summary
tts-test.ymltokokoro-tts-test.ymland polish to matchpocket-tts-test.ymlstyle (dependency caching, PR result comments, 45min timeout, explicit Swift 6.1 setup, ffmpeg install, remove unusedFLUIDAUDIO_ENABLE_TTSenv var)framework-app-store-validation.ymlandframework-validation.yml— both filter onSources/FluidAudio/Frameworks/**which no longer exists, so they never trigger.framework-app-store-validation.ymlalso references a nonexistentFrameworkLinkTeststest class.Test plan