feat: folder-config precedence + logs endpoint + ipynb streaming #25
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ory reduction Refactored Jupyter notebook interpreter to use pure streaming parsing: - Made ijson a required dependency (was optional/fallback) - Removed all full-load fallbacks that defeated streaming purpose - Optimized streaming parser to count all cells while limiting content sampling - Version bumped from 0.2.0 to 0.3.0 Memory efficiency improvements: - Achieved 97.9% memory reduction vs full-load parsing (far exceeds 40% target) - For 3.6MB notebook: streaming uses ~165KB vs ~8MB for full load - All cells counted accurately regardless of notebook size Tests added: - Small notebook memory efficiency test (< 1MB peak) - Large notebook memory reduction test (validates >=40% reduction) - Large notebook cell counting accuracy test (1500 cells) - Streaming extracts imports and headings correctly All ipynb-related tests pass. Resolves task:interpreters/refactor/ipynb-streaming. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Implements streaming parser for Jupyter notebooks with 97.9% memory reduction. - Refactored ipynb interpreter to use ijson streaming (no full-load fallbacks) - Added comprehensive memory profiling tests - Added tutorial documentation - Version bump to 0.3.0 Resolves task:interpreters/refactor/ipynb-streaming
… resolution Fixed two issues preventing per-folder configuration from working correctly: 1. **Python 3.10 compatibility**: Added tomllib/tomli fallback - Python 3.11+ has tomllib built-in - Python 3.10 needs tomli backport - Config loading was failing silently on Python 3.10 2. **Removed inline config reading in scans_service**: - Previously read `.scidk.toml` directly without precedence - Now uses `load_effective_config()` consistently - Properly walks up directory tree and merges configs - Closest config wins (child overrides parent) Test results: - ✅ test_folder_config_precedence_includes_excludes now passes - ✅ All 117 non-e2e tests pass - ✅ Per-folder rules honored for sibling directories Resolves task:interpreters/toggles/folder-config 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Fixes per-folder .scidk.toml precedence and Python 3.10 compatibility. - Added tomllib/tomli fallback for Python 3.10 - Fixed scans_service to use load_effective_config consistently - Test now passes: sibling directories honor their own configs Resolves task:interpreters/toggles/folder-config
Added operational logs browsing endpoint alongside existing /api/metrics: **Endpoint features:** - GET /api/logs with pagination (limit, offset) - Filtering by level (INFO, ERROR, etc.) - Filtering by timestamp (since_ts) - Returns: ts, level, message, context - Privacy: No sensitive file paths or user data exposed **Implementation:** - Queries logs table from SQLite - Max limit capped at 1000 entries - Results ordered by timestamp DESC (most recent first) - Graceful error handling **Tests added:** - Endpoint existence and structure validation - Pagination functionality - Level filter verification - Timestamp filter verification - Privacy guardrails (no sensitive fields exposed) All 122 non-e2e tests pass. Resolves task:ops/mvp/metrics-and-logs-endpoints. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Adds /api/logs endpoint for operational log browsing. - Pagination and filtering (level, since_ts) - Privacy guardrails (no sensitive data) - Comprehensive test coverage Resolves task:ops/mvp/metrics-and-logs-endpoints
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR includes three completed high-priority tasks from the roadmap:
1. Jupyter Notebook Streaming Parser (RICE 3.5) ✅
2. Per-Folder Config Precedence (RICE 3.4) ✅
.scidk.tomlprecedence to honor per-folder configstest_folder_config_precedencenow passes3. Operational Logs Endpoint (RICE 3.0) ✅
GET /api/logswith pagination and filteringTest Results
Changes Summary
scidk/interpreters/ipynb_interpreter.py- Streaming refactorscidk/core/folder_config.py- Python 3.10 compat + precedencescidk/services/scans_service.py- Use load_effective_configscidk/app.py- Added /api/logs endpointtests/test_ipynb_interpreter.py- Memory profiling teststests/test_logs_endpoint.py- Logs API testsdocs/ipynb-streaming-optimization.md- Tutorialpyproject.toml+requirements.txt- Added ijson dependencyMigration Notes
Resolves:
🤖 Generated with Claude Code