feat: Implement chunked file uploading #21
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit introduces chunked file uploading for large files, improving the reliability and user experience for uploads.
Key changes:
- **Server-side (app/routes/index.ts):**
- Added `POST /upload/chunk` endpoint to receive individual file chunks and store them temporarily in `uploads/tmp/<originalFilename>/<chunkIndex>`.
- Added `POST /upload/complete` endpoint to verify all chunks, assemble them into the final file in `uploads/`, and then integrate with existing media processing (database update, transcoding via `MediaProcesser`).
- Enhanced error handling for chunk upload and assembly processes.
- Multer configured for chunk handling.
- **Client-side (app/public/js/index.js):**
- Updated `FileUploader` class to slice large files (configurable threshold, e.g., >20MB) into smaller chunks (e.g., 5MB).
- Chunks are uploaded sequentially to `/upload/chunk`.
- After all chunks are sent, a request is made to `/upload/complete`.
- Progress indication is provided to you during chunked uploads.
- Error handling and retry mechanism for chunk uploads.
- **Error Handling & Cleanup (app/lib/cleanup.ts):**
- Added `cleanupOrphanedChunks` utility function to scan `uploads/tmp/` and remove old, incomplete chunk directories, preventing disk space issues. This function can be triggered by a scheduled job.
- **Testing (tests/chunkedUpload.test.ts):**
- Added a new test suite for the chunked upload functionality.
- Tests cover successful end-to-end uploads, handling of missing chunks, invalid metadata, and authentication for protected routes.
This feature addresses issue #6 (chunk uploading for large files).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit introduces chunked file uploading for large files, improving the reliability and user experience for uploads.
Key changes:
Server-side (app/routes/index.ts):
POST /upload/chunkendpoint to receive individual file chunks and store them temporarily inuploads/tmp/<originalFilename>/<chunkIndex>.POST /upload/completeendpoint to verify all chunks, assemble them into the final file inuploads/, and then integrate with existing media processing (database update, transcoding viaMediaProcesser).Client-side (app/public/js/index.js):
FileUploaderclass to slice large files (configurable threshold, e.g., >20MB) into smaller chunks (e.g., 5MB)./upload/chunk./upload/complete.Error Handling & Cleanup (app/lib/cleanup.ts):
cleanupOrphanedChunksutility function to scanuploads/tmp/and remove old, incomplete chunk directories, preventing disk space issues. This function can be triggered by a scheduled job.Testing (tests/chunkedUpload.test.ts):
This feature addresses issue #6 (chunk uploading for large files).