Check files in parallel with crossbeam #6783
Open
+251
−113
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Processing all files in parallel is more efficient than checking each repo in parallel. One issue with the current design of processing each repo in parallel is that threads that process smaller repos end early and don't help process files from larger repos. For example, r-l/rust is a large repo that takes a long time to check because there's only one thread working on it. Besides enabling us to process all files in parallel, `check_diff_for_file` return a `Result<(), (Diff, ..)>` instead of a `u8`, which is a more idiomatic way to represent any errors we find when checking for diffs.
Now we'll process each file in parallel instead of processing each repo in parallel, and after checking all repositories we'll report on any errors that we've found.
This option control how many threads process files in parallel.
rustbot
added
A-CI
Contributor
Author
|
@jieyouxu the |
Contributor
Author
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This improves on the current diff check process. It's more efficient to process each file in parallel instead of processing each repo in parallel. Also, a new
worker_threadsoption was added to control how many threads we want processing files at the same time.I tried to break this PR up into logical commits so it might be best to review this one commit at a time.