Performance: Skip already processed projects #41
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Should fix the issue observed in #37.
|
Performance issues persist. As shown in the image below, a large number of calls occur here. 
|
|
After I commented out that line of code, it loaded normally. |
Member
Author
|
That is interesting. Any chance you can share an abstract example? I looks like the dependency graph is very deep, i.e. lots of target projects in a dependency chain. Is that true? If removing this line helps, it means we are re-entering the method. @runchen0919 I will add more debug log output. Can you collect the log and attach it to the issue? |
|
You can add some logs. I've collected them here, and after hiding the project-related information, I'll publish the logs. |
|
@guw Below is a portion of the logs I extracted, which is consistent with the phenomena I observed during debugging. The consistent issue is the repeated execution of some underlying common dependency libraries. |
|
The following diagram shows the dependency graph of one of the services in the project, which reflects the depth of the service dependency graph. 
|
guw
force-pushed
the
perf
branch
2 times, most recently
from
38aba6f to
62758a9
Compare
Member
Author
|
@runchen0919 Thank you. One more update with a bit more detail. Can you get me the log again please? I am thinking the resolution might be unnecessary at this point and can probably be eliminated but figuring out what takes so much time would be nice. |
|
@guw Below are the logs from my run after it was blocked. |
Member
Author
|
@runchen0919 Does |
|
@guw src.java.com.project_name.base - base's deps only contain third-party dependencies and //src/thrift/project_name/common:utils |
Member
Author
|
Is this all from a single classpath container? I am missing the output of |
|
@guw The code never reached the following code block, so no log recording the execution time was obtained. |
|
The following log entries have been added: |
Member
Author
|
Ah perfect. Thanks @runchen0919. So the problem is the overwhelming number of projects in the It was a convenience anyway and never correct. I will document it. |
Member
Author
|
@runchen0919 Please let me know if that makes things better now. |
guw
changed the title
|
@guw The problem persists. My debugging revealed that the current blocking issue isn't related to the commented-out code, as the blocking logic occurs before that commented-out section. This is because my breakpoint at 
However, as shown below, based on my observation, the blocking code logic is currently still executed in the 
|
|
@guw Based on my observations, commenting out Do you think removing the |
Member
Author
|
@runchen0919 Thanks for your debugging. I think it is the pure complexity of the dependency graph that is triggering this. The |
guw
changed the title
|
@guw Thank you very much. It's working normally now. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As part of analysis in this PR it was discovered that a complex build graph can overwhelm the runtime classpath computation as observed in #37.
When a project was already processed there is no need to process is it again. It's classpath was resolved and added already. Thus, we now skip it, avoid re-entering branches which we already processed.