Improve document for IDE support and simplify doc build #654
+121
−38
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
Author
viirya
reviewed
Dec 25, 2025
README.md
Outdated
| --mount type=bind,source="/path/to/spark-website",target="/spark-website" \ | ||
| -w /spark-website \ | ||
| docs-builder:latest \ | ||
| /bin/bash -c "sh .github/run-in-container.sh" |
Member
There was a problem hiding this comment.
.github? Maybe it is better to be in dev?
cloud-fan
commented
Dec 25, 2025
cloud-fan
commented
Dec 25, 2025
cloud-fan
commented
Dec 25, 2025
cloud-fan
commented
Dec 25, 2025
HyukjinKwon
approved these changes
Dec 25, 2025
viirya
approved these changes
Dec 26, 2025
pan3793
reviewed
Dec 26, 2025
developer-tools.md
Outdated
| <h4>IntelliJ</h4> | ||
|
|
||
| While many of the Spark developers use SBT or Maven on the command line, the most common IDE we | ||
| use is IntelliJ IDEA. You can get the community edition for free (Apache committers can get |
Member
There was a problem hiding this comment.
there is no IDEA community edition now ...
https://blog.jetbrains.com/idea/2025/07/intellij-idea-unified-distribution-plan/
pan3793
reviewed
Dec 26, 2025
developer-tools.md
Outdated
| choose "Detect automatically" for `Generated source folders`, and choose "generate sources" for | ||
| `Phase to be used for folders update`. | ||
| - Go to `Settings -> Build, Execution, Deployment -> Compiler -> Scala Compiler -> Scala Compiler Server`, | ||
| pick a large enough number for `Maximum heap size, MB`, such as "16000". |
Member
There was a problem hiding this comment.
the default is 2g, which does not suffice for Spark, but 16g seems to be too aggressive, this might scare off many developers, I tested 4g also works
yaooqinn
reviewed
Dec 26, 2025
developer-tools.md
Outdated
|
|
||
| Make sure you have a clean start before setting up the IDE: A clean git clone of the Spark repo, install the latest | ||
| version of the IDE. If something goes wrong, clear the build outputs by `./build/sbt clean` and `./build/mvn clean`, | ||
| clear the m2 cache by `rm -rf ~/.m2/repository/*`, remove the IDE folder such as `.idea`, re-import the project into |
Member
There was a problem hiding this comment.
Looks it's IDEA-specific, can we move this to IntelliJ section?
pan3793
approved these changes
Dec 26, 2025
yaooqinn
approved these changes
Dec 26, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR does two things:
developer-tools.mdto give more details about IDE setup, w.r.t. my success tries