Skip to content

build: Set thread thresholds envs for spark test on macOS #2987

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wForget wants to merge 2 commits into
base: main
Choose a base branch
from
+16 −8
Open

build: Set thread thresholds envs for spark test on macOS #2987

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2d97cfb
build: Set thread thresholds envs for spark test on macOS
wForget Dec 26, 2025
09ea1be
add thread threshold conf in CometTestBase
wForget Dec 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions .github/workflows/pr_build_macos.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,14 +132,6 @@ jobs:
value: |
org.apache.spark.sql.CometToPrettyStringSuite

exclude:
# Skip fuzz suite for Spark 4.0
# https://github.com/apache/datafusion-comet/issues/2965
- profile:
name: "Spark 4.0, JDK 17, Scala 2.13"
suite:
name: "fuzz"

fail-fast: false
name: ${{ matrix.os }}/${{ matrix.profile.name }} [${{ matrix.suite.name }}]
runs-on: ${{ matrix.os }}
Expand All @@ -152,6 +144,14 @@ jobs:
jdk-version: ${{ matrix.profile.java_version }}
jdk-architecture: aarch64
protoc-architecture: aarch_64
- name: Set thread thresholds envs for spark test on macOS
# see: https://github.com/apache/datafusion-comet/issues/2965
shell: bash
run: |
echo "SPARK_TEST_SQL_SHUFFLE_EXCHANGE_MAX_THREAD_THRESHOLD=256" >> $GITHUB_ENV
echo "SPARK_TEST_SQL_RESULT_QUERY_STAGE_MAX_THREAD_THRESHOLD=256" >> $GITHUB_ENV
echo "SPARK_TEST_HIVE_SHUFFLE_EXCHANGE_MAX_THREAD_THRESHOLD=48" >> $GITHUB_ENV
echo "SPARK_TEST_HIVE_RESULT_QUERY_STAGE_MAX_THREAD_THRESHOLD=48" >> $GITHUB_ENV
- name: Java test steps
uses: ./.github/actions/java-test
with:
Expand Down
8 changes: 8 additions & 0 deletions spark/src/test/scala/org/apache/spark/sql/CometTestBase.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,14 @@ abstract class CometTestBase
// this is an edge case, and we expect most users to allow sorts on floating point, so we
// enable this for the tests
conf.set(CometConf.getExprAllowIncompatConfigKey("SortOrder"), "true")
// For spark 4.0 tests, we need limit the thread threshold to avoid OOM, see:
// https://github.com/apache/datafusion-comet/issues/2965
conf.set(
"spark.sql.shuffleExchange.maxThreadThreshold",
sys.env.getOrElse("SPARK_TEST_SQL_SHUFFLE_EXCHANGE_MAX_THREAD_THRESHOLD", "1024"))
conf.set(
"spark.sql.resultQueryStage.maxThreadThreshold",
sys.env.getOrElse("SPARK_TEST_SQL_RESULT_QUERY_STAGE_MAX_THREAD_THRESHOLD", "1024"))
conf
}

Expand Down