[SPARK-56619][SQL][TESTS] Add DSv2 repeated table access tests with internal/external changes #55462
Open
longvu-db wants to merge 4 commits into
Open
[SPARK-56619][SQL][TESTS] Add DSv2 repeated table access tests with internal/external changes #55462longvu-db wants to merge 4 commits into
longvu-db wants to merge 4 commits into
Conversation
longvu-db
force-pushed
the
dsv2-pr2-repeated-sql
branch
from
77c6144 to
242bdaa
Compare
longvu-db
changed the title
longvu-db
changed the title
longvu-db
changed the title
longvu-db
force-pushed
the
dsv2-pr2-repeated-sql
branch
from
4b5c3a5 to
489e519
Compare
andreaschat-db
approved these changes
May 12, 2026
Contributor
andreaschat-db
left a comment
andreaschat-db
left a comment
There was a problem hiding this comment.
LGTM. Please update the PR description with the correct number of tests added.
longvu-db
force-pushed
the
dsv2-pr2-repeated-sql
branch
from
a668d67 to
49fc102
Compare
…nternal/external changes Rebased on latest master. Removed CachingInMemoryTableCatalog.scala from the PR since it was already merged via SPARK-56643.
Move the 9 repeated-sql tests from DataSourceV2DataFrameSuite into a DSv2RepeatedSqlTests trait backed by DSv2ExternalMutationTestBase, following the same pattern as DSv2TempViewWithStoredPlanTests (PR apache#55571). This allows a future Connect suite to reuse the same tests by mixing in the trait and providing Connect-specific implementations. Co-authored-by: Isaac
longvu-db
force-pushed
the
dsv2-pr2-repeated-sql
branch
from
49fc102 to
7621d0d
Compare
Co-authored-by: Isaac
- Remove build/sbt-launch-1.12.8.jar.part (accidental partial download) - Fix DSv2ExternalMutationTestBase Scaladoc: DSv2RepeatedSqlTests -> DSv2RepeatedTableAccessWithExternalChangesTests Co-authored-by: Isaac
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Add 9 tests verifying that DSv2 tables reflect the latest state when accessed repeatedly via
sql()(without CACHE TABLE). Eachsession.sql("SELECT * FROM t")call creates a freshQueryExecution, so it always sees the most recent data, schema, and table identity.The tests are extracted into a shared trait
DSv2RepeatedTableAccessWithExternalChangesTestsbacked byDSv2ExternalMutationTestBase, following the same pattern asDSv2TempViewWithStoredPlanTests(PR #55571). This allows a future Connect suite to reuse the same tests by mixing in the trait and providing Connect-specific implementations.The tests cover three scenarios, each with a session-write, external-write, and caching-connector variant:
sql()query sees the new data.sql()query reflects the updated schema.sql()query sees the empty recreated table.For each scenario, the caching-connector variant (
cachingcat) demonstrates that a connector with its ownloadTablecache returns stale results untilREFRESH TABLEinvalidates the cache.External writes use direct catalog API calls (
loadTablewith write privileges,alterTable,dropTable/createTable), matching the pattern used by the existing temp view tests in the same suite.New files
DSv2ExternalMutationTestBase: Base trait with abstract methods (withTestSession,checkRows,getTableCatalog,withTestTableAndViews) and a concreteexternalAppendhelper. Identical to PR [SPARK-56643][CONNECT][TESTS] Add DSv2 temp view with stored plan tests using Spark Connect mode #55571's version for dedup on merge.DSv2RepeatedTableAccessWithExternalChangesTests: Shared trait containing all 9 tests, usingsession.sql(...)with.collect()calls (harmless in classic mode, required for Connect).Modified files
DataSourceV2DataFrameSuite: Mixes inDSv2RepeatedTableAccessWithExternalChangesTestsand implements the 4 classic-mode abstract methods.Why are the changes needed?
These tests document and lock down the expected behavior: repeated
sql()access without CACHE TABLE always sees the latest table state. This prevents regressions if internal resolution or caching logic changes. The trait extraction enables Connect-mode reuse without duplicating test logic.Does this PR introduce any user-facing change?
No. This PR is test-only.
How was this patch tested?
9 new tests in
DataSourceV2DataFrameSuite, all passing:```
build/sbt 'sql/testOnly *DataSourceV2DataFrameSuite -- -z "repeated sql()"'
```
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (claude-opus-4-6)