Skip to content

Backend API support for pagination #4948

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
ahkcs wants to merge 5 commits into
base: main
Choose a base branch
from
Draft

Backend API support for pagination #4948

ahkcs wants to merge 5 commits into from

Conversation

@ahkcs
Copy link
Contributor

@ahkcs ahkcs commented Dec 11, 2025 •
edited
Loading

Description

This PR adds offset-based pagination support for PPL queries executed through the Calcite engine. Users can now paginate through large result sets by specifying pageSize and offset parameters in the request body

  • Simple API: Pass pageSize and offset in the JSON request body
  • Respects user intent: User-specified commands like head X from Y are honored first

API Design

Request Format

  POST /_plugins/_ppl
  {
    "query": "source=logs | where status='error' | sort timestamp",
    "pageSize": 100,
    "offset": 0
  }

Parameters

Parameter Type Default Description
pageSize int 0 Number of rows per page. 0 = pagination disabled
offset int 0 Number of rows to skip (0-based)

Example: Paginating Through Results

  Page 1 (rows 0-99):
  { "query": "source=accounts | fields name, age | sort age", "pageSize": 100, "offset": 0 }

  Page 2 (rows 100-199):
  { "query": "source=accounts | fields name, age | sort age", "pageSize": 100, "offset": 100 }

  Page 3 (rows 200-299):
  { "query": "source=accounts | fields name, age | sort age", "pageSize": 100, "offset": 200 }

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • New functionality has javadoc added.
  • New functionality has a user manual doc added.
  • New PPL command checklist all confirmed.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff or -s.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Dec 11, 2025

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Comment @coderabbitai help to get the list of available commands and usage tips.

@LantaoJin LantaoJin added the enhancement New feature or request label Dec 12, 2025
LantaoJin
LantaoJin reviewed Dec 12, 2025
View reviewed changes
Copy link
Member

@LantaoJin LantaoJin left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add some tests in CalciteExplainIT?

Is this query

POST /_plugins/_ppl
  {
    "query": "source=logs | where status='error' | sort timestamp",
    "pageSize": 100,
    "offset": 1000
  }

equals to

POST /_plugins/_ppl
  {
    "query": "source=logs | where status='error' | sort timestamp | head 100 from 1000"
  }

?

core/src/main/java/org/opensearch/sql/calcite/plan/LogicalPaginationLimit.java Outdated
* <p>Unlike regular LogicalSort which may be merged with other sorts by Calcite optimizer, this
* class ensures pagination is applied as a final post-processing step on the query result.
*/
public class LogicalPaginationLimit extends Sort {
Copy link
Member

@LantaoJin LantaoJin Dec 12, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please reuse LogicalSystemLimit with type=pagination

Copy link
Contributor Author

@ahkcs ahkcs Dec 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated

core/src/main/java/org/opensearch/sql/executor/QueryService.java Outdated
RelNode optimized = optimize(relNode, context);
RelNode calcitePlan = convertToCalcitePlan(optimized);
executionEngine.execute(calcitePlan, context, listener);
AccessController.doPrivileged(
Copy link
Member

@LantaoJin LantaoJin Dec 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why add the doPrivileged back?

Copy link
Contributor Author

@ahkcs ahkcs Dec 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated to remove doPrivileged

@ahkcs
Copy link
Contributor Author

ahkcs commented Dec 12, 2025

Can you add some tests in CalciteExplainIT?

Is this query

POST /_plugins/_ppl
  {
    "query": "source=logs | where status='error' | sort timestamp",
    "pageSize": 100,
    "offset": 1000
  }

equals to

POST /_plugins/_ppl
  {
    "query": "source=logs | where status='error' | sort timestamp | head 100 from 1000"
  }

?

Added test cases in CalciteExplainIT

@ahkcs ahkcs force-pushed the feat/pagination_support branch from 39352de to 9a809b7 Compare December 12, 2025 19:39
ahkcs added 5 commits December 15, 2025 15:04
@ahkcs
API support for pagination
3473ea5 
Signed-off-by: Kai Huang <ahkcs@amazon.com>

# Conflicts:
#	core/src/main/java/org/opensearch/sql/executor/QueryService.java
@ahkcs
fix compile
3204bf1 
Signed-off-by: Kai Huang <ahkcs@amazon.com>
@ahkcs
Fix ITs
7287833 
Signed-off-by: Kai Huang <ahkcs@amazon.com>
@ahkcs
Update CalciteExplainIT, and fixes
4b9ee5c 
Signed-off-by: Kai Huang <ahkcs@amazon.com>

# Conflicts:
#	integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteExplainIT.java
@ahkcs
fix tests
ad1c15b 
Signed-off-by: Kai Huang <ahkcs@amazon.com>
@ahkcs ahkcs force-pushed the feat/pagination_support branch from 69f3cdb to ad1c15b Compare December 15, 2025 23:13
LantaoJin
LantaoJin reviewed Dec 17, 2025
View reviewed changes
Copy link
Member

@LantaoJin LantaoJin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this PR ready for review now?

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteExplainIT.java
*/
@Test
public void testExplainPaginationApi() throws IOException {
enabledOnlyWhenPushdownIsEnabled();
Copy link
Member

@LantaoJin LantaoJin Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this line since the PPL pagination feature should work when the DSL pushdown disabled. We should check the calcite plan without pushdown.

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteExplainIT.java
*/
@Test
public void testExplainPaginationApiWithHeadFrom() throws IOException {
enabledOnlyWhenPushdownIsEnabled();
Copy link
Member

@LantaoJin LantaoJin Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

@ahkcs
Copy link
Contributor Author

ahkcs commented Dec 18, 2025

Is this PR ready for review now?

Not yet, it's currently a POC PR, still need to finalize design

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@LantaoJin LantaoJin LantaoJin left review comments

@ps48 ps48 Awaiting requested review from ps48 ps48 will be requested when the pull request is marked ready for review ps48 is a code owner

@kavithacm kavithacm Awaiting requested review from kavithacm kavithacm will be requested when the pull request is marked ready for review kavithacm is a code owner

@derek-ho derek-ho Awaiting requested review from derek-ho derek-ho will be requested when the pull request is marked ready for review derek-ho is a code owner

@joshuali925 joshuali925 Awaiting requested review from joshuali925 joshuali925 will be requested when the pull request is marked ready for review joshuali925 is a code owner

@dai-chen dai-chen Awaiting requested review from dai-chen dai-chen will be requested when the pull request is marked ready for review dai-chen is a code owner

@YANG-DB YANG-DB Awaiting requested review from YANG-DB YANG-DB will be requested when the pull request is marked ready for review YANG-DB is a code owner

@mengweieric mengweieric Awaiting requested review from mengweieric mengweieric will be requested when the pull request is marked ready for review mengweieric is a code owner

@vamsimanohar vamsimanohar Awaiting requested review from vamsimanohar vamsimanohar will be requested when the pull request is marked ready for review vamsimanohar is a code owner

@Swiddis Swiddis Awaiting requested review from Swiddis Swiddis will be requested when the pull request is marked ready for review Swiddis is a code owner

@penghuo penghuo Awaiting requested review from penghuo penghuo will be requested when the pull request is marked ready for review penghuo is a code owner

@seankao-az seankao-az Awaiting requested review from seankao-az seankao-az will be requested when the pull request is marked ready for review seankao-az is a code owner

@MaxKsyunz MaxKsyunz Awaiting requested review from MaxKsyunz MaxKsyunz will be requested when the pull request is marked ready for review MaxKsyunz is a code owner

@Yury-Fridlyand Yury-Fridlyand Awaiting requested review from Yury-Fridlyand Yury-Fridlyand will be requested when the pull request is marked ready for review Yury-Fridlyand is a code owner

@anirudha anirudha Awaiting requested review from anirudha anirudha will be requested when the pull request is marked ready for review anirudha is a code owner

@forestmvey forestmvey Awaiting requested review from forestmvey forestmvey will be requested when the pull request is marked ready for review forestmvey is a code owner

@acarbonetto acarbonetto Awaiting requested review from acarbonetto acarbonetto will be requested when the pull request is marked ready for review acarbonetto is a code owner

@GumpacG GumpacG Awaiting requested review from GumpacG GumpacG will be requested when the pull request is marked ready for review GumpacG is a code owner

@ykmr1224 ykmr1224 Awaiting requested review from ykmr1224 ykmr1224 will be requested when the pull request is marked ready for review ykmr1224 is a code owner

@noCharger noCharger Awaiting requested review from noCharger noCharger will be requested when the pull request is marked ready for review noCharger is a code owner

@qianheng-aws qianheng-aws Awaiting requested review from qianheng-aws qianheng-aws will be requested when the pull request is marked ready for review qianheng-aws is a code owner

@yuancu yuancu Awaiting requested review from yuancu yuancu will be requested when the pull request is marked ready for review yuancu is a code owner

@RyanL1997 RyanL1997 Awaiting requested review from RyanL1997 RyanL1997 will be requested when the pull request is marked ready for review RyanL1997 is a code owner

At least 2 approving reviews are required to merge this pull request.

Assignees

No one assigned

Labels

enhancement New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ahkcs @LantaoJin