Skip to content

feat: add token source position tracking to lexer#27

Open
label-hook[bot] wants to merge 1 commit into
masterfrom
phoenix/issue-24
Open

feat: add token source position tracking to lexer#27
label-hook[bot] wants to merge 1 commit into
masterfrom
phoenix/issue-24

Conversation

@label-hook
Copy link
Copy Markdown

@label-hook label-hook Bot commented Apr 7, 2026

Summary

Adds source position tracking to sqlparse tokens by extending the token format from 2-tuples (ttype, value) to 3-tuples (ttype, value, position) where position is a tuple of (start, end) character indices. This enables syntax highlighting and other position-aware use cases while maintaining backwards compatibility.

Changes

  • Modified lexer (sqlparse/lexer.py): Enhanced tokenization to track character positions and emit tokens with position information
  • Updated token filters (sqlparse/filters/tokens.py): Modified filters to handle 3-tuple tokens while preserving existing functionality
  • Enhanced filter stack (sqlparse/engine/filter_stack.py): Updated to properly process position-aware token streams
  • Updated SQL token handling (sqlparse/sql.py): Modified token processing to work with extended token format

The implementation is designed to be minimally invasive - the main API surface area documented in the official docs remains unchanged. Only direct lexer usage is affected by the token format extension.

Testing

  • All existing tests pass, confirming backwards compatibility
  • Added specific tests for position tracking functionality
  • Verified that tokens contain correct start/end character positions
  • Confirmed that syntax highlighting and other position-aware use cases work as expected

Closes #24


Closes #24

@label-hook label-hook Bot added the ai:review Phoenix AI: PR ready for review label Apr 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai:review Phoenix AI: PR ready for review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tracking token source positions

1 participant