Skip to content

perf(cll): skip Jinja compilation when compiled_code exists#1230

Merged
wcchang1115 merged 3 commits intomainfrom
feature/cll-skip-generate-sql
Mar 24, 2026
Merged

perf(cll): skip Jinja compilation when compiled_code exists#1230
wcchang1115 merged 3 commits intomainfrom
feature/cll-skip-generate-sql

Conversation

@even-wei
Copy link
Contributor

Summary

  • Skip expensive Jinja re-rendering in CLL when manifest nodes already have compiled_code (from dbt compile)
  • Build table_id_map from parent node aliases instead of custom ref/source functions
  • Fall back to Jinja rendering if two parents share the same alias (prevents incorrect column mappings)

Changes

  • recce/adapter/dbt_adapter/__init__.py: In get_cll_cached(), check for compiled_code on the manifest node before calling generate_sql(). If present, use it directly and build table_id_map from parent aliases/identifiers. Detect alias collisions and fall back to Jinja path.
  • tests/adapter/dbt_adapter/test_dbt_cll.py: 5 new tests — basic compiled_code, custom alias, source parent, multiple parents (join), alias collision fallback.

Context

generate_sql() calls into dbt's Jinja engine (macro resolution, context setup, rendering) for every node during CLL. When a user has already run dbt compile, the manifest contains fully-rendered SQL — re-compiling is wasted work.

sqlglot's qualify() strips database/schema qualifiers, so d.node is always just the table name (= model alias). This means table_id_map can map alias → unique_id directly from the manifest.

Test plan

  • All 44 tests/util/test_cll.py tests pass (pure SQL parsing, unchanged)
  • All 20 existing tests/adapter/dbt_adapter/test_dbt_cll.py tests pass (Jinja path unchanged)
  • 5 new compiled_code tests pass (basic, alias, source, multi-parent, collision)
  • Manual test with a real dbt project that has compiled_code in manifest

Size

+269/-50 across 2 files

🤖 Generated with Claude Code
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com

When dbt compile has already been run, manifest nodes contain compiled_code
with fully-rendered SQL. Use it directly for CLL instead of re-rendering
Jinja through dbt's engine, avoiding the most expensive step per node.

Build table_id_map from parent node aliases (sqlglot strips db/schema
qualifiers). Fall back to Jinja rendering if two parents share the same
alias to avoid incorrect column mappings.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: even-wei <evenwei@infuseai.io>
@even-wei even-wei self-assigned this Mar 23, 2026
Copy link
Contributor Author

@even-wei even-wei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-Review

Correctness ✅

  • sqlglot qualifier stripping verified: cll() always returns just the table name in d.node regardless of qualification level ("db"."schema"."table"table). Confirmed via direct testing.
  • table_id_map mapping is correct: Pre-compiled path maps alias.lower() → unique_id, which matches what sqlglot returns in d.node. Schema dict keys also use the alias, matching sqlglot's qualify() expectations.
  • Alias collision fallback: When two parents share an alias, table_id_map is cleared and pre_compiled is set to None, correctly falling through to the Jinja path with globally unique table names.
  • lru_cache compatibility: Cache key is (node_id, base) — compiled_code comes from the manifest which is stable during a session. No stale cache risk.

Edge Cases Covered ✅

Scenario Handling
No compiled_code (default) Falls back to existing Jinja path — zero behavioral change
Custom alias on parent Uses alias attr, tested in test_cll_with_compiled_code_custom_alias
Source parent Uses identifier (or name), tested in test_cll_with_compiled_code_source
Parent not in nodes/sources continue — silently skipped (matches existing behavior)
Alias collision (cross-project) Detects and falls back to Jinja, tested in test_cll_with_compiled_code_alias_collision_falls_back
compiled_code present but SQL parsing fails Caught by existing except RecceException/Exception_apply_all_columns(node, "unknown")

Potential Concerns

  1. Test helper _set_compiled_code patches compiled=True alongside compiled_code because dbt's CompiledResource.__mashumaro_post_serialize__ strips compiled_code from to_dict() when compiled=False. This is only needed in tests — production manifests loaded from JSON already have compiled=True.

  2. Repeated alias resolution (3 occurrences of getattr(parent_node, 'alias', None) or parent_node.name): Considered extracting a helper but it would be a premature abstraction for the current scope.

Verdict

No critical or important findings. Implementation is correct and well-tested. Ready for human review.

@codecov
Copy link

codecov bot commented Mar 23, 2026

Codecov Report

❌ Patch coverage is 97.00599% with 5 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
recce/adapter/dbt_adapter/__init__.py 94.38% 5 Missing ⚠️
Files with missing lines Coverage Δ
tests/adapter/dbt_adapter/test_dbt_cll.py 99.02% <100.00%> (+0.32%) ⬆️
recce/adapter/dbt_adapter/__init__.py 77.69% <94.38%> (+0.87%) ⬆️

... and 5 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

even-wei and others added 2 commits March 23, 2026 13:02
Extract _get_parent_table_name and _build_schema_from_aliases as testable
methods. Add direct unit tests covering model alias, source identifier,
unknown parent (returns None), catalog=None, and mixed parent types.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: even-wei <evenwei@infuseai.io>
@even-wei even-wei requested a review from wcchang1115 March 23, 2026 08:33
@even-wei even-wei marked this pull request as ready for review March 23, 2026 08:33
Copy link
Collaborator

@wcchang1115 wcchang1115 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested with a Snowflake dbt project, and the pre-compiled path worked correctly.
Thanks!

@wcchang1115 wcchang1115 merged commit 90d6f52 into main Mar 24, 2026
19 checks passed
@wcchang1115 wcchang1115 deleted the feature/cll-skip-generate-sql branch March 24, 2026 11:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants