Move compilation orchestration from App into compiler module with plugin pipeline#6274
Move compilation orchestration from App into compiler module with plugin pipeline#6274FarhanAliRaza wants to merge 8 commits intoreflex-dev:mainfrom
Conversation
Replace 6 separate recursive tree walks (_get_all_imports, _get_all_hooks, _get_all_custom_code, _get_all_dynamic_imports, _get_all_refs, _get_all_app_wrap_components) with a single collect_component_tree_artifacts walk that gathers all compilation data in one pass. Wire the new collector into app.py, compiler.py, and utils.py. Add CompilerPlugin protocol, CompilerHooks dispatcher, and BaseContext/ PageContext/CompileContext types as foundations for the async plugin pipeline.
Refactor the monolithic plugins.py into a plugins/ package: - base.py: core framework (CompilerPlugin base class, CompilerHooks, contexts) - builtin.py: concrete plugins matching legacy recursive collectors - __init__.py: re-exports all public names Adds ApplyStylePlugin, DefaultPagePlugin, and Consolidate*Plugin classes that replicate legacy page compilation behavior. Expands test coverage with per-plugin verification and full pipeline integration tests.
…gin pipeline Extract the monolithic App._compile() method into compiler.compile_app(), routing page evaluation through the async CompileContext/CompilerHooks plugin pipeline. Remove ExecutorType enum and associated env vars (REFLEX_COMPILE_EXECUTOR, REFLEX_COMPILE_PROCESSES, REFLEX_COMPILE_THREADS) in favor of a simple ThreadPoolExecutor. Delete ExecutorSafeFunctions class, move _app_root() and page compilation helpers into compiler.py as standalone functions, and rename the old compile_app() to compile_app_root() to avoid collision.
Greptile SummaryThis PR moves the compilation orchestration that previously lived in Key changes:
Issues found:
Confidence Score: 4/5
Important Files Changed
Sequence DiagramsequenceDiagram
participant CLI as CLI / prerequisites.py
participant App as App._compile()
participant CC as compiler.compile_app()
participant Ctx as CompileContext.compile()
participant Hooks as CompilerHooks
participant Plugins as Plugin Pipeline
participant Thread as asyncio.to_thread
CLI->>App: _compile(prerender_routes, dry_run, use_rich)
App->>CC: compile_app(self, ...)
CC->>CC: _should_compile() check
CC->>CC: asyncio.run(_compile_with_context())
CC->>Ctx: async with compile_ctx → compile(apply_overlay=True)
loop For each page
Ctx->>Hooks: eval_page(page_fn, page=page)
Hooks->>Plugins: DefaultPagePlugin.eval_page → PageContext
Hooks-->>Ctx: PageContext
Ctx->>Hooks: compile_component(root_component)
Hooks->>Plugins: ApplyStylePlugin (pre-yield)
Hooks->>Plugins: ConsolidateCustomCodePlugin (pre-yield)
Hooks->>Plugins: [other plugins pre-yield]
Hooks->>Hooks: recurse structural children
Hooks->>Hooks: traverse prop components (in_prop_tree=True)
Hooks->>Plugins: [plugins post-yield in reverse]
Hooks-->>Ctx: compiled root component
Ctx->>Hooks: compile_page(page_ctx)
Hooks->>Plugins: ConsolidateImportsPlugin.compile_page
Ctx->>Thread: asyncio.to_thread(compile_page, route, component)
end
CC->>CC: compile_memo_components()
CC->>CC: _resolve_app_wrap_components()
CC->>CC: _build_app_root()
CC->>CC: compile_document_root()
CC->>CC: plugin.pre_compile() → collect save/modify tasks
CC->>CC: run save_tasks sequentially
CC->>CC: compile_contexts() + compile_app_root()
CC->>CC: write output_mapping to disk
CC-->>CLI: done
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| async def _compile_with_context() -> None: | ||
| async with compile_ctx: | ||
| await compile_ctx.compile( | ||
| apply_overlay=True, | ||
| evaluate_progress=lambda: progress.advance(task), | ||
| render_progress=lambda: progress.advance(task), | ||
| ) | ||
|
|
||
| with console.timing("Compile pages"): | ||
| asyncio.run(_compile_with_context()) |
There was a problem hiding this comment.
overlay_component silently dropped
_setup_overlay_component() is no longer called anywhere in the new compilation pipeline. That method issues a console.deprecate() with removal_version="0.9.0". The overlay is still applied via CompileContext.compile(apply_overlay=True), but the deprecation notice that guides users to migrate to extra_app_wraps is gone.
Users who set overlay_component will see no warning before it is removed in 0.9.0, breaking their apps silently.
_setup_overlay_component() should be called (or its deprecation logic replicated) before the overlay is applied:
# Emit the deprecation notice for overlay_component before applying it
if app.overlay_component is not None:
app._setup_overlay_component() # issues console.deprecate(...)
async def _compile_with_context() -> None:
async with compile_ctx:
await compile_ctx.compile(
apply_overlay=True,
...
)Or alternatively, call it on line 873 after app._add_optional_endpoints() to preserve the original call-site ordering.
| ) | ||
|
|
||
| with console.timing("Compile pages"): | ||
| asyncio.run(_compile_with_context()) |
There was a problem hiding this comment.
asyncio.run will fail when an event loop is already running
asyncio.run(_compile_with_context()) creates a brand-new event loop. If _compile / compile_app is ever invoked from within an already-running loop — e.g., from an async test, a Jupyter-style environment, or a framework that wraps the CLI in anyio.run — Python raises RuntimeError: This event loop is already running.
Current call sites are synchronous, so this works today, but the constraint is invisible and easy to violate. Consider using asyncio.get_event_loop().run_until_complete(...) with a fallback, or restructuring so the async context runs at a well-known top-level entry point:
try:
loop = asyncio.get_running_loop()
except RuntimeError:
loop = None
if loop is not None and loop.is_running():
# already in an async context; schedule as a task or use nest_asyncio
import nest_asyncio
nest_asyncio.apply()
loop.run_until_complete(_compile_with_context())
else:
asyncio.run(_compile_with_context())Or simply document that compile_app must be called from a synchronous context.
| or not isinstance(compiled_component, Component) | ||
| ): | ||
| return | ||
|
|
||
| hooks = {} | ||
| hooks.update(compiled_component._get_hooks_internal()) | ||
| if (user_hooks := compiled_component._get_hooks()) is not None: | ||
| hooks[user_hooks] = None |
There was a problem hiding this comment.
_collect_prop_custom_code traverses un-compiled prop children
_collect_prop_custom_code is a synchronous recursive helper called in the pre-yield (descending) phase of compile_component, before any plugin has had a chance to transform the prop component tree. Because the async plugin traversal for prop components happens after structural children are compiled and the generators unwind, any plugin that transforms a prop component's _get_custom_code() output in a later pass will not be reflected in the code accumulated here.
This won't matter for the current default plugin ordering, but it is a subtle contract violation that could silently produce wrong output if future plugins modify prop components. Consider collecting prop custom code in the post-yield phase (the ascending side of the generator), or deferring the collection to compile_page after the full tree has been walked.
1 participant
All Submissions:
Type of change
Please delete options that are not relevant.
New Feature Submission:
Changes To Core Features:
After these steps, you're ready to open a pull request.