Mistral Workflows integration + runtime_bridge extraction + MISTRAL_NATIVE router#954
Mistral Workflows integration + runtime_bridge extraction + MISTRAL_NATIVE router#954lchoquel wants to merge 100 commits into
Conversation
- Updated `mistralai` dependency to version 2.4.4 in `pyproject.toml`. - Added a new `TODOS.md` file for planning and progress tracking of Mistral Workflows integration. - Introduced new activities and bridge for running Pipelex pipes within Mistral Workflows. - Implemented execution modes for Pipelex pipes in Mistral Workflows. - Added exception handling specific to Mistral Workflows. - Updated various imports to align with the new `mistralai.client` structure. - Enhanced type checking and error handling in workflow scripts. - Added integration tests for the new Mistral Workflows activities.
- Introduced the `pipelex.plugins.mistralai_workflows` plugin, enabling invocation of Pipelex pipes within Mistral Workflows activities. - Added three execution modes: `DIRECT`, `TEMPORAL_BLOCKING`, and `TEMPORAL_FIRE_AND_FORGET`, allowing flexible integration options. - Created documentation for the new plugin and usage recipes, detailing installation and execution modes. - Implemented integration tests for both blocking and fire-and-forget modes to ensure functionality. - Updated `CHANGELOG.md` to reflect these additions and improvements.
- Introduced `pipelex_run_pipe_offloaded` activity to handle large payloads by leveraging Mistral's `ActivityInOutOffloadingInterceptor`, allowing seamless streaming of oversized data through blob storage. - Updated `CHANGELOG.md` to document the new offloaded activity and its usage. - Added integration tests for the offloaded activity to ensure correct payload handling and functionality. - Enhanced documentation with examples for using the new offloaded activity and its configuration requirements.
- Added `pipelex_run_pipe_streaming` activity to enable real-time progress tracking of Pipelex pipes within Mistral Workflows, emitting lifecycle events (`CustomTaskStarted`, `CustomTaskInProgress`, `CustomTaskCompleted`, `CustomTaskFailed`). - Enhanced `DIRECT` execution mode to provide per-step event updates, allowing detailed observability of pipe execution. - Updated `CHANGELOG.md` to document the new streaming activity and its features. - Created integration tests for the streaming activity to validate event emissions and functionality. - Improved documentation with examples for using the new streaming variant in workflows.
- Implemented per-step event updates for the `pipelex_run_pipe_streaming` activity in `DIRECT` execution mode, allowing detailed observability of pipe execution through `CustomTaskInProgress` events. - Updated `CHANGELOG.md` to reflect the new features and improvements in streaming support. - Enhanced integration tests to validate the emission of per-step events during multi-step pipe executions. - Improved documentation with examples for utilizing the new streaming features in workflows.
Remove the `output: PipelexPipeRunOutput | None = None` initializer and the trailing `assert output is not None` in `_run_streaming_with_per_step_events`. The narrowing was assert-based (stripped under `python -O`) and only existed to satisfy the type checker after a `try`/`finally` cleanup block. With the pre-init dropped, pyright narrows `output` from the `try` body alone, since the post-`try/finally` code is only reachable on success. Extend the drain comment to call out that the cleanup serves both the happy path (snapshot ordering) and the failure path (publishing pending in-progress events before `Task.__aexit__` emits `CustomTaskFailed`).
- Updated the `instructor` dependency to a new commit hash for improved functionality.
Relocate the in-tree TODOS.md to wip/mistral-workflows-sub-module.md and add wip/mistral-workflows-plugin-extract.md outlining the extraction of the Mistral Workflows integration into its own pip-installable plugin package scaffolded from pipelex-starter-python.
…ce docs and clarify package structure
…ning and clarification on line references
…gration - Introduced a new `pipelex.runtime_bridge` package with core components for executing Pipelex pipes in various modes (DIRECT, TEMPORAL_BLOCKING, TEMPORAL_FIRE_AND_FORGET). - Added `ensure_pipelex_booted` function for idempotent initialization of Pipelex. - Created input/output models (`PipelexPipeRunInput`, `PipelexPipeRunOutput`) for structured data handling. - Implemented error handling with custom exceptions for runtime bridge operations. - Developed comprehensive integration tests for the runtime bridge, covering direct execution and library crate handling. - Updated documentation and TODOs to reflect the new structure and functionality.
- Extracted Mistral Workflows integration from `pipelex` into a new package `pipelex-mistralai-workflows`, allowing for independent installation and usage. - Removed the `pipelex.plugins.mistralai_workflows` modules and updated the core runtime bridge to be framework-agnostic. - Updated `CHANGELOG.md` to reflect the changes and migration instructions for users. - Deleted outdated documentation and tests related to the previous plugin structure. - Ensured no behavioral changes; all existing functionalities remain intact under the new package structure.
… updates and clarifying remaining tasks for plugin extraction
…workflows # Conflicts: # CHANGELOG.md
…ipelex.temporal Move pipe_run_arg, hydration, submitter_hydration into a new pipelex.runtime_bridge.primitives package and extract framework-agnostic cores for graph assembly, delivery, and trace flushing. Add pipe_classification.is_controller_pipe / is_leaf_pipe helpers. The Temporal activity wrappers in pipelex.temporal.tprl_pipe now delegate to the shared primitives, so the upcoming Mistral-native execution mode can reuse the same bodies without coupling the plugin to pipelex.temporal. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Surface the new mode in PipelexExecutionMode + add the bridge dispatch branch (raises MissingMistralWorkflowsPluginError with an install hint until the plugin primitives land in the next phase). The branch lazy-imports pipelex_mistralai_workflows.primitives.pipe_run so the pipelex package itself stays free of the optional plugin dep. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Lets host runtimes (e.g. Mistral-native mode) install a custom pipe router for the duration of a controller body without touching the process-wide hub state. Tracked under primitives.md §10.5 — accepted option (b). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…leaves Pin the controller/leaf branching used by both pipelex.temporal and the mistral-native plugin to dispatch pipes correctly. Covers all four controller types (PipeSequence, PipeBatch, PipeCondition, PipeParallel) and four leaf operators (PipeLLM, PipeCompose, PipeFunc, PipeImgGen) via spec'd MagicMocks, since the classifier only inspects the type MRO. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Remove the entire "text then object" mechanism from PipeLLM down through cogt and Temporal layers. The StructuringMethod.PRELIMINARY_TEXT enum value stays so a future implementation can opt in; selecting it at runtime now raises NotImplementedError. Rename make_object_direct -> make_object and make_object_list_direct -> make_object_list since the "_direct" suffix only existed to contrast with the deleted text-then-object variants. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Audit confirms today's per-workflow uniqueness invariant holds for the collapse refactor: every operator-side ContentGeneratorProtocol method is invoked at most once per WfPipeRouter execution (mutually-exclusive branches in PipeLLM/PipeImgGen, single unconditional call in PipeExtract, no calls from PipeCompose/StructuredContentComposer). Strategy (i) is adopted with two mitigations for Phase 1: split the duplicate "craft-image" default between make_single_image/make_image_list, and construct distinct activity_ids inside make_extract_pages for its two inner activity dispatches. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…hild-spawn paths Phase 5 of the Temporal IDs/Naming redesign: replace deprecated dict-based search attributes with TypedSearchAttributes throughout the workflow layer. Adds five module-level SearchAttributeKey constants in observability.py and flips the type annotations on the WorkflowExecutor surface. As a follow-up, unifies the last raw workflow.execute_child_workflow call in wf_pipe_run.py to route through WorkflowExecutor.execute_child_workflow, matching the pattern already used by TemporalPipeRouter's child branch. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ath test - Replace catch-all except Exception in workflow_caller.py with named SDK exceptions (WorkflowAlreadyStartedError, RPCError, WorkflowFailureError on the client path; ChildWorkflowError-only on the child path). - Add failure-path integration test for WfPipeRun pinning the exception-type shift from ChildWorkflowError to WorkflowExecutionError introduced by the Phase 5 child-spawn-path unification. - Fix latent production hang: register WorkflowExecutionError via workflow_failure_exception_types on the production Worker (and the test Worker). Without this, any workflow re-raising WorkflowExecutionError triggers indefinite workflow-task retry instead of failing terminally, because WorkflowExecutionError is not a temporalio.exceptions.FailureError subclass. - Fix TODOS.md doc path: docs/under-the-hood/temporal-deployment.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Captures the deferred design work to make WorkflowExecutionError inherit from temporalio.exceptions.ApplicationError, which would remove the workflow_failure_exception_types Worker-side registration added in 117bbe0. Documents scope, open questions, and trigger conditions for the eventual cleanup. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…up CLI Flip the Temporal search-attribute boot check from warn-and-continue to hard-fail on reachable namespaces — the previous framing was dishonest because real clusters reject every workflow start that references an unregistered attribute. Add a [temporal.search_attributes] config block (master enabled toggle + opt-in subset of the five built-ins), and a new `pipelex setup-temporal-namespace` CLI that wraps the registration via the same connection config the worker uses, with a permission-denied fallback runbook for Temporal Cloud namespaces. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
TODOS.md, the WfPipeRun failure-path test docstring, the worker workflow_failure_exception_types comment, and the WorkflowExecutor child-spawn wrapper docstrings still described the briefly-unified state from the Phase 5 follow-up. The Phase 6 follow-up (commit ac8e233) reverted that unification for replay-determinism reasons — WorkflowExecutorFactory.create_executor seeds config-derived options that would be baked into the recorded StartChildWorkflowExecution command — but the surrounding prose was not updated to match. Updated narrative to reflect the current state: both child-spawn sites call workflow.execute_child_workflow(...) directly and wrap ChildWorkflowError as WorkflowExecutionError in-place; the unused WorkflowExecutor.execute_child_workflow / start_child_workflow wrapper methods now carry warnings explaining the in-workflow replay-determinism trap. No code-behavior changes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…the rest Move plans for shipped work (id-and-naming-plan pre-checkpoints, collapse-content-generation v2 plan+analysis+HTML, per-activity-queue-routing-v1, queue-options-and-worker-profiles plan+design, text-then-object brief, operators-as-activities analysis, workflow-and-activity-ids problem statement) into wip/archive/. Renumber the four surviving files with sortable prefixes: 00-temporal-id-primitives (evergreen reference), 01-id-and-naming-design (refreshed status: "Implemented"), 02-id-and-naming-plan (formerly top-level TODOS.md; refreshed status: "Phases 1-6 shipped"), 03-temporal-error-handling-revamp (the only deferred open item). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ng, point at Worker Versioning Document the rejected `TemporalDispatchSnapshot` approach (architectural inversion, payload bloat, defeats central-config purpose, not Temporal-idiomatic) and the three-option roadmap (docs+replay test / Worker Versioning / thin search-attrs-only snapshot) so future readers don't re-attempt the same wrong-shaped fix. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… claim, contradictory escape-hatch example, skill allowed-tools gaps + unbounded worker-start waits - docs/distributed-execution/task-routing.md: correct activity_queues path to [temporal.worker_config.activity_queues.*] throughout (per-activity routing, per-handle overlays, resolution-order list) - docs/under-the-hood/pipe-routing-and-execution.md: replace stale act_llm_gen_text-only claim with current resolve_dispatch behavior across all content-generation activities - wip/temporal-primitives/id-and-naming.html: flip escape-hatch example to enabled=false so it matches the "turn off custom attributes entirely" header - .claude/skills/temporal-e2e-validate/SKILL.md: add timeout/pkill/sleep/echo/tail/seq to allowed-tools; replace unbounded `until ... grep ...; do sleep 1; done` with bounded 30s waits that dump last 50 lines and exit 1 on timeout (both two-scoped-workers and single-worker blocks) - wip/temporal-next/01-deferred-items.md: cross-cite cubic-dev-ai on the replay-determinism deferral so both reviewers' flags are visible
…p with shared helper The _cv_job_iter helper was copy-pasted from pipe_job_from_library but dropped the open_registry/close_registry bracket and let build_pipe_job mint its own random pipeline_run_id, so --pipe-run-mode live runs would fail in reporting. Extended pipe_job_from_library with an optional working_memory_builder hook, deleted the duplicate, and routed cv_batch_screening_job through the single source of truth. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
# Conflicts: # CHANGELOG.md # TODOS.md # docs/building-methods/pipes/pipe-operators/PipeLLM.md # pipelex/cogt/templating/template_preprocessor.py # tests/unit/pipelex/cogt/templating/test_template_preprocessor.py # tests/unit/pipelex/pipe_operators/pipe_llm/test_llm_prompt_blueprint.py # tests/unit/pipelex/pipe_operators/pipe_llm/test_pipe_llm_blueprint.py
…xtract / ReasoningEffort fields Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…to feature/Mistral-workflows-merge-2 # Conflicts: # CHANGELOG.md # TODOS.md # pipelex/temporal/tprl_pipe/temporal_pipe_router.py # pipelex/temporal/tprl_pipe/wf_pipe_run.py # tests/unit/pipelex/runtime_bridge/primitives/test_temporal_pipe_router_child_observability.py
…ows-merge-2 # Conflicts: # docs/building-methods/pipes/pipe-operators/PipeLLM.md
Reconcile dev into the Mistral Workflows feature branch. dev carried the review-driven refinements of the shared Temporal/PipeStructure work (released in v0.29.0) plus the error-handling overhaul, dataclass/pathlib cleanups, and the v0.29-v0.30.3 releases; this branch carries the Mistral Workflows integration, the runtime_bridge.primitives extraction, the MISTRAL_NATIVE contextvar router, and the Temporal IDs/observability redesign. Conflict resolution: - Temporal config (pipelex.toml x3 + config_temporal.py): adopt dev's reviewed defaults (search_attributes off, empty queue_options, default worker-runtime profile, frozen DispatchOptions); drop HEAD's shipped rate-cap block and its guard test. - Interpreter/core: take dev's relocated BundleElaboratorError; keep HEAD's hub.py contextvar pipe-router override (MISTRAL_NATIVE). - Temporal runtime (workflow_caller, temporal_error, wf_pipe_*, act_*): weave HEAD's runtime_bridge.primitives extraction with dev's ErrorReport propagation (recover_error_report, convert_pipelex_errors, per-workflow WorkflowLog request_id). The execute_delivery primitive now forwards error_report and request_id end-to-end. - Error-handling: adopt dev (observe-and-reraise in pipe_run, CogtError propagation in the router protocol, JobMetadata.request_id, jinja2.exceptions rename, relocated json error classes). - Mistral workers: keep mistralai 2.x imports/structure; port dev's InferenceErrorCategory/UserAction enrichment onto the 2.x calls. Repoint dev-only tests and mistral_factory off the 1.x SDK paths. - Docs/CHANGELOG: union; [Unreleased] = dev's entries plus the Mistral Workflows entry (the Temporal/PipeStructure entries stay under dev's v0.29.0 release). - uv.lock regenerated against the merged pyproject (mistralai 2.4.4). Verified: make tb, make agent-check (ruff/pyright/mypy clean), pipelex validate --all, make agent-test (full suite), and the Temporal error-handling integration tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Greptile SummaryThis PR adds runtime bridge support and a Mistral-native execution path. The main changes are:
Confidence Score: 2/5These issues should be fixed before merging.
Focus on Important Files Changed
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1db5ecb987
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
24 issues found across 114 files
You’re at about 92% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name=".claude/skills/workflows/references/getting-started/value-proposition.mdx">
<violation number="1" location=".claude/skills/workflows/references/getting-started/value-proposition.mdx:111">
P2: Broken link: `../appendices/streaming` points to a non-existent `appendices/` directory. Readers will get a 404.</violation>
</file>
Note: This PR contains a large number of files. cubic only reviews up to 100 files per PR, so some files may not have been reviewed. cubic prioritizes the most important files to review.
On a pro plan you can use ultrareview for larger PRs.
Re-trigger cubic
…bridge crate path Address unresolved review-agent findings on PR #954: - _scoped_library_for_crate now uses hub.scoped_current_library() to capture and restore the caller's current-library ContextVar instead of unconditionally clearing it, so a bridge call made from within an already-scoped library no longer clobbers the caller's context. - Seed a per-call ClassRegistry pre-loaded from the global registry and attach it to the scoped library before load_from_crate, so classes generated from a crate's inline structured concepts no longer leak into or collide in the global Kajson registry across crate versions. Mirrors the Temporal worker hydration path in wf_pipe_router.py. - mistral_list_available_models raises MistralModelListingError when the type-filtered model list is empty, matching its two prior empty checks. Adds regression tests for library-context restore and global-registry isolation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… Mistral Workflows docs Triage of cubic-dev-ai review comments on PR #954 (verified against installed mistralai 2.4.4): - rate-limiting: import Mistral from mistralai.client, Depends from mistralai.workflows; drop non-existent Messages, type messages concretely - scheduling: add workflows import + entrypoint decorator; SchedulePolicy fields catchup_window_seconds/overlap (all occurrences) - activities: from datetime import timedelta (was datetime.timedelta) - streaming: inject llm via Depends so snippets don't NameError - python-sdk: add import os; fix "synchronous" mislabel - your-first-workflow: valid JSON input example - links: repoint payload-encoding/task-events to existing pages; remove dangling appendices/ links (streaming-architecture, worker-versioning, configuration-reference) across the references tree Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
What this brings to
devLands the Mistral Workflows integration and its supporting runtime refactors, and reconciles the latest
dev.Headline features
pipelex-mistralai-workflows); the framework-agnostic core (boundary types,run_pipe_via_bridge, execution modes,ensure_pipelex_booted) is promoted topipelex.runtime_bridge.*so any host runtime — not just Mistral Workflows — can embed Pipelex.runtime_bridge.primitivesextraction — activity logic (delivery, hydration, trace flush, graph assembly, submitter hydration, pipe-run-arg) lifted out ofpipelex.temporalso both the Temporal track and the Mistral-native track share one body.MISTRAL_NATIVEexecution mode with a contextvar-backedpipe_routeroverride on the hub, so a host runtime can turn sub-pipe calls into its own child workflows/activities.workflow_idderived frompipeline_run_id,TypedSearchAttributes, per-activity/per-handle task-queue routing, named worker-runtime profiles, and per-call summaries in the dashboard.dev reconciliation
Merges current
dev(error-handling overhaul, dataclass/pathlib cleanups, the v0.29–v0.30.3 releases). The shared Temporal/PipeStructure work dev already shipped in v0.29.0 is deduplicated, with this branch's net-new work layered on top. Woven points worth noting:runtime_bridgedelivery primitive (execute_deliverynow forwardserror_report/request_id); Temporal activities keep the@convert_pipelex_errorsboundary andrecover_error_reportrecovery.queue_options, default worker-runtime profile, frozenDispatchOptions).InferenceErrorCategory/UserActionerror-classification enrichment ported onto the 2.x calls.Verification
make tb,make agent-check(ruff / plxt / pyright / mypy all clean),pipelex validate --all,make agent-test(full suite), and the Temporal error-handling integration tests all pass.🤖 Generated with Claude Code
Summary by cubic
Extracts Mistral Workflows into
pipelex-mistralai-workflowsand introduces a framework‑agnosticpipelex.runtime_bridge, enabling DIRECT, Temporal, and Mistral‑native execution of Pipelex pipes with shared primitives and updated deps. Also fixes bridge scoping by preserving the caller’s library context and isolating per‑call class registries, and updates the vendored Mistral Workflows docs with corrected examples and links.New Features
pipelex.runtime_bridgewithrun_pipe_via_bridgeand execution modes:direct,temporal_blocking,temporal_fire_and_forget, andmistral_native.hydration,submitter_hydration,delivery,trace_flush,graph_assembly,pipe_classification); Temporal wrappers now thin adapters.pipe_routeroverride in the hub to route sub-pipes through a host runtime (used bymistral_native).ClassRegistrywhen loading crates to prevent global Kajson leaks; tests cover library-context restore and registry isolation.mistral_list_available_modelsnow raises when no models remain after type filtering.Migration
pipelex[mistralai-workflows]extra withpip install pipelex-mistralai-workflowsand import frompipelex_mistralai_workflows.*.mistralai>=2.4.4(new import pathmistralai.client.*) andtemporalio==1.24.0.>=3.12.Written for commit d5c90e3. Summary will update on new commits.