Commit 20b3338
docs: user guide page + runnable examples for distributing expressions
Wraps up the Expr-pickle work with the user-facing material:
* docs/source/user-guide/io/distributing_work.rst — new user guide
page covering the multiprocessing, Ray, and datafusion-distributed
patterns. Includes the Security section that is the canonical home
for the cloudpickle / pickle.loads threat model.
* docs/source/user-guide/io/index.rst — toctree entry.
* examples/multiprocessing_pickle_expr.py — runnable example: a
Pool.map of a closure-capturing UDF across processes, with worker
context registration in the initializer.
* examples/ray_pickle_expr.py — Ray actor analogue.
* examples/datafusion-ffi-example/python/tests/_test_pickle_strict_ffi.py
— exercises the strict-mode refusal end to end against an FFI
capsule scalar UDF (kept under the FFI example crate because the
test needs that crate's compiled artifacts).
* examples/README.md — index entries for the new files.
Also tightens three docstrings that previously duplicated the
security warning so they point at the canonical Security section
instead:
* PythonLogicalCodec::with_python_udf_inlining (rustdoc): one-line
summary plus a relative pointer to distributing_work.rst and the
upstream Python pickle module security warning.
* SessionContext.with_python_udf_inlining: one-sentence summary plus
:doc: link to the user guide.
* datafusion.ipc module docstring: cross-reference to the user guide
for the full pattern.
The crate-level codec.rs module rustdoc also updates "pure-Python
scalar UDFs" to "scalar / aggregate / window UDFs" now that all three
are covered.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>1 parent d680e12 commit 20b3338
7 files changed
Lines changed: 763 additions & 13 deletions
File tree
- crates/core/src
- docs/source/user-guide/io
- examples
- datafusion-ffi-example/python/tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
22 | | - | |
23 | | - | |
24 | | - | |
25 | | - | |
26 | | - | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
| |||
217 | 217 | | |
218 | 218 | | |
219 | 219 | | |
220 | | - | |
221 | | - | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
222 | 224 | | |
223 | 225 | | |
224 | 226 | | |
| |||
369 | 371 | | |
370 | 372 | | |
371 | 373 | | |
372 | | - | |
| 374 | + | |
373 | 375 | | |
374 | 376 | | |
375 | 377 | | |
| |||
496 | 498 | | |
497 | 499 | | |
498 | 500 | | |
499 | | - | |
500 | | - | |
501 | | - | |
502 | | - | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
503 | 505 | | |
504 | 506 | | |
505 | 507 | | |
| |||
698 | 700 | | |
699 | 701 | | |
700 | 702 | | |
701 | | - | |
| 703 | + | |
702 | 704 | | |
703 | 705 | | |
704 | 706 | | |
| |||
0 commit comments