feat(python/sedonadb): add DataFrame.filter / DataFrame.where

[jiayuasu]

Wire the Expr layer into the existing lazy DataFrame so users can filter rows without writing SQL predicates.

This is the fourth and final small PR completing Phase P1 of #791, building on #807 (Expr foundation), #823 (operators), and #832 (DataFrame.select).

What's new

from sedonadb.expr import col

df.filter(col("x") > 0)
df.filter(col("x") > 0, col("y") < 10)              # multiple preds → AND
df.filter((col("x") > 0) & col("y").is_not_null())  # explicit & also fine
df.filter(col("x") > 1).filter(col("x") < 4)        # chained: two filter nodes

Multiple predicates are AND-combined into a single composed Expr via DataFusion's conjunction helper, giving the optimizer one conjunction to reason about rather than N stacked filter nodes. Chained .filter().filter() calls produce two filter nodes in the plan but the same result.

Decision worth flagging — rejecting Literal 🛎️ cc @paleolimbot

filter() rejects both str and Literal arguments at the Python boundary.

• Strings are not interpreted as SQL predicates — that's a separate query()-style feature, not this PR's surface.
• Literal is rejected because filter(lit(True)) (or any filter(lit(value))) is almost always a typo: DataFusion would silently accept WHERE 7 as truthy, producing a filter that looks present in the source but is actually a no-op. If a user genuinely wants a constant predicate they can write col("flag") == lit(True), which forces the intent.

This is more restrictive than select, which does accept Literal. The asymmetry is intentional: a literal as a projection is meaningful (a constant column), but a literal as a filter is almost never what you want. Happy to revisit if you'd rather have filter(lit(...)) accepted and the resulting no-op surface as an optimizer-time warning instead.

Implementation

Rust: InternalDataFrame::filter(Vec<PyExpr>) — empty list errors at the Python boundary; multi-element lists are folded via datafusion_expr::utils::conjunction, matching the pattern used elsewhere in the workspace (sedona-pointcloud, sedona-query-planner). Step-by-step comments explain the conjunction choice.

Python: DataFrame.filter(*exprs: "Expr") -> DataFrame with explicit isinstance checks rejecting str, Literal, and other types.

Test plan

12 tests in tests/expr/test_dataframe_filter.py:

• Positive: simple predicate, multi-AND, explicit &, |, ~, isin, chained filter().filter() matches multi-AND.
• Lazy: filter returns a DataFrame without materializing.
• Errors: empty filter → ValueError; string arg → TypeError; Literal arg → TypeError with the actionable suggestion; unknown column → SedonaError whose message includes the valid field names.

Each test builds its own input df inline from the smallest pd.DataFrame({...}) that exercises the predicate. Existing select/expression/literal tests all still pass. ruff format and doctests both clean.

What's next

This completes the issue-tracker Phase P1 (Expr layer + select/filter on DataFrame). Phase P2 in the design doc covers Series / Scalar, groupby, merge, the curated sd.st module, and the cookbook track.

[Copilot]

Pull request overview

Adds pandas-style row filtering to the Python sedonadb.DataFrame API by wiring the existing Expr layer into the lazy DataFusion-backed DataFrame, including a .where() alias.

Changes:

• Add DataFrame.filter(*exprs: Expr) and DataFrame.where (alias), with Python-side type checks and clear error paths (empty args, string args, Literal args).
• Add Rust InternalDataFrame::filter(Vec<PyExpr>) that AND-combines multiple predicates into a single DataFusion filter expression.
• Add a dedicated pytest module covering positive behavior, laziness, and key error contracts.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
python/sedonadb/python/sedonadb/dataframe.py	Adds DataFrame.filter() implementation + where alias with Python-boundary validation and docs.
python/sedonadb/src/dataframe.rs	Exposes InternalDataFrame.filter() and folds multi-predicate calls via Expr::and before calling DataFusion.
python/sedonadb/tests/expr/test_dataframe_filter.py	Introduces unit tests for filter/where behavior, composition, laziness, and error messages.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[paleolimbot]

Can we not add an alias for filter here? We can specifically add a pyspark compatibility later at a different time if there is interest.

Suggested change

	where = filter

[jiayuasu]

Done in 92afb9b — where = filter is gone and test_where_alias_produces_same_output along with it. PR description updated to drop the where mentions.

[paleolimbot]

There's a helper for this in DataFusion (I think it's called conjunction!(); there may be one for both physical and logical expressions and you'll need the logical one).

[jiayuasu]

Switched in 92afb9b — now using datafusion_expr::utils::conjunction, matching the pattern in sedona-pointcloud and sedona-query-planner. unwrap/expect is safe because the empty case is already rejected at the Python boundary.

[paleolimbot]

Can you inline this into each test? Probably pd.DataFrame({"x": [1, 2, 3]}) would be sufficient.

[jiayuasu]

Done in 92afb9b — _xy_df helper removed; each test builds its own df inline from the smallest pd.DataFrame({...}) sufficient for the assertion. Most tests use {"x": [1, 2, 3]}; tests that need a second column use {"x": ..., "y": ...}.