Add FFI type coverage and implementation pattern to check-upstream skill

Document the full FFI type pipeline (Rust PyO3 wrapper → Protocol type →
Python wrapper → ABC base class → exports → example) and catalog which
upstream datafusion-ffi types are supported, which have been evaluated as
not needing direct exposure, and how to check for new gaps.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: timsaucer

.claude/skills/check-upstream/SKILL.md

@@ -196,6 +196,150 @@ def new_method(self, param: str) -> DataFrame:
     return DataFrame(self.ctx.new_method(param))
 ```
 
+### Adding a New FFI Type
+
+FFI types require a full pipeline from C struct through to a typed Python wrapper. Each layer must be present.
+
+**Step 1: Rust PyO3 wrapper class** in a new or existing file under `crates/core/src/`:
+```rust
+use datafusion_ffi::new_type::FFI_NewType;
+
+#[pyclass(from_py_object, frozen, name = "RawNewType", module = "datafusion.module_name", subclass)]
+pub struct PyNewType {
+    pub inner: Arc<dyn NewTypeTrait>,
+}
+
+#[pymethods]
+impl PyNewType {
+    #[staticmethod]
+    fn from_pycapsule(obj: &Bound<'_, PyAny>) -> PyDataFusionResult<Self> {
+        let capsule = obj
+            .getattr("__datafusion_new_type__")?
+            .call0()?
+            .downcast::<PyCapsule>()?;
+        let ffi_ptr = unsafe { capsule.reference::<FFI_NewType>() };
+        let provider: Arc<dyn NewTypeTrait> = ffi_ptr.into();
+        Ok(Self { inner: provider })
+    }
+
+    fn some_method(&self) -> PyResult<...> {
+        // wrap inner trait method
+    }
+}
+```
+Register in the appropriate `init_module()`:
+```rust
+m.add_class::<PyNewType>()?;
+```
+
+**Step 2: Python Protocol type** in the appropriate Python module (e.g., `python/datafusion/catalog.py`):
+```python
+class NewTypeExportable(Protocol):
+    """Type hint for objects providing a __datafusion_new_type__ PyCapsule."""
+
+    def __datafusion_new_type__(self) -> object: ...
+```
+
+**Step 3: Python wrapper class** in the same module:
+```python
+class NewType:
+    """Description of the type.
+
+    This class wraps a DataFusion NewType, which can be created from a native
+    Python implementation or imported from an FFI-compatible library.
+    """
+
+    def __init__(
+        self,
+        new_type: df_internal.module_name.RawNewType | NewTypeExportable,
+    ) -> None:
+        if isinstance(new_type, df_internal.module_name.RawNewType):
+            self._raw = new_type
+        else:
+            self._raw = df_internal.module_name.RawNewType.from_pycapsule(new_type)
+
+    def some_method(self) -> ReturnType:
+        """Description of the method."""
+        return self._raw.some_method()
+```
+
+**Step 4: ABC base class** (if users should be able to subclass and provide custom implementations in Python):
+```python
+from abc import ABC, abstractmethod
+
+class NewTypeProvider(ABC):
+    """Abstract base class for implementing a custom NewType in Python."""
+
+    @abstractmethod
+    def some_method(self) -> ReturnType:
+        """Description of the method."""
+        ...
+```
+
+**Step 5: Module exports** — add to the appropriate `__init__.py`:
+- Add the wrapper class (`NewType`) to `python/datafusion/__init__.py`
+- Add the ABC (`NewTypeProvider`) if applicable
+- Add the Protocol type (`NewTypeExportable`) if it should be public
+
+**Step 6: FFI example** — add an example implementation under `examples/datafusion-ffi-example/src/`:
+```rust
+// examples/datafusion-ffi-example/src/new_type.rs
+use datafusion_ffi::new_type::FFI_NewType;
+// ... example showing how an external Rust library exposes this type via PyCapsule
+```
+
+**Checklist for each FFI type:**
+- [ ] Rust PyO3 wrapper with `from_pycapsule()` method
+- [ ] Python Protocol type (e.g., `NewTypeExportable`) for FFI objects
+- [ ] Python wrapper class with full type hints on all public methods
+- [ ] ABC base class (if the type can be user-implemented)
+- [ ] Registered in Rust `init_module()` and Python `__init__.py`
+- [ ] FFI example in `examples/datafusion-ffi-example/`
+- [ ] Type appears in union type hints where accepted (e.g., `Table | TableProviderExportable`)
+
+### 7. FFI Types (datafusion-ffi)
+
+**Upstream source of truth:**
+- Crate source: https://github.com/apache/datafusion/tree/main/datafusion/ffi/src
+- Rust docs: https://docs.rs/datafusion-ffi/latest/datafusion_ffi/
+
+**Where they are exposed in this project:**
+- Rust bindings: various files under `crates/core/src/` and `crates/util/src/`
+- FFI example: `examples/datafusion-ffi-example/src/`
+- Dependency declared in root `Cargo.toml` and `crates/core/Cargo.toml`
+
+**Currently supported FFI types:**
+- `FFI_ScalarUDF` — `crates/core/src/udf.rs`
+- `FFI_AggregateUDF` — `crates/core/src/udaf.rs`
+- `FFI_WindowUDF` — `crates/core/src/udwf.rs`
+- `FFI_TableFunction` — `crates/core/src/udtf.rs`
+- `FFI_TableProvider` — `crates/core/src/table.rs`, `crates/util/src/lib.rs`
+- `FFI_TableProviderFactory` — `crates/core/src/context.rs`
+- `FFI_CatalogProvider` — `crates/core/src/catalog.rs`, `crates/core/src/context.rs`
+- `FFI_CatalogProviderList` — `crates/core/src/context.rs`
+- `FFI_SchemaProvider` — `crates/core/src/catalog.rs`
+- `FFI_LogicalExtensionCodec` — multiple files
+- `FFI_ExtensionOptions` — `crates/core/src/context.rs`
+- `FFI_TaskContextProvider` — `crates/core/src/context.rs`
+
+**Evaluated and not requiring direct Python exposure:**
+These upstream FFI types have been reviewed and do not need to be independently exposed to end users:
+- `FFI_ExecutionPlan` — already used indirectly through table providers; no need for direct exposure
+- `FFI_PhysicalExpr` / `FFI_PhysicalSortExpr` — internal physical planning types not expected to be needed by end users
+- `FFI_RecordBatchStream` — one level deeper than FFI_ExecutionPlan, used internally when execution plans stream results
+- `FFI_SessionRef` / `ForeignSession` — session sharing across FFI; Python manages sessions natively via SessionContext
+- `FFI_SessionConfig` — Python can configure sessions natively without FFI
+- `FFI_ConfigOptions` / `FFI_TableOptions` — internal configuration plumbing
+- `FFI_PlanProperties` / `FFI_Boundedness` / `FFI_EmissionType` — read from existing plans, not user-facing
+- `FFI_Partitioning` — supporting type for physical planning
+- Supporting/utility types (`FFI_Option`, `FFI_Result`, `WrappedSchema`, `WrappedArray`, `FFI_ColumnarValue`, `FFI_Volatility`, `FFI_InsertOp`, `FFI_AccumulatorArgs`, `FFI_Accumulator`, `FFI_GroupsAccumulator`, `FFI_EmitTo`, `FFI_AggregateOrderSensitivity`, `FFI_PartitionEvaluator`, `FFI_PartitionEvaluatorArgs`, `FFI_Range`, `FFI_SortOptions`, `FFI_Distribution`, `FFI_ExprProperties`, `FFI_SortProperties`, `FFI_Interval`, `FFI_TableProviderFilterPushDown`, `FFI_TableType`) — used as building blocks within the types above, not independently exposed
+
+**How to check:**
+1. Compare the upstream `datafusion-ffi` crate's `lib.rs` exports against the lists above
+2. If new FFI types appear upstream, evaluate whether they represent a user-facing capability
+3. Check against the "evaluated and not requiring exposure" list before flagging as a gap
+4. Report any genuinely new types that enable user-facing functionality
+
 ## Important Notes
 
 - The upstream DataFusion version used by this project is specified in `crates/core/Cargo.toml` — check the `datafusion` dependency version to ensure you're comparing against the right upstream version.