[FEAT][Rust] Support Rust stubgen.#603
Conversation
Refine stubgen tests and build setup refactor stubgen array usage suppress non-snake-case warnings Improve stubgen typing and Map support Format Rust sources Use toml serializer for Cargo.toml Add AnyValue for typed Any support Avoid AnyValue TryFrom overlap Extend typed macro arity Improve stubgen wrappers and output Add stubgen fields; first commit to build TileLang IR Refine stubgen layout and warnings fix(stubgen): update test to match _tvm_ffi_stubgen_detail layout The stubgen test was expecting functions.rs at src/functions.rs, but the generator actually writes it to src/_tvm_ffi_stubgen_detail/functions.rs. This commit updates the test to check the correct path. Also includes formatting fixes from cargo fmt. Co-authored-by: Cursor <cursoragent@cursor.com> feat(rust): implement repr(C) zero-cost stubgen with safe Deref-based subtyping 实现 repr(C) 零开销 Rust FFI wrapper 生成,使用 Deref trait 建模继承, 生成 100% safe 的用户代码。 - **subtyping.rs (新文件)**: 子类型转换辅助函数 - `upcast<From, To>`: 安全的消费型向上转换 - `try_downcast<From, To>`: 安全的消费型向下转换(使用 runtime type check) - `is_instance_of`: 继承关系运行时检查 - **macros.rs**: 新增 `impl_object_hierarchy!` 宏 - 自动生成 Deref/From/TryFrom impls 用于继承链 - 零开销引用 upcast(通过 Deref auto-coercion) - 安全的消费型转换(使用标准库 trait) - **object_wrapper.rs**: 移除 `is_instance_type`(移至 subtyping.rs) - **derive(ObjectRef)**: 修复为委托给 ObjectRef 的 AnyCompatible 实现, 消除对 `pub(crate) unsafe_::inc_ref` 的直接调用 - **derive(Object)**: 将 `proc_macro_error::abort!` 替换为 `panic!`, 避免在外部 crate 编译时失败 - **repr_c.rs (新文件)**: C 兼容性检查逻辑 - `check_repr_c`: 验证类型是否可用 repr(C) 表示 - 根据 field size 映射精确的 Rust POD 类型(i8/i16/i32/i64, f32/f64) - 改进 alignment padding 验证 - **generate.rs**: 全新 repr(C) 代码生成路径 - 生成 `#[repr(C)] *Obj` struct(零开销字段访问) - 生成 `#[repr(C)] *Ref` wrapper(user-facing 类型) - 调用 `impl_object_hierarchy!` 自动生成子类型转换 - 只为直接字段生成 getter(继承字段通过 Deref 自动可用) - 过滤内建类型(ffi.*)避免重复定义 - 对 repr(C) 类型使用 `self as &ObjectRef`(deref coercion) - 对 fallback 类型使用 `self.as_object_ref()` - **model.rs**: 扩展 TypeGen - 新增 `ancestor_chain` 字段存储继承链 - 修改 `call_expr` 使用 `Into<ObjectRef>` 而非 `as_object_ref().clone()` ✅ **100% Safe Rust**: 无任何 `unsafe` 关键字出现在生成的 stub 代码中 ✅ **零开销字段访问**: repr(C) 类型直接访问字段无 FFI 调用 ✅ **类型安全子类型转换**: 使用标准库 Deref/From/TryFrom trait ✅ **符合 Rust 惯用法**: 自动 upcast、fallible downcast 使用 tvm_ffi_testing.so 生成 stub crate,编译通过,无任何 unsafe 代码。 Closes: stubgen repr(C) redesign initiative Co-authored-by: Cursor <cursoragent@cursor.com> stubgen 第二轮修复:子类型转换、check_repr_c、fallback 与导出清理 - impl_object_hierarchy!: 补充 TryFrom<DirectParent> for Self,覆盖根类型 ObjectRef 下转 - subtyping: upcast/try_downcast 改为 #[doc(hidden)] pub,供宏展开使用,用户统一走 From/TryFrom - check_repr_c: 仅允许对齐 padding (field.offset == align_up(pos, alignment)),并补齐 Any 映射以支持 Map/Array<Any> - define_object_wrapper!: 增加 From<Wrapper> for ObjectRef,保证 .into() 兼容 - generate: is_builtin_type 增加 None;render_facade_module 跳过空模块,避免空 pub mod ffi Co-authored-by: Cursor <cursoragent@cursor.com> fix(stubgen): use type_ancestors[type_depth-1] for direct parent check_repr_c previously used ancestor[0] as parent, which for multi-level inheritance returns the root type (ffi.Object). This caused parent_total_size=24 while direct field offset=56, falsely rejected as layout gap. Use ancestor[type_depth-1] for direct parent so TestObjectDerived passes repr(C). Co-authored-by: Cursor <cursoragent@cursor.com> docs(stubgen): split user guide and harden runnable examples Move the Rust stubgen usage guide into docs/packaging so it is grouped with packaging workflows, and keep the crate README focused on generated interface/design details. Update the stubgen integration test to validate the documented usage flow with raw-string test source and stable APIs. Co-authored-by: Cursor <cursoragent@cursor.com> feat(stubgen): resolve object methods via type metadata Switch generated object method wrappers from global registry lookup to type reflection method resolution, and map constructor wrappers to Rust-style `new` while removing legacy `c_ffi_init`. Extend integration tests and docs to validate typed constructors and Cxx inheritance roundtrip behavior. Co-authored-by: Cursor <cursoragent@cursor.com> chore(stubgen): remove dead code and unused repr-c fields Drop unused helper and stale repr-c struct fields to keep the Rust workspace warning-free after the method lookup and constructor naming refactor. Co-authored-by: Cursor <cursoragent@cursor.com> [stubgen] support multi-prefix generation for single-crate multi-namespace output Allow --init-prefix to be specified multiple times. With a single prefix, behavior is unchanged (prefix stripped, items at crate root). With multiple prefixes, no stripping occurs and each prefix becomes a top-level module, enabling a single generated crate to cover multiple namespaces (e.g. tl, ir, tir, script). Also fixes cross-module repr(C) parent type resolution by using absolute paths into the types module, and adds `impl`/`mod` to the keyword escape list in sanitize_ident. Made-with: Cursor feat(stubgen): format generated crates by default Run cargo fmt on generated Rust stub crates by default and add a --no-format escape hatch for raw generator debugging. Also relax into_typed_fn! so rustfmt-produced typed signatures remain compilable. Made-with: Cursor feat(stubgen): gap-filling repr_c layout and relaxed parent resolution Rewrite check_repr_c to use a gap-filling strategy instead of strict layout validation. Byte ranges not covered by registered fields are emitted as `[u8; N]` padding members, handling C++ tail padding, vtable pointers, and unregistered fields uniformly. Key changes: - Replace strict contiguous-field checks with gap-filling layout builder - Parent types that are not in type_map or fail check_repr_c no longer cause the child to fall back; the parent region becomes a gap after the Object header - Fields whose type schema cannot be mapped to Rust are skipped and covered by gaps instead of failing the entire type - Handle Optional and ffi.Array with no type args (map to ObjectRef) - Remove alignment inference (infer_alignment_from_size, align_up) - Add log + env_logger for persistent debug/trace diagnostics (RUST_LOG=debug or RUST_LOG=trace) This reduces define_object_wrapper fallbacks from 160 to ~11 (93%) without any changes to the C++ TVM codebase. Made-with: Cursor style(stubgen): cargo fmt repr_c.rs Made-with: Cursor docs(stubgen): update README for gap-filling repr_c strategy Made-with: Cursor fix(stubgen): rename base-class field to __tvm_ffi_object_parent Avoids name collision when a C++ struct has a data field also named "parent". Update README examples and the inherited getter access path in build_getter_specs to match the new field name. docs(stubgen): add TODO section for known open issues - Ancestor chain is truncated when direct parent is not repr(C)-mappable: the chain collapses to [ObjectRef], losing intermediate mappable ancestors and breaking upcast/downcast through those types. - Common interface needed between fallback (define_object_wrapper!) and repr(C) paths to avoid source-compatibility breaks when stubgen version changes cause a type to move between generation strategies. Made-with: Cursor refactor(stubgen): generalize parent_range_fields to non_layout_fields Track ALL registered ObjectDef fields that cannot become direct repr(C) struct members in a unified non_layout_fields list: - Parent-range fields (offset < parent_total_size), with or without mappable schema. - Own-range fields whose schema is not mappable to a Rust type. Generate FieldGetter accessors for every entry: - Typed fields (mappable schema): get_xxx() -> T via FieldGetter::get(). - Untyped fields (unmappable schema): get_xxx() -> tvm_ffi::Any via get_any(), with tvm_ffi::Any as the FieldGetter<T> type parameter. Both variants use the same infallible get_* naming convention as direct struct-field getters, preserving interface compatibility across access paths. Made-with: Cursor docs(stubgen): document non_layout_fields and unified getter interface - Field Accessor Style: new section explaining the two accessor paths (direct struct field vs FieldGetter) with the unified get_* convention, return-type rules, and a debugging recipe. - repr(C) Decision Rules: expand soft-handling list to cover parent-range fields and unmappable-schema fields, describing the non_layout_fields mechanism and the emitted get_* accessors. - TODO: add "Parent-range field layout override" item (phase 2 work); update "Common interface" item to reflect current repr(C) state. Made-with: Cursor refactor(rust): upgrade to edition 2024, syn 2.x, fix clippy - Upgrade all Rust crates from edition 2021 to edition 2024 - Set rust-version = "1.85" (MSRV for edition 2024) - Add rust-toolchain.toml (stable + rustfmt/clippy components) - Add .rustfmt.toml with edition 2024 formatting rules - Migrate tvm-ffi-macros from syn 1.x to syn 2.x - Remove unmaintained proc-macro-error dependency - Fix all clippy warnings across tvm-ffi, tvm-ffi-sys, tvm-ffi-macros - Allow unsafe_op_in_unsafe_fn at crate level for incremental migration - Update stubgen template to emit edition 2024 + clippy allows Made-with: Cursor format. Signed-off-by: Linzhang Li <yuchuan.7streams@gmail.com> remove the lint check. Signed-off-by: Linzhang Li <yuchuan.7streams@gmail.com> remove the update. Signed-off-by: Linzhang Li <yuchuan.7streams@gmail.com> test. Signed-off-by: Linzhang Li <yuchuan.7streams@gmail.com> test. Signed-off-by: Linzhang Li <yuchuan.7streams@gmail.com> fix. Signed-off-by: Linzhang Li <yuchuan.7streams@gmail.com> fix test. Signed-off-by: Linzhang Li <yuchuan.7streams@gmail.com> rust output. Signed-off-by: Linzhang Li <yuchuan.7streams@gmail.com> locate the error. Signed-off-by: Linzhang Li <yuchuan.7streams@gmail.com> fix the loading. Signed-off-by: Linzhang Li <yuchuan.7streams@gmail.com> remove temparary fix. Signed-off-by: Linzhang Li <yuchuan.7streams@gmail.com> test. Signed-off-by: Linzhang Li <yuchuan.7streams@gmail.com> Revert "test." This reverts commit 4e5b74d. test. Signed-off-by: Linzhang Li <yuchuan.7streams@gmail.com> test. Signed-off-by: Linzhang Li <yuchuan.7streams@gmail.com> test. Signed-off-by: Linzhang Li <yuchuan.7streams@gmail.com> fix borro. Signed-off-by: Linzhang Li <yuchuan.7streams@gmail.com> revert the test. Signed-off-by: Linzhang Li <yuchuan.7streams@gmail.com> check. Signed-off-by: Linzhang Li <yuchuan.7streams@gmail.com> finish. Signed-off-by: Linzhang Li <yuchuan.7streams@gmail.com>
There was a problem hiding this comment.
Code Review
This pull request introduces a Rust stub generator (tvm-ffi-stubgen) that generates Rust stubs from TVM-FFI reflection metadata, alongside supporting changes in tvm-ffi and tvm-ffi-sys such as a new Map collection, subtyping casting utilities, and macro expansions. The review feedback identifies three critical issues in the code generator: a potential null pointer dereference when traversing type ancestors, a double Obj suffix bug in parent type formatting, and an invalid non-primitive cast in the generated method wrappers.
| unsafe { | ||
| for i in 1..info.type_depth { | ||
| let parent = *info.type_acenstors.add(i as usize); | ||
| if parent.is_null() { | ||
| continue; | ||
| } | ||
| let parent = &*parent; | ||
| if parent.fields.is_null() || parent.num_fields <= 0 { | ||
| continue; | ||
| } | ||
| let parent_fields = | ||
| std::slice::from_raw_parts(parent.fields, parent.num_fields as usize); | ||
| for field in parent_fields { | ||
| callback(field); | ||
| } | ||
| } |
There was a problem hiding this comment.
In for_each_field_info, info.type_acenstors is dereferenced via .add(i as usize) without checking if the pointer is null. If info.type_depth > 1 but info.type_acenstors is null, this will cause undefined behavior or a crash. A null check should be added before accessing info.type_acenstors.
unsafe {
if !info.type_acenstors.is_null() {
for i in 1..info.type_depth {
let parent = *info.type_acenstors.add(i as usize);
if parent.is_null() {
continue;
}
let parent = &*parent;
if parent.fields.is_null() || parent.num_fields <= 0 {
continue;
}
let parent_fields =
std::slice::from_raw_parts(parent.fields, parent.num_fields as usize);
for field in parent_fields {
callback(field);
}
}
}| let parent_rust = _type_map | ||
| .get(parent_key) | ||
| .cloned() | ||
| .unwrap_or_else(|| format!("{}Obj", sanitize_ident(parent_key, IdentStyle::Type))); |
There was a problem hiding this comment.
If parent_key is not found in _type_map, parent_rust is formatted with the Obj suffix appended. However, at line 961, format!("{}Obj", types_path) is called, which appends Obj again. This results in a double Obj suffix (e.g., TestingTestCxxClassBaseObjObj), causing a compilation error in the generated code. The fallback should return the base type name without the Obj suffix.
| let parent_rust = _type_map | |
| .get(parent_key) | |
| .cloned() | |
| .unwrap_or_else(|| format!("{}Obj", sanitize_ident(parent_key, IdentStyle::Type))); | |
| let parent_rust = _type_map | |
| .get(parent_key) | |
| .cloned() | |
| .unwrap_or_else(|| sanitize_ident(parent_key, IdentStyle::Type)); |
| if ty.repr_c_info.is_some() { | ||
| writeln!( | ||
| out, | ||
| "{} views.push(AnyView::from(self as &tvm_ffi::object::ObjectRef));", |
There was a problem hiding this comment.
The cast self as &tvm_ffi::object::ObjectRef is an invalid non-primitive cast between different struct references in Rust and will cause a compilation error in the generated code. Since Self implements AnyCompatible (via the derived AnyCompatible trait), AnyView::from(self) is completely valid and does not require any cast.
| "{} views.push(AnyView::from(self as &tvm_ffi::object::ObjectRef));", | |
| "{} views.push(AnyView::from(self));", |
This PR is based on #434 to support stubgen in Rust. This PR fixed issues such as the module lifecycle issue on Windows, lint problem, PATH resolution, and so on.
TODO List: