Skip to content

[FEAT][Rust] Support Rust stubgen.#603

Draft
Seven-Streams wants to merge 1 commit into
apache:mainfrom
Seven-Streams:main-dev/2026-05-31/rust_stubgen
Draft

[FEAT][Rust] Support Rust stubgen.#603
Seven-Streams wants to merge 1 commit into
apache:mainfrom
Seven-Streams:main-dev/2026-05-31/rust_stubgen

Conversation

@Seven-Streams
Copy link
Copy Markdown

This PR is based on #434 to support stubgen in Rust. This PR fixed issues such as the module lifecycle issue on Windows, lint problem, PATH resolution, and so on.

TODO List:

  • further tests, including e2e and modules.
  • further human reviews.
  • document alignment.

Refine stubgen tests and build setup

refactor stubgen array usage

suppress non-snake-case warnings

Improve stubgen typing and Map support

Format Rust sources

Use toml serializer for Cargo.toml

Add AnyValue for typed Any support

Avoid AnyValue TryFrom overlap

Extend typed macro arity

Improve stubgen wrappers and output

Add stubgen fields; first commit to build TileLang IR

Refine stubgen layout and warnings

fix(stubgen): update test to match _tvm_ffi_stubgen_detail layout

The stubgen test was expecting functions.rs at src/functions.rs, but the
generator actually writes it to src/_tvm_ffi_stubgen_detail/functions.rs.
This commit updates the test to check the correct path.

Also includes formatting fixes from cargo fmt.

Co-authored-by: Cursor <cursoragent@cursor.com>

feat(rust): implement repr(C) zero-cost stubgen with safe Deref-based subtyping

实现 repr(C) 零开销 Rust FFI wrapper 生成,使用 Deref trait 建模继承,
生成 100% safe 的用户代码。

- **subtyping.rs (新文件)**: 子类型转换辅助函数
  - `upcast<From, To>`: 安全的消费型向上转换
  - `try_downcast<From, To>`: 安全的消费型向下转换(使用 runtime type check)
  - `is_instance_of`: 继承关系运行时检查

- **macros.rs**: 新增 `impl_object_hierarchy!` 宏
  - 自动生成 Deref/From/TryFrom impls 用于继承链
  - 零开销引用 upcast(通过 Deref auto-coercion)
  - 安全的消费型转换(使用标准库 trait)

- **object_wrapper.rs**: 移除 `is_instance_type`(移至 subtyping.rs)

- **derive(ObjectRef)**: 修复为委托给 ObjectRef 的 AnyCompatible 实现,
  消除对 `pub(crate) unsafe_::inc_ref` 的直接调用
- **derive(Object)**: 将 `proc_macro_error::abort!` 替换为 `panic!`,
  避免在外部 crate 编译时失败

- **repr_c.rs (新文件)**: C 兼容性检查逻辑
  - `check_repr_c`: 验证类型是否可用 repr(C) 表示
  - 根据 field size 映射精确的 Rust POD 类型(i8/i16/i32/i64, f32/f64)
  - 改进 alignment padding 验证

- **generate.rs**: 全新 repr(C) 代码生成路径
  - 生成 `#[repr(C)] *Obj` struct(零开销字段访问)
  - 生成 `#[repr(C)] *Ref` wrapper(user-facing 类型)
  - 调用 `impl_object_hierarchy!` 自动生成子类型转换
  - 只为直接字段生成 getter(继承字段通过 Deref 自动可用)
  - 过滤内建类型(ffi.*)避免重复定义
  - 对 repr(C) 类型使用 `self as &ObjectRef`(deref coercion)
  - 对 fallback 类型使用 `self.as_object_ref()`

- **model.rs**: 扩展 TypeGen
  - 新增 `ancestor_chain` 字段存储继承链
  - 修改 `call_expr` 使用 `Into<ObjectRef>` 而非 `as_object_ref().clone()`

✅ **100% Safe Rust**: 无任何 `unsafe` 关键字出现在生成的 stub 代码中
✅ **零开销字段访问**: repr(C) 类型直接访问字段无 FFI 调用
✅ **类型安全子类型转换**: 使用标准库 Deref/From/TryFrom trait
✅ **符合 Rust 惯用法**: 自动 upcast、fallible downcast

使用 tvm_ffi_testing.so 生成 stub crate,编译通过,无任何 unsafe 代码。

Closes: stubgen repr(C) redesign initiative
Co-authored-by: Cursor <cursoragent@cursor.com>

stubgen 第二轮修复:子类型转换、check_repr_c、fallback 与导出清理

- impl_object_hierarchy!: 补充 TryFrom<DirectParent> for Self,覆盖根类型 ObjectRef 下转
- subtyping: upcast/try_downcast 改为 #[doc(hidden)] pub,供宏展开使用,用户统一走 From/TryFrom
- check_repr_c: 仅允许对齐 padding (field.offset == align_up(pos, alignment)),并补齐 Any 映射以支持 Map/Array<Any>
- define_object_wrapper!: 增加 From<Wrapper> for ObjectRef,保证 .into() 兼容
- generate: is_builtin_type 增加 None;render_facade_module 跳过空模块,避免空 pub mod ffi

Co-authored-by: Cursor <cursoragent@cursor.com>

fix(stubgen): use type_ancestors[type_depth-1] for direct parent

check_repr_c previously used ancestor[0] as parent, which for multi-level
inheritance returns the root type (ffi.Object). This caused parent_total_size=24
while direct field offset=56, falsely rejected as layout gap.
Use ancestor[type_depth-1] for direct parent so TestObjectDerived passes repr(C).

Co-authored-by: Cursor <cursoragent@cursor.com>

docs(stubgen): split user guide and harden runnable examples

Move the Rust stubgen usage guide into docs/packaging so it is grouped with packaging workflows, and keep the crate README focused on generated interface/design details. Update the stubgen integration test to validate the documented usage flow with raw-string test source and stable APIs.

Co-authored-by: Cursor <cursoragent@cursor.com>

feat(stubgen): resolve object methods via type metadata

Switch generated object method wrappers from global registry lookup to type reflection method resolution, and map constructor wrappers to Rust-style `new` while removing legacy `c_ffi_init`. Extend integration tests and docs to validate typed constructors and Cxx inheritance roundtrip behavior.

Co-authored-by: Cursor <cursoragent@cursor.com>

chore(stubgen): remove dead code and unused repr-c fields

Drop unused helper and stale repr-c struct fields to keep the Rust workspace warning-free after the method lookup and constructor naming refactor.

Co-authored-by: Cursor <cursoragent@cursor.com>

[stubgen] support multi-prefix generation for single-crate multi-namespace output

Allow --init-prefix to be specified multiple times. With a single prefix,
behavior is unchanged (prefix stripped, items at crate root). With multiple
prefixes, no stripping occurs and each prefix becomes a top-level module,
enabling a single generated crate to cover multiple namespaces (e.g. tl, ir,
tir, script).

Also fixes cross-module repr(C) parent type resolution by using absolute
paths into the types module, and adds `impl`/`mod` to the keyword escape
list in sanitize_ident.

Made-with: Cursor

feat(stubgen): format generated crates by default

Run cargo fmt on generated Rust stub crates by default and add a --no-format escape hatch for raw generator debugging. Also relax into_typed_fn! so rustfmt-produced typed signatures remain compilable.

Made-with: Cursor

feat(stubgen): gap-filling repr_c layout and relaxed parent resolution

Rewrite check_repr_c to use a gap-filling strategy instead of strict
layout validation.  Byte ranges not covered by registered fields are
emitted as `[u8; N]` padding members, handling C++ tail padding, vtable
pointers, and unregistered fields uniformly.

Key changes:
- Replace strict contiguous-field checks with gap-filling layout builder
- Parent types that are not in type_map or fail check_repr_c no longer
  cause the child to fall back; the parent region becomes a gap after
  the Object header
- Fields whose type schema cannot be mapped to Rust are skipped and
  covered by gaps instead of failing the entire type
- Handle Optional and ffi.Array with no type args (map to ObjectRef)
- Remove alignment inference (infer_alignment_from_size, align_up)
- Add log + env_logger for persistent debug/trace diagnostics
  (RUST_LOG=debug or RUST_LOG=trace)

This reduces define_object_wrapper fallbacks from 160 to ~11 (93%)
without any changes to the C++ TVM codebase.

Made-with: Cursor

style(stubgen): cargo fmt repr_c.rs

Made-with: Cursor

docs(stubgen): update README for gap-filling repr_c strategy

Made-with: Cursor

fix(stubgen): rename base-class field to __tvm_ffi_object_parent

Avoids name collision when a C++ struct has a data field also named
"parent".  Update README examples and the inherited getter access path
in build_getter_specs to match the new field name.

docs(stubgen): add TODO section for known open issues

- Ancestor chain is truncated when direct parent is not repr(C)-mappable:
  the chain collapses to [ObjectRef], losing intermediate mappable
  ancestors and breaking upcast/downcast through those types.
- Common interface needed between fallback (define_object_wrapper!) and
  repr(C) paths to avoid source-compatibility breaks when stubgen version
  changes cause a type to move between generation strategies.

Made-with: Cursor

refactor(stubgen): generalize parent_range_fields to non_layout_fields

Track ALL registered ObjectDef fields that cannot become direct repr(C)
struct members in a unified non_layout_fields list:

- Parent-range fields (offset < parent_total_size), with or without
  mappable schema.
- Own-range fields whose schema is not mappable to a Rust type.

Generate FieldGetter accessors for every entry:
- Typed fields (mappable schema): get_xxx() -> T via FieldGetter::get().
- Untyped fields (unmappable schema): get_xxx() -> tvm_ffi::Any via
  get_any(), with tvm_ffi::Any as the FieldGetter<T> type parameter.

Both variants use the same infallible get_* naming convention as direct
struct-field getters, preserving interface compatibility across access paths.

Made-with: Cursor

docs(stubgen): document non_layout_fields and unified getter interface

- Field Accessor Style: new section explaining the two accessor paths
  (direct struct field vs FieldGetter) with the unified get_* convention,
  return-type rules, and a debugging recipe.
- repr(C) Decision Rules: expand soft-handling list to cover parent-range
  fields and unmappable-schema fields, describing the non_layout_fields
  mechanism and the emitted get_* accessors.
- TODO: add "Parent-range field layout override" item (phase 2 work);
  update "Common interface" item to reflect current repr(C) state.

Made-with: Cursor

refactor(rust): upgrade to edition 2024, syn 2.x, fix clippy

- Upgrade all Rust crates from edition 2021 to edition 2024
- Set rust-version = "1.85" (MSRV for edition 2024)
- Add rust-toolchain.toml (stable + rustfmt/clippy components)
- Add .rustfmt.toml with edition 2024 formatting rules
- Migrate tvm-ffi-macros from syn 1.x to syn 2.x
- Remove unmaintained proc-macro-error dependency
- Fix all clippy warnings across tvm-ffi, tvm-ffi-sys, tvm-ffi-macros
- Allow unsafe_op_in_unsafe_fn at crate level for incremental migration
- Update stubgen template to emit edition 2024 + clippy allows

Made-with: Cursor

format.

Signed-off-by: Linzhang Li <yuchuan.7streams@gmail.com>

remove the lint check.

Signed-off-by: Linzhang Li <yuchuan.7streams@gmail.com>

remove the update.

Signed-off-by: Linzhang Li <yuchuan.7streams@gmail.com>

test.

Signed-off-by: Linzhang Li <yuchuan.7streams@gmail.com>

test.

Signed-off-by: Linzhang Li <yuchuan.7streams@gmail.com>

fix.

Signed-off-by: Linzhang Li <yuchuan.7streams@gmail.com>

fix test.

Signed-off-by: Linzhang Li <yuchuan.7streams@gmail.com>

rust output.

Signed-off-by: Linzhang Li <yuchuan.7streams@gmail.com>

locate the error.

Signed-off-by: Linzhang Li <yuchuan.7streams@gmail.com>

fix the loading.

Signed-off-by: Linzhang Li <yuchuan.7streams@gmail.com>

remove temparary fix.

Signed-off-by: Linzhang Li <yuchuan.7streams@gmail.com>

test.

Signed-off-by: Linzhang Li <yuchuan.7streams@gmail.com>

Revert "test."

This reverts commit 4e5b74d.

test.

Signed-off-by: Linzhang Li <yuchuan.7streams@gmail.com>

test.

Signed-off-by: Linzhang Li <yuchuan.7streams@gmail.com>

test.

Signed-off-by: Linzhang Li <yuchuan.7streams@gmail.com>

fix borro.

Signed-off-by: Linzhang Li <yuchuan.7streams@gmail.com>

revert the test.

Signed-off-by: Linzhang Li <yuchuan.7streams@gmail.com>

check.

Signed-off-by: Linzhang Li <yuchuan.7streams@gmail.com>

finish.

Signed-off-by: Linzhang Li <yuchuan.7streams@gmail.com>
@Seven-Streams Seven-Streams changed the title Add Rust stubgen and integration tests [FEAT][Rust] Add Rust stubgen and integration tests Jun 1, 2026
@Seven-Streams Seven-Streams changed the title [FEAT][Rust] Add Rust stubgen and integration tests [FEAT][Rust] Support Rust stubgen. Jun 1, 2026
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a Rust stub generator (tvm-ffi-stubgen) that generates Rust stubs from TVM-FFI reflection metadata, alongside supporting changes in tvm-ffi and tvm-ffi-sys such as a new Map collection, subtyping casting utilities, and macro expansions. The review feedback identifies three critical issues in the code generator: a potential null pointer dereference when traversing type ancestors, a double Obj suffix bug in parent type formatting, and an invalid non-primitive cast in the generated method wrappers.

Comment on lines +241 to +256
unsafe {
for i in 1..info.type_depth {
let parent = *info.type_acenstors.add(i as usize);
if parent.is_null() {
continue;
}
let parent = &*parent;
if parent.fields.is_null() || parent.num_fields <= 0 {
continue;
}
let parent_fields =
std::slice::from_raw_parts(parent.fields, parent.num_fields as usize);
for field in parent_fields {
callback(field);
}
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

In for_each_field_info, info.type_acenstors is dereferenced via .add(i as usize) without checking if the pointer is null. If info.type_depth > 1 but info.type_acenstors is null, this will cause undefined behavior or a crash. A null check should be added before accessing info.type_acenstors.

    unsafe {
        if !info.type_acenstors.is_null() {
            for i in 1..info.type_depth {
                let parent = *info.type_acenstors.add(i as usize);
                if parent.is_null() {
                    continue;
                }
                let parent = &*parent;
                if parent.fields.is_null() || parent.num_fields <= 0 {
                    continue;
                }
                let parent_fields = 
                    std::slice::from_raw_parts(parent.fields, parent.num_fields as usize);
                for field in parent_fields {
                    callback(field);
                }
            }
        }

Comment on lines +954 to +957
let parent_rust = _type_map
.get(parent_key)
.cloned()
.unwrap_or_else(|| format!("{}Obj", sanitize_ident(parent_key, IdentStyle::Type)));
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

If parent_key is not found in _type_map, parent_rust is formatted with the Obj suffix appended. However, at line 961, format!("{}Obj", types_path) is called, which appends Obj again. This results in a double Obj suffix (e.g., TestingTestCxxClassBaseObjObj), causing a compilation error in the generated code. The fallback should return the base type name without the Obj suffix.

Suggested change
let parent_rust = _type_map
.get(parent_key)
.cloned()
.unwrap_or_else(|| format!("{}Obj", sanitize_ident(parent_key, IdentStyle::Type)));
let parent_rust = _type_map
.get(parent_key)
.cloned()
.unwrap_or_else(|| sanitize_ident(parent_key, IdentStyle::Type));

if ty.repr_c_info.is_some() {
writeln!(
out,
"{} views.push(AnyView::from(self as &tvm_ffi::object::ObjectRef));",
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The cast self as &tvm_ffi::object::ObjectRef is an invalid non-primitive cast between different struct references in Rust and will cause a compilation error in the generated code. Since Self implements AnyCompatible (via the derived AnyCompatible trait), AnyView::from(self) is completely valid and does not require any cast.

Suggested change
"{} views.push(AnyView::from(self as &tvm_ffi::object::ObjectRef));",
"{} views.push(AnyView::from(self));",

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants