PGO errors on native with symbols generated by `wit_bindgen::generate!`

[tliron]

While trying to create a PGO build of a complex program that uses wasmtime as well as wasmtime::component::bindgen I encountered a linker error during the PGO optimization phase:

 "/home/xxx/.cargo/target/x86_64-unknown-linux-gnu/release/deps/{libregex-c07ab95555f39da0,libregex_automata-7ee923d5f6c2717f,libaho_corasick-5c8f4fd45dbfd81d,libmemchr-6bf3bd84cc706660,libregex_syntax-29ae9a4cd46cd534,libpuccini_plugin_sdk_tosca_2_0-e5bae6e4238f19b8,libchrono-cdb2d612e7931659,libiana_time_zone-e1ddac5c8d3f6e30,libbase64-bc218090048adb9e,libfloria_plugin_sdk-2d3041ed2aa76b6f,libwit_bindgen-d74a0375665663a5,libwaki-0efdb38898b964aa,libform_urlencoded-b370f73aedd7d6ac,libpercent_encoding-3f263064c2841206,libwit_bindgen-fb809008eb3c2160,libwit_bindgen_rt-0a4c48ecc7933c57,libbitflags-50780dc367a8909d,libhttp-25d9a9cb8324f3c8,libitoa-6dd9b0b770f89707,libbytes-fe6d9b7a8a9571c5,libanyhow-8b765c7e03803d5b,libordered_float-1bbdc6eb007024dd,libserde-6a94ee0c090e296d,libserde_core-67855228a5a4e304,libnum_traits-24fe799717f9791d}.rlib" "<sysroot>/lib/rustlib/x86_64-unknown-linux-gnu/lib/{libstd-*,libpanic_unwind-*,libobject-*,libmemchr-*,libaddr2line-*,libgimli-*,libcfg_if-*,librustc_demangle-*,libstd_detect-*,libhashbrown-*,librustc_std_workspace_alloc-*,libminiz_oxide-*,libadler2-*,libunwind-*,liblibc-*,librustc_std_workspace_core-*,liballoc-*,libcore-*,libcompiler_builtins-*}.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-L" "/tmp/rustctoZu3J/raw-dylibs" "-B<sysroot>/lib/rustlib/x86_64-unknown-linux-gnu/bin/gcc-ld" "-fuse-ld=lld" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "<sysroot>/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "/home/xxx/.cargo/target/x86_64-unknown-linux-gnu/release/deps/libpuccini_plugin_tosca_2_0.so" "-Wl,--gc-sections" "-shared" "-Wl,-z,relro,-z,now" "-Wl,-O1" "-Wl,--strip-debug" "-nodefaultlibs"
  = note: some arguments are omitted. use `--verbose` to show all linker arguments
  = note: rust-lld: error: /tmp/rustctoZu3J/list:4: ; expected, but got cabi_post_floria:plugins/dispatch
          >>>     cabi_post_floria:plugins/dispatch#[method]custom-resource.replica;
          >>>     ^
          collect2: error: ld returned 1 exit status

As you can see, it seems to have something to do with the symbol name for the C ABI. I opened this issue first on cargo-pgo but it's a head-scratcher.

Maybe we can try to figure this out together?

Steps to Reproduce

Not trivial, but the best way is this:

Clone and install the khutulun-dev repository. And then try to PGO Puccini:

cd puccini
scripts/build
scripts/build-examples
PGO=true WASM_PRECOMPILE=false LTO=false scripts/build -r

(We don't actually need to disable Wasm precompile and LTO, but they don't make a different for this bug and this will get you the results faster)

For reference, here is the WIT file used.

Versions and Environment

Wasmtime version or commit: 41.0.0

Operating system: Fedora 43

Architecture: x86-64

[alexcrichton]

Thanks for the report! I think this is probably due to wit-bindgen, however, since that symbol looks like it was generated by wit-bindgen. Would it be possible to exclude the dependency using wit-bindgen on native platforms? That's still something that should probably be fixed in wit-bindgen itself but for now excluding that dependency might work

[tliron]

Thanks, though I'm not entirely sure I follow.

I am not exactly using wit-bindgen. Well, it's there when I build the Wasm blob, and that blob is included in this binary (using include_bytes!), which is then compiled at runtime by cranelift in wasmtime. But that's indirect and I don't think that would be related to PGO of the program itself. Specifically, when I run cargo pgo optimize there is no Wasm blob involved.

But maybe you mean something else? Where do you intend for me disable a wit-bindgen dependency?

[alexcrichton]

The symbol name in question, cabi_post_floria:plugins/dispatch#[method]custom-resource.replica, is a symbol generated by wit-bindgen. I agree wit-bindgen is intended for in-wasm use primarily, but it's possible for it to leak in to the crate graph nonetheless in some situations for native compilations.

For example the linker command line you pasted here contains libfloria_plugin_sdk-2d3041ed2aa76b6f,libwit_bindgen-d74a0375665663a5 which looks like it's including an SDK that was intended for wasm and wit-bindgen itself is being linked in. This is someting we can fix in wit-bindgen (generate different symbol names on native that are more amenable to native linkers), but if you're able to exclude the dependency on floria-plugin-sdk that'd probably also help here.

[tliron]

Good catch!

The issue seems to be with the cargo-pgo tool. It was trying to optimize everything in the [workspace] project, which included the Wasm blob, and that was where I got the failure. After some hacking I was able to get it to get it to only optimize the native binary, and there was no failure there.

I will continue the issue in cargo-pgo, but I wonder if it's worth pursuing the secondary issue of PGO for wit-bindgen Wasm builds. I would think that it's an important use case to have PGO there. You say that it might possible to produce more compatible symbols there. How do you recommend we proceed? Open an issue with wit-bindgen?

[alexcrichton]

Agreed yeah would still be good to fix! I've transferred this issue to the wit-bindgen reopsitory and renamed it now