Goal to cut down uses of unsafe?

[ValorZard]

I know that this codebase is mostly an LLM translation of Lua to Rust, but is there any intention to go back and try to remove as much usages of unsafe once everything is stabalized?

I would be down to help contribute to such an effort if necessary

[CppCXY]

Currently, there are several hundred uses of unsafe, most of which are for performance considerations. I have over 1,000 commits. In fact, I have tried various designs without unsafe. Here are some conclusions I can give you:

if GC objects use GCId instead of pointers, the cost of fixing GCId either involves using a hashmap, which significantly degrades the performance of dereferencing GC objects, or adopting an arena approach, which causes GC performance to drastically decline as the number of GC objects grows, due to traversing a large number of holes.

The main reason for using unsafe in execute loops is to bypass Rust's automatic bounds checking. For large functions with over 2,000 lines, this bounds checking significantly bloats the function's stack frame size, making CPU cache misses more likely. Additionally, this checking incurs a performance loss of over 10%, and it is unnecessary because its safety is already guaranteed by Lua's compiler part.

The last major chunk of unsafe code comes from the design of the table. Initially, I tried a combination of Vec and HashMap. First, its performance was less than one-third of the current implementation. Additionally, it was very difficult to implement a performant next function, even when using the hashbrown version of HashMap, because they are typically designed for large amounts of data, whereas Lua most commonly uses small tables. Moreover, after Lua 5.5, a completely new array design was adopted. I think this design is excellent, but it is very hard to implement directly using Vec. Furthermore, the Vec + HashMap combination has excessively high memory overhead for small tables.

The remaining uses of unsafe are mostly cases where I have added warnings that users need to know what they are doing before using them. This project prioritizes performance above all else. You can find many Rust implementations of Lua on GitHub, most of which are not truly complete. Among those that are preliminarily complete are rilua and tsuki. If you are interested in performance, you can compare them yourself. I can offer a personal conclusion: they more or less suffer from insufficient performance due to being safe.

[ValorZard]

That makes sense! Do you use something like https://github.com/rust-lang/miri/ to test for soundness/UB?

[CppCXY]

That's a great suggestion, I'll give it a try. At first glance, it seems to be caused by some of my optimizations and the way I store pointers.

[ValorZard]

adding miri to the CI/CD might make it easier to catch like soundness/memory bugs in the future as well!
https://github.com/taiki-e/github-actions/blob/main/.github/workflows/miri.yml

[CppCXY]

Through simple testing and effort, I believe miri's judgment is essentially wrong. If you're interested, you can try to fix it.

[CppCXY]

I passed the Miri test under relatively relaxed conditions and have used Miri tests on the CI for a few simple tests. Running a full Miri test with cargo test or Lua benchmarks takes an extremely long time, so for now I don't plan to run the full test suite on CI.