Async watchers for multi-threading.

[ImagicTheCat]

Hi,

Currently the async watcher API doesn't seem to handle the use case of multi-threading synchronization (waking up the event loop from a different thread). Could it be implemented with the Lua C API in a flexible way to interface with threading libraries ? (open question)

[brimworks]

Unfortunately, I no longer use this library that I created and have been maintaining, so I don't have any interest in adding this support, however pull requests will be reviewed and accepted!

[brimworks]

Thanks for the PR! My biggest concern is that we have an unsafe API that is taking an arbitrary Lua string and casting it. Therefore, if you pass in a string of the correct size, you could cause an illegal address error which would bomb the entire process. Therefore, if anyone uses this library to provide users with a "sandbox", this "sandbox" could be used to create a denial of service attack.

I assume the intention of this is to make it so a Lua_async in one Lua VM can be used from an event loop in a separate Lua VM (but the same process).

How are you getting data from one Lua VM to the other?

[ImagicTheCat]

I assume the intention of this is to make it so a Lua_async in one Lua VM can be used from an event loop in a separate Lua VM (but the same process).

Yes, the most relevant use of async watchers is being able to wakeup the event loop from another thread.
In this case, it is just about serializing pointers from one thread to another. It doesn't serialize Lua data.

Therefore, if anyone uses this library to provide users with a "sandbox", this "sandbox" could be used to create a denial of service attack.

Yes, this is an unsafe API. I personnally came to the conclusion that sandboxing is not something that should be designed at the lower levels, but carefully built around existing features; also not all sandboxes have equal requirements.

I don't see a "safe" alternative to this, unless the API provides its own multi-threading features.

[ImagicTheCat]

As an alternative, I didn't know that I could access userdata with LuaJIT using the FFI, so I implemented an "extension" with a similar API in pure Lua which elegantly suit my needs.

It was probably the intent of the async watcher feature contributor to use the userdata from custom code, but I didn't know how to do that sanely.

Gist of the "extension".

[brimworks]

@ImagicTheCat can you answer this question?

How are you getting data from one Lua VM to the other?

[brimworks]

As mentioned in the PR, I'm happy to accept the contribution that supports the send API (and the Async constant export), and maybe documentation on how to do the rest using FFI with an explanation about how you can crash your process if a loop (or async) is garbage collected prematurely.

[brimworks]

The only safe way I can think of to do this is to have memory stored outside of the Lua VM which contains the loops and async watchers which are reference counted. This could be stored in a pre-allocated array with some arbitrary limit. Our loop create logic and async create logic would be modified so the user-data specifies which index in this array contains the loop/async, and the array elements would contain the actual loop/async object along with a reference counter/type field. We could then "send" the index, and the receiver would verify the type and increment the reference counter ...and of course we would need to decrement the reference counter when the user-data is being garbage collected.

Of course, we also need a way to provide mutual exclusion access to read/write to this array, and we would need to provide the user with an error if this array is exhausted (no more empty slots).

It also implies that creation/deletion of loop and async logics will be slower as it requires mutual exclusion. Therefore, we should probably have a compile time flag to indicate if the library is being built with "pthreads" (assuming, that is what we use for mutual exclusion), and if NOT being built with "pthreads", then the send/receive methods should produce an error to indicate that this functionality is disabled due to the library not being built with pthreads.

What do you think?

[ImagicTheCat]

@ImagicTheCat can you answer this question?

I did, it is serialization of a structure containing the pointers.

What do you think?

I don't see anything wrong with the idea, but it's probably too much troubles (re-implementing some part of a multi-threading Lua library) for a feature that most users don't know about or would find a way if they do.

I will think about making another PR with selected changes (edit: #28).

[brimworks]

Thank you for the contribution. I added a general disclaimer about using the FFI approach along with a suggestion on how we could safely implement this if someone wants to tackle this in the future:

https://github.com/brimworks/lua-ev/blob/master/contrib/ev-async.lua

Thanks!