Windowing & event-loops: who owns what? (Internal vs external event loops)

[tychedelia]

I'd like to discuss a bit our high-level library design. There are basically two different ways that we could have structured our library that comes down to the main event loop that drives the user's application forward: internal or external. Right now, we are structuring things in a way where we expect the external caller/FFI implementer to drive things forward. I think this is the right architecture but it's worth articulating the design trade-offs here.

Internal event loop and the callback handler architecture

In the internal event loop model, the "Processing runtime" owns the window and drives everything forward. What this means concretely from an API design perspective is that rather than having a series of imperative FFI functions, we'd rely on the user to provide us function pointers to callbacks that we fire whenever something happens: e.g. setting up your sketch, drawing each frame, receiving user input, etc.

This is how I built things for Nannou. Basically, we store a bunch of function pointers for the user's sketch and then call their handlers each frame when asked to do so by the engine. More specifically, here we leave frame pacing to the engine and by default it's using Bevy's game oriented "run as fast as possible" pacing.

In other words, the user doesn't decide when to present to a surface, they instead exist inside an event loop that drives all their handlers when the engine decides to.

There are advantages to this design in that it means that all the integration between windowing and the user's application can be contained and doesn't need to be exposed. This is also the most performant way to write a modern application and allows decoupling concerns. For example, we can enable a low-power / reactive event loop where we only re-render on user input quite easily here because the event loop is telling us when to render, not the other way around.

From an FFI perspective, however, this has a big drawback in that it requires expressing everything in terms of function pointers. Typically when codegening bindings in other languages, this makes things a bit more annoying because we have to wrap our functions rather than just having the tooling generate a function we can call. It also means defining a ton more data structures at the FFI layer; you have to have types for all the events, for the arguments to each function handler, etc.

External event loop and the procedural style

For this reason, when scaffolding libprocessing, it made sense to go for an imperative or procedural style with an external event loop. In this case, the implementer calls into our library at each logical step of the event loop. Our functions are exposed in terms of different operations that update some kind of draw/rendering state but are themselves stateless in their sense of time; you can call "start frame" and then wait an hour and call "end frame" and from the perspective of the library that's totally fine. The caller decides when to drive things forward.

Of course, any application that needs a window is still subject to an event loop, but rather than an application handler it can be implemented more via a polling/pumping mechanism like we do in GLFW. That is, GLFW doesn't tell us when to render, we just run in a loop and check in with GLFW to see if there are any new events.

In terms of the FFI boundary, this greatly simplifies things, and means that we can expose the API in terms of procedures. For the most part, we don't even need to define any custom FFI structs, as most things can be simple primitive arguments to functions, or getters for some logical API object's properties. This, in turn, makes it very easy to codegen. The implementer will get a list of a bajillion functions that they need to integrate with, but for the most part, their implementation will just be a pretty small layer over calling these functions. As demonstrated by our examples, setting up a basic GLFW loop is just a handful of lines of code.

The engine-eventing interface

So what would the advantage of integrating with winit in Bevy be? First, it would mean that implementers would also not have to worry about managing a window and setting up a surface. Instead, they would pull events from us each frame in a similar fashion to glfwPollEvents. This would be nice because it would mean that we wouldn't rely on implementers having to figure out what windowing libraries are available in their language and could reliably just expect us to manage the window for them.

But, secondly, it would also mean that events are more easily propagated within the engine itself. Right now, if we want to react to user input in Rust, we have to create an API to feed events into libprocessing. For the core engine, this isn't necessary because outside of window resizes there's no engine code that depends on certain user actions (button presses, etc.). However, there are potentially engine features in the future that may require such events. For example, render picking which requires casting a ray from the pointers position, etc.

The current limit in Bevy

Right now, Bevy uses an internal event loop, and although winit has support for external event loops it's not easy to bridge. I feel confident that we want to continue to maintain an external event loop. The entire Processing API is designed around the assumption that you can call certain methods like beginDraw/endDraw on a graphics object, and this only makes sense in the context where we imperatively drive things forward / decide when to present. Although this is a certain kind of inefficiency with respect to super high performance graphics, for most art and art installations it's conceptually way simpler.

So, for now, I feel confident that external event loop is the right design, but we'll eventually probably want to bring windowing into libprocessing so that users don't have to deal with it!

[catilac]

Thank you so much for this detailed write up!

I see how an external event loop will shape the work of someone using it. eg the python prototype I'm working on.
And the possibility of codegen is really promising. There is also a lot of desire, especially from p5.js land for render picking. Which is another discussion possibly.

What are your thoughts around bringing windowing into libprocessing? Making winit a dependency on our side, and managing its relationship to the bevy renderer manually?

I'm always happy to see that it's possibly for us to take ownership of certain things without having to toss out bevy. One of the reasons why I wanted to use it was because of its modular design.

[limzykenneth]

I think I generally also lean very much towards the external event loop, partly is because of a bit of sunk cost I designed into p5.js 2.x, but partly it is going back to the point of having the simpler set of FFI implementation.

The eventing and windowing side of things are a bit lower level than what I usually work with so I may not be making sense here, so feel free to correct or ignore. Ideally it would be great for libprocessing to handle windowing fully, which for me creates a split in terms of the host language will handle the Processing/p5.js style runtime loop of setup() and draw() etc, then libprocessing will handle windowing and having events that the host language can listen to.

This leads me to think (which may be the part that make the least sense here) that would it be possible/efficient to decouple the runtime loop of the host language and libprocessing so that when calls into libprocessing happens depends entirely on the host language, while when the surface gets updated depends entirely on libprocesssing? This builds on the assumption that libprocessing can know when is the best time to update a surface given a series of draw calls while the host language would only be concerned with passing along draw calls to libprocessing.

[tychedelia]

What are your thoughts around bringing windowing into libprocessing? Making winit a dependency on our side, and managing its relationship to the bevy renderer manually?

So this is partly what I was prototyping recently. We absolutely can/should do this. The blocker right now is just that the winit ApplicationHandler trait requires a lot of glue to integrate with Bevy. None of it's particularly complicated, but it's not very well factored ATM to be used by other projects that want to integrate with winit themselves. Let's talk more if you run into problems with GLFW and Python.

This leads me to think (which may be the part that make the least sense here) that would it be possible/efficient to decouple the runtime loop of the host language and libprocessing so that when calls into libprocessing happens depends entirely on the host language, while when the surface gets updated depends entirely on libprocesssing? This builds on the assumption that libprocessing can know when is the best time to update a surface given a series of draw calls while the host language would only be concerned with passing along draw calls to libprocessing.

I definitely am interested in any/all improvements here that would help with implementing things. We definitely have some heuristics around when we need to present. Basically, we can accumulate draw state until the user either asks for readback via loadPixels (which requires materializing the draw on the render target although not necessarily presenting to the swapchain) or ends the frame. Currently, ending the frame is a blocking operation, which is inefficient. Ideally, we could return as soon as the user ends the frame and submit in a separate thread, but this has some additional complexity we've decided to forego for now.

The difficulty here is that we need some kind of transaction boundary for a frame in the users sketch, because the immediate mode graphics style has a semantic guarantee that what you draw will show up on the window in the frame in which you draw it.. We could imagine, for example, a model in which the sketch runs at a certain rate and we submit draw commands on queue and the main window/render event loop consumes that queue when asked to present and draws whatever is there. This would likely work in many situations but would result in weird flicker/dropped frames/other bugs in the event that the sketch and render loops ran ahead or behind each other.

This isn't as much as a problem in games, for example, where physics can run in a fixed time step and there are a variety of mechanisms to deal with objects popping in and out of existence (level of detail, geometry streaming, etc). In a complicated 3d scene, gamers are used to seeing a bit of jank as things take a moment settle when loading into a new level, etc.

So yeah, just some random thoughts. I definitely think we should be guided here by any pain during implementation. If there are ways we can improve the core Processing API we should! One thing that comes to mind is that right now doing multi-window rendering is a bit of a pain, but this is something I'd love to support.

[limzykenneth]

@tychedelia I agree it is probably more concrete to work out the problem during implementation as well to avoid the discussions being too abstract. I mentioned this to @catilac the other day already but I'm definitely interested in testing out potential implementation on p5.js side so do let me know whenever there is a proof of concept or libprocessing has got to a place where I can start playing around with it.