[FEATURE] Interactive viewer plugins

[Milotrince]

Description

nice-to-have feature, allows users to integrate interactive viewer plugins.

Related Issue

N/A

Motivation and Context

• ViewerPlugin enables directly interfacing with pyviewer mouse/keyboard controls.
• Replaces pynput
• Keybind callback registration

How Has This Been / Can This Be Tested?

added tests/test_viewer.py

Screenshots (if appropriate):

Screen.Recording.2025-11-17.at.18.03.51.mov

Checklist:

• I read the CONTRIBUTING document.
• I followed the Submitting Code Changes section of CONTRIBUTING document.
• I tagged the title correctly (including BUG FIX/FEATURE/MISC/BREAKING)
• I updated the documentation accordingly or no change is needed.
• I tested my changes and added instructions on how to test it for reviewers.
• I have added tests to cover my changes.
• All new and existing tests passed.

[duburcqa]

This feature looks very convenient!

[Milotrince]

• Should we keep under genesis/ext/pyrender/interaction/plugins/ or genesis/vis/plugins ?
• Do we want to use Vec3, Quat, Color, Ray, etc. classes in the codebase?
• now that pyrender viewer mouse/keyboard interaction is exposed, we can get rid of pynput in examples

[duburcqa]

Should we keep under genesis/ext/pyrender/interaction/plugins/ or genesis/vis/plugins ?

Do you consider this as a "universal" plugin or something pyrender-specific? This would give you the answer :)

Do we want to use Vec3, Quat, Color, Ray, etc. classes in the codebase?

I'm not a fan of these objects. Overly complicated and not very pythonic in my view.

now that pyrender viewer mouse/keyboard interaction is exposed, we can get rid of pynput in examples

That would be great! But we need a way to expose this in a clean way.

[duburcqa]

@Milotrince If you could finish this PR I would be happy to review it and merge it :)

[Milotrince]

@duburcqa Could I get your opinion on the state of the PR? there are some design decisions I'd like your input on!

• removed pynput, see examples/* changes for new usage
• I kept the mouse spring interaction implementation and moved it as a plugin
• there existed a register key system in pyrender viewer but was not exposed in genesis viewer. I made a Keybind/Keybindings class and exposed register_keybind to viewer

I still need to add tests.

[duburcqa]

Could I get your opinion on the state of the PR? there are some design decisions I'd like your input on!

Sure ! I will do it this week ! Could you fix the conflicts in the meantime? Thank you :)

[duburcqa]

Small comment on snapshots PR: https://huggingface.co/datasets/Genesis-Intelligence/snapshots/discussions/6

[duburcqa]

Could you add from pyglet.window import key in your new keybindings module, so that users can do:

from genesis.ext.pyrender.interaction.keybindings import key

This would avoid breaking encapsulation of pyglet, that should be an implementation details for users.

Similarly, it would be great to expose your keybindings module outside ext, that is also supposed to be an implementation details for the end user. This would be relaying not the same mechanism I just mentioned. Ideally the user should be able to replace genesis.ext.pyrender.interaction.keybindings by genesis.vis.keybindings

[Milotrince]

leaving a note here: on mac (untested with other machines), when i import pyglet or pyglet.window.key at the top level before viewer does, I get

*** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '-[NSApplication macOSVersion]: unrecognized selector sent to instance 0x1618b7df0'

*** First throw call stack:
(
        0   CoreFoundation                      0x0000000198f498ec __exceptionPreprocess + 176
        1   libobjc.A.dylib                     0x0000000198a22418 objc_exception_throw + 88
        2   CoreFoundation                      0x000000019902b7e8 +[NSObject(NSObject) instanceMethodSignatureForSelector:] + 0
...
        87  python3.12                          0x000000010436e1f0 builtin_exec + 404
        88  python3.12                          0x00000001042bb45c cfunction_vectorcall_FASTCALL_KEYWORDS + 92
        89  python3.12                     
libc++abi: terminating due to uncaught exception of type NSException

By claude's suggestion I added it as a lazily loading module to safely expose key at genesis.vis.keybindings.

[duburcqa]

Could you split the introduction of ViewerPlugin feature, with the ability to interactively add markers on geometries from the viewer? Because this PR is getting quite bug already and both features are quite orthogonal. It would make my life easier for the review.

You can open a (stacked) PR for the second part right away, merging on top of this branch instead of main (with draft status).

[duburcqa]

Why FIXME disappeared? You actually implemented the fix?!

[Milotrince]

I removed it because I think the FIXME note isn't quite accurate, raycaster.py does ray-AABB tests while LBVH.query() checks AABB-AABB. There might be a bit of duplicated logic but i think it'd be more complicated to split it up into smaller ti.funcs so may as well leave as is.