Add a Way to Map Specific Face-Tracked Data to Drive Specific Blendshapes

[Chylum]

Is your feature request related to a problem?

[Jinxxy] Melanie

this is a *current limitation of basis, for vrc they generate the initial fx animator and you would go in and tune the visemes to respond how you desire, we currently only directly drive blendshapes based on the output of openlipsync. to fit this use case we would need to add a mapping/config setup for visemes.

Nobody has built a script/system to define and drive non-linear mappings for visemes. Also yes, this is a common technique to do billboard expressions when you dont want to modify the base mesh or have a specific mesh you want to show.

its more that basis drives visimes' blendshapes with the direct output from the lip sync system
it will output 20-80 for a full match of the viseme which works find for continuous blendshapes

you can get good looking viseme responses with it [avatars with billboard expressions instead of continuous visemes] still, it just requires selecting the highest value out of the viseme output and only showing showing that blendshape at 100

Otherwise, if not implemented, avatars with non-liner visemes have limited options:

probably recording the performance as an animation, then go back through with audio and manually map the blendshapes. lol

Describe your preferred solution

Writing the system for visemes mapping.

Describe any considered alternatives

Resonite has a good way of doing it: https://wiki.resonite.com/Component:AvatarExpressionDriver

Additional Context

No response

[mriise]

something along the lines of

public struct VisemeProfile
{
    public int BlendShape;
    public float Gain;
    [Range(0f, 1f)] public float Threshold; // >0 = binary cutoff
    public float Attack;
    public float Release;
}