Skip to content

Add support for signing detection using yolo11 model#2

Open
pdufour wants to merge 9 commits intomainfrom
paul.dufour/test-local-trans-sign-lang
Open

Add support for signing detection using yolo11 model#2
pdufour wants to merge 9 commits intomainfrom
paul.dufour/test-local-trans-sign-lang

Conversation

@pdufour
Copy link
Copy Markdown
Owner

@pdufour pdufour commented Mar 16, 2026

Description

This adds the ability to detect letters you are signing via ASL in realtime via a YOLO model trained on an ASL image dataset and convert it into text.

i.e. any of these

image

are converted into their letter equivalents when you type /signletter and do one of the signs

The architecture to the similar to the voice transcription one. It uses hf_hub crate to download models, so that way if they are already cached you can re-use the same cached models.

Everything is contained within the app, meaning you do not need to spin up any other programs. This offers a lot of benefits like easier installs for users and more control of what models are allowed.

Example of what it captures:
image

Video demo

564455988-1f583bb0-ad3b-45a1-a912-ceb0c894d1af.mp4

Specific Changes

  • It adds two commands
/sign-model - allows you to select from different onnx models trained on ASL sign letters - there are some I uploaded here - https://huggingface.co/pdufour/asl-yolo-models-onnx/tree/main. They are downloaded on demand
/signletter - opens webcam and allows you sign letters, the Yolo model that runs locally automatically detects when your signs are and transcribes it into text.

Future Enhancements

  • Support all of ASL and just not just lettering
  • You could let the user decide between using a local model and a remote hosted one.
  • There are also other model architectures that might offer better performance like this transformer architecture model https://www.nature.com/articles/s41598-025-31558-1

Test Plan

Sign letter

  • Open codex
  • Run /sign-model to select from a Yolo model - the larger ones take more time to download
  • Run signletter
  • Make a sign and see the it outputs the letter you expecte

@pdufour pdufour force-pushed the paul.dufour/test-local-trans-sign-lang branch from dec7d5c to cfe2960 Compare March 16, 2026 23:30
@pdufour pdufour changed the title Add support for signing detection using yolo11 local LLM Add support for signing detection using yolo11 model Mar 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant