This repo is for exploring surface electromyography (sEMG) data and training models associated with the paper "A generic non-invasive neuromotor interface for human-computer interaction".
The dataset contains sEMG recordings from 100 participants in each of the three tasks described in the paper: discrete_gestures, handwriting, and wrist. This repo contains implementations of the models in the paper as well as code for training and evaluating the models.
First, clone this repository and navigate to the root directory.
git clone https://github.com/facebookresearch/generic-neuromotor-interface.git
cd generic-neuromotor-interfaceNow setup the conda environment and install the local package.
# Setup and activate the environment
conda env create -f environment.yml
conda activate neuromotor
# Install this repository as a package
pip install -e .To download the full dataset to ~/emg_data for a given task, run:
python -m generic_neuromotor_interface.scripts.download_data \
--task $TASK_NAME \
--output-dir ~/emg_datawhere $TASK_NAME is one of {discrete_gestures, handwriting, wrist}.
Alternatively, you can download and extract a smaller version of the dataset with only 3 participants per task to quickly get started:
# NOTE: `--small-subset` downloads only 3 users per task
python -m generic_neuromotor_interface.scripts.download_data \
--task $TASK_NAME \
--output-dir ~/emg_data \
--small-subsetTo download pretrained checkpoints for a task, run:
python -m generic_neuromotor_interface.scripts.download_models \
--task $TASK_NAME \
--output-dir ~/emg_modelsThe extracted output contains a .ckpt file and a model_config.yaml file.
Use the explore_data.ipynb notebook to see how data can be loaded and plotted:
jupyter lab notebooks/explore_data.ipynbTrain a model via:
python -m generic_neuromotor_interface.train \
--config-name=$TASK_NAMENote that this requires downloading the full_data dataset as described earlier.
You can also launch a small test run (1 epoch on the small_subset dataset) via:
python -m generic_neuromotor_interface.train \
--config-name=$TASK_NAME \
trainer.max_epochs=1 \
trainer.accelerator=cpu \
data_module/data_split=${TASK_NAME}_mini_splitAfter training, the model checkpoint will be available at ./logs/<DATE>/<TIME>/lightning_logs/<VERSION>/checkpoints/, and the model config will be available at ./logs/<DATE>/<TIME>/hydra_configs/config.yaml.
Model evaluation on the validation and test sets is automatically performed in the training script after training is complete.
We also provide interactive notebooks to run model evaluation on any given trained model. Please see the evaluation notebooks:
jupyter lab notebooks
# see:
# notebooks/discrete_gestures_eval.ipynb
# notebooks/handwriting_eval.ipynb
# notebooks/wrist_eval.ipynbThese notebooks also provide some visualizations of the model outputs.
| Quantity | Discrete Gestures | Handwriting | Wrist | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Train | Val | Test | Train | Val | Test | Train | Val | Test | |
| Number of Users | 80 | 10 | 10 | 80 | 10 | 10 | 80 | 10 | 10 |
| Number of Datasets | 80 | 10 | 10 | 618 | 135 | 54 | 142 | 20 | 20 |
| Hours | 51.4 | 6.2 | 6.4 | 98.8 | 31.5 | 10.4 | 60.2 | 8.7 | 8.3 |
| Datasets Per User | 1.0 | 1.0 | 1.0 | 7.7 | 13.5 | 5.4 | 1.8 | 2.0 | 2.0 |
| Hours Per User | 0.6 | 0.6 | 0.6 | 1.2 | 3.2 | 1.0 | 0.8 | 0.9 | 0.8 |
We are releasing data from 100 data collection participants for each task: 80 train, 10 validation, and 10 test participants. Train participants correspond to those from the 80 participant data point in Figures 2e-g (except for Handwriting, where we randomly selected 80 participants from the 100 participant data point). The 10 validation and train participants were randomly selected from the full set of validation and test participants.
Evaluation metrics may deviate slightly from the published results due to subsampling of the test participants (as there is considerable variability across participants) and variability across model seeds.
Each recording is stored in an .hdf5 file, and there can be multiple recordings per participant. There is also a .csv file for each task (${TASK_NAME}_corpus.csv) documenting the recordings included for each participant, start and end times for each relevant "stage" from the experimental protocol (see below), and their assignment to the train / val / test splits. This .csv file is downloaded alongside the data by the download script described above.
sEMG is recorded at 2 kHz and is high pass filtered at 40 Hz. Timestamps are expressed in seconds. A stages dataframe is included in each dataset that encodes the time of each stage of the experiment (see explore_data.ipynb for more details). Specifics for each task are as follows:
Datasets include the name of each gesture and the time at which it occurred. Stage names include the types of gestures performed in each stage, as well as the posture (e.g. static_arm_in_front, static_arm_in_lap, ...)
Handwriting datasets include the start and end time of each prompt. start is the time the prompt appears, and end is the time at which participants marked as having finished writing the prompt. Stage names describe the types of prompts in each stage (e.g. words_with_backspace, three_digit_numbers, ...).
Wrist angle datasets also include wrist angle measurements, which are upsampled to match the 2 kHz EMG sampling rate. Stage names include information about the type of task and movement in each stage (e.g. cursor_to_target_task_horizontal_low_gain_screen_4, smooth_pursuit_task_high_gain_1, ...).
The dataset and the code are CC-BY-NC-4.0 licensed, as found in the LICENSE file.
@article{generic_neuromotor_interface_2025,
title = {A generic non-invasive neuromotor interface for human-computer interaction},
author = {Kaifosh, Patrick and Reardon, Thomas R. and CTRL-labs at Reality Labs},
journal = {Nature},
year = {2025},
doi = {10.1038/s41586-025-09255-w},
issn = {1476-4687},
url = {https://www.nature.com/articles/s41586-025-09255-w},
}