Decoding High-Dimensional Finger Motion from EMG Using Riemannian Features and RNNs
Decoding high-dimensional finger motion using Riemannian features and RNNs, TRR achieves 9.79° error on EMG-FK dataset.
Key Findings
Methodology
The paper presents an end-to-end framework combining an 8-channel EMG armband, a single webcam, and an automatic synchronization procedure to collect the EMG-FK dataset. This dataset includes synchronized EMG and 15 finger joint angles from 20 participants performing rich, unconstrained right-hand motions. The Temporal Riemannian Regressor (TRR), a lightweight GRU-based model, is introduced to decode finger motion using sequences of multi-band Riemannian covariance features.
Key Results
- On the EMG-FK dataset, the TRR model achieves an average absolute error of 9.79°±1.48 in intra-subject evaluation and 16.71°±3.97 in cross-subject evaluation.
- TRR also outperforms state-of-the-art methods on the emg2pose benchmark, demonstrating robustness across different subjects and sessions.
- Real-time deployment of the TRR model on a Raspberry Pi 5 shows efficiency on embedded devices, achieving nearly 10 predictions per second.
Significance
This research lowers the complexity of decoding high-dimensional finger motion, providing more natural and intuitive control for EMG-based systems. By utilizing consumer-grade hardware and innovative data acquisition methods, the study reduces deployment costs and complexity outside laboratory conditions. These contributions pave the way for natural interaction in fields like hand prostheses, AR/XR interfaces, and teleoperation.
Technical Contribution
The technical contributions include: 1) Introducing a novel regression model combining Riemannian geometric features with recurrent neural networks; 2) Developing a low-cost data acquisition framework supporting high-quality EMG-kinematics recordings; 3) Demonstrating real-time inference on embedded devices, showcasing the efficiency and applicability of the TRR model.
Novelty
The TRR model is the first to integrate multi-band covariance matrices projected from the Riemannian manifold to the Euclidean tangent space with a recurrent neural network for finger joint-angle regression. Compared to existing methods, TRR offers significant improvements in accuracy and real-time performance.
Limitations
- The model's performance in cross-subject configurations still has room for improvement due to the high sensitivity of EMG signals to individual differences.
- The data acquisition process requires participants to perform prolonged finger movements, which may lead to fatigue and affect data quality.
- In practical applications, environmental lighting and camera positioning may affect the accuracy of motion capture.
Future Work
Future research could explore more robust cross-subject models to reduce the impact of individual differences on performance. Additionally, more efficient feature extraction methods could be developed to further enhance the model's real-time capabilities and accuracy. The study could also be expanded to different hand movement scenarios to validate the model's generalizability.
AI Executive Summary
Accurately decoding high-dimensional finger motion is crucial for natural interaction in fields like hand prostheses, AR/XR interfaces, and teleoperation. However, due to the complexity of human hand gestures and the entanglement of forearm muscles, existing methods often rely on classification-based machine learning, limiting controllable degrees of freedom and compromising natural interaction.
This paper presents an end-to-end framework that combines an 8-channel EMG armband, a single webcam, and an automatic synchronization procedure to collect the EMG-FK dataset. This dataset includes synchronized EMG and 15 finger joint angles from 20 participants performing rich, unconstrained right-hand motions. The Temporal Riemannian Regressor (TRR), a lightweight GRU-based model, is introduced to decode finger motion using sequences of multi-band Riemannian covariance features.
The TRR model performs exceptionally well on the EMG-FK dataset, achieving an average absolute error of 9.79°±1.48 in intra-subject evaluation and 16.71°±3.97 in cross-subject evaluation. Additionally, TRR outperforms state-of-the-art methods on the emg2pose benchmark, demonstrating robustness across different subjects and sessions. Real-time deployment of the TRR model on a Raspberry Pi 5 shows efficiency on embedded devices, achieving nearly 10 predictions per second.
These contributions lower the barrier to reproducible, real-time EMG-based decoding of high-dimensional finger motion, paving the way for more natural and intuitive control of embedded EMG-based systems. By utilizing consumer-grade hardware and innovative data acquisition methods, the study reduces deployment costs and complexity outside laboratory conditions.
However, the model's performance in cross-subject configurations still has room for improvement due to the high sensitivity of EMG signals to individual differences. Future research could explore more robust cross-subject models to reduce the impact of individual differences on performance. Additionally, more efficient feature extraction methods could be developed to further enhance the model's real-time capabilities and accuracy. The study could also be expanded to different hand movement scenarios to validate the model's generalizability.
Deep Analysis
Background
Gesture recognition technology has gained significant attention in recent years, particularly in fields like augmented reality, teleoperation, and hand prosthesis control. Traditional gesture recognition methods mainly rely on cameras and computer vision techniques, tracking joint positions to achieve recognition. However, these methods often suffer from performance degradation under hand occlusion, poor lighting conditions, or during object manipulation. Additionally, these systems depend on cameras positioned in the environment, which may constrain mobility, increase setup complexity, and raise privacy concerns. Electromyography (EMG) offers a promising alternative by equipping users with physiological sensors around the forearm or wrist to measure muscle activity, which can be processed to infer hand and finger movements. EMG-based interaction reduces dependence on external sensing infrastructure and is less affected by lighting conditions or occlusion, making it a minimally intrusive and robust solution.
Core Problem
Despite the advantages of EMG technology in gesture recognition, accurately decoding high-dimensional finger motion remains challenging due to the complexity of human hand gestures and the entanglement of forearm muscles. Existing methods often rely on classification-based machine learning, limiting controllable degrees of freedom and compromising natural interaction. Furthermore, EMG signals are highly sensitive to individual differences, including muscle size, gesture patterns, and mental state, making it necessary for each EMG-based system to undergo a dedicated data acquisition phase to develop a customized model that accurately reflects the application's needs.
Innovation
The core innovation of this paper lies in introducing a novel regression model that combines Riemannian geometric features with recurrent neural networks—Temporal Riemannian Regressor (TRR). This model utilizes sequences of multi-band Riemannian covariance features to decode finger motion, significantly improving accuracy and real-time performance. Additionally, the paper develops a low-cost data acquisition framework that supports high-quality EMG-kinematics recordings. By utilizing consumer-grade hardware and innovative data acquisition methods, the study reduces deployment costs and complexity outside laboratory conditions.
Methodology
The methodology of this paper includes the following key steps:
- �� Data Acquisition: An 8-channel EMG armband and a single webcam, combined with an automatic synchronization procedure, are used to collect the EMG-FK dataset.
- �� Feature Extraction: Multi-band Riemannian covariance features are extracted from EMG signals and projected to the Euclidean tangent space.
- �� Model Design: A lightweight GRU-based model, TRR, is designed to decode finger motion using the extracted feature sequences.
- �� Experimental Validation: The performance of TRR is validated on the EMG-FK and emg2pose benchmarks, and its real-time deployment on embedded devices is demonstrated.
Experiments
The experimental design includes validation on the EMG-FK and emg2pose datasets. The EMG-FK dataset comprises synchronized EMG and 15 finger joint angles from 20 participants performing rich, unconstrained right-hand motions. We use 10-fold cross-validation to evaluate the model's intra-subject performance and leave-one-subject-out cross-validation for cross-subject performance. Baseline models used in the experiments include vemg2pose, TDF, and CRNN. Key hyperparameters include the number of GRU layers, feature extraction frequency bands, and learning rate.
Results
The experimental results show that the TRR model performs exceptionally well on the EMG-FK dataset, achieving an average absolute error of 9.79°±1.48 in intra-subject evaluation and 16.71°±3.97 in cross-subject evaluation. Additionally, TRR outperforms state-of-the-art methods on the emg2pose benchmark, demonstrating robustness across different subjects and sessions. Real-time deployment of the TRR model on a Raspberry Pi 5 shows efficiency on embedded devices, achieving nearly 10 predictions per second.
Applications
The applications of this research include hand prosthesis control, AR/XR interfaces, and teleoperation. By utilizing consumer-grade hardware and innovative data acquisition methods, the study reduces deployment costs and complexity outside laboratory conditions. These contributions pave the way for more natural and intuitive control of embedded EMG-based systems.
Limitations & Outlook
Despite the TRR model's exceptional performance in decoding high-dimensional finger motion, its performance in cross-subject configurations still has room for improvement due to the high sensitivity of EMG signals to individual differences. Additionally, the data acquisition process requires participants to perform prolonged finger movements, which may lead to fatigue and affect data quality. In practical applications, environmental lighting and camera positioning may affect the accuracy of motion capture. Future research could explore more robust cross-subject models to reduce the impact of individual differences on performance.
Plain Language Accessible to non-experts
Imagine your hand is like a band, and your brain is the conductor. Every time you want to make a move, like waving or grabbing something, your brain gives the command, and each instrument (muscle) in the band starts playing (contracting). These sounds (EMG signals) can be recorded by a special device. Researchers are like music analysts, trying to guess what song (finger movement) the band is playing by analyzing these sounds.
However, the challenge is that different people have different bands, and each person's playing style is unique, making analysis difficult. To better understand this music, researchers developed a new method, like equipping each instrument with a microphone, so they can hear each instrument's sound more clearly. This method not only allows for more accurate identification of the song being played but also performs well across different bands.
Through this approach, researchers hope to enable people who have lost their hands to control prosthetics through the band's music, just as naturally as their own hands. This technology can also be used in virtual reality and remote control, allowing people to interact with technology without using traditional control devices.
ELI14 Explained like you're 14
Imagine you're playing a super cool game, but this time you don't need a controller or keyboard. Instead, you just move your fingers, and the game character mimics your actions! Sounds magical, right? That's what scientists are working on!
They use a technology called electromyography (EMG) to capture the muscle signals on your arm. Like listening to music, these signals are like the beats, telling the game character how to move. Scientists have developed a new method that can decode these signals faster and more accurately, making the game character's movements smoother.
This new method isn't just for games; it can also help people who need prosthetics. They can control the prosthetics by moving the muscles on their arms, just like controlling their own hands. This technology can also be used in virtual reality, allowing you to explore virtual worlds freely without any bulky equipment.
However, there are some challenges, like different people's muscle signals might be different, so scientists are still working hard to make it better. In the future, we might see more of this cool technology making our lives more convenient and fun!
Glossary
Electromyography (EMG)
EMG is a technique for recording muscle activity by measuring the electrical signals of muscles through electrodes placed on the skin. These signals can be used to infer hand and finger movements.
In this paper, EMG is used to capture forearm muscle activity to decode finger motion.
Riemannian Features
Riemannian features are a feature extraction method based on Riemannian geometry, used to capture covariance patterns in signals. This method provides a richer representation of complex biosignals.
Riemannian features are used in this paper to extract information from EMG signals to improve decoding accuracy.
Recurrent Neural Network (RNN)
RNN is a neural network architecture designed to handle sequential data, capable of capturing dynamic patterns in time series.
RNN is used in this paper to process sequences of EMG signals to predict finger motion.
GRU
GRU is a variant of RNN with fewer parameters and faster training speed, suitable for real-time applications.
GRU is used in this paper to build the lightweight TRR model for real-time finger motion decoding.
Covariance Matrix
A covariance matrix is a matrix used to describe the relationships between multiple variables, capturing joint activation patterns in signals.
Covariance matrices are used in this paper to represent muscle activation patterns in EMG signals.
Euclidean Tangent Space
Euclidean tangent space is a linear approximation of the Riemannian manifold, allowing for simple linear operations on complex geometric structures.
Covariance matrices are projected from the Riemannian manifold to the Euclidean tangent space in this paper to simplify computations.
Dataset
A dataset is a collection of data used for training and evaluating machine learning models, typically containing input signals and corresponding labels.
The EMG-FK dataset and emg2pose benchmark are used in this paper to evaluate model performance.
Benchmark
A benchmark is a standard test used to evaluate the performance of algorithms or models, often used to compare the effectiveness of different methods.
The emg2pose benchmark is used in this paper to validate the performance of the TRR model.
Real-time Inference
Real-time inference refers to making predictions and decisions as data streams in, requiring models to have fast computational capabilities.
The real-time inference capability of the TRR model on Raspberry Pi 5 is demonstrated in this paper.
Cross-subject Evaluation
Cross-subject evaluation is a method for assessing a model's generalization ability across different individuals, typically used to test model robustness.
Cross-subject evaluation is used in this paper to validate the generalization performance of the TRR model.
Open Questions Unanswered questions from this research
- 1 How can the TRR model's performance in cross-subject configurations be further improved? Although TRR performs well in intra-subject settings, its performance in cross-subject scenarios still has room for improvement due to individual differences in EMG signals. More robust feature extraction methods and model architectures are needed to reduce the impact of these differences.
- 2 How can the accuracy of motion capture be improved in practical applications? Environmental lighting and camera positioning may affect the accuracy of motion capture, especially outside laboratory settings. More robust motion capture techniques are needed to adapt to different application scenarios.
- 3 How can participant fatigue be reduced during data acquisition? Prolonged finger movements may lead to participant fatigue, affecting data quality. More efficient data acquisition methods are needed to reduce the burden on participants.
- 4 How can the TRR model be extended to accommodate different hand movement scenarios? The current study focuses on unconstrained right-hand movements; future research should validate the model's generalizability in other hand movement scenarios.
- 5 How can the real-time capabilities of the TRR model be further optimized on embedded devices? Although TRR achieves real-time inference on Raspberry Pi 5, further optimization of the model's computational efficiency is needed to accommodate a wider range of embedded applications.
Applications
Immediate Applications
Hand Prosthesis Control
The TRR model can be used for natural control of hand prostheses by decoding EMG signals to achieve precise finger movements. This will help individuals with hand function loss perform daily activities more effectively.
AR/XR Interfaces
In augmented and extended reality, the TRR model can be used for gesture recognition, providing a more natural user interaction experience and reducing dependence on traditional input devices.
Teleoperation
The TRR model can be used in teleoperation systems to control remote devices by decoding the operator's hand movements, enabling more precise operations.
Long-term Vision
Smart Home Control
In the future, the TRR model could be used in smart home systems to control appliances through gesture recognition, achieving more convenient home automation.
Medical Rehabilitation
The TRR model could be used in medical rehabilitation to monitor patients' hand movements and provide real-time feedback, helping patients perform more effective rehabilitation exercises.
Abstract
Continuous estimation of high-dimensional finger kinematics from forearm surface electromyography (EMG) could enable natural control for hand prostheses, AR/XR interfaces, and teleoperation. However, the complexity of human hand gestures and the entanglement of forearm muscles make accurate recognition intrinsically challenging. Existing approaches typically reduce task complexity by relying on classification-based machine learning, limiting the controllable degrees of freedom and compromising on natural interaction. We present an end-to-end framework for continuous EMG-to-kinematics regression using only consumer-grade hardware. The framework combines an 8-channel EMG armband, a single webcam, and an automatic synchronization procedure, enabling the collection of the EMG Finger-Kinematics dataset (EMG-FK), a 10-h dataset of synchronized EMG and 15 finger joint angles from 20 participants performing rich, unconstrained right-hand motions. We also introduce the Temporal Riemannian Regressor (TRR), a lightweight GRU-based model that uses sequences of multi-band Riemannian covariance features to decode finger motion. Across EMG-FK and the public emg2pose benchmark, TRR outperforms state-of-the-art methods in both intra- and cross-subject evaluation. On EMG-FK, it reaches an average absolute error of $9.79 °\pm 1.48$ in intra-subject and $16.71 °\pm 3.97$ in cross-subject. Finally, we demonstrate real-time deployment on a Raspberry Pi 5 and intuitive control of a robotic hand; TRR runs at nearly 10 predictions/s and is roughly an order of magnitude faster than state-of-the-art approaches. Together, these contributions lower the barrier to reproducible, real-time EMG-based decoding of high-dimensional finger motion, and pave the way toward more natural and intuitive control of embedded EMG-based systems.
References (20)
Simultaneous Estimation of Hand Joints’ Angles Toward sEMG-Driven Human–Robot Interaction
Wang Hai, Tao Qing, Sushkova Na et al.
emg2pose: A Large and Diverse Benchmark for Surface Electromyographic Hand Pose Estimation
Sasha Salter, Richard Warren, Collin Schlager et al.
Estimating finger joint angles on surface EMG using Manifold Learning and Long Short-Term Memory with Attention mechanism
Cries Avian, Setya Widyawan Prakosa, Muhamad Faisal et al.
Linear non-conservative unsupervised domain adaptation for cross-subject EMG gesture recognition
Martin Colot, C. Simar, A. C. Alvarez et al.
Machine learning for hand pose classification from phasic and tonic EMG signals during bimanual activities in virtual reality
C. Simar, Martin Colot, A. Cebolla et al.
Accurate Continuous Prediction of 14 Degrees of Freedom of the Hand from Myoelectrical Signals through Convolutive Deep Learning
Raul C. Sîmpetru, M. Osswald, Dominik I. Braun et al.
Design of upper limb prosthesis using real-time motion detection method based on EMG signal processing
N. N. Unanyan, A. Belov
Virtual/Augmented Reality for Rehabilitation Applications Using Electromyography as Control/Biofeedback: Systematic Literature Review
C. Toledo-Peral, G. Vega-Martínez, J. A. Mercado-Gutiérrez et al.
Gaussian mixture regression on symmetric positive definite matrices manifolds: Application to wrist motion estimation with sEMG
Noémie Jaquier, S. Calinon
LEAP Hand: Low-Cost, Efficient, and Anthropomorphic Hand for Robot Learning
Kenneth Shaw, Ananye Agarwal, Deepak Pathak
NeuroPose: 3D Hand Pose Tracking using EMG Wearables
Yilin Liu, Shijia Zhang, Mahanth K. Gowda
Sign Language Recognition Using the Electromyographic Signal: A Systematic Literature Review
Amina Ben Haj Amor, Oussama El Ghoul, Mohamed Jemni
A general model based on Riemannian manifold for stable decoding movement trajectory from ECoG signals
Reza Eyvazpour, B. Farrokhi, Abbas Erfanian
Hand Tracking for Immersive Virtual Reality: Opportunities and Challenges
G. Buckingham
Maestro: An EMG-driven assistive hand exoskeleton for spinal cord injury patients
Youngmok Yun, Sarah Dancausse, Paria Esmatloo et al.
A deep Kalman filter network for hand kinematics estimation using sEMG
Tianzhe Bao, Yihui Zhao, Syed Ali Raza Zaidi et al.
Teleoperated robotic arm movement using electromyography signal with wearable Myo armband
Hussein F. Hassan, S. Abou-Loukh, I. Ibraheem
Classification of covariance matrices using a Riemannian-based kernel for BCI applications
A. Barachant, S. Bonnet, M. Congedo et al.
Effect of User Practice on Prosthetic Finger Control With an Intuitive Myoelectric Decoder
Agamemnon Krasoulis, S. Vijayakumar, K. Nazarpour
Finger Angle Estimation From Array EMG System Using Linear Regression Model With Independent Component Analysis
S. Stapornchaisit, Yeongdae Kim, Atsushi Takagi et al.