From Wrist to Finger: Hand Pose Tracking Using Ring-Watch Wearables

Fri, 25 Apr 2025 00:00:00 +0000

Project Overview (CHI 2025 Late-Breaking Work)

Hand pose tracking is a cornerstone for advancing human-computer interaction applications, from virtual reality to prosthetics control. However, existing vision-based systems and wearable devices often face limitations in portability, usability, and practicality. In this CHI 2025 Late-Breaking Work, we introduce a novel multimodal hand pose tracking framework that integrates data from an IMU-equipped ring and EMG sensors embedded in a wrist-worn device.

This work represents an important step towards enabling more practical and accessible hand tracking solutions for everyday use.

Key Innovations & Progress:

1. Novel Ring-Watch Wearable Design

We propose a compact and ergonomic wearable system that combines a single IMU-equipped ring (worn on the thumb) with EMG sensors integrated into a smartwatch. This design prioritizes wearability and comfort, addressing the common limitations of bulky or multi-device tracking systems while accurately capturing intricate finger and hand motion data.

2. Multi-Sensor Fusion for Precise Tracking

Our framework leverages the complementary strengths of both motion dynamics (from the IMU) and muscle activity (from the EMG sensors). This deep learning-based sensor fusion approach achieves precise 3D hand pose reconstruction, providing robust performance even in complex or high-speed gestures. We developed a transformer-based model with time encoding and cross-modal attention mechanisms for optimal data integration.

Media & Resources:

Paper: Extended Abstract (ACM DL)
Video: Project Video

Did you find this page helpful? Consider sharing it 🙌

PressInPose: Integrating Pressure and Inertial Sensors for Full-Body Pose Estimation in Activities

Thu, 21 Nov 2024 00:00:00 +0000

Project Overview

Accurate human body posture assessment through wearable technology has significant implications across various fields, including sports science, clinical diagnostics, rehabilitation, and VR interaction. Traditional methods often face limitations due to complex setups or environmental constraints. To address these challenges, we developed PressInPose, an innovative system that integrates pressure and inertial sensors for precise full-body pose estimation in dynamic activities. This work was published in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT) and will be presented at UbiComp 2025.

Key Innovations & Contributions:

1. Novel Multi-Sensor Fusion

PressInPose employs an advanced shoe insole embedded with pressure sensors and an Inertial Measurement Unit (IMU), coupled with a single wrist-mounted IMU. This unique multi-modal sensor fusion approach allows for a comprehensive analysis of human biomechanics, capturing intricate body dynamics that traditional single-sensor systems often miss.

2. LLM-Powered Virtual Data Augmentation

To enhance the robustness and generalization of our system, we leveraged large language models (LLMs) to generate virtual human motion sequences. These sequences were utilized to create synthetic IMU data for data augmentation, effectively addressing the challenge of limited real-world data availability and variability, especially for complex and dynamic movements.

3. Physical Kinematics Modeling & Deep Learning Network

Our approach uniquely combines physical kinematics modeling based on pressure data with a multi-region human posture estimation network. This integration allows PressInPose to accurately capture interactions and dependencies between different body parts, leading to superior accuracy in pose reconstruction.

Media & Resources:

Paper: Full Paper (ACM DL)

Did you find this page helpful? Consider sharing it 🙌

Wearable Computing | Yang Gao

From Wrist to Finger: Hand Pose Tracking Using Ring-Watch Wearables

Project Overview (CHI 2025 Late-Breaking Work)

Key Innovations & Progress:

1. Novel Ring-Watch Wearable Design

2. Multi-Sensor Fusion for Precise Tracking

Media & Resources:

PressInPose: Integrating Pressure and Inertial Sensors for Full-Body Pose Estimation in Activities

Project Overview

Key Innovations & Contributions:

1. Novel Multi-Sensor Fusion

2. LLM-Powered Virtual Data Augmentation

3. Physical Kinematics Modeling & Deep Learning Network

Media & Resources: