From Wrist to Finger: Hand Pose Tracking Using Ring-Watch Wearables

Fri, 25 Apr 2025 00:00:00 +0000

Project Overview (CHI 2025 Late-Breaking Work)

Hand pose tracking is a cornerstone for advancing human-computer interaction applications, from virtual reality to prosthetics control. However, existing vision-based systems and wearable devices often face limitations in portability, usability, and practicality. In this CHI 2025 Late-Breaking Work, we introduce a novel multimodal hand pose tracking framework that integrates data from an IMU-equipped ring and EMG sensors embedded in a wrist-worn device.

This work represents an important step towards enabling more practical and accessible hand tracking solutions for everyday use.

Key Innovations & Progress:

1. Novel Ring-Watch Wearable Design

We propose a compact and ergonomic wearable system that combines a single IMU-equipped ring (worn on the thumb) with EMG sensors integrated into a smartwatch. This design prioritizes wearability and comfort, addressing the common limitations of bulky or multi-device tracking systems while accurately capturing intricate finger and hand motion data.

2. Multi-Sensor Fusion for Precise Tracking

Our framework leverages the complementary strengths of both motion dynamics (from the IMU) and muscle activity (from the EMG sensors). This deep learning-based sensor fusion approach achieves precise 3D hand pose reconstruction, providing robust performance even in complex or high-speed gestures. We developed a transformer-based model with time encoding and cross-modal attention mechanisms for optimal data integration.

Media & Resources:

Paper: Extended Abstract (ACM DL)
Video: Project Video

Did you find this page helpful? Consider sharing it 🙌

SandTouch: Empowering Virtual Sand Art in VR with AI Guidance and Emotional Relief

Fri, 25 Apr 2025 00:00:00 +0000

Project Overview

In today’s data-rich world, simply having data isn’t enough – we need to understand it, communicate it, and extract insights from it. This course empowers you to transform raw numbers and complex datasets into compelling, insightful visual narratives. We’ll blend art with science, learning how to design interactive visualizations that not only look good but also enable powerful data exploration and decision-making. Get ready to become a data storyteller!

Sand painting is a unique and valuable art form, but it’s often constrained by physical equipment and a steep learning curve. To address these challenges, we developed SandTouch, a novel VR sand painting system that offers an immersive and intuitive experience, closely mirroring real-world sand manipulation. This project was presented at CHI 2025 (the premier conference in Human-Computer Interaction).

Key Features & Contributions:

1. Realistic Hand-Sand Interaction

We designed SandTouch to create a highly realistic and natural interaction between the user’s hands and virtual sand, allowing direct manipulation without external devices or controllers. This approach provides an intuitive, device-free interface and emphasizes the authenticity of the interaction. Our system captures fine sensations of real sand manipulation, complemented by realistic sound feedback.

2. AI Guidance for Creative Flow

A pioneering aspect of SandTouch is the integration of an AI agent, powered by a large language model (LLM). This AI intelligently interprets users’ creative intentions in real-time, offering contextually relevant artistic suggestions. This feature simplifies the creative process, enhances interactivity, and helps users (especially beginners) refine their artwork through technique recommendations and composition analysis.

3. Emotional Relief & Immersion

Beyond artistic creation, SandTouch prioritizes user well-being. It incorporates calming and responsive soundscapes that react to user gestures, reinforcing a relaxing atmosphere. Comprehensive evaluations demonstrated a significant increase in user engagement and immersion, with the realistic sound feedback enhancing emotional relief and deepening the painting experience. This highlights its potential for therapeutic applications.

Media & Resources:

Paper: Full Paper (ACM DL)
Video: Project Video

Did you find this page helpful? Consider sharing it 🙌

VR | Yang Gao

From Wrist to Finger: Hand Pose Tracking Using Ring-Watch Wearables

Project Overview (CHI 2025 Late-Breaking Work)

Key Innovations & Progress:

1. Novel Ring-Watch Wearable Design

2. Multi-Sensor Fusion for Precise Tracking

Media & Resources:

SandTouch: Empowering Virtual Sand Art in VR with AI Guidance and Emotional Relief

Project Overview

Key Features & Contributions:

1. Realistic Hand-Sand Interaction

2. AI Guidance for Creative Flow

3. Emotional Relief & Immersion

Media & Resources: