HCI | Yang Gao

From Wrist to Finger: Hand Pose Tracking Using Ring-Watch Wearables

Fri, 25 Apr 2025 00:00:00 +0000

Project Overview (CHI 2025 Late-Breaking Work)

Hand pose tracking is a cornerstone for advancing human-computer interaction applications, from virtual reality to prosthetics control. However, existing vision-based systems and wearable devices often face limitations in portability, usability, and practicality. In this CHI 2025 Late-Breaking Work, we introduce a novel multimodal hand pose tracking framework that integrates data from an IMU-equipped ring and EMG sensors embedded in a wrist-worn device.

This work represents an important step towards enabling more practical and accessible hand tracking solutions for everyday use.

Key Innovations & Progress:

1. Novel Ring-Watch Wearable Design

We propose a compact and ergonomic wearable system that combines a single IMU-equipped ring (worn on the thumb) with EMG sensors integrated into a smartwatch. This design prioritizes wearability and comfort, addressing the common limitations of bulky or multi-device tracking systems while accurately capturing intricate finger and hand motion data.

2. Multi-Sensor Fusion for Precise Tracking

Our framework leverages the complementary strengths of both motion dynamics (from the IMU) and muscle activity (from the EMG sensors). This deep learning-based sensor fusion approach achieves precise 3D hand pose reconstruction, providing robust performance even in complex or high-speed gestures. We developed a transformer-based model with time encoding and cross-modal attention mechanisms for optimal data integration.

Media & Resources:

Paper: Extended Abstract (ACM DL)
Video: Project Video

Did you find this page helpful? Consider sharing it 🙌

SandTouch: Empowering Virtual Sand Art in VR with AI Guidance and Emotional Relief

Fri, 25 Apr 2025 00:00:00 +0000

Project Overview

In today’s data-rich world, simply having data isn’t enough – we need to understand it, communicate it, and extract insights from it. This course empowers you to transform raw numbers and complex datasets into compelling, insightful visual narratives. We’ll blend art with science, learning how to design interactive visualizations that not only look good but also enable powerful data exploration and decision-making. Get ready to become a data storyteller!

Sand painting is a unique and valuable art form, but it’s often constrained by physical equipment and a steep learning curve. To address these challenges, we developed SandTouch, a novel VR sand painting system that offers an immersive and intuitive experience, closely mirroring real-world sand manipulation. This project was presented at CHI 2025 (the premier conference in Human-Computer Interaction).

Key Features & Contributions:

1. Realistic Hand-Sand Interaction

We designed SandTouch to create a highly realistic and natural interaction between the user’s hands and virtual sand, allowing direct manipulation without external devices or controllers. This approach provides an intuitive, device-free interface and emphasizes the authenticity of the interaction. Our system captures fine sensations of real sand manipulation, complemented by realistic sound feedback.

2. AI Guidance for Creative Flow

A pioneering aspect of SandTouch is the integration of an AI agent, powered by a large language model (LLM). This AI intelligently interprets users’ creative intentions in real-time, offering contextually relevant artistic suggestions. This feature simplifies the creative process, enhances interactivity, and helps users (especially beginners) refine their artwork through technique recommendations and composition analysis.

3. Emotional Relief & Immersion

Beyond artistic creation, SandTouch prioritizes user well-being. It incorporates calming and responsive soundscapes that react to user gestures, reinforcing a relaxing atmosphere. Comprehensive evaluations demonstrated a significant increase in user engagement and immersion, with the realistic sound feedback enhancing emotional relief and deepening the painting experience. This highlights its potential for therapeutic applications.

Media & Resources:

Paper: Full Paper (ACM DL)
Video: Project Video

Did you find this page helpful? Consider sharing it 🙌

Data Visualization & Interaction (Offered Every Fall)

Sun, 01 Sep 2024 00:00:00 +0000

In our increasingly data-rich world, simply having data isn’t enough – we need to understand it, communicate it effectively, and extract actionable insights from it. This course empowers you to transform raw numbers and complex datasets into compelling, insightful visual narratives. We’ll blend the art of design with the science of data, learning how to create interactive visualizations that not only look good but also enable powerful data exploration and decision-making. Get ready to become a data storyteller!

What You’ll Explore:

1. Visual Perception & Cognition 👁️🧠

The human element is key! We’ll start by understanding how our eyes and brains process visual information, exploring principles of pre-attentive attributes, Gestalt psychology, and cognitive biases. This lays the groundwork for designing effective and intuitive visualizations.

2. Data Fundamentals & Preparation 🧹

Before we visualize, we clean and structure! Learn about different data types (quantitative, categorical, ordinal), data quality issues, and essential techniques for wrangling, cleaning, and transforming your data to make it suitable for visualization.

3. Visualizing Spatial Data 🗺️

Explore techniques for representing geographic locations and spatial relationships. From simple scatter plots on maps to choropleth maps, heatmaps, and flow maps, we’ll learn to convey location-based insights effectively.

4. Visualizing Tree Structures 🌲

Dive into hierarchical data! Learn how to effectively visualize relationships and structures in organizational charts, file systems, category taxonomies, and other tree-like datasets using methods like treemaps, sunburst charts, and node-link diagrams.

5. Visualizing Graph Structures (Networks) 🕸️

Uncover connections! Master techniques for visualizing complex networks, social graphs, co-occurrence relationships, and other graph data to reveal hidden patterns, communities, and central nodes.

6. Visualizing Time-Varying Data 📈

See trends unfold! Learn how to effectively represent changes over time, from traditional line charts and area charts to more advanced streamgraphs, sparklines, and animated timelines that show evolution and patterns over temporal dimensions.

7. Visualizing Text Data 💬

Transform words into insights! Explore methods for visualizing textual information, including word clouds, n-gram charts, sentiment analysis visualizations, topic models, and network graphs derived from text.

8. Geospatial Information Systems (GIS) & Advanced Mapping 🌍

Beyond basic maps, delve into the power of GIS for analyzing and visualizing complex geographical datasets with real-world applications in urban planning, environmental science, and public health.

9. Visualizing High-Dimensional Data ✨

Tame complexity! Learn strategies to visualize datasets with many attributes (dimensions), employing techniques like parallel coordinates, scatter plot matrices, dimensionality reduction (e.g., PCA, t-SNE), and glyph-based representations.

10. Interactive Views & Dashboard Design 🕹️

Bring your data to life! Explore principles of interactive design, enabling users to filter, zoom, pan, drill down, and explore data dynamically. We’ll also cover best practices for designing effective and user-friendly dashboards.

By the end of this course, you’ll not only be able to create beautiful and effective visualizations but also to communicate complex data stories with clarity and empower users to gain actionable insights. This is a critical skill for anyone navigating our data-driven world!

Did you find this page helpful? Consider sharing it 🙌