Multimodal LLM | Yang Gao

Two New Papers Accepted to ACM IMWUT 2026

Sat, 04 Jul 2026 00:30:00 +0800

Table of Contents

Overview

We are excited to share that two of our papers have been accepted to Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (ACM IMWUT) and will be presented at UbiComp 2026.

HoloHand: Bidirectional Motion–Language Modeling for Semantic Hand Interaction in Immersive Environments
EmbodiedRecall: A Ring-to-Glasses System for Preserving Valuable, Fleeting Moments in Daily Activities

More details will be added as the final publication information and conference program become available.

HMotionGPT Accepted to ACM IMWUT 2026

Wed, 17 Jun 2026 00:00:00 +0000

Table of Contents

Overview

We are excited to share that our paper, HMotionGPT: Aligning Hand Motions and Natural Language for Activity Understanding with Smart Rings, has been accepted to Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (ACM IMWUT).

The paper will be presented at UbiComp 2026 in Shanghai, China.

HMotionGPT studies how smart-ring motion signals can be aligned with natural language to support hand-centric activity understanding, including activity classification, motion captioning, and instruction-following interaction with wearable sensing data.

Publication page: HMotionGPT

Project page: HMotionGPT Open-Source Project

HMotionGPT: Aligning Hand Motions and Natural Language for Activity Understanding with Smart Rings

Tue, 07 Apr 2026 00:00:00 +0000

Project Overview

HMotionGPT is an open-source multimodal framework that bridges smart-ring inertial measurement unit (IMU) signals and natural language for hand-centric activity understanding. By aligning motion representations with a language model, HMotionGPT supports downstream tasks including activity classification, motion captioning, and instruction-following activity understanding.

The project accompanies our paper, HMotionGPT: Aligning Hand Motions and Natural Language for Activity Understanding with Smart Rings, published in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (PACM IMWUT / UbiComp 2026).

Open-Source Release

The public repository provides the core model construction and training pipeline under the MIT License, including:

Stage-1 alignment training for the IMU projector
Stage-2 supervised fine-tuning with a frozen projector
Configurable language-model backbones
Minimal runnable examples for smoke testing

Key Contributions

Motion-Language Alignment

HMotionGPT converts hand-motion sequences captured by smart rings into representations that can be aligned with a large language model, connecting wearable sensing with flexible natural-language reasoning.

Two-Stage Training Pipeline

The framework first trains an IMU projector while freezing the language model, then performs supervised fine-tuning to improve instruction following and task-specific generation.

Hand-Centric Activity Understanding

The aligned model supports several forms of activity understanding, from recognizing actions to producing natural-language descriptions of hand-object interactions.

Resources

Open-Source Code: SCUT-HAI/HMotionGPT on GitHub
Paper: ACM Digital Library
License: MIT License