HoloHand: Bidirectional Motion–Language Modeling for Semantic Hand Interaction in Immersive Environments

Sep 30, 2026·
Yingjing Xiao
,
Wolin Liang
,
Zhengte Cai
,
Yang Gao
,
Di Wu
,
Zhanpeng Jin
HoloHand describes a two-handed cup-opening action and generates a ghost-hand preview from a language instruction. Figure 1 from the paper.
Abstract
HoloHand connects hand motion and natural language in both directions for mixed-reality interaction. It describes observed hand actions in terms of intent and coordination, and turns language-level intentions into ghost-hand previews. Discrete motion tokenization and a shared motion–language interface support semantic interpretation and feedback in an MR prototype.
Type
Publication
Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (PACM IMWUT / UbiComp 2026)

† Equal contribution: Yingjing Xiao and Wolin Liang.