HoloHand: Bidirectional Motion–Language Modeling for Semantic Hand Interaction in Immersive Environments
Figure 1 from the paper.Abstract
HoloHand connects hand motion and natural language in both directions for mixed-reality interaction. It describes observed hand actions in terms of intent and coordination, and turns language-level intentions into ghost-hand previews. Discrete motion tokenization and a shared motion–language interface support semantic interpretation and feedback in an MR prototype.
Type
Publication
Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (PACM IMWUT / UbiComp 2026)
† Equal contribution: Yingjing Xiao and Wolin Liang.