SandTouch: Empowering Virtual Sand Art in VR with AI Guidance and Emotional Relief

Fri, 25 Apr 2025 00:00:00 +0000

Project Overview

In today’s data-rich world, simply having data isn’t enough – we need to understand it, communicate it, and extract insights from it. This course empowers you to transform raw numbers and complex datasets into compelling, insightful visual narratives. We’ll blend art with science, learning how to design interactive visualizations that not only look good but also enable powerful data exploration and decision-making. Get ready to become a data storyteller!

Sand painting is a unique and valuable art form, but it’s often constrained by physical equipment and a steep learning curve. To address these challenges, we developed SandTouch, a novel VR sand painting system that offers an immersive and intuitive experience, closely mirroring real-world sand manipulation. This project was presented at CHI 2025 (the premier conference in Human-Computer Interaction).

Key Features & Contributions:

1. Realistic Hand-Sand Interaction

We designed SandTouch to create a highly realistic and natural interaction between the user’s hands and virtual sand, allowing direct manipulation without external devices or controllers. This approach provides an intuitive, device-free interface and emphasizes the authenticity of the interaction. Our system captures fine sensations of real sand manipulation, complemented by realistic sound feedback.

2. AI Guidance for Creative Flow

A pioneering aspect of SandTouch is the integration of an AI agent, powered by a large language model (LLM). This AI intelligently interprets users’ creative intentions in real-time, offering contextually relevant artistic suggestions. This feature simplifies the creative process, enhances interactivity, and helps users (especially beginners) refine their artwork through technique recommendations and composition analysis.

3. Emotional Relief & Immersion

Beyond artistic creation, SandTouch prioritizes user well-being. It incorporates calming and responsive soundscapes that react to user gestures, reinforcing a relaxing atmosphere. Comprehensive evaluations demonstrated a significant increase in user engagement and immersion, with the realistic sound feedback enhancing emotional relief and deepening the painting experience. This highlights its potential for therapeutic applications.

Media & Resources:

Paper: Full Paper (ACM DL)
Video: Project Video

Did you find this page helpful? Consider sharing it 🙌

PressInPose: Integrating Pressure and Inertial Sensors for Full-Body Pose Estimation in Activities

Thu, 21 Nov 2024 00:00:00 +0000

Project Overview

Accurate human body posture assessment through wearable technology has significant implications across various fields, including sports science, clinical diagnostics, rehabilitation, and VR interaction. Traditional methods often face limitations due to complex setups or environmental constraints. To address these challenges, we developed PressInPose, an innovative system that integrates pressure and inertial sensors for precise full-body pose estimation in dynamic activities. This work was published in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT) and will be presented at UbiComp 2025.

Key Innovations & Contributions:

1. Novel Multi-Sensor Fusion

PressInPose employs an advanced shoe insole embedded with pressure sensors and an Inertial Measurement Unit (IMU), coupled with a single wrist-mounted IMU. This unique multi-modal sensor fusion approach allows for a comprehensive analysis of human biomechanics, capturing intricate body dynamics that traditional single-sensor systems often miss.

2. LLM-Powered Virtual Data Augmentation

To enhance the robustness and generalization of our system, we leveraged large language models (LLMs) to generate virtual human motion sequences. These sequences were utilized to create synthetic IMU data for data augmentation, effectively addressing the challenge of limited real-world data availability and variability, especially for complex and dynamic movements.

3. Physical Kinematics Modeling & Deep Learning Network

Our approach uniquely combines physical kinematics modeling based on pressure data with a multi-region human posture estimation network. This integration allows PressInPose to accurately capture interactions and dependencies between different body parts, leading to superior accuracy in pose reconstruction.

Media & Resources:

Paper: Full Paper (ACM DL)

Did you find this page helpful? Consider sharing it 🙌

AI | Yang Gao

SandTouch: Empowering Virtual Sand Art in VR with AI Guidance and Emotional Relief

Project Overview

Key Features & Contributions:

1. Realistic Hand-Sand Interaction

2. AI Guidance for Creative Flow

3. Emotional Relief & Immersion

Media & Resources:

PressInPose: Integrating Pressure and Inertial Sensors for Full-Body Pose Estimation in Activities

Project Overview

Key Innovations & Contributions:

1. Novel Multi-Sensor Fusion

2. LLM-Powered Virtual Data Augmentation

3. Physical Kinematics Modeling & Deep Learning Network

Media & Resources: