VR-based Simulation for Clinical Training

An AI agent-driven platform integrating immersive VR simulation for automated debriefing and comprehensive learning analytics, alongside adaptive instruction. Our AI agent system supports moment-by-moment debriefing, delivers comprehensive learning analytics, and provides adaptive, personalized feedback with explainable reasoning to enhance clinical competency, decision-making, and team performance across diverse VR-based clinical task trainings.

🎯 Adaptive Learning 🧠 AI-Powered Analysis 📊 Real-time Feedback 🔬 Evidence-Based 💡 Explainable AI 🎓 Personalized Instruction
Demo

Module 1

The AI-Driven Debriefing System

A human instructor-like AI agent for post-simulation debriefing. By tracking multimodal VR traces (e.g., actions, conversations, and gaze patterns) from VR training, one AI instructor agent works like a human instructor: it presents evidence and prompts the trainee through guided questions to uncover what they were thinking when they took action, using evidence-based delivery to facilitate the debriefing process.

Key Capabilities:

  • Evidence-Based Prompting: Uses captured actions, conversations, and gaze data to prompt debriefing discussions
  • Behavioral Evidence Presentation: Shows trainees what they did, said, and looked at during critical moments
  • Guided Reconstruction: Facilitates trainee-led reconstruction of clinical reasoning using multimodal evidence
  • Socratic Dialogue: Asks questions about observed behaviors to prompt reflection and self-analysis
  • Thought Process Articulation: Prompts trainees to explain their thinking behind observed actions and attention patterns

Module 2

GUIDE System

GUIDE (GUided Instruction for Diagnostic/Evaluative Collaborative Clinical Reasoning) System is an evidence-grounded human–AI platform for VR-based clinical simulation. It detects collaborative clinical reasoning (CCR) from multimodal VR traces and translates them into adaptive, personalized instructional guidance for trainees through an AI agentic workflow with complementary analytic methods.

Key Capabilities:

  • AI Agentic Workflow: LLM-based agents combined with statistical and ML analytics interpret multimodal VR traces in a traceable way; modality-specialized agents and a fusion coordinator detect CCR episodes, characterize competence, and generate teachable moments with evidence and uncertainty estimates.
  • Instructional Translation: Converts detected CCR patterns into evidence-linked teachable moments and maps them to pedagogically appropriate responses delivered through the trainee-facing AI assistant.
  • Human–AI Collaboration: Expert-facing verification and authoring workspace lets clinicians and educators review outputs, refine rubrics, calibrate uncertainty, and validate feedback.
  • Multi-Channel Replay & Analytics: VR video, patient vitals, actions, conversations, gaze tracking; critical moment markers and timeline analysis.

AI-Driven Features

The AI-powered Clinical Simulation Platform supports advanced research in medical education by integrating cutting-edge AI capabilities with immersive VR technology. Our platform enables comprehensive data collection and analysis across multiple dimensions of learner performance and clinical reasoning.

Multimodal Analytics

Integration of verbal communication, physical actions, physiological data, and VR interaction patterns

Collaborative Learner Model

Continuous tracking of knowledge state, skill development, and cognitive patterns over time

Behavioral Sequence Mining

Pattern discovery in clinical decision-making sequences and temporal action analysis

Cognitive Load Assessment

Real-time evaluation of mental workload and attention allocation during critical tasks

Team Dynamics Analysis

Quantitative assessment of communication patterns, leadership, and collaboration effectiveness

Adaptive Feedback

Dynamic adjustment of instructional strategies based on individual learning trajectories

Longitudinal Tracking

Cross-session performance monitoring and competency progression analytics

Error Pattern Recognition

Automated identification and classification of clinical errors with evidence-based remediation

Supporting evidence-based research in medical education, simulation training, and AI-enhanced learning systems

Developed by the Technology & Innovation in Medical Education (TIME) Lab, University of Michigan.

AI Assistant