Hand tracking for surgical skills... with just a webcam

Motion capture systems are notoriously complex. I'm on a quest to try to achieve hand tracking with as minimalistic of a setup as possible; ideally, a single camera from the optimal vantage point – the surgeon's head. Here's my progress so far.

For the uninitiated, please see my prior work here on the importance of the head-mounted camera setups and the first-person perspective for capturing operative video.

This setup currently is not fully head-mounted, but does work with the overhead camera on the boom arm.

Codepen: https://codepen.io/mediapipe/pen/RwGWYJw

The big picture of what's next for this project:

• Learn how to save / export data after capture
• Figure out how to analyze
• Capture various skill levels: no experience, novice, intern, chief, attending, 'master'
• Figure out how to predict experience level given a new recording