Trailer

Learnings

The main goal of this project is to explore the potential of using machine learning to create game animation directly from video footage, without the burden of motion trackers. Is it possible to import video footage, using machine learning to output smooth game animation directly attached to a 3D model? The short answer is maybe. The long answer is it has great potential, but the current machine learning model has lots of limitations in what type of footage can be processed well.

After researching multiple models, we chose OpenPose for its accuracy.

This is our initial planned process. In the end, we had to manually adjust the animation due to model limitations.

The primary takeaway from this project revolves around the considerations when working with ML technology. Obtaining comprehensive depth information with ML models can be challenging, making it more suitable for 2D animation at the current stage. Additionally, ML technology lacks the ability to assess spine position accurately, which presents challenges when dealing with dances like the Paul Taylor Back Exercise that heavily focus on the spine. Furthermore, it is crucial to always be mindful of the training data used for ML models. Most available models are trained on ordinary individuals performing simpler dances. Therefore, when applied to professional modern dance routines, the animation results did not meet our expectations.

A valuable design lesson learned pertains to the difficulties people encounter when learning mirrored movements. While we made efforts to address this issue in our design, it remains a challenging skill to master completely.

Please see more details on the project website.