Kinect Cartwheel Training

Finally, something concrete to be released. I've created my own Kinect Unity plugin and some utilities, and this prototype will act as a starting point for Leo Holsti, a circus artist / programmer who will join our group in June. A big thanks to Aki Martikainen for the comments on the visuals!

We will be investigating what kind of new digital sports and sports training environments we can create to make more people interested in sports and provide new kinds of challenges and fun. In addition to Kinect(s), we'll probably use floor projections, ropes, trampolines, trampoline projections, auditory feedback etc. The project continues my older work on motor skill training, visualization and computer vision. As shown in the video below, I was exploring how to make the learning cycle (perform, evaluate, improve) faster with computer generated feedback.

Lessons learned from the cartwheel training prototype:

• Clear numerical feedback indeed seems to motivate training, but as always, gamification can lead to exploitation. The simple flatness scoring can be gamed to some extend by keeping one's feet tucked in and not straight. The straightness of feet should be included in the score, but at the moment it's difficult to implement because the OpenNI/NITE tracking breaks when both feet are off the ground. The flatness scoring works without the tracker because it's computed directly from the user pixels (fortunately, the OpenNI/NITE user blob extraction doesn't break during a cartwheel.) I hope I have time to implement my own tracker or find some other tracker with an easy to use pipeline for defining the range of tracked moves. Totally unlimited tracking/motion capture using a single depth sensor is still a difficult problem even offline, let alone in real time.
• Floor graphics and shadows are more clear than separate windows/views and can be perceived while also watching one's movement. We are of course tuned to sense our surroundings based on shadow and lighting. Projecting graphics on the floor in the real world would probably be even better than the virtual floor graphics, but they would make downloading and trying out the software much more tedious. I tried having a bird's eye camera as a separate window, but it felt confusing, especially since the user mesh is actually just a thin layer lacking the back of the body.
• I also tried making the experience more gamelike by having walls between which you must roll with the cartwheel, however the lack of tactile feedback and stereo vision made it almost impossible to estimate one's precise position in relation to the walls.

One thing I'm not sure about yet is whether the user should explicitly start recording/analysis with a gesture or if the system should automatically recognize the start of the movement, or if there even should be separate recording and visualization modes. Depends on the case. Automatic recognition and mode change is handy but only if it is really robust. It also limiting to have separate modes for each move. For freeform training of gymnastics and tricking, the system should perhaps constantly capture all sorts of metrics, recognize the moves and what metrics are relevant to each move, and display the info for the latest chain of moves when the user stops to look at the screen(s). When the user is moving, most of the screen should probably be dedicated to the live camera view, although it seems that one can simultaneously move and look at the screen only in some movements. However, auditory feedback is probably an entirely different story.

You can download the PC prototype (CartwheelTrainer.zip) from my Google Drive. If you don't have Kinect stuff installed, also run the OpenNI, NITE, sensor and SensorKinect installers (.msi) from the same folder. I admit it's quite a hassle, but I don't have the time to learn to make a proper installer right now.

Edit 5th June 2012: embedded the video mirrors video as well.