Infrared5 Ultimate Coder Week Six: GDC, Mouth Detection Setbacks, Foot Tracking and Optimizations Galore

April 3rd, 2013 by Chris Allen

For week six Chris and Aaron made the trek out to San Francisco to the annual Game Developers Conference (GDC) where they showed the latest version of our game Kiwi Catapult Revenge. The feedback we got was amazing! People were blown away at the head tracking performance that we’ve achieved, and everyone absolutely loved our unique art style. While the controls were a little difficult for some, that allowed us to gain some much needed insight into how to best fine tune the face tracking and the smartphone accelerometer inputs to make a truly killer experience. There’s nothing like live playtesting on your product!

Not only did we get a chance for the GDC audience to experience our game, we also got to meet some of the judges and the other Ultimate Coder competitors. There was an incredible amount of mutual respect and collaboration among the teams. The ideas were flowing on how to help improve each and every project in the competition. Chris gave some tips on video streaming protocols to Lee so that he will be able to stream over the internet with some decent quality (using compressed JPEGs would have only wasted valuable time). The guys from Sixense looked into Brass Monkey and how they can leverage that in their future games, and we gave some feedback to the Code Monkeys on how to knock out the background using the depth camera to prevent extra noise that messes with the controls they are implementing. Yes, this is a competition, but the overall feeling was one of wanting to see every team produce their very best.

The judges also had their fair share of positive feedback and enthusiasm. The quality of the projects obviously had impressed them, to the point that Nicole was quoted saying “I don’t know how we are going to decide”. We certainly don’t envy their difficult choice, but we don’t plan on making it any easier for them either. All the teams are taking it further and want to add even more amazing features to their applications before the April 12th deadline.

The staff in the Intel booth were super accommodating, and the exposure we got by being there was invaluable to our business. This is a perfect example of a win-win situation. Intel is getting some incredible demos of their new technology, and the teams are getting exposure and credibility by being in a top technology company’s booth. Not only that, but developers now get to see this technology in action, and can more easily discover more ways to leverage the code and techniques we’ve pioneered. Thank you Intel for being innovative and taking a chance on doing these very unique and experimental contests!

While Aaron and Chris were having a great time at GDC the rest of the team was cranking away. Steff ran into some walls with mouth detection for the breathing fire controls, but John, Rebecca and Elena were able to add more polish to the characters, environment and game play.

John added on a really compelling new feature – playing the game with your feet! We switched the detection algorithm so that it tracks your feet instead of your face. We call it Foot Tracking. It works surprisingly well, and the controls are way easier this way.

Steff worked on optimizing the face tracking algorithms and came up with some interesting techniques to get the job done.

This week’s tech tip and code snippet came to us during integration. We were working hard to combine the head tracking with the Unity game on the Ultrabook, and ZANG we had it working! But, there was a problem. It was slow. It was so slow it was almost unplayable. It was so slow that it definitely wasn’t “fun.” We had about 5 hours until Chris was supposed to go to the airport and we knew that the head tracking algorithms and the camera stream were slowing us down. Did we panic? (Don’t Panic!) No. And you shouldn’t either when faced with any input that is crushing the performance of your application. We simply found a clever way to lower the sampling rate but still have smooth output between frames.

The first step was to reduce the number of times we do a head tracking calculation per second. Our initial (optimistic) attempts were to update in realtime on every frame in Unity. Some computers could handle it, but most could not. Our Lenovo Yoga really bogged down with this. So, we introduced a framesToSkip constant and started sampling on every other frame. Then we hit a smoothing wall. Since the head controls affect every single pixel in the game window (by changing the camera projection matrix based on the head position), we needed to be smoothing the head position on every frame regardless of how often we updated the position from the camera data. Our solution was to sample the data at whatever frame rate we needed to preserve performance, save the head position at that instant as a target, and ease the current position to the new position on every single frame. That way, your sampling rate is down, but you’re still smoothing on every frame and the user feels like the game is reacting to their every movement in a non-jarring way. (For those wondering what smoothing algorithm we selected: Exponential Smoothing handles any bumps in the data between frames.) Code is below.

Feeling good about the result, we went after mouth open/closed detection with a vengeance! We thought we could deviate from our original plan of using AAM and POSIT, and lock onto the mouth using a mouth specific Haarcascade on the region of interest containing the face. The mouth Haarcascade does a great job finding and locking onto the mouth if the user is smiling – which is not so good for our purposes. We are still battling with getting a good lock on the mouth using a method that combines depth data with RGB, but we have seen why AAM exists for feature tracking. It’s not just something you can cobble together and have confidence that it will work well enough to act as an input for game controls.

Overall, this week was a step forward even with part of the team away. We’ve got some interesting and fun new features that we want to add as well. We will be sure to save that surprise for next week. Until then, please let us know if you have any questions and/or comments. May the best team win!

, , , , , , ,

Leave a Reply