Wrapping Up: IR5′s Team Looks Back On Kiwi Katapult Revenge

April 23rd, 2013 by Rosie

This week marks the wrap up on Intel’s Ultimate Coder Challenge, a seven week showdown between the worlds top developers and agencies. Intel’s hand picked teams were given seven weeks to work with Intel’s new perceptual computing SDK to create a cutting edge experience. The team at Infrared5 worked to put together Kiwi Katapult Revenge, a one of a kind game that leverages Brass Monkey’s controller and head tracking technology to create an incredible interactive experience.

I spoke with each of our team members to hear about the ups and downs of creating this game under such a tight schedule.

What was the most fun and exciting thing about being a part of this competition?

Elena: The whole time that we were making this game I was amazed at the pace we were able to keep at. Maybe it was the thrill of the contest or the fact that we had such a short deadline, but we were able to cram what I thought would take months into a few weeks. I think the most fun aspect of it was the creativity we were able to have with the game. The encouraging feedback from the judges and other team members was really nice to hear too. It made working on the art for the game exciting as we kept improving it.

Chris: One of the most exciting things about this challenge was how our team collaborated on the concepts that were first brought to the table. I had come into it thinking of something very specific. I had envisioned a post apocalyptic driving/shooting game.

The thing was though, the art team had been thinking of something very different. They wanted to create a paper world, made up of items in the style of pop-up books and paper dolls. How could a Mad Max style game fit into this vision? I’m not quite sure how it came up, but I had seen several articles via Twitter that had gone viral on the subject of a New Zealand Ecologist that wanted to eradicate all the domestic cats in his country.  Apparently the cats were killing off the native species of flightless birds like the Kiwi. I brought up the concepts of using this as our theme, and it was something that resonated with everyone. Plus it would work perfectly with Rebecca and Aaron’s art style that they wanted to create. And as an added bonus the flat paper style had the advantage of really accentuating the 3D effect of peering around objects in the game. A little more brainstorming, a weekend writing up a crazy story, and Kiwi Katapult Revenge was born. In many ways, after looking back, it’s amazing how much of the core mechanics stayed exactly the same as what I had originally planned.

Steff: For me, the most exciting part was working with the Intel and OpenCV libraries and the new camera. I definitely got kind of a charge the first time I got a good image from the depth camera.

John: One of many fun features is seeing Kiwi in the rearview mirror.  The mirror itself is an organic shape so we used a render texture on the face of the mesh and this worked quite nicely.  However, dealing with where fire and lasers actually shoot from in reality, as opposed to how they look coming out of the model of the kiwi in the mirror, was a bit of a challenge.  We had to set up a separate camera for the render texture of course, but also had to assign separate features to show the fire coming out of the mouth and lasers out of the eyes so that it would look consistent.  On top of all of that, we had to then deal with the parenting and rotation of the head now that Steff’s code was coming in to deal with the main camera’s view matrix.  I think the result looks convincing and the game is certainly fun!”

Aaron: Being a fulfillment house, it’s not very often that we get to just roll with our own idea. Even though it wasn’t exactly a blank check, it was fun to explore a style not often used in modern 3D games.

Rebecca: The most exciting aspect was getting to have complete creative freedom with the concept through to the final product. I really enjoyed being able to have the paper cut out style implemented on a project after so many discussion with Aaron about that idea. I am just so happy with the end results. The creative team worked positively together to ensure that this style worked well and looked amazing. Really proud of it!

Like all truly exciting projects, Kiwi Katapult Revenge came with its share of questions and struggles. I asked the team to tell me what the most challenging part of their experience was.

Elena: The most challenging aspect for me was diving straight to work with only some idea of how things were going to turn out. I certainly learned a lot more about 3D work and Unity than I had known previously. The project required a lot of flexibility as things kept changing. In the end, I’m still happy with how it came out.

Chris: I think the biggest challenge was retaining maximum flexibility while keeping some semblance of a process in place. As any game designer knows, what you think will be fun is often not the case, and you have to go through a highly iterative process of trying things out. Doing that in a short seven week time frame posed many challenges, especially when we were dealing with cutting edge technology like the perceptual computing depth camera and computer vision algorithms. I do think our team managed quite well even though the experience was incredibly chaotic. I also admit that my style of always pushing and never quite being satisfied was probably a bit difficult to work with. In the end though, we made an incredible game. It’s something that’s fun to play, and has quite a bit of depth for being built in such a short time period. The game is far from complete however, and we look forward to building on it further in the near future.

Steff: When we had to dump all the code in Unity dealing with the camera because our C# port of OpenCV was not going to work and I had to essentially restart the project from scratch in Visual Studio. That was daunting!

John: The game has some unique control features that include flight with relative head rotation for firing/aiming.  In and of itself, this is basic gaming, but combined with the perceptual camera code which worked with the main camera’s view matrix, we had our work cut out for us.  On top of that, since the typical renderer skybox feature didn’t work when you changed out the view matrix in Unity at runtime, we had to create a simple cube that follows the player around on x and z axis (simple and effective) and thankfully, we didn’t hit too many other roadblocks because of the direct access to the main camera’s view matrix.

Rebecca: The most challenging aspect was working on a complete game in such a short time frame. We only had 7 weeks from beginning to end. In that time, we had to have a game concept created, design a 3D world, create numerous characters, develop a full featured game and integrate perceptual computing and Brass Monkey. It was a large task and the team had a few bumps along the road. However, we all persevered and managed to get it done. Lots of lessons learned. :)

Here at Infrared5, we would like to congratulate everyone that participated in the Ultimate Coder Challenge. It has been an amazing ride!

Infrared5 Participating in the Intel Ultimate Coder Challenge

April 18th, 2013 by admin

Perceptual Computing Challenge Winner to be Announced Next Week

Boston, MA – April 18, 2013 – Infrared5, an innovative Boston-based interactive studio, today
announced that the company is one of only seven developer teams participating in the Intel® Ultimate
Coder Challenge, and the only East Coast team.

The seven teams used the latest Intel Ultrabook™ devices, the Intel Perceptual Computing Software
Developers Kit (SDK), and Creative® Interactive Gesture Camera to build a special application prototype
in just seven weeks. The Ultimate Coder teams, including the Infrared5 team, blogged about their
experiences online and a panel of experts will crown the Ultimate Coder on April 24.

The Intel Ultimate Coder Challenge is designed to promote the use of perceptual computing. Perceptual
computing creates more immersive software applications by incorporating 2D/3D object tracking, user
gesture control, facial analysis and recognition, eye control, and voice recognition. Perceptual computing
is more than just the use of touch screens; it is a new area of pioneering development.

“The Intel Ultimate Coder Challenge provides a great opportunity for participants to innovate around
perceptual computing, and learn from peers in a few short weeks,” said Bob Duffy, community manager
for the Intel Developer Zone at Intel.

“We’re very comfortable with pioneering applications of technology,” said Rebecca Smith Allen, Infrared5
CEO and part of the Challenge team. “Perceptual computing is a new frontier we’re excited to explore.”

“The combination of the immersive properties of Brass Monkey and the power of perceptual computing
allowed our team to give the judges a unique experience that will hopefully earn us the Ultimate Coder
title,” said Chris Allen, the project leader. Allen is CEO of BrassMonkey, a next generation browser-based
game system created by the Infrared5 team.

In this game, the player uses a phone to fly a character around a 3D world. When game players turn
their head, the perceptual computing camera tracks what the player is looking at, causing the scene to
shift. This effect allows the player to peer around obstacles, giving the game a real sense of depth and

, , , , , ,

Mass High Tech’s 2013 Women to Watch

April 17th, 2013 by admin

More than ever, women are leading the way in the latest advancements in technology and life sciences. We are extremely proud to announce that our very own Rebecca Smith Allen has been featured as one of Mass High Tech’s 2013 Women to Watch. This list honors women in tech and life sciences who are at the forefront of their field, and shaping the future of these developing industries. Rebecca is in good company, sharing the list with women such as Susan Bornstein of Pfizer and MaryAlice Brady of MosaicHub. We would like to thank Mass High Tech and congratulate all of the inspiring women who made the list.

, , ,

Infrared5 Ultimate Coder Week Seven: The Final Countdown

April 10th, 2013 by admin

This the final week in the contest, and we think that we’ve got a pretty solid game out of the six weeks we’ve been at this. This week we will focus on adding that little bit of polish to the experience. We know that Nicole suggested not adding more features, but we simply couldn’t resist. Our character Karl really needed to be able to breathe fire properly, and for the player to truly experience this we felt it necessary to control it with your voice. So, to breathe fire in Kiwi Catapult Revenge you can choose to yell “aaaaaahhhhh”, or “firrrrrrrre”, and the bird will throw flames from his mouth. This feature allows the player to also be able to shoot lasers at the same time doubling your fire power. Beware of timing though, as currently there’s a slight delay. We plan on optimizing this feature as much as we can before our Friday deadline.

We’ve been playing with “mouth open” detection from the perceptual computing camera as well, but the funny thing is that it might have been a lot of work for not much gain. We found that using audio detection was a bit more interesting, and we still have more polish on mouth detection to make it really work well. That said, Steff gave a little overview of the technique we are using for mouth detection in this week’s video.

We also have some words from our protagonist Karl Kiwi, shots from GDC, footage of fire breathing via voice and more insight from our team on how things have gone during the Ultimate Coder contest.

Optimizations Galore

The game we’ve produced is really taxing the little Lenovo Yoga Ultrabook we were given for the contest. The integrated graphics and the overall low power on this little guy doesn’t allow us to do too much while running the perceptual computing algorithms. What runs great on our development PCs really kills the Ultrabook, so now we are optimizing as much as we can. Last week we showed some of the techniques Steff came up with to fine tune the computer vision algorithms so that they are highly optimized, but we didn’t talk much about the other things we are doing to make this game play well on such a low powered device.

Unity is a great IDE to create games like this. It’s not without its struggles, of course, but considering what we’re trying to accomplish, it has to be the best development environment out there. The world (New Zealand) is constructed using Unity’s terrain editor, and the 3D assets were created largely in Blender (we love open source software!). We’ve been able to gain performance with tools like Beast lightmapping that allow us to have a rich looking scene with nice baked shadows and shading. We’ve had some decent hurdles with multiple cameras for the mainview, UniSWF gui and the rearview mirror (render to texture to accommodate organic shape of mirror object), but we’ve been able to handle this just fine. Most of the optimizations, so far, have been concerning draw calls and lighting issues. We typically build apps for iOS/Android, so we tend to keep code/assets lean from the get go. Still, we’ve got a bit more to go before we hand Kiwi Katapult Revenge to the judges.

This week’s Web Build

We are excited to share with you the latest build of the game. Since this is a web build it won’t allow you to use the perceptual computing camera, but we are building an installer for that piece which will be delivered to the judges at the end of the week. With this build you can still fly around using your phone as the controller via the Brass Monkey app for Android or iOS, and you can breath fire by yelling. There are new updates to the UI and environment, and overall the only thing that’s missing from this build are powerups and the perceptual computing camera input. We will have three kinds of power ups for the final build of the game that we deliver on Friday.

Check out the playable demo of Kiwi Catapult Revenge!

What’s Next for Karl Kiwi and the Intel Perceptual Computing SDK?

We really had a great time using the new hardware and SDK from Intel and we are definitely going to keep building on what we have started (remember, we did promise to release this code as open source). We have some optimization to do (see above). And looking back at our plan from the early weeks of the competition, we were reaching for robust feature tracking to detect if the mouth was open or closed, the orientation of the head, and the gaze direction right from the pupils. All three of the above share the same quality that makes them difficult: in order for specific feature tracking to work with the robustness of a controller in real time, you need to be confident that you are locked onto each feature as the user moves around in front of the camera. We have learned that finding trackable points and tracking them from frame to frame does not enable you to lock onto the targeted feature points that you would need to do something like gaze tracking. As the user moves around, the points slide around. How to overcome the lack of confidence in the location of the trackable points? Active Appearance Model (AAM).

So, next steps are to see what kind of a boost we get on our face detection and head tracking using the GPU methods built into OpenCV. HaarCascades, feature detection and optical flow should all benefit from utilizing the GPU. Then, we are going to implement AAM with and without face training to get a good lock on the mouth and on the eyes. The idea behind implementing AAM without face training (or a calibration step) is to see if how good it works without being trained to a specific face in the hope that we can avoid that step so people can just jump in and play the game. With a good lock on the face features using AAM, we will isolate the pupil locations in the eyes and calculate if the gaze vector intersects with the screen and viola! Robust gaze tracking!

Where can you play the game once it’s done? Well we’ve not yet decided on how we want to distribute the game, and there are a ton more features that we would like to add before we are ready for a production release. We are considering putting it on Steam Greenlight. So, with that, you will have to wait a little while before you can play Kiwi Katapult Revenge in all its glory. Let us know your thoughts, how would you distribute such a crazy game if it were up to you?

Parting Words

This has been a great experience for our whole team, and we want to thank Intel for having us be a part of the contest. We really like the game we came up with, and are excited to finish it and have it available for everyone to play. Thanks too to the judges for all the time and effort you’ve given us. Your feedback has only made our game better. Now best of luck choosing a winner! For those of you that have been following us during the contest, please do stay in touch and follow us on our own blog. From the team at Infrared5, over and out!

Infrared5 Ultimate Coder Week Six: GDC, Mouth Detection Setbacks, Foot Tracking and Optimizations Galore

April 3rd, 2013 by Chris Allen

For week six Chris and Aaron made the trek out to San Francisco to the annual Game Developers Conference (GDC) where they showed the latest version of our game Kiwi Catapult Revenge. The feedback we got was amazing! People were blown away at the head tracking performance that we’ve achieved, and everyone absolutely loved our unique art style. While the controls were a little difficult for some, that allowed us to gain some much needed insight into how to best fine tune the face tracking and the smartphone accelerometer inputs to make a truly killer experience. There’s nothing like live playtesting on your product!

Not only did we get a chance for the GDC audience to experience our game, we also got to meet some of the judges and the other Ultimate Coder competitors. There was an incredible amount of mutual respect and collaboration among the teams. The ideas were flowing on how to help improve each and every project in the competition. Chris gave some tips on video streaming protocols to Lee so that he will be able to stream over the internet with some decent quality (using compressed JPEGs would have only wasted valuable time). The guys from Sixense looked into Brass Monkey and how they can leverage that in their future games, and we gave some feedback to the Code Monkeys on how to knock out the background using the depth camera to prevent extra noise that messes with the controls they are implementing. Yes, this is a competition, but the overall feeling was one of wanting to see every team produce their very best.

The judges also had their fair share of positive feedback and enthusiasm. The quality of the projects obviously had impressed them, to the point that Nicole was quoted saying “I don’t know how we are going to decide”. We certainly don’t envy their difficult choice, but we don’t plan on making it any easier for them either. All the teams are taking it further and want to add even more amazing features to their applications before the April 12th deadline.

The staff in the Intel booth were super accommodating, and the exposure we got by being there was invaluable to our business. This is a perfect example of a win-win situation. Intel is getting some incredible demos of their new technology, and the teams are getting exposure and credibility by being in a top technology company’s booth. Not only that, but developers now get to see this technology in action, and can more easily discover more ways to leverage the code and techniques we’ve pioneered. Thank you Intel for being innovative and taking a chance on doing these very unique and experimental contests!

While Aaron and Chris were having a great time at GDC the rest of the team was cranking away. Steff ran into some walls with mouth detection for the breathing fire controls, but John, Rebecca and Elena were able to add more polish to the characters, environment and game play.

John added on a really compelling new feature – playing the game with your feet! We switched the detection algorithm so that it tracks your feet instead of your face. We call it Foot Tracking. It works surprisingly well, and the controls are way easier this way.

Steff worked on optimizing the face tracking algorithms and came up with some interesting techniques to get the job done.

This week’s tech tip and code snippet came to us during integration. We were working hard to combine the head tracking with the Unity game on the Ultrabook, and ZANG we had it working! But, there was a problem. It was slow. It was so slow it was almost unplayable. It was so slow that it definitely wasn’t “fun.” We had about 5 hours until Chris was supposed to go to the airport and we knew that the head tracking algorithms and the camera stream were slowing us down. Did we panic? (Don’t Panic!) No. And you shouldn’t either when faced with any input that is crushing the performance of your application. We simply found a clever way to lower the sampling rate but still have smooth output between frames.

The first step was to reduce the number of times we do a head tracking calculation per second. Our initial (optimistic) attempts were to update in realtime on every frame in Unity. Some computers could handle it, but most could not. Our Lenovo Yoga really bogged down with this. So, we introduced a framesToSkip constant and started sampling on every other frame. Then we hit a smoothing wall. Since the head controls affect every single pixel in the game window (by changing the camera projection matrix based on the head position), we needed to be smoothing the head position on every frame regardless of how often we updated the position from the camera data. Our solution was to sample the data at whatever frame rate we needed to preserve performance, save the head position at that instant as a target, and ease the current position to the new position on every single frame. That way, your sampling rate is down, but you’re still smoothing on every frame and the user feels like the game is reacting to their every movement in a non-jarring way. (For those wondering what smoothing algorithm we selected: Exponential Smoothing handles any bumps in the data between frames.) Code is below.

Feeling good about the result, we went after mouth open/closed detection with a vengeance! We thought we could deviate from our original plan of using AAM and POSIT, and lock onto the mouth using a mouth specific Haarcascade on the region of interest containing the face. The mouth Haarcascade does a great job finding and locking onto the mouth if the user is smiling – which is not so good for our purposes. We are still battling with getting a good lock on the mouth using a method that combines depth data with RGB, but we have seen why AAM exists for feature tracking. It’s not just something you can cobble together and have confidence that it will work well enough to act as an input for game controls.

Overall, this week was a step forward even with part of the team away. We’ve got some interesting and fun new features that we want to add as well. We will be sure to save that surprise for next week. Until then, please let us know if you have any questions and/or comments. May the best team win!

, , , , , , ,