Multithreading Perceptual Computing Applications in Unity3d

August 21st, 2013 by Steff Kelsey

Once again we are excited about being featured on the Intel Developer Zone blog for a post done by Infrared5 Senior Developer Steff Kelsey. This post Multithreading Perceptual Computing Application in Unity3d talks about the challenges the Infrared5 team faced while participating in the Intel Ultimate Coder Challenge. Follow the link to hear the steps we went through to make Kiwi Catapult Revenge a reality! As always, we love hearing your feedback and feel free to share with others.

, , , , ,

What our team at Infrared5 is watching in emerging technology?

July 10th, 2013 by Adam Doucette

Here at Infrared5, we know we have one of the best development teams in the business. Our team of designers and developers know the ins and outs of (and have even written textbooks on) programs like Flash, Red5, Unity, and many other programs we use on a daily basis. But being in an industry where technology changes often and quickly, our team realizes they have chosen careers that require lifelong learning and understand the need for adapting to new technologies constantly. This past week I decided to ask some of our team members what technology they are keeping their eyes on and are excited to learn more about.

Read the rest of this entry »

, , , , , , , , ,

Infrared5 Participating in the Intel Ultimate Coder Challenge

April 18th, 2013 by admin

Perceptual Computing Challenge Winner to be Announced Next Week

Boston, MA – April 18, 2013 – Infrared5, an innovative Boston-based interactive studio, today
announced that the company is one of only seven developer teams participating in the Intel® Ultimate
Coder Challenge, and the only East Coast team.

The seven teams used the latest Intel Ultrabook™ devices, the Intel Perceptual Computing Software
Developers Kit (SDK), and Creative® Interactive Gesture Camera to build a special application prototype
in just seven weeks. The Ultimate Coder teams, including the Infrared5 team, blogged about their
experiences online and a panel of experts will crown the Ultimate Coder on April 24.

The Intel Ultimate Coder Challenge is designed to promote the use of perceptual computing. Perceptual
computing creates more immersive software applications by incorporating 2D/3D object tracking, user
gesture control, facial analysis and recognition, eye control, and voice recognition. Perceptual computing
is more than just the use of touch screens; it is a new area of pioneering development.

“The Intel Ultimate Coder Challenge provides a great opportunity for participants to innovate around
perceptual computing, and learn from peers in a few short weeks,” said Bob Duffy, community manager
for the Intel Developer Zone at Intel.

“We’re very comfortable with pioneering applications of technology,” said Rebecca Smith Allen, Infrared5
CEO and part of the Challenge team. “Perceptual computing is a new frontier we’re excited to explore.”

“The combination of the immersive properties of Brass Monkey and the power of perceptual computing
allowed our team to give the judges a unique experience that will hopefully earn us the Ultimate Coder
title,” said Chris Allen, the project leader. Allen is CEO of BrassMonkey, a next generation browser-based
game system created by the Infrared5 team.

In this game, the player uses a phone to fly a character around a 3D world. When game players turn
their head, the perceptual computing camera tracks what the player is looking at, causing the scene to
shift. This effect allows the player to peer around obstacles, giving the game a real sense of depth and

, , , , , ,

Plan of Attack and Face Tracking Challenges Using the Intel Perceptual Computing SDK

February 19th, 2013 by Chris Allen

This post was featured on Intel Software’s blog, in conjunction with Intel’s Ultimate Coder Challenge. Keep checking back to read our latest updates!

We are very excited to be working with the Intel Perceptual Computing (IPC) SDK and to be a part of the Ultimate Coder Challenge! The new hardware and software that Intel and its partners have created allows for some very exciting possibilities. It’s our goal to really push the boundaries of what’s possible using the technology. We believe that perceptual computing plays a huge role in the future of human-to-computer interaction, and isn’t just a gimmick shown in movies like Minority Report. We hope to prove out some of the ways that it can actually improve the user experience with the game that we are producing for the competition.

Before we begin with the bulk of this post, we should cover a little bit on the makeup of our team and the roles that each of us play on the project. Unlike many of the teams in the competition, we aren’t a one man show, so each of our members play a vital role in creating the ultimate application. Here’s a quick rundown of our team:

Kelly Wallick – Project Manager

Chris Allen – Producer, Architect and Game Designer
Steff Kelsey – Tech Lead, Engineer focusing on the Intel Perceptual Computing SDK inputs
John Grden – Game Developer focusing on the Game-play

Rebecca Allen – Creative Director
Aaron Artessa – Art Director, Lead Artist, Characters, effects, etc.
Elena Ainley – Environment Artist and Production Art

When we first heard about the idea of the competition we started thinking about ways that we could incorporate our technology (Brass Monkey) with the new 3D image tracking inputs that Intel is making accessible to developers. Most of the examples being shown with the Perceptual Computing SDK focus on hand gestures, and we wanted to take on something a bit different. After much deliberation we arrived at the idea of using the phone as a tilt-based game controller input, and using head and eye tracking to create a truly immersive experience. We strongly believe that this combination will make for some very fun game play.

Our art team was also determined not to make the standard 3D FPS shoot-em-up game that we’ve seen so many times before, so we arrived at a very creative use of the tech with a wonderful background story of a young New Zealand Kiwi bird taking revenge on the evil cats that killed his family. To really show off the concept of head-tracking and peering around items in the world, we decided on a paper cutout art style. Note that this blog post and the other posts we will be doing on the Ultimate Coder site are really focused on the technical challenges and processes we are taking, and much less on the art and game design aspects of the project. After all, the competition is call the Ultimate Coder, not the Ultimate Designer. If you are interested in the art and design of our project, and we hope that you are, then you should follow our posts on our company’s blogs that will be covering much more of those details. We will be sure to reference these posts on every blog post here as well so that you can find out more about the entire process we are undertaking.

The name of the game that we’ve come up with for the competition is called Kiwi Catapult Revenge.

So with that, let’s get right to the technical nitty gritty.

Overview of the Technology We are Using


As we wanted to make a 3D game for the competition we decided to use Unity as our platform of choice. This tool allows for fast prototyping, ease of art integration and much more. We are also well versed in using Unity for a variety of projects at Infrared5, and our Brass Monkey SDK support for it is very polished.

Brass Monkey

We figured that one of our unique advantages in the competition would be to make use of the technology that we created. Brass Monkey SDK for Unity allows us to turn the player’s smartphone into a game controller for Unity games. We can leverage the accelerometers, gyroscopes and touch screens of the device as another form of input to the game. In this case, we want to allow for steering your Kiwi bird through the environment using tilt, and allow firing and control of the speed via the touch screen on the player’s phone.

Intel Perceptual Computing SDK

We decided to leverage the ICP SDK for head tracking, face recognition and possibly more. In the case of Kiwi Catapult Revenge the payer will use his eyes for aiming (the player character can shoot lasers from his eyes). The environment will also shift according to the angle in which the user is viewing causing the scene to feel like real 3D. Take a look at this example using a Wiimote for a similar effect that we are going for. In addition, our player can breath fire by opening his or her mouth in the shape of an “o” and pressing the fire button on the phone.

There are certainly other aspects of the SDK we hope to leverage, but we will leave those for later posts.


We are going to use this C-based library for more refined face tracking algorithms. Read more to find out why we chose OpenCV( to work in conjunction with the IPC SDK. Luckily, OpenCV is also developed by Intel, so hopefully that gets us additional points for using two of Intel’s libraries.

Head Tracking

The biggest risk item in our project is getting head tracking that performs well enough to be a smooth experience in game play, so we’ve decided to tackle this early on.

When we first started looking at the examples that shipped with the IPC SDK there were very few dealing with head tracking. In fact it was really only in the latest release where we found anything that was even close to what we proposed to build. That, and it was in this release that they exposed these features to the Unity version of the SDK. What we found are examples that simply don’t perform very well. They are slow, not all that accurate, and unfortunately just won’t cut it for the experience we are shooting for.

To make matters worse, the plugin for Unity is very limited. It didn’t allow us to manipulate much, if anything, with regards to head tracking or face recognition algorithms. As a Unity developer you either have to accept the poor performing pre-canned versions of the algorithms the SDK exposes, or get the raw data from the camera and do all the calculations yourself. What we found is that face tracking with what they provide gives us sub 3 frame per second performance that wasn’t very accurate. Now to be clear, the hand gesture features are really very polished, and work well in Unity.  It seems that Intel’s priority has been on those features, and head/face detection is lagging very much behind. This presents a real problem for our vision of the game, and we quickly realized that we were going to have to go about it differently if we were going to continue with our idea.


When we realized the current version of the IPC SDK wasn’t going to cut it by itself, we started looking into alternatives. Chris had done some study of OpenCV (CV stands for computer vision) a while back, and he had a book on the subject. He suggested that we take a look at that library to see if anyone else had written more effective head and eye tracking algorithms using that tool-set. We also discovered what looked like a very polished and effective head tracking library called OpenTL . We got very excited with what we saw, but when we went to download the library, we discovered the OpenTL isn’t so open after all. It’s not actually open source software, and we didn’t want to get involved with licensing a 3rd party tool for the competition. Likewise the FaceAPI from SeeingMachines looked very promising, but it also carried with it a proprietary license in order for us to use it.  Luckily what we found using OpenCV appeared to be more than capable of doing the job.

Since OpenCV is a C library we needed to figure out how to get it to work within Unity. We knew that we would need to compile a dll that would expose the functions to the Mono based Unity environment, or find a version out on the Internet that had already done this. Luckily we found this example, and incorporated it into our plans.

Use of the Depth Camera

The other concern we had was that all the examples we saw of face tracking in real-time didn’t make use of any special camera. They all used a simple webcam, and we really wanted to leverage the unique hardware that Intel provided us for the challenge. One subtle thing that we noticed with most of the examples we saw was they performed way better with the person in front of a solid background. The less noise the image had the better it would perform. So, we thought, why not use the depth sensor to block out anything behind the user’s head, essentially guaranteeing less noise in our images being processed regardless of what’s behind our player. This would be a huge performance boost over traditional webcams!

Application Flow and Architecture

After carefully considering our tools we finally settled on an architecture that spelled out how all the pieces would work together. We would use the Unity IPC SDK for the camera frames as raw images, and for getting the depth sensor data to block out only the portions of the image that had the person’s head. We would then leverage OpenCV for face tracking algorithms via a plugin to Unity.

We will be experimenting with a few different combinations of algorithms until we find something that give us the performance we need to implement as a game controller and (hopefully) also satisfy the desired feature set of tracking the head position and rotation, identifying if the mouth is open or closed, and tracking the gaze direction of the eyes.  Each step in the process is done to set up the the steps that follow.

In order to detect the general location of the face, we propose to use the Viola-Jones detection method.  The result of this method will be a smaller region of interest (ROI) for mouth and eye detection algorithms to sort through.

There are few proposed methods to track the facial features and solve for the rotation of the head.  The first method is to take use the results from the first pass to define 3 new ROIs and to search specifically for the mouth and the eyes using sets of comparative images designed specifically for the task.  The second method is to use the Active Appearance Model (AAM) to find match a shape model of facial features in the region.  We will go into more detail about these methods in future posts after we attempt them.

Tracking the gaze direction will be done by examining the ROI for each eye and determining the location of the iris and pupil by the Adaptive EigenEye method.

Trackable points will be constrained with Lucas-Kanade optical flow.  The optical flow compares the previous frame with the current one and finds the most likely locations of tracked points using a least squares estimation.

Summing it Up

We believe that we’ve come up with an approach that leverages the unique capabilities of the Perceptual Computing Camera and actually adds to the user experience of our game concept. As we start in on the development it’s going to be interesting to see how much this changes over the next seven weeks. We already have several questions about how it’s going to go: How much did we think would actually work will? What performance tuning will we need to do? How many of the other features of the IPC SDK can we leverage to make our game more engaging? Will we have enough time to pull off such an ambitious project in such a short time frame?

Wow! That turned out to be a long post! Thanks for taking the time to read what we are up to.

We are also curious to hear from you, other developers out there. What would you do differently given our goals for the project? If you’ve got experience with computer vision algorithms, or even just want to chime in with your support, we would love to hear from you!

, , , , , , , , , , , ,