I recently took a trip down to NYC a few weeks ago to pick up my new Google Glass. I figured now that we’ve had a bit of time to play with it here at Infrared5 and Brass Monkey, it’s time to write a post on our findings and what we think.
We definitely lost one of our nine lives this week with integrating face tracking into our game, but we still have our cat’s eyes, and are still feel very confident that we will be able to show a stellar game at GDC. On the face tracking end of things we had some big wins. We are finally happy with the speed of the algorithms, and the way things are being tracked will work perfectly for putting into Kiwi Catapult Revenge. We completed some complex math to create very realistic perspective shifting in Unity. Read below on those details, as well as for some C# code to get it working yourself. As we just mentioned, getting a DLL that properly calls update() from Unity and passes in the tracking values isn’t quite there yet. We did get some initial integration with head tracking coming into Unity, but full integration with our game is going to have to wait for this week.
On the C++ side of things, we have successfully found the 3D position of the a face in the tracking space. This is huge! By tracking space, we mean the actual (x,y,z) position of the face from the camera in meters. Why do we want the 3D position of the face in tracking space? The reason is so that we can determine the perspective projection of the 3D scene (in game) from the player’s location. Two things made this task interesting: 1) The aligned depth data for a given (x,y) from the RGB image is full of holes and 2) the camera specs only include the diagonal field of view (FOV) and no sensor dimensions.
We got around the holes in the aligned depth data by first checking for a usable value at the exact (x, y) location, and if the depth value was not valid (0 or the upper positive limit), we would walk through the pixels in a rectangle of increasing size until we encountered a usable value. It’s not that difficult to implement, but annoying when you have the weight of other tasks on your back. Another way to put it: It’s a Long Way to the Top on this project.
The z-depth of the face comes back in millimeters right from the depth data, the next trick was to convert the (x, y) position from pixels on the RGB frame to meters in the tracking space. There is a great illustration here of how to break the view pyramid up to derive formulas for x and y in the tracking space. The end result is: TrackingSpaceX = TrackingSpaceZ * tan(horizontalFOV / 2) * 2 * (RGBSpaceX – RGBWidth / 2) / RGBWidth) TrackingSpaceY = TrackingSpaceZ * tan(verticalFOV / 2) * 2 * (RGBSpaceY – RGBHeight / 2) / RGBHeight)
Where TrackingSpaceZ is the lookup from the depth data, horizontalFOV, and verticalFOV are are derived from the diagonal FOV in the Creative Gesture Camera Specs (here). Now we have the face position in tracking space! We verified the results using a nice metric tape measure (also difficult to find at the local hardware store – get with the metric program, USA!)
From here, we can determine the perspective projection so the player will feel like they are looking through a window into our game. Our first pass at this effect involved just changing the rotation and position of the 3D camera in our Unity scene, but it just didn’t look realistic. We were leaving out adjustment of the projection matrix to compensate for the off-center view of the display. For example: consider two equally-sized (in screen pixels) objects at either side of the screen. When the viewer is positioned nearer to one side of the screen, the object at the closer edge appears larger to the viewer than the one at the far edge, and the display outline becomes trapezoidal. To compensate, the projection should be transformed with a shear to maintain the apparent size of the two objects; just like looking out a window! To change up our methods and achieve this effect, we went straight to the ultimate paper on the subject: Robert Koomla’s Generalized Perspective Projection. Our port of his algorithm into C#/Unity is below.
//float n = 0.01f;
//float f = 1000f;
//Resolution curRes = Screen.currentResolution;
// all below in world space
// screen's bottom left corner
// screen's bottom right corner
// screen's top left corner
// head position (use mouse cursor for now)
// TODO temp translate it to a percentage of a 1x1 screen where 0,0 is the center
// Find the distance from the eye to screen plane.
// Find the extent of the perpendicular projection.
// Load the perpendicular projection.
// glFrustum(l, r, b, t, n, f);
// Rotate the projection to be non-perpendicular.
// Move the apex of the frustum to the origin.
The code follows the mouse pointer to change perspective (not a tracked face) and does not change depth (the way a face would). We are currently in the midst of wrapping our C++ libs into a DLL for Unity to consume and enable us to grab the 3D position of the face and then compute the camera projection matrix using the face position and the position of the computer screen in relation to the camera.
Last but not least we leave you with this week’s demo of the game. Some final art for UI elements are in, levels of increasing difficulty have been implemented and some initial sound effects are in the game.
As always, please ask if you have any questions on what we are doing, or if you just have something to say we would love to hear from you. Leave us a comment! In the meantime we will be coding All Night Long!
Last Thursday, Chris Allen (one of my business partners and husband) and I headed on a train to New York City for the first inaugural conference called Start. We were one of 23 startups invited to show off our product, Brass Monkey, to the highly curated group of 500 attendees. Hands down, it has to be one of the best events I have ever attended. From the moment we arrived at Centre 548 in Chelsea at 7:30am Friday morning until we left at 6:30pm that evening, it was one great conversation after another. Paddy Cosgrave and his amazing team of organizers at f.ounders did an outstanding job. We were honored to be selected as an exhibitor and excited to be amongst such innovative products and applications. Here are a few of my favorites: LittleBits , 3Doodler, BrandYourself and Magisto. LittleBits is an open source library of electronic modules that snap together with magnets for prototyping, learning and fun. Such a cool product that totally hits on so much that we love: open source technology, education, fun and creativity!
Since Chris and I were managing our booth, we were unable to attend the round tables and talks that happened throughout the day. We are excited that the talks were recorded, and Chris and I will be spending some quality time going through all of this great content. We had a fabulous day and would recommend to anyone that’s into Startups to attend Start 2014 when it comes around next year. I look forward to making it to WebSummit, f.ounders other event in the fall. Dublin, here we come!
This post was featured on Intel Software’s blog, in conjunction with Intel’s Ultimate Coder Challenge. Keep checking back to read our latest updates!
Week three seemed to go much better for the Infrared5 team. We are back on our feet with head tracking, and despite the judges lack of confidence in our ability to track eyes, we still believe that we’ve got a decent chance of pulling it off. Yes, it’s true as Nicole as said in her post this week, that the Intel Perceptual Computing (IPC) SDK isn’t yet up to the task. She had an interview with the Perceptual computing team and they told her “that eye tracking was going to be implemented later”. What’s funny about the lack of eye tracking and even decent gaze tracking in the IPC SDK is that the contest is showing this:
Yes we know it’s just marketing, but it is a pretty misleading image. They have a 3D mesh over a guy’s face giving the impression that the SDK can do AAM and POSIT. That would be so cool! Look out FaceAPI! Unfortunately it totally doesn’t do that. At least not yet.
This isn’t to say that Intel is taking a bad approach with the IPC SDK beta either. They are trying out a lot of things at once and not getting lost in the specifics of just a few features. This allows developers to tell them what they want to do with it without spending tremendous effort on features that wouldn’t even be used.
The lack of decent head, gaze and eye tracking is what’s inspired us on to eventually release our tracking code as open source. Our hope is that future developers can leverage our work on these features and not have to go through the pain we did in this contest. Maybe Intel will just merge our code into the IPC SDK and we can continue to make the product better together.
Another reason we are sticking with our plan on gaze and eye tracking is that we feel strongly, as do the judges, that these features are some of the most exciting aspects of the perceptual computing camera. A convertible ultrabook has people’s hands busy with typing, touch gestures, etc. and having an interface that works using your face is such a natural fit for this kind of setup.
Latest Demo of Kiwi Catapult Revenge
Check out the latest developments with the Unity Web Player version. We’ve added a new fireball/flamethrower style effect, updated skybox, sheep and more. Note that this is still far from final art and behavior for the game, but we want to continue showing the process we are going through by providing these snapshots of the game in progress. This build requires the free Brass Monkey app for iOS or Android.
A Polished Experience
In addition to being thoroughly entertained by the judges’ video blooper this week, one thing we heard consistently from them is that they were expecting more polished apps from the non-individual teams. We couldn’t agree more! One advantage that we have in the contest is that we have a fantastic art and game design team. That’s not to say our tech skills are lacking either. We are at our core a very technically focused company, but we tend not to compartmentalize the design process and the technology implementation in projects we take on. Design and technology have to work together in harmony to create an amazing user experience, and that’s exactly what we’re doing in this challenge.
Game design is a funny, flexible and agile process. What you set out to do in the beginning rarely ends up being what you make in the end. Our initial idea started as a sort of Mad Max road warrior style driving and shooting game (thus Sascha thinking ours was a racing game early on), but after having read some bizarre news articles on eradicating cats in New Zealand we decided the story of Cats vs. Kiwis should be the theme. Plus Rebecca and Aaron really wanted to try out this 2D paper, pop-up book style, and the Kiwi story really lends itself to that look.
Moving to this new theme kept most of the core game mechanics as the driving game. Tracking with the head and eyes to shoot and using the phone as a virtual steering wheel are exactly the same in the road warrior idea. Since our main character Karl Kiwi has magical powers and can fly, we made it so he would be off the ground (unlike a car that’s fixed to the ground). Another part of the story is that Karl can breathe fire like a dragon, so we thought that’s an excellent way to use the perceptual computing camera by having the player open their mouth to be able to shoot fire. Shooting regular bullets didn’t work with the new character either, so we took some inspiration from funny laser cats memes, SNL and decided that he should be able to shoot lasers from his eyes. Believe it or not, we have been wanting to build a game involving animals and lasers for a while now. “Invasion of the Killer Cuties” was a game we concepted over two years ago where you fly a fighter plane in space against cute rodents that shoot lasers from their eyes (initial concept art shown below).
Since Chris wrote up the initial game design document (GDD) for Kiwi Catapult Revenge there have been plenty of other changes we’ve made throughout the contest. One example: our initial pass at fire breathing (a spherical projectile) wasn’t really getting the effect we wanted. In the GDD it was described as a fireball so this was a natural choice. What we found though is that it was hard to hit the cats, and the ball didn’t look that good either. We explored how dragon fire breathing is depicted in movies, and the effect is much more like how a flamethrower works. The new fire breathing effect that John implemented this week is awesome! And we believe it adds to the overall polish of our entry for the contest.
Another aspect of the game that wasn’t really working so far was that the main character was never shown. We chose a first person point of view so that the effect of moving your head and peering around items would feel incredibly immersive, giving the feeling that you are really right in this 3D world. However, this meant that you would never see Karl, our protagonist.
Enter the rear view mirror effect. We took a bit of inspiration from the super cool puppets that Sixense showed last week, and this video of an insane wingsuit base jump and came up with a way to show off our main character. Karl Kiwi will be fitted with a rear view mirror so that he can see what’s behind him, and you as the player can the character move the same as you. When you tilt your head, Karl will tilt his, when you look right, so will Karl, and when you open your mouth Karl’s beak will open. This will all happen in real time, and the effect will really show the power of the perceptual computing platform that Intel has provided.
Head Tracking Progress Plus Code and Videos
It wouldn’t be a proper Ultimate Coder post without some video and some code, so we have provided you some snippets for your perusal. Steff did a great job of documenting his progress this week, and we want to show you step by step where we are heading by sharing a bit of code and some video for each of these face detection examples. Steff is working from this plan, and knocking off each of the individual algorithms step by step. Note that this week’s example requires the OpenCV library and a C compiler for Windows.
This last week of Steff’s programming was all about two things: 1) switching from working entirely in Unity (with C#) to a C++ workflow in Visual Studio, and 2) refining our face tracking algorithm. As noted in last week’s post, we hit a roadblock trying to write everything in C# in Unity with DLL for the Intel SDK and OpenCV. There were just limits to the port of OpenCV that we needed to shed. So, we spent some quality time setting up in VS 2012 Express and enjoying the sharp sting of pointers, references, and those type of lovely things that we have avoided by working in C#. However there is good news, we did get back the amount of lower level control needed to refine face detection!
Our main refinement this week was to break through the limitations of tracking faces that we encountered when implementing the Viola-Jones detection method using Haar Cascades. This is a great way to find a face, but it’s not the best for tracking a face from frame to frame. It has limitations in orientation; e.g. if the face is tilted to one side the Haar Cascade no longer detects a face. Another drawback is that while looking for a face, the algorithm is churning through images per every set block of pixels. It can really slow things down. To break through this limitation, we took inspiration from the implementation by the team at ROS.org . They have done a nice job putting face tracking together using python, OpenCV, and an RGB camera + Kinect. Following their example, we have implemented feature detection with GoodFeaturesToTrack and then tracked each feature from frame to frame using Optical Flow. The video below shows the difference between the two methods and also includes a first pass at creating a blue screen from the depth data.
This week, we will be adding depth data into this tracking algorithm. With depth, we will be able to refine our Region Of Interest to include an good estimate of face size and we will also be able to knock out the background to speed up Face Detection with the Haar Cascades. Another critical step is integrating our face detection algorithms into the Unity game. We look forward to seeing how all this goes and filling you in with next week’s post!
We are also really excited about all the other teams’ progress so far, and in particular we want to congratulate Lee on making a super cool video last week! We had some plans to do a more intricate video based on Lee’s, but a huge snowstorm in Boston put a bit of a wrench in those plans. Stay tuned for next week’s post though, as we’ve got some exciting (and hopefully funny) stuff to show you!
For you code junkies out there, here is a code snippet showing how we implemented GoodFeaturesToTrack and Lucas-Kanada Optical Flow:
using namespace cv;
using namespace std;
static void help()
// print a welcome message, and the OpenCV version
cout << "\nThis is a demo of Robust face tracking use Lucas-Kanade Optical Flow,\n"
"Using OpenCV version %s" << CV_VERSION << "\n"
cout << "\nHot keys: \n"
"\tESC - quit the program\n"
"\tr - restart face tracking\n" << endl;
// function declaration for drawing the region of interest around the face
We are very excited to be working with the Intel Perceptual Computing (IPC) SDK and to be a part of the Ultimate Coder Challenge! The new hardware and software that Intel and its partners have created allows for some very exciting possibilities. It’s our goal to really push the boundaries of what’s possible using the technology. We believe that perceptual computing plays a huge role in the future of human-to-computer interaction, and isn’t just a gimmick shown in movies like Minority Report. We hope to prove out some of the ways that it can actually improve the user experience with the game that we are producing for the competition.
Before we begin with the bulk of this post, we should cover a little bit on the makeup of our team and the roles that each of us play on the project. Unlike many of the teams in the competition, we aren’t a one man show, so each of our members play a vital role in creating the ultimate application. Here’s a quick rundown of our team:
Kelly Wallick – Project Manager
Chris Allen – Producer, Architect and Game Designer
Steff Kelsey – Tech Lead, Engineer focusing on the Intel Perceptual Computing SDK inputs
John Grden – Game Developer focusing on the Game-play
Rebecca Allen – Creative Director
Aaron Artessa – Art Director, Lead Artist, Characters, effects, etc.
Elena Ainley – Environment Artist and Production Art
When we first heard about the idea of the competition we started thinking about ways that we could incorporate our technology (Brass Monkey) with the new 3D image tracking inputs that Intel is making accessible to developers. Most of the examples being shown with the Perceptual Computing SDK focus on hand gestures, and we wanted to take on something a bit different. After much deliberation we arrived at the idea of using the phone as a tilt-based game controller input, and using head and eye tracking to create a truly immersive experience. We strongly believe that this combination will make for some very fun game play.
Our art team was also determined not to make the standard 3D FPS shoot-em-up game that we’ve seen so many times before, so we arrived at a very creative use of the tech with a wonderful background story of a young New Zealand Kiwi bird taking revenge on the evil cats that killed his family. To really show off the concept of head-tracking and peering around items in the world, we decided on a paper cutout art style. Note that this blog post and the other posts we will be doing on the Ultimate Coder site are really focused on the technical challenges and processes we are taking, and much less on the art and game design aspects of the project. After all, the competition is call the Ultimate Coder, not the Ultimate Designer. If you are interested in the art and design of our project, and we hope that you are, then you should follow our posts on our company’s blogs that will be covering much more of those details. We will be sure to reference these posts on every blog post here as well so that you can find out more about the entire process we are undertaking.
The name of the game that we’ve come up with for the competition is called Kiwi Catapult Revenge.
So with that, let’s get right to the technical nitty gritty.
Overview of the Technology We are Using
As we wanted to make a 3D game for the competition we decided to use Unity as our platform of choice. This tool allows for fast prototyping, ease of art integration and much more. We are also well versed in using Unity for a variety of projects at Infrared5, and our Brass Monkey SDK support for it is very polished.
We figured that one of our unique advantages in the competition would be to make use of the technology that we created. Brass Monkey SDK for Unity allows us to turn the player’s smartphone into a game controller for Unity games. We can leverage the accelerometers, gyroscopes and touch screens of the device as another form of input to the game. In this case, we want to allow for steering your Kiwi bird through the environment using tilt, and allow firing and control of the speed via the touch screen on the player’s phone.
Intel Perceptual Computing SDK
We decided to leverage the ICP SDK for head tracking, face recognition and possibly more. In the case of Kiwi Catapult Revenge the payer will use his eyes for aiming (the player character can shoot lasers from his eyes). The environment will also shift according to the angle in which the user is viewing causing the scene to feel like real 3D. Take a look at this example using a Wiimote for a similar effect that we are going for. In addition, our player can breath fire by opening his or her mouth in the shape of an “o” and pressing the fire button on the phone.
There are certainly other aspects of the SDK we hope to leverage, but we will leave those for later posts.
We are going to use this C-based library for more refined face tracking algorithms. Read more to find out why we chose OpenCV(opencv.org) to work in conjunction with the IPC SDK. Luckily, OpenCV is also developed by Intel, so hopefully that gets us additional points for using two of Intel’s libraries.
The biggest risk item in our project is getting head tracking that performs well enough to be a smooth experience in game play, so we’ve decided to tackle this early on.
When we first started looking at the examples that shipped with the IPC SDK there were very few dealing with head tracking. In fact it was really only in the latest release where we found anything that was even close to what we proposed to build. That, and it was in this release that they exposed these features to the Unity version of the SDK. What we found are examples that simply don’t perform very well. They are slow, not all that accurate, and unfortunately just won’t cut it for the experience we are shooting for.
To make matters worse, the plugin for Unity is very limited. It didn’t allow us to manipulate much, if anything, with regards to head tracking or face recognition algorithms. As a Unity developer you either have to accept the poor performing pre-canned versions of the algorithms the SDK exposes, or get the raw data from the camera and do all the calculations yourself. What we found is that face tracking with what they provide gives us sub 3 frame per second performance that wasn’t very accurate. Now to be clear, the hand gesture features are really very polished, and work well in Unity. It seems that Intel’s priority has been on those features, and head/face detection is lagging very much behind. This presents a real problem for our vision of the game, and we quickly realized that we were going to have to go about it differently if we were going to continue with our idea.
When we realized the current version of the IPC SDK wasn’t going to cut it by itself, we started looking into alternatives. Chris had done some study of OpenCV (CV stands for computer vision) a while back, and he had a book on the subject. He suggested that we take a look at that library to see if anyone else had written more effective head and eye tracking algorithms using that tool-set. We also discovered what looked like a very polished and effective head tracking library called OpenTL . We got very excited with what we saw, but when we went to download the library, we discovered the OpenTL isn’t so open after all. It’s not actually open source software, and we didn’t want to get involved with licensing a 3rd party tool for the competition. Likewise the FaceAPI from SeeingMachines looked very promising, but it also carried with it a proprietary license in order for us to use it. Luckily what we found using OpenCV appeared to be more than capable of doing the job.
Since OpenCV is a C library we needed to figure out how to get it to work within Unity. We knew that we would need to compile a dll that would expose the functions to the Mono based Unity environment, or find a version out on the Internet that had already done this. Luckily we found this example, and incorporated it into our plans.
Use of the Depth Camera
The other concern we had was that all the examples we saw of face tracking in real-time didn’t make use of any special camera. They all used a simple webcam, and we really wanted to leverage the unique hardware that Intel provided us for the challenge. One subtle thing that we noticed with most of the examples we saw was they performed way better with the person in front of a solid background. The less noise the image had the better it would perform. So, we thought, why not use the depth sensor to block out anything behind the user’s head, essentially guaranteeing less noise in our images being processed regardless of what’s behind our player. This would be a huge performance boost over traditional webcams!
Application Flow and Architecture
After carefully considering our tools we finally settled on an architecture that spelled out how all the pieces would work together. We would use the Unity IPC SDK for the camera frames as raw images, and for getting the depth sensor data to block out only the portions of the image that had the person’s head. We would then leverage OpenCV for face tracking algorithms via a plugin to Unity.
We will be experimenting with a few different combinations of algorithms until we find something that give us the performance we need to implement as a game controller and (hopefully) also satisfy the desired feature set of tracking the head position and rotation, identifying if the mouth is open or closed, and tracking the gaze direction of the eyes. Each step in the process is done to set up the the steps that follow.
In order to detect the general location of the face, we propose to use the Viola-Jones detection method. The result of this method will be a smaller region of interest (ROI) for mouth and eye detection algorithms to sort through.
There are few proposed methods to track the facial features and solve for the rotation of the head. The first method is to take use the results from the first pass to define 3 new ROIs and to search specifically for the mouth and the eyes using sets of comparative images designed specifically for the task. The second method is to use the Active Appearance Model (AAM) to find match a shape model of facial features in the region. We will go into more detail about these methods in future posts after we attempt them.
Tracking the gaze direction will be done by examining the ROI for each eye and determining the location of the iris and pupil by the Adaptive EigenEye method.
Trackable points will be constrained with Lucas-Kanade optical flow. The optical flow compares the previous frame with the current one and finds the most likely locations of tracked points using a least squares estimation.
Summing it Up
We believe that we’ve come up with an approach that leverages the unique capabilities of the Perceptual Computing Camera and actually adds to the user experience of our game concept. As we start in on the development it’s going to be interesting to see how much this changes over the next seven weeks. We already have several questions about how it’s going to go: How much did we think would actually work will? What performance tuning will we need to do? How many of the other features of the IPC SDK can we leverage to make our game more engaging? Will we have enough time to pull off such an ambitious project in such a short time frame?
Wow! That turned out to be a long post! Thanks for taking the time to read what we are up to.
We are also curious to hear from you, other developers out there. What would you do differently given our goals for the project? If you’ve got experience with computer vision algorithms, or even just want to chime in with your support, we would love to hear from you!
Infrared5 and Brass Monkey are excited to announce their participation in Intel Software’s Ultimate Coder Challenge, ‘Going Perceptual’! The IR5/Brass Monkey team, along with six other teams from across the globe, will be competing in this seven week challenge to build the ultimate app. The teams will be using the latest Ultrabook convertible hardware, along with the Intel Perceptual Computing SDK and camera to build the prototype. The competitors range from large teams to individual developers, and each will take a unique approach to the challenge. The question will be which team or individual can execute their vision with the most success under such strict time constraints?
Here at Infrared5/Brass Monkey headquarters, we have our heads in the clouds and our noses to the grindstone. We are dreaming big, hoping to create a game that will take user experience to the next level. We are combining game play experiences like those available on Nintendo’s Wii U and Microsoft Kinect. The team will use the Intel Perceptual Computing SDK for head tracking, which will allow the player to essentially peer into the tablet/laptop screen like a window. The 3D world will change as the player moves his head. We’ve seen other experiments that do this with other technology and think it is really remarkable. This one using Wii-motes by Johnny Lee is one of the most famous. Our team will be exploring this effect and other uses of the Intel Perceptual Computing SDK combined with the Brass Monkey’s SDK (using a smartphone as a controller) to create a cutting edge, immersive experience. Not only that, but our creative team is coming up with all original IP to showcase the work.
Intel will feature documentation of the ups and downs of this process for each team, beginning February 15th. We will be posting weekly on our progress, sharing details about the code we are writing, and pointing out the challenges we face along the way. Be sure to check back here as the contest gets under way.
What would you build if you were in the competition? Let us know if you have creative ideas on how to use this technology; we would love to hear them.
We would like to thank Intel for this wonderful opportunity and wish our competitors the best of luck! Game on!