Pittsburgh Disney Research recently developed a novel touch interaction method that not only detects a on/off touch capacitance, but can also detect a wide range of touch gestures through capacitive frequency sweeping [Disney]. This coupled with a simple Box Jumping game inspired from [BoxJump] pushed us to create a multi-gesture fluid, interactive, and dynamic game controller.

Touché for Arduino [instructable] samples 160 frequencies to develop a gesture profile. This is 1/10th the data points for our gesture recognition than the Disney Research experiment which uses a digital signal processing (DSP) chip with the ability to capture over 2,000 frequencies. Furthermore, due to limitations of the Arduino's onboard RAM, we had to reduce the number of data points to 100 per gesture. As such, we don't have nearly the same granularity of touch interactions and resorted to working with damp substances or water. Though Touché is touted for working in all mediums, this particular medium resulted in the best accuracy.

BoxJump, a game where a moving box must avoid obstacles, was a simple, yet refined game that provided inspiration for creating gestures. Since only a minimal amount of the code was available for BoxJump, we further developed the game. This included adding in levels, making the style match LessMilk, and most importantly, added three different commands into the game: jump, high jump, and pause. We ultimately envisioned someone tapping their drink with one finger to jump, two fingers to high jump, and a full grasp to take a drink to pause the game. Unfortunately this turned out to be too ambitious with such a small amount of points available for gesture sensing and recognition.

BoxJump game adapted from

Our goal was to create an input device that is adaptable for many different types of games and our design was built incrementally. The first stage consisted of a breadboard implementation with an Arduino Uno, graphed touch gestures, and construction of a schematic to support LED and button interaction with the device:


Next, we found a non-conductive box in the HCIL Hacker-Space and drilled holes for LED's, for the store and detect buttons on the side of the device, for the usb connector to the Arduino, and for the wire for touch interaction. Next, we replaced the UNO for the Leonardo so that we would have inherent HID compatibility with our computer. In order to fit both the Arduino and the breadboard into the box, we decided to solder the components:


We strategically separated the resistors for the LEDs and the buttons from the Touché circuit in an attempt to reduce interference, although due to some inexperience with soldering and the close proximity of the soldered board to the Arduino, some noise still exist in our touch interactions. We attempted to mitigate some of the interference by placing a foam piece between the soldered board and the Arduino.


Gestures, as mentioned earlier, are stored in 100 data points. In order to determine the most important range of data points for our gestures, we graphed our points and visually inspected which range was more pertinent to the gestures we were interested in. For us, this meant that we stored points 50-150 out of the range 0-160. In order to recognize the touch gesture, we set a +-10 threshold for each data point. If any point exceeds the tolerance, it is not considered a gesture. We also experimented with comparing observed touch sensing readings to each gesture profile and selecting the profile with minimum error. However, we found that the simplest method of thresholding, was the most effective at recognizing the correct gesture.

Many gesture detection errors took us by surprise. For instance, printing out too much to the Serial port would slow down gesture detection enough to be unable to play a split second interactive game. Alternatively, pressing the buttons resulted in many down signals getting sent to the Arduino simultaneously and resulting in too many button presses being sent to the computer. Many of the optimizations for the device suffered from the Goldilocks complex until we got it "just right."

Our biggest challenge with the device is that it is a touch sensing device, as such, it turns everything into a touch sensor. Even putting your hand in proximity of the wiring can create a capacitive effect. Given the chance to redo the device, we would add shielding to all the wiring and do a better job at shielding each individual components.