Happy Feet


During performance, ambient sounds usually come from sources other than the performer. This project’s goal was to use synesthetic design between movement and sound to enhance walking during performance in order to create a more immersive and realistic experience. As performers walk, their legs would produce sounds that mimic their movement through the imagined environment (e.g. if the performance takes place on a hiking trail during the fall, the audience would hear the performer crunch through leaves as they walked). In improv, performers often react to audience input, so we wanted an audience member to be able to directly manipulate the performer's footsteps as a way of controlling the direction of the story.

Although this was the idea we hoped to implement, we thought that if we had enough time, we could implement one of several add-ons to further enhance the audience and performer experience. We wanted audience members to able to choose sound-clips as they watched the performance, so we planned to build a small Android app that would let an audience member control sound from their phone. We also wanted to let audience members record short sound clips themselves, and directly send the sound to the wearable to play. Additionally, we thought about including a mike in the wearable itself, and giving performers the opportunity to record clips of themselves walking, and then playing back these clips at a later time.

Unfortunately, our team ran into several issues during the project, which prevented us from being able to implement this seamless integration of wireless and cable-less interaction. We decided to show our interaction using muppets with the setup on the performer's arms instead of legs.

2x Arduino Yun with microSD https://www.sparkfun.com/products/12053
2x Flexible Speakers and amplifier https://www.sparkfun.com/products/12793
2x USB-audio adapter https://www.sparkfun.com/products/8639
Conductive thread https://www.sparkfun.com/products/10867
2x Battery pack and AA battery https://www.sparkfun.com/products/8248 (did not show in demo)
2x male-to-male mono audio cables
conductive thread https://www.sparkfun.com/products/10867
conductive fabric https://www.sparkfun.com/products/10070
Soldering iron
Electrical Tape


Design Process and Challenges


We wanted to create dynamic audience participation, so the initial focus of the project was getting user input over WiFi. Being a wearable, we briefly thought about using a Lilypad Xbee to send the entire file over the serial port to play, but decided against this when we found that the Arduino Uno had a SD card, which would prevent us from having to buffer sound data. We first tested the individual components: connecting to the WPA2 WiFi, playing sound to the flexible speaker from the WiFi Shield, and tracking footsteps using the accelerometer. Although the components worked individually, when we tried to connect them the WiFi shield gave us errors to update the firmware, which we tried to do multiple times. Since most of the Hackerspace's WiFi Shields weren't working properly, we thought that it might be a hardware issue, and ended up buying WiFi shield's from Microcenter to test out. Once we realized that the Uno Shields wouldn't work, we decided to try the Arduino Yun, which had both the WiFi and SD built into the microprocessor.

With the Yun, we encountered a whole slew of new problems. First, the Wifi could not connect to a enterprise network, so we relied on hotspotting from our phones and using the spotty Hackerspace network. Every time we tested the device through the Wifi, we had to first connect to the Arduino Yun, and then connect it and our laptop or phone to the Hackerspace network, which could take as many as 20-30 minutes. Since the Yun is in beta, there were many unresolved issues that that may have been caused by the board itself.

The Yun also used completely different libraries than the Uno, and played music through the USB port. We rewrote our code to fit the Yun, but when we tried connecting through the USB, we soon realized that we needed an adapter that would convert the digital signal from the USB to the analog audio signal. We bought this adapter on another trip to microcenter, but when we tried to play music through the USB, the speaker audio was barely audible, which had not been a problem when had tested the speaker through the Uno. We thought that there might have been a power issue and spent a long time trying to fix this, however the problem *magically* resolved itself.

Making it Wearable

The images above depict the evolution of our design. The first image is of the Arduino Uno using an accelerometer to measure footsteps using the serial port to select audio and playing through a laptop. The second design uses the Arduino Yun, the flexible speaker, a battery pack, and a low-tech button underfoot instead of accelerometer that delivers signal over WiFi. In image 3, our team decided to showcase our design using
puppets (whose "feet" were our elbows) in order to overcome several battery power and connection issues.

In order to make the sound more believable, we needed to duplicate our setup for each leg. This provided to be one of the biggest challenges we encountered. We found that each leg setup worked on its own, but debugging both legs together made the task 10x more difficult and simply didn't work. For the purpose of the demonstration, we ended up using just one of the two systems.

Measuring Footsteps

When we envisioned this project, we first thought about using an accelerometer to measure steps. We found several papers that solve the problem (See Libby (2008)) and got the accelerometer working on the WiFi Shield. When we started working with the Yun, we were worried about not having enough time to rewrite the accelerometer code, so we decided to use pressure sensors. Although we had used them in previous projects, we found that the pressure sensors did not work well on the foot because users put pressure on their feet differently during different steps, and often changed pressure while shifting their feet.

We decided to go for the simplest solution, buttons. We first used conductive thread with a miniature button on the bottom of the shoe, but then realized that it would make walking uncomfortable. Instead, we found a low -tech alternative to create a shoe button: we separated 2 pieces of conductive fabric with a piece of felt with small holes cut into it: when the user stepped down, the conductive parts would touch and complete the circuit.


The images above depict our evolution of footstep detection. Image 1 shows the 3D accelerometer we used, images 2 and 3 show different pressure sensor configurations we tested, image 4 shows the mini push button and image 5 shows the low-tech felt and conductive fabric button.


Moving Forward:
Although our project encountered many problems, we believe that the interaction is both meaningful and interesting. We've decided to rethink our design and look for smaller easier alternatives to implement it. Although we were trying to be lightweight by using the flexible speaker, an alternative might be a small speaker and amplifier that sits on the top of your shoe.

We chose to duplicate the setup for each leg because we thought that connecting Arduino down two legs would be impractical, difficult to put on and take off. Instead, we might think about a way to connect both speakers in a way that simplifies the setup while still being easy to put on and take off.

One alternative we *very* recently discovered was the Lilypad Mp3 Player board which has a built in SD card, is Mp3 compatible and has a built in amplifier. Combined with the Lilypad Xbee, this might be a much cleaner and simpler solution for this project.


Whose Line is it Anyway: Audience sound effects influence Performers
Libby(2008). A simple method for reliable footstep detection on Embedded Sensor Platforms.
BeatFeet: Manipulating music using pressure sensors on feet
Audience Changes Background and Performer Interacts with it