In a collaboration with the Philharmonic Orchestra of PUCRS, we designed a 3D user interface to allow a conductor to play an interactive game during an orchestral concert. Our objective was to raise public attendance and satisfaction by providing a more interactive approach to the performance, which on this special occasion was themed on iconic video games songs.
A Unity Engine application obtained live music through a set of microphones positioned around the stand, and the 3D position of the conductor’s baton through the use of a Polhemus Fastrak device. By using digital signal processing to extract the behavior of the frequency spectrum, artifacts would be generated on the screen according to the music. These artifacts, which could have different formats, from sound notes to rings from the Sonic franchise, should be hit by the conductor using his baton. Each artifact grabbed counted towards a final score. The application was employed during a 1-hour live concert and in front of an audience of 1600 people. We later provided public demonstrations where students and visitors could experience the game.