The performer & new interfaces for  musical expression
Traditionally, the live performance of acoustic instruments has been controlled by simple human gestures - blowing, striking, pressing. This enables the audience to appreciate a direct âcauseâ and âactionâ link between action and sound. However, as musicians rapidly embrace technology such as the portable laptop into their musical compositions, this link becomes less transparent. This performative complication has led to the design of new musical interfaces for musical expression, which can be classified into three categories: gestural interfaces, software interfaces, and abstract interfaces (not discussed here). While this essay is not an exhaustive analysis of the available interfaces, it gives a brief overview of developing projects in the field.Â
A Brief History of Instrumentation
Traditional orchestral instruments can be traced back to the early 15th century, where their makers relied largely upon mechanical technology for the production of sound. The musical interface was an integral and inseparable part of the instrument, and the predominant point of sonification. Each uniquely crafted instrument produced different musical tone colours, often within a certain dynamic and tonal range, and heavily reliant upon the players physical effort for the production of sound. For instance, âthe instruments of the string family share a mechanism for retaining the string, which sees the strings attached to the tail piece, run over the bridge which sits on the sound board, and run to the peg which holds in the scroll, and allows for variations in tuning. The performer causes the transfer of spatial and temporal information, through their physical and intimate interaction with the instrument, to produce certain tone colours.â (Garth Paine, Towards Unified Design Guidelines for New Interfaces for Musical Expression)Â
In 1914, Erich von Hornbostel and Curt Sachs (adoptng Mahillonâs scheme) published a new scheme for instrument classification in Zeitschrift fur Ethnologue. Hornbostel-Sachs classified instruments into four main groups;Â
Idiophones, which produce sound by vibrating the body of the instrument itself (e.g xylophone, rattle)
Membraphones, which produce sound by a vibrating stretched membrane (e.g drums, kazoo)
Chordophones, which produce sound by vibrating one or some strings (e.g piano, guitar)
Aerophones, which produce sound by vibrating a column of air (e.g flute, trumpet)
With the advent of new means by which to produce sound, Sachs later added a 5th category;Â
Electrophones, which produce sounds by electronic means.Â
The auto-mechanisation of musical instruments first surfaced in 17th century Europe, with the introduction of the music box in hand-cranked street organs, and later with the enormity of the Salomon de Causâs pegged organ. This mechanical organ is based upon a series of pipes, by which wind moves to produce sounds. The pipes are divided and operated with the players hands or feet. This type of instrumentation still required a direct correlation between the human body and instrument during performance. The interface had a inseparable relationship with the production of sound, with the method of sound production located within the interface.Â
The electrical automated instrument has a much more recent history, dating back to the late 19th century with Cahillâs Telharmonum electromechanical synthesiser. Instrumentation moved rapidly from electro-mechanical to purely electronic instruments. Dr Freidrich Adolf Trautwein was one of the first to explore instruments of a solely electronic nature with the trautonium. The trautonium assigns the left hand to the control of the of amplitude, and the right to the control of the pitch. Pitched variation of notes is achieved through modifying the shape of the right hand. However, as the synthesis engine remains unchanged, the timbre remains fixed.Â
The advent of electrical instrumentation required an expansion in the allowable musical sounds accepted by audiences. Electric instruments would not only simulate sounds made by acoustic instruments, but created harmonically and timberally complex sounds of their own. This type of instrumentation escaped the harmonic and locational restrictions of the acoustic interfaces of the past, and expected audiences to take on a more experimental approach to sound.Â
First developed in the mid 20th century, electric digital computers soon became not just a new force behind mathematical and office efficiently, but a new interface for musical expression. Musicians first begun to use computers for musical production in the early 50s, with mathematician Geoff Hill programing the CSIRAC to play popular musical melodies, and later as controller instruments for external components, such as MIDI controllers. Computers were used for sound synthesis, sound design and digital signal processing, amongst other things. With the advent of personal computing, and the wide availability of audio-synthesis programs, musicians begun using computers in their live musical performances, for a myriad of audio techniques such as live playback, looping, processing, and external MIDI components, etc. The computer is mapped to speakers to project the sound.Â
The computer laptop has been widely embraced by live musician as an additional or solo instrument. While there are numerous reasons for embrace, predominately they focus around the portability and multi-functionality of the laptop, its ability to perform a vast number of instrumental tone colours from the one device in real-time, and the new and exciting tonal possibilities of audio-processing. Rather than accepting the costs of instrument transportation, or the plight of keeping four musicians in the same room long enough to practice, computer musicians will often be seen as the sole performer on stage, accompanied with nothing more than their laptop, a powerful audio-synthesis program, and a library of samples.
While the computer as an instrument may be sonically impressive to the blind listener, audience members and academics have widely criticised the laptop as an instrument for live music performance, due to the loss of human agency, its relatively passive gestural requirements, and the dislocation of sound created by the removal of the interface as the position of sonic output. While arguments have been made that audiences will become accustomed to the laptop instrument, efforts have been taken to create a more seamless interaction between performers and their audience.Â
The loss of human agency is interrelated with the gestural and physical effort audiences are accustomed to in acoustic instruments. They expect a particular musical âeffectâ to be the result of a gestural âcauseâ made by the performer, e.g blowing air into a flute will produce the sonic and tonal qualities of a flute note. With a computer, this same âcauseâ and âeffectâ is not as obvious, not only because the audience cannot see what is happening clearly on the computer screen, but often sounds that come out of a computer emulate those of an acoustic instrument, which the audience does not appreciate the same as if the performer is actually playing the instrument. Rather, the audience sees the laptop as performing much of the âcauseâ and the âeffectâ, which renders them unappreciative of the talents of the performer.Â
While it can be argued that laptop performers shouldnât strive to make their computers more like acoustic instruments, and embrace the glow of the laptop screen to make an inevitable evolution in contemporary, computer performances, counter arguments have been made that that the traditional notions of gesture should be maintained by laptop performers, and the same technology should be used in a more communicative manner. In an acoustic music performance, musicians use their gestures to engage their audience, communicate authenticity, and as an expression of their emotional interpretation. Gesturally, the acoustic performer gives the audience a sense that there is an immediacy in the music making, that they may have a chance to witness something extraordinary that they canât get on off a CD, that there are some risks taken by the performer, and that the performance has an element of physical prowess. While the best laptop performers may put on an exciting sonic performance, they will struggle to be as physically engaging as a great saxophone player as there is little transparency of the mapping between the input to the laptop, and the corresponding output. This had led to the development of the controller, which creates a new external interfaced to control the laptop, giving the laptop performer more gestural capabilities.Â
A New Approach to Musical Interfaces for Live Performance
âInvention is the mother of necessityâÂ
Moving into the 21st century, musicians are not only rapidly embracing technological developments into their music, but designing a whole new type of interface to best communicate their music. Rather than the static laptop performer, designers are creating new musical interfaces which allow musicians to independently move around the stage, and communicate counterintuitive gestures which audiences can appreciate. Alternatively, musicians have embraced more portable technology for their instruments, such as the iPad, designing software specifically for its touch screen capabilities. Lastly, as scientists learn more about the brain, musicians employ this technology for new methods of compositional practice, using the brain to control and generate musical ideas (not discussed here).
Although this field is still in its experimental stages, with each new development it is rapidly becoming common musical practice for the 21st century performer.
Drawing largely on the performance practices of acoustic instrument musicians, external laptop controllers have become a popular method of electronic music performance. A study undertaken by University of Western Sydney professors Paine, Stevenson and Pearce (Garth Paine, Towards Unified Design Guidelines for New Interfaces for Musical Expression), sought to define the fundamental control parameter utilised by expert musicians on traditional acoustic instruments, in order to help define how electronic controllers should be designed. They sought information through interviews about how many discrete control parameters trained musicians consciously exercise in normal performance conditions. An analysis of the results led to the identification of four primary physical controls used by performers;
Using these results, the study outlined a model for interactive electronic music performances, containing a control surface, processor and effector mechanism.
âThe parts of the instrument that are directly controlled or manipulated by parts of the body, and to which information is directly transfered, are called the control interface.The parts that actually produce sound are called the effector mechanism. Intervening between the control interface and effector mechanisms is often the processor of some kind that converts information in control format to effector format.....Interaction between the person and the instrument takes place through the aural (visual) feedback loop an the performer makes decisions on that basis in real time.â (Garth Paine, Towards Unified Design Guidelines for New Interfaces for Musical Expression)
3.1.1 Case Study 1: Thomas Mitchell & Imogen Heap, âThe SoundGrasp: A Gestural Interface for the Performance of Live Musicâ (2011)
A paper presented at the International Conference of New Interfaces for Musical Expression (Oslo, 2011) by University of West England professor Thomas Mitchell, and recording artist Imogen Heap, introduced the âSoundGraspâ, a data glove for the purpose of musical performance, which responds to gestures and body movements. It was their intention to âenable a performer to manipulate digital music processes without having to defer audience engagement to undertake subtle interactions with machineryâ. The glove incorporates âclear sound producing or ancillary gestures into performanceâ to enhance audience engagement and performer communication.Â
The data glove was first developed in the late 1970s to enable a database of files to be searched and played back using simple hand gestures. To date, the glove has been programmed to enable musicians to âfreezeâ single notes when the finger and thumb are pressed together (VAMP), to modulate harmony and amplitude through the flexion and accelerator sensors attached to the glove, and to hold the incoming audio.Â
Mitchell & Heap propose to expand on this technology, and design a highly intuitive glove of their own. Their device is represented as a three part system: the gestural controller, the audio processing unit, and the mapping that exists between the two (via C++). Also connected to the glove (gestural controller) is a wireless lavaliere microphone to enable the recording of live input.Â
Posture identity is an essential part of the design, requiring the user to learn the positions, such like a guitar player will learn the position of chords. Each gesture is mapped to a particular audio process, which affect the sound in real-time and increase visual transparency for the audience. The gestures are summarised below:
3.1.2 Case Study 2: Garth Paine & the Nintendo Wii Remote. âEncounterâ (2008)
The Nintendo Wii Remote & Nunchuck (WiiMote) has been a major focus of attention as an external music controller for live performance. The WiiMote can be controlled using an OSC bridge in various software applications (e.g Max/MSP, Quartz Composer). Musicians have been attracted to the technology due to the large amount of control it affords, containing eleven buttons, and three-dimensional accelerometers. Additionally, all buttons on the interface can be used in parallel, meaning that actions can be performed independently and simultaneously.Â
The WiiMote interface also allows for the two primary goals of a gesturally based electronic instrument:
Increased performability; and
Increased communication with the audience
Paine illustrates the capabilities of his interface in a composition entitled âEncounterâ, which comprises of a large number of synthesis variables. Some of the variables are automated in real-time while others are pre-set. The parameters under real time control are:
The control of buffer recordings (4 separate buffers)
The rate and density of granulation
The playback of a single sample
The control of two frequency/pitch variables
The same-and-hold rate of two oscillators; and
The composition itself draws on influences from John Cage and Terry Riley (in C), and the idea that musical decisions are determined through structural improvisation. The piece is made up of of staccato and legato phrases, as well as pitched and noise tones, in order to have a wide range of musical possibilities for the WiiMote to control. The composition also uses non-linear control of events, so that the performance has a wide range of compositional potentials, and is not confined to a distinct structural form.Â
3.2 The Software Performer
Advances in portable, handheld computing hardware & software have hastily been received by the live music performer. The technologies portability, multi-functionality, third-party software application development, and touch screen capabilities, have made items such as the iPad and the mobile phone a highly attractive interface for musical expression and performance. Devices also often contain âadd-onsâ such as microphones, bluetooth enabled page turning foot pedals, accelerometers, and touch screen velocity sensors, which are made use of by music software developers.Â
The portability and large accessibility of the devices, means that performers can make their performances more interactive, exploring the concept of using the audience as a source of sound. G. Levinâs âDial Tonesâ (2001) was among the first works to use the audience as an interactive sound source. Tanaka and Gemeinboeckâs installation piece ânet dâeriveâ (2006) used GPS and wireless communication to trace and display position information of moving audience members in the streets. Yamahaâs âau Design Project X) explored transforming the iPhone into a musical instrument, creating mobile phones that can be split into two for two drum sticks (Sticks in the Air).Â
While still in development, these portable software devices are allowing for new and unheard of ways of musical expression and live composition. By not imposing a ceiling on allowable software, developers are open to any musical possibility, only limited by their own creativity.Â
3.2.1 Case Study 1: Milne, Xambo, Laney, Sharpe, Prechtl & Holland, âHex Player - A Virtual Musical Controllerâ (2011)
In a paper delivered at the International Conference on New Interfaces for Musical Expression (Oslo, 2011), various members of The Open University UK showcased a new software application for the iPad 2 - âHex Playerâ. Their interface take the general keyboard interface, and transforms it into an array of virtual touch-sensitive buttons. The spacial locations of the buttons are transformable by shears and rotations, and their colour can be change to reflect their musical function in different scales. While developed to aid beginner piano players, the application can assist composers and musicians âseeking to explore the universe of unfamiliar microtonal scalesâ.Â
Rather then traditional piano keys, the interface consists of a lattice of hexagonal buttons. Each hexagon send a MIDI note which is mapped to; software, or external hardware, or a synthesiser. The blank sections either side of the hexagons are a control surface used to control a number of expressive parameters, such as timbre, volume, vibrato and tuning. This means that an expressive lead part can be played with one hand, and a bass line or chords by the other (rather then one hand to play the melody, and the other to operate the pitch-blend on a keyboard).Â
The Hex Player also offers a large array of possible pitch arrangements, with not just conventional western scales, but a variety of microtonal generalised diatonic scales. When a user choses a scale, the hexagonal buttons will automatically re-arrange themselves so that their spatial height corresponds with their pitch. Within a scale, the most useable scale steps run along row, the notes within the scale are light-coloured, while those outside the scale are dark or removed.Â
This application retains all the possible capabilities of a large, bulky keyboard in a small tablet device, but provides many more functional and practical tonal and structural capabilities.Â
3.2.2 Case Study 2: Oh, Herrera, Bryan, Dahl & Wang, âEvolving the Mobile Phone Orchestraâ (Sydney, 2011)
The Stanford Mobile Phone Orchestra, formed in 2007, is an ensemble of musicians with iPhones and wearable speakers. Taking advantage of the mobility and interactive capability of smartphone technology, the orchestra offers a new software interface design in a move away from the âlaptop orchestraâ.Â
Originally performed on Nokia N95 smartphones, the orchestra has moved to Apple iPhones due to its superior computing power and on-board sensors. To provide amplification, each member has wearable speakers, tested as a necklace, jacket or glove design, finally deciding on the glove speakers. The glove speakers allow each performer to âeasily change the directionality of the amplified sound by making simple hand or arm gesturesâ. The close distance between the speakers and phone also allow for the interface to be viewed as being more acoustic, as the dislocation of sound is less apparent.Â
The software for the iPhone uses an Objective-C programming language (XCode), and a graphically designed simple user interface. Â
Their first concert in December 2009 showcased the possibilities of the Mobile Phone Orchestra, performing the software instrument âColoursâ by Jieun Oh. Rather than structurally setting up in a traditional concert formation, the orchestra (due to the mobility of their instruments), surrounded the audience, and performers walked around in the performance space, âmaking various physical gestures from bouncing an imaginary ball to swinging armsâ.Â
Performers triggered polyphonic sounds by touching the screen in five different locations, moving in both horizontal and vertical directions, and given visual feedback through coloured rectangular markers. During a performance, the players may also walk up to another player to carry out a âsonic conversationâ, conveying different emotions through physical interaction.Â
The orchestra also showcased âinterVâ by Jorge Herrera, which uses the iPhone accelerometer as its principle method of control. âTwo axes of acceleration are mapped to the volume of two notes simultaneously playâ. The players controlled the sounds using gestures, allowing the audience a visual understanding of the playback mechanism.Â
While the amalgamation of music and technology is not a recent partnership, efforts to develop new and more effective ways of communicating electronic music to audiences are still in their early stages. Audiences are known to appreciate grand gestures, a direct coloration between âcauseâ and âeffectâ, physical effort and raw emotion between a performer and his/her instrument. Research into new gestural and software interfaces has produced a wealth of new designs for best communicating electronic music, and the field is growing. With each passing year, new interface design is being presented to the music community, and implemented into live music performance.Â
Garth Paine, Towards Unified Design Guidelines for New Interfaces for Musical Expression, Cambridge University Press (2009)
Thomas Mitchell & Imogen Heap, âThe SoundGrasp: A Gestural Interface for the Performance of Live Musicâ, NIMEâ 11 (Oslo, 2011)
Milne, Xambo, Laney, Sharpe, Prechtl & Holland, âHex Player - A Virtual Musical Controllerâ, NIME 11â (Oslo, 2011)
Oh, Herrera, Bryan, Dahl & Wang, âEvolving the Mobile Phone Orchestraâ NIMEâ 10 (Sydney, 2010)
E R Miranda, Brain-Computer Music Interface for Generative Music, Plymouth UK (2006)
Paul Doornbusch, Instruments from Now into the Future: The Disembodied Voice, Journal of the Australian Music Centre (2003)
Collins, dâEscrivan & Rincon, The Cambridge companion to electronic music, Cambridge University Press (2007)
Holmes, Electronic and experimental music: pioneers in technology and composition, Routledge (2002)