WEAVE 03.2011 – Interview: Making Music With Muscles
The new media and sonic artist, performer and teacher Marco Donnarumma (http://marcodonnarumma.com) works hard to develope an OpenSource Framework for augmented muscles sounds – the so called »Xth Sense Biosensing Wearable Technology«! WEAVE talked to him about how it all came, what the technological basis of »Xth Sense« is, and what his future plans are.
When did you have the idea for that framework? And how did that happen?
Well, the long version is… It would be difficult to say when it did happen; the Xth Sense project represents a further stage of an ongoing research focused on the augmentation of the human body and its environment. In this context I’ve been developing reactive computing systems which were deployed in live media performances, participatory works and generative applications.
In 2007 I began the implementation of C::NTR::L (http://cntrl.sourceforge.net/), an audiovisual framework for real time human computer interaction based on pitch tracking and score following.
At the time I simply felt the need to create a specific tool which would allow me to better concentrate on the performative act rather than the technological aspect of a live performance. So I used C::NTR::L to augment a traditional electric bass guitar in order to control audio and video processing in real time simply playing my favourite instrument and forgetting about the laptop. I had been working on this project for almost 3 years and, although the system had reached a good level of interactivity, at some point I felt again constrained. Playing, or even improvising with a traditional musical instrument requires the performer to act within a wide but still fixed set of references: hitting a percussion, plucking a chord are universal gesture which, even if performed unconventionally, contribute to an expected and recognizable somatic behaviour of the performer.
I spent summer 2010 studying the possibilities of physical computing and gestural control of music, and in September I started researching biofeedback applications for Sound Design at The University of Edinburgh under Dr. Martin Parker. During our early conversations I started growing the idea of approaching the human body as an actual instrument. Augmented musical instruments, physical computing techniques are generally based on the relation user>controller>system: the performer can interact with a control interface (a physical controller or sensor systems) and modify results and/or rules of a computing system.
However, I find that sometime this approach can confine and perhaps drive the kinetic expression of a performer, leaving less room for his/her physical energy and non-verbal communication. Besides, being that often the sonic outcome of such performances is digitally synthesised, sometime the overall performance can lack of “liveness”. The short version is… While researching biofeedback techniques, I collected several publications in the area of biomedical computing and eventually learned that muscles produce mechanical vibrations, which are nothing but sounds (MMG). It was surprising to realize how, in computer music and general interactive systems, muscle sounds seem to be overlooked; what also fascinated me was the idea that muscle sounds could retain a meaningful sonic vocabulary. Overall the topic seemed to be worth further efforts and research and I finally realized that it could have been the basis for the experimental paradigm of musical performance I wanted to explore. Human body as a musical instrument, and its kinetic energy as its sound. What is your intention by developing »Xth Sense«?
Generally, I’m interested in exploring experimental modalities of performance in digital environments. More specifically, I’m concerned with the use and abuse of technology and the modalities by which we can preserve, and perhaps improve, the primal, essential and unique features of human body in the context of digital interaction. I have a controversial approach towards technological advances, and the paradigms I explore tend to avoid a mere celebration of technology itself, but attempts at using new technologies to enhance our own perceptive experience and physical behaviours.
In the era of smart-phones and the Internet of things, apparently we are now able to put less and less effort in our daily interactions with computing systems, and this phenomenon seems to be mirrored in digital creativity and performance practice too. Even though I don’t deny that such concept is motivating successful and useful advances in digital interaction, I believe we shouldn’t forget about the bodily experience of a concert, the expressive and metaphorical potential of human body, and the inner and intimate perceptual adventure we undertake everytime we perform, interact with a system or attend to a spectacle. There’s a good quote by Martin Scorsese, who once said “I love the energy of concerts. Music and performing is a primal form of communication, more so that any kind of visual medium.”
Do you develop the framework on your own?
As mentioned earlier, I’m implementing the project within a research at The University of Edinburgh; here I found a strongly creative environment and very good resources. Although I’m the only author, the hardware development was initially supported by Dorkbot ALBA (http://dorkbot.noodlefactory.co.uk/wiki) and the EdinburghHacklab (http://edinburghhacklab.com/). Together with them I could develop a first circuit for the biophysical sensor which I later improved and then embedded in a wearable device.
The reactive system and its underlying structure are a branch of C::NTR::L, the software mentioned above, and even if several colleagues shared with me inspiring insights and suggestions, I’m the only programmer and artist involved in the planning and realization of the project. Recently, for the production of a new participatory work based on the Xth Sense and commissioned by Inspace (http://inspace.mediascot.org/research/non-bioboom), Edinburgh, I teamed up with Brendan F. Doyle, a talented American sound and light engineer.
The residency was a success, so now we are collaborating on my next solo performance; we want to develop a more complex show with a peculiar sound setup for eight channels; we aim at expanding the control of the performer over the stage, enabling an augmentation of the whole surrounding environment; the project will involve biophysical control of sound, lights and live visuals for a full augmentation of performer’s body, and will provide the audience with an immersive, enveloping experience.
What is the software basis of your framework?
Being that the use of open source tools is an integral part of the research, the framework is being developed on a Linux operating system. As for most of my other works, the software is programmed in Pure Data (http://puredata.info/), a real-time graphical programming environment for audio, video, and graphical processing, originally developed by Miller Puckette and now collaboratively maintained and expanded by a wonderful community of artists, programmers and musicians. I use Pure Data since more than five years and I truly love this tool and its community; it’s a great and powerful project and being an active member of the community is fully rewarding in terms of inspirations, exchange, support and also social relationships.
On a side note I would like to mention that the next Pure Data convention is taking place at the Bauhaus University at Weimar this August (http://bit.ly/dXnRpU); this is the best chance to discover all the different ideas and project which uses Pure Data and personally get in touch with maintainers, contributors and artists involved in the project. The Xth Sense computing system is actually the most important part of the framework; it includes several features such as a muscle sounds features extraction and calibration system, multi-layered mapping of biophysical signals to control data and real time audio processing of muscle sounds. The software comes with its own library and a user-friendly interface designed to facilitate a fast workflow and easy prototyping.
How does it work on the hardware side?
The hardware consists of a custom MMG sensor embedded in a wearable device. The sensor is very accurate and can be positioned in many different points on the body. It captures muscle sounds and transmits them to the laptop. Presently the performer needs to “be wired” with a sound cable, but together with some fellow researchers at The School of Informatics at Edinburgh University, I’m looking into the implementation of a wireless system, which would further enhance the capabilities of the framework. During this summer a paper of mine which discusses the hardware and software design will be published in the proceedings of the Linux Audio Conference 2011, and I will be presenting a related short paper at the ICMC too; I look forward to these conferences as I will have further chances to discuss the project in depth and gather useful feedbacks.
The piece of you that I saw sounded quite spherical. Can one choose which kind of sound one wants to make? Or is this simply the sound of this »instrument«?
The piece you mention is titled Music for Flesh II (http://marcodonnarumma.com/works/music-for-flesh-ii/) and it’s the first performance I designed with the Xth Sense system. The piece explores a coupling between muscle sounds and theatrical gesture; it is based on several sound/gesture, namely specific movements which produce a correspondent sound. Given the nature of this technology the sonic outcome is strictly related to the kinetic behaviour of the performer. My performance is an improvisation describing a haunting sonic space, but it represents a personal aesthetic choice; different postures and gestures would produce different sonic responses, and the sound processing chains also play an important role in the final sound material.
At the same time, muscle sounds are mechanical vibrations exerted by a living body, thus they retain peculiar characteristics, such as fast dynamics and complex sound spectrum, which make them distinctive. The Xth Sense resembles some aspects of a traditional musical instrument; it has a specific timbre, it produces sound when excited and requires training; in fact an awareness of the use of one’s body is fundamental. When I give presentations it can happen that somebody who tries the instrument for the first time can’t produce any useful sound! Once you learned how to use it, the Xth Sense is fairly intuitive because it is based on a very simple process, one’s kinetic energy becomes sound.
What are your future plans: do you want to sell Xth Sense some day? Or do you treat it as a non-commercial art or research project?
I’m still in the process of exploring this technology and its paradigm, thus I find difficult to classify it other than a “research”; it is a strongly interdisciplinary investigation which opens exciting prospects in diverse fields, like music technology, digital art, dance, theater. I will keep on the research and will be touring with a new performance and a Xth Sense in-depth workshop; it’s an intensive course focused on biophysical generation and control of music.
Participants build their own biosensors, learn comprehensively the software tools and are trained in gestural and bodily musical performance. I just taught this workshop at NK in Berlin and I have to say it was an exciting experience both for the students and myself. At the moment I’m not interested in selling the Xth Sense; one of my main goal is to release it as an open source framework for biophysically-driven performances. I think that the DIY world community and awesome projects like Arduino or Pure Data keep showing the reliability and of open source and sharing models when it comes to the creation of innovative artworks and technologies. The open source ethos underpins my artistic research both in the processes I develop and the outcomes I aim for.
The state of art of biosensing systems for computer music includes devices which are generally expensive and difficult to build for someone without a specific knowledge. Moreover, there exists a trend today, which sees off-the-shelves, proprietary devices being used in unconventional ways or hacked in order to fit more artistic purposes, such as Apple iPhone and Microsoft Kinect.
I’m going to be critic (without wanting to get political) about this practice and I’m aware many readers will possibly not agree me. However, it’s my belief that when using a proprietary tool, especially for an artistic, creative scope, we foster a standardization of contents and processes, leaving increasingly less room to those truly innovative ideas which usually born out of a challenging, inspiring and heterogeneous milieux.
That said, I don’t mean that the Xth Sense project will provide the ultimate solution for an open source framework for augmented body performance, because the beauty of the open source community is its surprising variety of solutions to a given issue. What I do aim for is an efficient and accurate framework for biophysical performance which would be low cost, re-distributable, customizable and easy to set up.
