(synaesthesia and contemporary live art practice)

Misuse can mean the crossing of wires, both literally or figuratively. "Mind, the gap" is a practice-based research project dedicated to the development of collaborative, interdisciplinary, performative live artworks that are influenced by the notion of synaesthesia - the cross wiring of sensory perceptions.

Synaesthesia (Greek, syn = together + aisthesis = perception) roughly translates as "the bringing together of the senses" and refers to the phenomenon that the stimulation of one sense modality gives rise to a simultaneous sensation in another sense modality. In medical terms synaesthesia is often regarded as a disorder, while people with the synaesthetic condition often regard it as a gift.

Despite considerable scientific research there is little clarity as to what causes synaesthesia. Some studies seek to prove that everybody is born with synaesthetic abilities that diminish as a result of social conditioning, but give little evidence of why others retain their abilities.

The notion of synaesthesia caused a buzz in both scientific and artistic spheres in the early 20th century. Experiencing a revival in the 1960's, synaesthesia was connected to hallucinative experiences (often induced by drugs), the arts, science, pop/rock culture, new inventions - in short, the pursuit of a radically new way of perceiving the world. In the late 1980's Ecstasy intensified the sensations of ravers and clubbers to the brink of pre-hallucinogenic synaesthesia - a regurgitation of the 60's. Currently, psychologists approach synaesthesia foremost as a neurological phenomenon, while generally artists are exploring digital devices to elicit synaesthesia. In the case of the latter example most work has been carried out in the audio-visual sphere, and less focus has been given to more complex modes of performance where more sensory combinations are involved. (While several international institutions, such as the SIGGRAPH 2004 Conference Exhibition in Los Angeles and the ICA in London, have featured synaesthesia as an exhibition theme that reflects the current in interest in the topic, there is little evidence of work that stems from a performance perspective in these events.)

The development and accessibility of digital and communications technologies have increased the potential for artists to work in a multi-sensory mode. Current art practices utilizing digital technology build "instruments" in a "modular" fashion where digitized inputs from the analogue world can be broken down to component properties and those parameters can control or affect other properties of media. As such, they mimic synaesthesia. Networked environments allow collaborating artists to affect each other’s mediated inputs and outputs creating a cacophony of cross-wired connections that affect the total output.

The integration of digital and communications technologies in everyday life creates possibilities to experience the interaction and direct manipulation of combined sensory activity - a potential readily grasped upon by mass media and market industries. Additionally, prosthetic industrial technologies such as ultrasound (that renders surface images of depth via sound) simultaneously draw upon more than one sensory mode to reveal that which is otherwise hidden, and can be considered in synaesthetic terms.

The title of my project, "Mind, the gap", can be read as a warning sign. If synaesthetic influences are everywhere, and are commonly experienced, consciously or subliminally, by everyone, what is the point of conducting a research based on a focus on something as general as synaesthesia? While I must admit to be fascinated by the phenomenon of "genuine" synaesthesia, I am not concerned with issues such as whether or not notorious 19/20th century artists like Alexander Scriabin and Wassily Kandinsky possessed genuine synaesthetic abilities. However, I am interested in the social and historical context in which they formed their ideas and synaesthetic systems.

My background is in dance where cross-sensory integration was daily fodder throughout my education. I have worked professionally as co-director of the artist collective "Motherboard" for the past 9 years creating collaborative, hybrid, live art events, performances and installations that utilize and "misuse" digital and communications technologies in process and realisation. From a retrospective perspective several of these productions relate strongly to the synaesthetic notion.

The synaesthetic notion cuts across the imaginary frontiers of the senses and offers the potential of escape from the segregation of art genres. It embraces elicited synaesthesia via cross-wiring analogue, digital, poetic and conceptual devices.

"Mind, the gap" focuses on a formal/structural artistic approach to the subject to inform the development of a performative live artwork where synaesthetic strategies are employed to evoke both unusual sensory connections and experiences and new aesthetic potentials.

3.1 Questions

01) What methods can be found to identify and structure synaesthetic traits in performance/artworks?
02) Can an artwork be described as "successful" in synaesthetic terms in relation to:
a) Process, input and output - from an artist perspective?
b) Output - influencing the way an audience perceives a work?
03) What is the relationship between stimuli (input), feedback and perceptions (output), and time/space structures?

3.2 Process and product
While my aim is to produce an artwork at the end of my research period, emphasis is placed on hands-on process and experimentation infused with analytic and theoretic investigations.

I will conduct my research through the construction of an instrument with synaesthetic abilities called "The Emotion Organ" (described below in point 3.3). It will initially function as a research tool, and evolve as an artwork. The gradual construction of "The Emotion Organ" will feed into the research project "Mind, the gap" and vice versa. The diagram below represents the feedback effect of process and product.

My main digital tool for hands-on experimenting is Keyworx, a multi-user networked environment and media-synthesizing software currently in development at WAAG (The Society for Old and New Media, www.keyworx.org) in Amsterdam. Keyworx allows real-life gestures to enter and affect the collaborative environment via periphery devices such as cameras, microphones and sensors. I have been using Keyworx for both creating, sketching and prototyping artworks since 2000. When more refinement becomes a necessity, more specialised hard- and softwares will be utilized and expert advice and competences will be sought, drawing on national and international resources in the field of both the arts and science, and the existing network that has grown through my work with Motherboard.

By adopting a "working alone-together" approach I am able to remain relatively autonomous in my special field of interest while working collaboratively on specific areas of development and production.The diagram below represents my current networking partners:

3.3 "The Emotion Organ"
I plan to transform a donated Ostlind and Almquist pump organ from 1901 into a synaesthetic testing instrument, exploring the potential of rebuilding its existing mechanisms with electrical, mechanical and digital devices.

"The Emotion Organ" is inspired by a futuristic simulacra contraption of the same name that features in Philip K. Dick's novel, "We can build you" (1972). When played, the emotion organ could evoke the deepest of emotions in the shallowest of people. My Emotion Organ will be treated as a responsive body whose nervous system is made up of a networked combination of hard- and software connections. I intend to extend the organ’s inputs and outputs such that the player can "compose through experiencing", with and through combinations of elements such as sound/music, projected imagery, lights, scents/smells, and tactile and kinetic elements. As a networked instrument, the potential to include stimulus from the world beyond shall be explored.
Theoretically, the Emotion Organ should function as

a) A device for driving/inspiring initial experiments.
The organ is a reed instrument. Air is pumped in and out of its bellows via 2 foot-pedals. It has 61 reeds corresponding to 61 keys that route the flow of air to produce different tones, while 8 sliding knobs and 2 knee-levers produce mood changes. The organ’s existing logic will be used as a point of departure for extending its abilities by connecting diverse sensors and devices to its various parts, which will feed information into a computer program. The analogue input is translated to digital information. It passes through custom made digital "emotion filters" which modulate or transform various other predominantly live medias (for example, the video image of the player) and control other additional physical devices (such as the valves on airbrush bottles containing various aromas) to create a combination of diverse sensory outputs - that can also be used as "new" inputs. The challenge, then, is to design a system that is both functional and expressive. Early experiments will therefore explore how combinations of sensory information (primarily touch, sight, sound and smell) can be structured to elicit synaesthetic responses and produce mood changes, and to find out how these can be organized in terms of non-linear dramaturgy.

b) A diagnostic tool for "testing" a player's perception of the synaesthetic output.
Guests will be invited to take part in a variety of sensory experiments and prototype tests of the organ. Their responses will be used as feedback to inform the development of the organ itself. This will also help to define the risk factors involved regarding artistic value, technical functionality and time scheduling. The multitudes of possibilities will be sharpened down towards the final presentation of my project. Additionally, it may be feasible that the organ is presented publicly as a synaesthetic testing machine.

c) A potential live art "object" for players of any age, cultural background and ability.
The physical appearance of the organ is envisaged as an eclectic mix of ancient and modern gadgetry - simultaneously nostalgic and futuristic; a keyboard and a menagerie of sliders, knobs, buttons, switches, lights, bottles and tubes, including a built-in computer connected to the internet, a projector and a responsive piano stool. It should be seductive, inviting people to touch and play with it. When played with, the organ will release a synergy of sensory information, which, having passed through digital "emotion filters", evoke roller-coaster states of emotion that are not necessarily controllable, but never the less make sense. "The Emotion Organ" will therefore be a spectacle as an art object, and simultaneously the main instrument in a live performance.

d) A potential instrument for live performances by virtuoso players.

Imagine that a pianist is invited to play Beethoven’s "5th Symphony" on "The Emotion Organ" - and experiences very unusual results!

3.4 Presentation and documentation
"The Emotion Organ" will be presented publicly in the form of a live art performance/ installation in November/December 2006. The main format for the documentation of "Mind, the gap" is a comprehensive website containing:

a) Texts - project notes, references and reflections.
b) Visual representations - diagrams, illustrations and photographs.
c) Audio-visual media – reference and documentation media.
d) Relevant links.

The design shall clearly represent my process and promote easy use in terms of structure and navigation, and shall evolve in relation to my progress. The final version will also contain a print-friendly reflection of the presentation of "The Emotion Organ" based on my own assessments and the reactions of the public. The website should also function as a resource for a wider public.

3.5 References: articles and papers

(Syn)aesthetics and Disturbance - A Preliminary Overview
Josephine Machon, Body, Space and Technology Journal,
Vol 1 No.2, Brunel University

KeyWorx: A Working-Alone-Together Reflection - Collaborative practice on a collaborative product
Sher Doruff, Waag Society for Old and New Media, Amsterdam

Digital Space: A Summary
David Dunn and Woody Vasulka

Hypermedia and mediation
James C. Morrison, Media Ecology Association Publications, 2000

Tele-synaesthesia: presentation of a hypothesis
Dr. Hugo, 1997

Synaesthetic Performance In The Club Scene
Annet Dekker, Netherlands Media Art Institute, Montevideo/Time Based Arts Amsterdam

Kinetic Synaesthesia: Experiencing Dance in Multimedia Scenographies
Marc Boucher, Contemporary Aesthetics, Vol 2, 2004

New Directions in Audience Research. Advances in Reception and Audience Research 2
Willmar Sauter (ed), Tijdschrift voor Teaterwetenschap, 1988

Synesthesia: Phenomenology And Neuropsychology. A Review of Current Knowledge
Richard E. Cytowic, PSYCHE, 2 (10), 1995

[see the "links" section for URLS to several of these items]


Since my employment at the Norwegian Theatre Academy in May 2002 I have experimented with various forms of networked environments in relation to performance/live art situations. I have used and tested this research by attending and speaking at international symposiums, in collaborative performances and live art events with (amongst others) Motherboard, by leading workshops in Norway and Uganda, and producing a video work that has been screened internationally, as well as publishing articles. Through these activities and many conversations with colleagues, the notion of synaesthesia has gradually emerged as an important factor for how I approach issues of interactivity in such environments and appeals to my interest in cross-wired sensory connections.

Earlier this year I collaborated with my colleague Per Platou to develop a project called "The 8th Sister" that is, in principle, strongly connected to the notion of synaesthesia. This project has successfully received funding from Arts Council Norway and will hopefully be realized as a Motherboard event in the summertime, 2005. As "The 8th Sister" is relevant to my work leading up to this application, it will be included in "Mind, the gap" as research material and documentation.

This is my proposed schedule for the next 2 years.

December 2004
Refinement and public release of the website "Mind, the gap".

Spring/summer 2005
Identify and prototype key areas and difficult processes. Conceptual/artistic development. Website.

Autumn/winter 2005
"Soft" reconstruction work. Prototype tests. Decide on the final hard- and software platform. Conceptual/artistic development. Website.

Spring/summer 2006
Hardcore reconstruction of the organ, programming and testing. Conceptual/artistic development. Website.

Autumn/winter 2006
Focus on the artistic output of the work, adjustments and tests. Website.

November/December 2006
Public presentation of The Emotion Organ.
Presentation of the Emotion Organ and final documentation of my research project "Mind, the gap" for a selected evaluation committee.

The organ transformed by accessible, modern day technology draws a red line from the early 20th C, a period of rapid developments in the arts, science and technology, to the current day.

As an experimental interdisciplinary artwork heavily influenced by my background in the performing arts it should generate both national and international interest.
Today artists engaged in issues of how technology is affecting our environment and the way we live our lives are a driving force in offering alternative (and resistant) visions of the way our world is perceived. I hope that through my research project "Mind, the gap" and the consequent development of "The Emotion Organ" as an artwork, I can contribute to the expanding research in this field.

Amanda Steggell
(Oslo, 08.11.04)