DECEMBER 04KEYWORX TRIALS
During December I experimented with various media control posibilites and dramturgic scenarios using the Keyworx software by:
- Using the left and right envelope properties/data of different sound files (mp3s) to generate and manipulate visual media.
- Using the sound envelope properties to control the speed of the playback of audio files (feedback).
- Using video tracking of visual media to control the volume of audio playback.
- Using the video tracking data to turn visual layers on and off.
Another aim was to look at varying degress of subjectivity in terms of input and output by experimenting with both abstract and concrete imagery.
Example 1. "Academic synaesthesia"
Media:
Computer generated hues/colour
1 jpg image
Quicktime instruments/piano
1 mp3 file (looped)
Control:
Left and right envelope of the mp3 file (passed through an echo filter)
Video tracking (object size)
This example uses abstract imagery only. Data derived from the audio file is used to generate different hues of colour and horizontal scale in the underlying video layer (the vertical "stripes"). The audio file was first put through an echo filter to obtain additional "echoing" data to make the visual imagery more lively. The data derived from the left and right audio envelope was inverted and also used to control the piano velocity and pitch of the piano sound in quick time instruments (midi). The mp3 file was not played back (not heard).
The jpg image is the black/green/yellow rectangle in the centre of the imagery, and constitutes the foremost video layer. The size of the image is controlled by inverting the data derived from the audio file. The video layer is turned on and off via video tracking of a red hue in the underlying video layer. As I have applied both luminance keying and transparency to both video layers, they merge, and both remain visible, even when the foremost layer is turned off.
"Academic synaesthesia as QT movie [320x240, 2.7mb, 0.50 min] here
Example 2: "How does it make you feel?"
Media:
1 qt movie (looped)
1 mp3 (looped)
Control:
Video tracking on the Y axis controls the volume of the mp3 file
My idea with this example was to pursue the notion of the abstract derived from the concrete in visual terms. I searched the net for QT movies with a simple colour range where it would be easy to track a specific colour. My search criteria was that the imagery should be:
- figurative (or concrete),
- uncomplex in terms of form
- could be associated with abstract forms
- clear directional movement.
The best example I found that reflected these conditions was a video recording of 9/11 - a plane crashing into a tower and emerging the other side of the building. The visual composition is:
- a dark blue tower on a light blue background.
- as the plain flies through the building (horizontal axis),
- flashes of orange appear as square pixels
- the flames follow the smoke clouds as they ascend (vertical axis).
When the video tracking data registers orange pixels as an "object" the volume of the Air soundtrack "How does it make you feel" is turned on to maximum volume. A glitch occurs as this happens, causing the soundtrack to make a loud crackle before it plays "normally", increasing the impact of the plane crashing. As the object (orange pixels) moves vertically upwards, the volume is turned down. Additionally, the object "numbers" are displayed and the tracked areas are outlined in green in the visual imagery.
"How Does it make you feel" as QT movie [320x240, 8.6mb, 2.33 min] here
Example 3: Vivaldi's Nightmare
Media:
Computer generated yellow guages
3 qt movies (looped)
1 mp3 file (looped)
Control:
Left and right envelope of the mp3 file (passed through an echo filter)
Video tracking on an object size
The object of this exercise was to make something that could be experienced as Faustian claustraphobia by inverting an existing classical compostion/dramaturgy that is generally considered to express something else.
As the main sound input I used Vivaldi's "Four Seasons/Spring". I wanted to see if I could use the compositional/dramaturgic structure Vivaldi's music and, by extracting data derived from the audio envelopes, apply it to the visual dramturgy, add an extra sound (a baby crying) and thereby invert the expression conveyed (emotional output). In other words, using and inverting the data derived from the original composition to invert the dramaturgy of the new composition.
The Spring mp3 file was passed through an echo filter and the left and right envelope was used to control the horizontal and vertical offset and scale of the main visual content (a qt movie sample of an animated man's face). These values were inverted to control the horizontal and vertical offset of another qt movie - a train light appearing as a blinking red light (thought: what if the red light in the visual imagery was also manifested as a "physical" red light?). The scale of this movie was controlled by the left and right envelope values of the baby crying mp3 . As the envelope data reached a high rate, the animated head movie was swapped for another movie sampled from the same source material, and mirrored. When the left and right envelope data exceeded a given value the audio data was represented visually as yellow guages. The speed and volume of the baby crying mp3 file was controlled by the envelope data from the Vivaldi/Spring audio file.
"Vivaldi's Nightmare" as QT movie [320x240, 12.8mb, 3.49 min] here
The above examples are just a few of the scenarios I worked with. Once opened, the Keyworx patches are self-running and demand no other input. Within each Keyworx patch I experimented with different audio and visual media content to see how the expression changed. I also considered how the various controls could be substituted with other controllers/sensors in relation to the technification of the pump organ. This would mean breaking the tight connection between the audio and visual media, and, to some extent, handing it over to the player.
MEDIA CREDITS
"Academic synaesthesia"
MP3: Sound effects/breaking glass.
"How does it make you feel"
Mov: http://mteww.com/9_11/index.html
MP3: "How does it make you feel", Air.
"Vivaldi's Nightmare"
Mov: 2 samples from "I'm not Mad", Kon Koumis, 2004.
URL: http://www.hash.com/animationcontest/Q0204/04.mov
1 sample from "Little Movies - Binary Code", Lev Manovitch, 1994-97.
URL: http://www.manovich.net/little-movies/menu3.html
MP3s: The Four Seasons, Spring, Vivaldi. Sound effects/baby crying.