CONTENTS

Schwarzchild Program Note

Creating Schwarzchild

Stills from Schwarzchild

Schwarzchild Program Note

"Schwarzchild explores the similarities possible between spatialised sonic and visual phenomena. It consists of objects and scenes that are determined by their sonic constituents. The production process incorporated a digital wind controller, a sampler, analogue synthesisers, phase vocoding techniques, a didgeridoo, an fm tuner, procedural 3D animation and visual effects software, 3D sound equipment and a supercomputing facility. The music was composed first, being largely an improvisation on a midi wind controller. This was then layered and edited with various other instruments until there were five seperate parts. These parts were then fed to the 3D animation software Houdini, which processed the audio and midi data to create all motion in the 3D models. The trajectories for each of the five parts were then exported to Lake Technologies 3D sound equipment. The result is a black and white stereoscopic 3D animation with a synthesised and spatialised soundtrack."

Composed, performed and animated by Andrew Lyons.

http://www.tstex.com

Duration - 14.3 minutes.

Audio - Binaural Encoding

Video - Stereoscopic Red/Blue Anaglyph Format

Performance Requirements - A darkened room, red/blue anaglyph glasses, stereo headphones and considerable volume.

3D animation Created using Houdini 3D animation software.

http://www.sidefx.com/

3D sound created using Lake Technologies.

http://www.lakedsp.com/index.htm

Thanks to:

The Sydney Conservatorium of Music: Dr. Gregory Schiemer.

 

Sydney Vislab: Ben Simons, Chris Willing.

NSWCPC: Youzhen Cheng, Dr.Russell Standish.

Lake Technologies: Roger Butler, Leonard Layton.

Side Effects Software: Greg Hermanovic.

Australian Research Council (ARC)

Melissah Kochel, my Parents, Brothers, and Sisters.

Creating Schwarzchild

1 Introduction

For as long as I can remember, my perception of music has involved mental imagery. This eidetic imagery occurs for many people in varying degrees of detail. The assignment of visual qualities to musical material is a key factor in my approach to music composition and improvisation. I often gauge music by its ability to precipitate coherent and spontaneous mental imagery and I augment my memory of works by the imagery with which I associate them.

There are three types of musical representation:

• Acoustical, i.e. based on the physical properties of sound (e.g. sonograms, spectrograms).
• Subsymbolic, i.e. based on the known behaviour of the human hearing system and how the human brain processes auditory information.
• Symbolic, i.e. based on the manipulation of symbols (e.g. Common Music Notation).

Although it explores acoustical representations, Schwarzchild is primarily concerned with symbolic representations. Ultimately, it searches for trans-cultural spatio-temporal metaphores for musical sound. In doing so it makes use of the sensory modalities of sight and hearing, and the three attributes common to all sensory modalities - intensity, duration and location.

2 Production Overview

In some ways, the original design brief for Schwarzchild was to represent music using 3D animation tools. Schwarzchild might be seen to follow in the tradition of such abstract film makers as Leopold Survage, Oskar Fischinger, the Whitney Brothers and Norman McLaren more than in the tradition of narrative and character based representations pursued in the Disney Fantasia films. Being part of a post-graduate research degree, it was necessary that the creative work referred to research undertaken in the preparation of a thesis rather than be of great entertainment value. This has been largely a serious, academic effort to represent music visually, although it has been tempered considerably at times by artistic license.

The brief therefore may be seen to have grown out of the development of a semiology describing the music, which was all that originally existed of Schwarzchild. These sound symbols then needed to be translated into visible, concrete correspondents. Towards this goal the tripartite stages of musical semiology, the poietic, the neutral and the esthesic levels were isolated for analysis. The phenomenological approach to perception utilised in Gestalt Psychology was reserved as a central epistemology for the analyisis of poietic and esthesic levels. These levels were understood to involve perceptual artifacts of both high and low level cognitive processes. Various de-differentiated modes of perception such as Synesthesia, Physiognomic Perception, and Eidetic Imagery were studied for what they described about the low level cognition of music and the universal nature of human cross-modal correspondance. The understanding of high level human metaphorical capacity involved the study of various musicological works dealing with the nature of mental imaging in response to sound. The analysis of the neutral level was completed largely by applying Houdini CHOP tools to midi and spectral analysis information.

In some cases objects were designed largely using the results of analysis of low level perceptual styles, other times intuitive eidetic imaging was the primary approach to 3D design. In some cases the channels themselves and the possibilities of the software suggested ideas. The emphasis shifted continuously from section to section.

One additional design criterior for Schwarzchild was that it in some way use 3D sound and stereoscopic reproduction equipment. Whilst this was initially a secondary requirement, the recognition of location as an important universal component of all sense datum was eventually understood to make the use of this equipment essential. It eventually became apparent that the visual representation of sound could not be satisfactorily attempted without the use of 'immersive' reproduction technology. This is supported by the Gestalt writings of Max Wertheimer regarding the omnipotence of "Common Fate" in Gestalt grouping of phenomena.

The completion of 15 minutes of 3D animation without the use of any key framing may be regarded as novel in the world of 3D animation. Schwarzchild is also notable as perhaps one of, if not the first colour music work to explore synthesised spatial reproduction techniques. The development of the piece Schwarzchild and the related research involved numerous stages. These are in summary:

2.1 Stage 1 ( Early 1998 )

1. Houdini CHOP familiarisation.
2. Preliminary composition tests using Houdini's CHOPS.
3. Preliminary animation tests using Houdini's CHOPS.
4. Evaluation of early product. Recognition of basic difficulties, and need for more research to avoid hopeless subjectivity and meaningless pastiche combinations. Intensive reading resumed.

2.2 Stage 2 ( Early 1999 )

1. Composition of Schwarzchild begins as daily experimentation and improvisation with a Yamaha WX5/ Ensoniq Sampler combination.
2. 3D sound familiarisation continues. Acquisition of intuited understanding of circumambient sound, its strengths, disadvantages and special design requirements, through extensive experimentation within the Sydney Conservatorium Lake Huron 3D sound laboratory. Experiments with 3D sound effect changes in the over-all composition. New sections are added and others removed.
3. Composition of music for Schwarzchild completed using ARP Axxe, Moog Prodigy, Yamaha WX5 midi wind controller, Ensoniq ASR10 sampler, FM radio tuner, Didgeridoo, and Csound.
4. Experiments with stereoscopic techniques. Red/blue anaglyph technique isolated as most portable. Stereo camera configurations explored. Optimal inter-occular distances and colour combinations developed. Black and white images decided upon due to their effectiveness using red/blue anaglyph techniques, and also to avoid the involvement of colour, for which there are no universal perceptual correspondences to sound. A colour's emotional significance and its association with any particular sound or music varies widely from person to person. This will be explored perhaps in future works.
5. Development of Houdini script to ouptut Lake "Loc" scripts for input into Lake Software.
6. Preliminary black and white test animation completed in which primitives were assigned to points, the location of which which were then exported to the Lake software and Huron hardware. Initial performance of animation in conjunction with 3D sound generated by Lake Huron equipment. Recognition of lack of aesthetic depth in animations based purely on the results of low level psychological research. Research into high level metaphore and eidetic approaches expands.

2.3 Stage 3 ( Mid 1999 )

1. Repeated eidetic observation of completed music conducted whilst in hypnogogic states.
2. Comparison of various ideas developed during different eidetic sittings. Rough script outline drawn up using this information and understanding of perceptual constants from research.
3. CHOP nets for each track developed. Midi file data imported and spectral analysis completed using chop trees developed using scripts developed in C. Later the spectrum chop became available.
4. Trajectories developed for each voice using Chops.
5. Modelling begins simultaneously with the development of chop tree's for each voice.
6. Rendering begins.
7. Rendering time on Vislab 20 cpu Power Challenge successfully applied for. Repeated extensions granted. At times 32 cpu's rendering at once around Sydney Vislab. Hscript usage refined greatly in render script writing and manipulation of large hip files already local on Power Challenge file system.
8. Development of panning patch for eight speakers developed in Houdini using Chops. Eight tracks successfully written out using Houdini to CDROM for performance at conference.
9. Development of compositing networks.
10. Rendering of dropped frames continues.

2.4 Stage 4 ( Late 1999 )

1. Composite trees revised.
2. Dropped frames re-rendered.
3. New Lake Huron software "Sonic Animator" arrives at Conservatorium 3D sound laboratory. Experimentation continues there.
4. Composite of bed components completed. Disk usage reduced greatly.
5. New 3D components developed and rendered.
6. Final composite of body of work completed.
7. Credit and title sequences developed and rendered.
8. Performance of Schwarzchild takes place at Conservatorium 3D sound laboratory in front of staff and students of the composition unit. All audience members sat within an 8 speaker cage and watched Schwarzchild projected on a large screen wearing red/blue anaglyph glasses.
9. Conversion of Masters degree to PhD subsequently approved.
10. Binaural components of audio created using Lake Huron equipment. Final stereo/binaural mix completed using components treated using Lake, Csound and Silicon Graphics software.

3 The role of Houdini in the creation of Schwarzchild

3.1 Chop Art

Schwarzchild could not have been completed using any 3D software other than Houdini. Colour music works have been regarded as a "new art", by numerous artists this century. Houdini is the best available tool for the creation of Stereoscopic 3D colour music works. I consider Houdini to be a "Creative Paradigm" rather than a software package. It is perhaps too delimited to be categorised as the latter. Its openness and extroversion make it a powerful hub for any creative audio-visual project.

The seamless integration of audio visual components is grossly under exploited within many media. Numerous studies, targetted at advertising markets especially, have indicated that sonic presentation is 30% successful in delivering a message, visual presentation is 40% successful, but the seamless combination of both can be as high as 90% successful in its communicative ability. The ease and power of Chops in the creation of such media is a major marketing point for Houdini.

3.2 Hscript Usage

The rendering of 200,000 frames by one animator was greatly facilitated by the power provided by hscript rendering scripts. These scripts made the seperate rendering of numerous components possible using one hip file. Other helpful software included perl scripts that sorted frames by file size, allowing partly rendered frames to be isolated and deleted. This process was followed by the use of scripts that wrote lists of frames missing from sequences. These were then easily converted into hscript render scripts and added to queues. The use of hscript also facilitated the automation of repetetive audio treatment tasks and the development of chopnets to create them.

3.3 Summary

The development of Schwarzchild is so irrevocably entwined with my use of Side Effects Software that is hard for me to isolate aspects for which Houdini should be credited. Since first learning to use Prisms in 1995, my thinking about 3D animation has been developed with the power and flexibility of Side Effects Software in mind. I have made use of many other 3D software packages but found them to be largely impotent creatively. Creativity was possible with these packages, but largely in order that their inherent restrictions in applicability could be transcended. Their insulting presumption of user idiocy alienated me. Mountains of GUI fluff do not help this situation.

There is so much rich and intuitively resonant time based information locked within music, and the harnessing of this information by a visual arts software package is ingenious to say the least. The cinematic tradition has maintained many stylistic characteristics which were initially merely dictated by the technical limitations of its original technology. As a time based art, it has much to learn from music, which has a heritage many thousands of years older. The use of hscript, chops, dynamic systems and expressions has enabled me to set up 15 minutes of animation using information sourced entirely from my music, over a period of about a month. ( The rendering and compositing then took about four months. ) Whilst the depth of the work is perhaps adversely effected by the time devoted to animation, a work of this length would have been inconceivable in any other package in the same amount of time. Furthermore, it is a test work, and will be followed by vastly superior ones.

4 Conclusion

In developing this system of metaphor, the doctrine of Phenomenology and techniques of Gestalt Psychology have provided an epistemological basis for the research underlying Schwarzchild. Musical semiology has provided a structure for the organisation of the research. Useful data in the analysis of cross-modal metaphor has been obtained from Psychological studies of musical perception and de-differentiated perceptual styles. These perceptual styles include the pathological condition known as Synesthesia, in which stimuli sourced in one sensory modality gives rise to the perception of phenomena in another. Related conditions include physiognomic perception, in which secondary sensory qualities from one modality are applied to that of another. (eg. high and low pitch).

Also, according to Foucault's collateral, correlative, and complementary spaces, and in Deleuzian and Guattarian terms as well, audio-visual metaphor results from the deterritorialisation of the aural and visual senses. A Foucaultian statement in the context of Schwarzchild, is not to be associated with the transmission of particular elements presupposed of it, but with the shape of the whole curve with which sonic and visual phenomena are related. In Schwarzchild, this curve may be regarded as "energy shape", plotted over time.

Schwarzchild has provided considerable insights into the difficulties involved in conceptualising and realising the symbolic integration of 3D audio and 3D image. These insights include:

• Issues related to the simultaneous and integrated conception of visual and sonic elements.
• An intuited understanding of issues related to spatial composition such as visual occlusion and sonic proximity.
• Motion and its relationship with perceived sonic and visual mass.
• The brightness, hue and saturation of colour and their relationship to frequency attenuation, doppler shifting, and source to observer vector information.
• The sonic headroom provided by circumambient reproduction technology and its facilitation of the composition and combination of complex textural sonorities rendered cacaphonic by domestic reproduction equipment.

My research in the years to come will continue to develop a semiology of cross modal metaphor and techniques with which to implement this semiology succesfully within spatio-temporal works. Increasing support for my work at Sydney Vislab has included the purchase of midi interfaces for graphics supercomputers with which I can interface electronic instruments for realtime experimentation. The resultant increase in the rate of experimental feedback should accelerate research developments greatly.

Stills From Schwarzchild

The following frames are quarter resolution stills from the completed 14 minute sequence. If viewed wearing red/blue anaglyph glasses, some stereo effect will be perceivable despite the low resolution. The animation is badly decontextualised when isolated from the music.

Top