Prof to investigate interactive audio

by Angela Roorda, from the Arts Research Update newsletter

Having just arrived in town, Karen Collins is getting her lab set up and her pencils sharpened in readiness for the work ahead. Joining the Canadian Centre of Arts and Technology as its new Tier II Canada Research Chair in Communication and Technology, Collins faces an ambitious research program: not only will she be conducting the theoretical work of developing a semiotics of sound that better accounts for the increasing interactivity of new media, but she will be using this theory as a basis for developing a software engine capable of composing real-time audio for interactive multimedia applications (educational software, video games, interactive museums and the like).

She is no newcomer to the theoretical side of sound, and she comes to Waterloo well versed in the field of music semiotics (the study of music as a symbolic language). In her doctoral studies at the University of Liverpool, she sought to better understand the function of sound and music in film; however, she found the existing semiotic models to be wanting.

Semiotics of sound is new

“With a few exceptions, the vast majority of work in music semiotics has focused on music as an entity separate from other forms of expression,” she explains. “The semiotics of sound in media is a relatively new and as yet largely unexplored field. It is a field that was opened up in 1979 with the publication of Philip Tagg?s groundbreaking study, Kojak: 50 Seconds of Television Music, a book that updated traditional musicology to account for the impacts of popular music and mass media.”

Working with Tagg, and with a research focus on the role of music and sound in science fiction cinema, Collins succeeded in expanding existing models so that they would better account for the additional layers of meaning and connotation generated by mixing visual and auditory cues. In post-doctoral studies at Carleton University, she took the next step by moving from linear cinema to interactive media.

As Canada Research Chair, she will continue this research, further refining her models to account for the impact of sound in media environments where audiences play an active role in co-creating the audio-visual experience. This is no simple task. Interactive media’s non-linearity (and hence unpredictability) is a challenge not only to semioticians but to designers as well. On top of this, there is the fact that sound and music have a deep connection to human emotion and can produce profound, if sometimes unrecognized, affective impacts on media users.

Towards an on-the-spot digital composer

Collins hopes to develop a system that will begin to codify some of these affective dimensions of sound. By doing this, she will not only pave the way to better understanding their function in interactive media, but she will be laying the groundwork for media developers to more seamlessly, and more effectively, incorporate sound elements into their products. Eventually, the plan is to use these findings to create a sophisticated software engine capable not only of selecting appropriate segments of audio in real time, but of composing, on the spot and in response to user cues, affectively appropriate and musically coherent sequences of audio for interactive applications.

To do this, Collins will work with technical experts but also draw upon her own extensive industry experience as a web developer and digital designer.

As well as playing a central role in the multi-billion dollar video games industry — which has been surprisingly slow in incorporating the latest research findings on sound — interactive audio is increasingly being adopted in a wide range of other applications, including computers, cars, mobile phones, web sites, theme parks, museum exhibits, education centres, and shopping checkout systems. “It’s crucial that humanities and social sciences researchers participate in and help shape this burgeoning new field of media and communications,” says Collins. “My research will help bridge the gap between academia and the new cultural industries.”

Collins is an assistant professor in UW’s Department of Drama and Speech Communication. Funding for the new interactive audio lab will be provided through a CFI infrastructure grant of more than $100,000.

Back to top

Notes from near and far

Gatherings here at home...

The Office of Research and the Graduate Studies Office are hosting a “Fall Faculty Workshop, Lunch and Tradeshow” on Friday, September 7, 11:30 am – 1:30 pm, in MC room 2017. It’s an opportunity for faculty members — especially new ones, but not exclusively — to meet staff from the Office of Research and the Graduate Studies Office and learn about services that these units provide to faculty, as well as about funding resources. RSVP by email to smsanti@uwaterloo.ca by August 24.

... and on the far side of the world

Virginia McLellan writes, once again, from the other side of the globe, where she expects to be until September 1. “I am back in India again this year,” says McLellan, who is UW’s manager, international marketing and recruitment, “to participate in the Canadian Higher Education Committee high school visits. The tour includes 14 other Canadian universities and we will be visiting 22 top national and international high schools in 5 cities — Bangalore, Mumbai, Pune, Delhi, and Dehra Dun.

“In addition to these school visits I have also organized two UW information sessions at the Canadian Education Centres in Bangalore and Delhi. We have invited prospective undergraduate and graduate students to attend these sessions.” She will also be meeting five UW alumni at a gathering in Delhi.

India is a top priority for UW’s internationalization efforts, McLellan writes. “This will be my second visit to the country, but it follows a number of initiatives over the past years, including David Johnston’s visit to the region last winter with the premier’s delegation, and agreements with a number of institutions, including six Indian institutes of engineering and technology (Bombay, Delhi, Kanpur, Kharagpur, Madras, Roorke), Chennai Mathematical Institute, Jawaharlal Nehru University, Tata Consultancy Services, and Thapar Institute of Engineering and Technology.”

CPA staff

Back to top