New project combines real world with digital elements

Digital technology and a technique known as augmented reality are being used at York University to transform a classic German play about a controversial murder trial into an interactive musical, with the audience in charge of the order of the scenes.

Graduate student Rebecca Rouse and a team of collaborators have created the installation in the Future Cinemas Lab at York. The play, Woyzeck, was written in 1835 -1836 by German playwright Georg Buchner, who died before finishing it. However, the story of Woyzeck, the soldier and drifter who murdered his female companion in a fit of jealousy in 1821 Leipzig, has survived and has been interpreted many ways.

Unlike virtual reality, which uses only computer-generated media, augmented reality combines the real world with digital elements. It has been used most often for military, medical and other scientific applications. In this case, however, it is being used for art, in a lab designed to explore how new technologies can change storytelling.

An audience member who visits the project – and only one can visit at a time – enters a dim room which appears empty except for a number of objects hanging in mid-air: dolls, a dollhouse, a book, a shirt, a document and a life-sized mannequin. The audience member is wearing an augmented reality headset which enables him to see a projected scene – perhaps dancers – and he may hear music or, when he touches an object, may hear lines of dialogue.

The augmented reality system consists of a tracking grid mounted on the ceiling of the installation space, sending out ultrasonic and inertial pulses to sensors worn on the head and hand of the audience member. The audience member sees and hears different scenes depending how he moves. Video and audio clips are sent from a computer to the headset and goggles.

"Most virtual reality and augmented reality projects are developed to highlight the technology’s capabilities, with the story sitting in second place," says Rouse, who recently graduated with a master of arts degree from the Ryerson/York Communications & Culture Program. "In contrast, in Woyzeck, the main character struggles with insanity and hallucinations, which can be particularly well evoked through augmented reality at this stage in the technology’s development. Augmented reality images often appear ghostly or hallucinatory."

Right: Rebecca Rouse with augmented reality headset and lead character Woyzeck

A two-year project, Rouse acted as director, translated the play from German, and distilled the play from 24 to 13 scenes because augmented reality work is so time-consuming and labor-intensive. A graduate of Brown University in Providence, Rhode Island, she developed the music with her long-time musical theatre writing collaborator Brendan Padgett at Columbia University in New York, and worked with modern dance choreographer Kyle Shepard, also of New York City.

Rouse’s augmented reality installation is the first project to come out of the Future Cinemas Lab in York’s Faculty of Fine Arts, said Rouse’s thesis supervisor York Professor Caitlin Fisher, Canada Research Chair in Digital Culture. With a run-time of one hour, it is one of the largest augmented reality projects ever done in the arts and entertainment field. The Future Cinemas Lab is the first lab in Canada dedicated to exploring how new technologies transform conventional experiences of fiction creation.

The installation ends at York this week and viewings are fully booked. However, a Flash version will be available by the end of the year. For more information, visit the Woyzeck Project Web site.