The Gonzo Scientist

Play It Again, Robot

See allHide authors and affiliations

Science  21 Mar 2008:
Vol. 319, Issue 5870, pp. 1613
DOI: 10.1126/science.319.5870.1613b

John Bohannon

A series of reports on connections between science, culture, and the arts from Science Contributing Correspondent John Bohannon, who, in true gonzo style *, will participate in the events he covers.

Hardly anyone noticed when the robots started playing the music. But then, they did face an uphill battle for attention. There were no chairs for the audience. So the 100 or so humans were all on their feet, chatting with each other, laughing, and swigging from bottles of beer purchased at the entrance. And even if they had been seated, there was no stage to focus on. Instead, the dozens of machine performers were everywhere—on the ground in the corners, attached to the walls, clamped to scaffolding overhead—and it was difficult to distinguish them from the rest of the electronics crammed into the space. About 10 minutes into the performance, when the robot sounds briefly rose above the human noise in a great clanging, drumming, screeching cacophony, a young woman nearby paused in her conversation to exclaim, “God, that's annoying!” Then, after scanning the room, she said, “Wait, is this part of the show?”

The show in question was the first of two concerts I attended last month to assess the state of the art of robot music. This one was the output of a collaboration between (human) composers and an ensemble of (machine) musicians called LEMURbots. The composers spent the month of January as the first artists in residence here at the LEMURplex, a laboratory-cum-performance-space in Brooklyn, New York. (Where else?)

To judge the output, I planted three roboticists and two professional musicians in the audience. Two of the roboticists, Daniel Paluska and Jeff Lieberman, were taking a breather from their own robot studio. They were putting the finishing touches on a machine ensemble known as the Absolut Quartet, the next stop on my robo-tour. My plan was to feed their bots with the music now being played by these bots—to see what it might sound like when artistic machines start talking to each other.

What's Ahead

All illustrations by Katrien Kolenberg

The LEMURbots

The piece ended just as it began, not with a bang but an almost imperceptible, otherworldly whimper. I noticed the composer, Drew Krause, a member of the music faculty at New York University, standing anonymously in an alcove to the side. He wore an expression of calm amusement, apparently untroubled that few were listening.

One thing is certain about the LEMURbots: They look and sound nothing like classic humanoid robots. In fact, most people probably wouldn't guess they're robots at all. Take the cylinder thumping above people's heads, for example. The name says it all. The Bucketbot is nothing more than a bucket and a mallet driven by an electric spring-loaded actuator called a solenoid. The small cluster of bells ringing near people's elbows—the TibetBot—uses the same simple mechanism. The largest LEMURbots stood near my corner. The XyloBot, also known as the Ill-Tempered Clangier, is a chest-high arc of mallets poised over lengths of metal tubing. The GuitarBot, a shiny metal grill that loomed over the audience, is essentially a giant four-stringed guitar with motor-driven pluckers on drawer sliders. All the bots are hooked up to a single computer.

“Simplicity is the point,” says Eric Singer, the creator of most of the LEMURbots, “because none of us are really roboticists.” Singer describes himself as a musician with an engineering background. Ten years ago, after attending the Burning Man festival in Nevada, “I started blowing things up,” he says. As a member of a Brooklyn-based art collective known as the Madagascar Institute—which has little to do with the country—Singer used flame-throwers and explosives in performance art. After 2001, he switched gears to “musical instruments that play themselves.” As a nod to the primates of Madagascar—the country—Singer named his own mechanical art collective LEMUR, the League of Electronic Musical Urban Robots.

Before the show, I had asked Krause what it was like working with the bots. “A challenge,” he said, because you can't just talk to them to explain how you want a certain note played. Instead, Krause talked to them through a programming language called LISP. One advantage the LEMURbots have “is that I could hear the performance while I was working on the piece,” he says, because the bots don't need coffee breaks. Another plus “was that I could get rhythmic ensemble performances that would be impossible for human players, such as unison rhythms following sine-wave curves to eight decimal places.” But because the only thing I could clearly hear were the twangs of the nearby GuitarBot, those complexities were lost. (In my imagination, a LASERbot was exterminating the audience so I could listen properly.)

Learning from Krause's misfortune, the next LEMURplex artist in residence, Taylor Kuffner, picked up a microphone and seized control. He made half of the audience sit down on the floor so the rest could see, and then he introduced his machine. While living in Indonesia, Kuffner, who described himself as a “laptop musician,” became fascinated by the classical Asian ensemble music known as gamelan. To see what gamelan might sound like through the interface of a laptop, Kuffner cannibalized parts from the LEMURbots and four different gamelan instruments—a mongang from Java and a trompong, gender, and slonding from Bali—to build a self-playing robo-gamelan. Whereas Krause had written a complete score note by note—the traditional Western approach—Kuffner created a few dozen rhythmic and melodic modules on his laptop to mix and match during the performance.

In some ways, the result was a typical DJ show. The most visible element was Kuffner himself—an American with a surfer's blond mop and a soul patch—up on a platform twiddling with his laptop in front of a screensaver-esque film projection. (The fact that he was burning incense and wearing some traditional Indonesian clothing did little to change that.) The structure of the music also felt familiar: a hypnotic buildup of rhythms with occasional hyperkinetic spurts borrowed directly from the Drum & Bass tradition of electronic music. But had this been a typical DJ show, the music would have all been digital and blaring from speakers. Instead, each and every sound was acoustic, created by a robot's mallets striking real instruments. And just moments into the 20-minute performance, that robot began to fall apart before our very eyes.

It was funny but also rather touching to see Kuffner perform in vivo surgery on his creation as it played. Each time a solenoid-powered mallet drifted off-target, replacing one of the gamelan tones with a wheezing rattle, Kuffner scrambled to find and fix it. When one of the drums started malfunctioning far out on the edge of the contraption, he recruited a member of the audience to hold it in place. To his credit, Kuffner looked unruffled and even, like Krause, gently amused by the unpredictable output of his month in the machine world. He told me that he came to love the robot's physical quirks, “the imperfections that make it feel more human.”

The Hoffman Test

Afterward over dinner, my test audience roundly panned the show. “Nothing new” was the verdict of Lieberman and Paluska, both recent alumni of MIT robotics groups who now work halfway between academia and the art world. Autonomous musicmaking machines have a very long pedigree, said Paluska, from the self-playing paper-roll pianos that amazed 19th century audiences to the musical “kinetic sculptures” of Alexander Calder and Jean Tinguely. “But technology should be thought of as just another paintbrush,” said Lieberman. He was disappointed to see “more of the same old robotics” rather than “a new expression.”

“So then, what's not a robot?” asked Vijay Iyer, a New York-based composer and jazz pianist. “How is a record player not a robot?” The roboticists nodded and shrugged. “Well, robots can't improvise,” said Lieberman. “But then you have to ask yourself, what exactly is improvisation?” shot back Iyer. The musicians and roboticists wrestled over these slippery concepts into the night, generating more questions than answers.

“By the way, I actually didn't think the show was all bad,” said Guy Hoffman, a roboticist at the MIT Media Lab, as the table was being cleared. “Distributing the robots, surrounding the audience with sound, I liked that.” But everyone agreed that Krause's piece was impossible to judge. Violinist Lara St. John chalked it up to a classic mistake of performers, be they man or machine: “If you ignore your audience, they ignore you.”

As for Kuffner's robo-gamelan, St. John found it “sort of interesting,” but instead of a machine, she wondered “if it wouldn't have been a lot better to have a bunch of percussionists.” Iyer's assessment was the harshest. “It was the colonial narrative all over again,” he said. “It's all about cultural domination and total control—now of the musicians themselves.” (To be fair, Kuffner was aware of the culture clash, admitting that his Indonesian teachers “would have laughed to see this, since gamelan is supposed to be about kerja bersama, doing it together as a community.”)

In spite of the negative reviews, Hoffman saw the show as a step in the right direction for artificial intelligence, although not by traditional standards. “When people wonder when machines will be truly intelligent,” he said, “they still talk about the Turing test.” It began as a thought experiment described by computer pioneer Alan Turing in the 1950s: When a machine can chat with humans—on any subject—and is indistinguishable from human conversationalists, that machine would have the equivalent of human intelligence. Each year, the $2000 Loebner prize is given to the computer program that comes the closest to that ultimate goal. (You can chat yourself with last year's winner, UltraHAL, to get a sense of just how far the machines have to go.) But the true recipe for intelligence—from reasoning and emotion to physical embodiment—is now hotly debated by scientists and philosophers.

According to Hoffman, whose research is on machine-human interaction, the performing arts may be a better testing ground for artificial intelligence than conversation. In Hoffman's version of the Turing test, “a machine must be able to rehearse with humans and give a convincing performance on stage, not just following a fixed program but interpreting and reacting to the humans in real-time.” The hardest version of the Hoffman test is acting in a play. But because natural language comprehension is a colossal hurdle for machines, “music is a field where many milestones can be achieved along the way.” Hoffman doesn't think scientists can do it alone. “Artists should get their hands dirty with robotics” in efforts like these, he said.

“Robotic art is still in its puberty,” concluded Lieberman. “It takes so much effort to deal with even simple mechanisms like solenoids, it leaves little room for artistic expression.” Paluska sighed, checking his watch. He still had a date with his own robot.

The Absolut Quartet

Lieberman and Paluska's studio, just a subway ride away in Greenwich Village, is remarkably similar to the LEMURbots' lair, minus the clutter. The only thing to see is the Absolut Quartet itself, a hulking machine far at the back of the elongated space. With its section of open mechanical guts sandwiched between long rows of stemware and what look like glowing plastic teeth, there was no guessing what it might do. Suddenly, a brief synthetic piano melody played from a nearby speaker. After a pause, several white balls launched from the machine's guts, flying in a 2-meter arc through the air and landing between the plastic teeth to precisely strike the keys of an underlying marimba, echoing the melody. Then the thing came alive with an explosion of flying balls, spinning glass, and tapping robo-fingers.

Iyer and Krause joined me to see Lieberman and Paluska's robot concert—although calling it a concert isn't quite right considering that it has been going nonstop since it came to life on 31 January. Krause brought the computer recipe for his LEMURbots composition. Kuffner e-mailed some of his robo-gamelan music directly to Paluska. That music was explicitly coded for the LEMURbots, with their instrumentation and constraints. It wasn't clear how the Absolut Quartet would translate it for itself. For example, because it must capture the balls and reload them into the launcher, there is a 1-second delay between notes played on each key of the marimba—the quartet transposes notes to different octaves when it must play quickly.

The ensemble gets the first part of its name from its financial sponsor, the Swedish vodka-producing company. Lieberman and Paluska call it a quartet because it consists of three robotic musicians—the ballistic marimba, a cluster of robotic percussion instruments at shin level, the spinning fingers-on-glasses known as the Wino—“and the fourth member is you,” said Paluska. The machine isn't taking musical orders from its creators. Instead, it listens to the Internet.

“Give it a try,” said Paluska, indicating a laptop chained to a pedestal. The Absolut Quartet's Web page was open on a browser. Iyer entered his name and e-mail address. (A video of each performance is automatically recorded by microphones and cameras in the space and e-mailed to the user.) Then a virtual piano keyboard filled the screen. “So now you just give it a melody,” said Paluska. Iyer began playing. The computer cut him off halfway through and Iyer looked up. Nothing happened. “It's sending it to the server,” said Paluska.

After about 10 seconds, Iyer's melodic fragment played from the speaker. Then the machine played the melody with a volley of balls, followed by a 3-minute improvisation. To me, the result sounded like the theme music of a zany Saturday morning children's cartoon about the adventures of a friendly alien insect. When I fed it the melody of “Mary Had a Little Lamb,” the output was a completely new improvisation that nonetheless sounded much the same.

“It's still learning,” said Paluska. The musical brain of the Absolut Quartet works with algorithmic “recipes” provided by human composers. “So far, it has input from 12 composers,” he said. As more composers work with the machine—providing musical code through a Web page maintained by Lieberman—the quartet's playing will grow more stylistically complex. “We have no idea how it will eventually sound,” said Paluska. “That's the idea.”

Paluska disappeared behind a panel to feed the quartet Krause's LEMURbot composition. “I had to modify it quite a bit,” said Krause. With no warning, a tidal wave of balls launched into the air, sprinkling the marimba with a complex but precise rhythm for about 20 seconds. Krause listened carefully. “Great. Yeah, that was it,” he said with a smile. Then came Kuffner's robo-gamelan music. The marimba has a very different sound color, but the quartet had no problem capturing Kuffner's riffs on the first try. “Machines do some things very well,” said Paluska, particularly things we find impossible.

Like the LEMURbots, Lieberman and Paluska's machines are clearly miles away from passing Hoffman's test. But the element of human-machine interaction does seem to be a step forward. “Although,” said Iyer, “as the user, you're not really making music.” Feeding the machine a melody over the Internet “is more like entering a PIN code,” he said. “But it's very impressive, especially in terms of the visual spectacle.” Iyer asked Paluska how he could make a compositional recipe for the machine. Behind them, dozens of balls cascaded as it improvised on the theme of “Frère Jacques.”

View Abstract

Navigate This Article