Electrifying Duets Mathematician Strives to Teach PCs to Play Accompaniment

By Paul Eng

Oct. 4 - Could a computer ever play music as well as a human musician?

Some would argue that computers already far surpass any human performer when it comes to producing pristine tunes. Music described in the absolute digital code of ones and zeros guarantees that a computer will play a particular score the same way every time.

But for Christopher Raphael, an accomplished oboist, such an unyielding musical performer would hardly be an ideal partner for a human musician to practice and possibly perform with. The human player would always have to conform to the computer's pace of performance, for example, and thus stifling the individual's musical interpretation.

So Raphael, a professor in the department of mathematics and statistics at the University of Massachusetts at Amherst, has been working on a software program that would essentially give computers human-like flexibility when it comes to playing alongside flesh-and-blood performers.

Listen, Analyze, Predict, and Play

His software, Music Plus One, is composed of over 40,000 lines of complex computer code developed over ten years of research. And how it works is based on how a real musician thinks and processes audio information.

First, data that describes the musical score to be played by the human and the accompaniment part to be played by the computer is keyed into the machine. In other words, the computer is programmed with the digital equivalent of sheet music which outlines every note of a musical performance. As the human instrumentalist plays his part, the computer picks up the sounds via a microphone. Then by using processing techniques similar to those used to recognize speech, the computer analyzes what it "hears," and determines what notes have been played and when they were played.

After several practice sessions, the computer develops an idea of how a person plays that particular piece of music. What's more, artificial intelligence algorithms in the Music Plus One software allows the computer to "predict" how that performer will play any particular note at any given point in the piece. In turn, the computer can then anticipate how it needs to play its corresponding musical part.

Real-Time, Real Easy

Raphael says the predictive capability of Music Plus One is what sets his approach from other accompaniment programs out there.

"If you're going to synchronize playing with another musician, you just can't listen and react," says Raphael. "You need to do what humans do: Take information about what you heard in the past and predict what you need to do in the future."

And Raphael says the software can do all of these tasks in "real time." In other words, the program is listening, analyzing, predicting and playing its part all in the same instant the human instrumentalist is performing.

What's more, Raphael says the Music Plus One software doesn't require heavy-duty computer systems. He claims to run the program, whimsically named in relation to the old Music Minus One accompaniment records he use to practice with, on a plain four-year old laptop with a standard Pentium-class microprocessor.

Roger Dannenberg, a research computer scientist at Carnegie Mellon University in Pitsburgh, Penn., has seen Raphael's work at a recent gathering of the Acoustical Society of America and is impressed.

"Chris' work is some of the more advanced in terms of developing performance systems," says Dannenberg who has done similar research work several years ago and now holds several patents in related music systems. "His system works and I find it striking how it works so well."

Is It Live Or Is It...?

Raphael says that for now, Music Plus One remains a beloved research project. "My main purpose is to see how far I can go with this," he says. "My belief is that it can go a lot further."

How much further?

According to Raphael's research paper, the ultimate milestone for Music Plus One would be to "pass the musical equivalent of the Turing Test." In other words, could the software be refined to such a degree that it can fool the listener into believing it's a human and not a machine performing alongside a live performer?

To hear examples of Music Plus One in action, Raphael has samples on his Web site: http://fafner.math.umass.edu/music_plus_one.

But in addition to working toward that goal, Raphael says it's possible a version of Music Plus One could become available for sale to musicians - some day. "I've worked very hard on this and would like the world to use what I build," he says.

For now, however, he hasn't broached any software companies to convert his research into a commercial product.