How does MPO Predict the Future?

A basic tenet of my approach to musical accompaniment systems is that one must anticipate the future evolution of the music to achieve good ensemble. This is, of course, one of the reasons musicians rehearse together -- so they share a collective notion of the way the music will unfold. The computer accompanist also benefits from rehearsal, in my case, by training a probabilistical model that allows for accurate prediction of future note onsets times. I have given many technical talks on this subject which have essentially explained the ideas about probabilistic modeling of music presented in my papers on Bayesian belief networks. The interested reader is welcome to explore these, though I take a completely different approach here.

I have prepared a movie that demonstrates the essential ideas of how my system coordinates through prediction. In this movie one can see a visual depiction of the audio in the form of a spectrogram, with frequency content evolving over time. As the audio is played, my system will occasionally detect notes in the solo part. In the movie these are shown in green. Note that, while these note onsets are usually detected reasonably accurately, the detections are made with a small amount of latency. This is because the computer (and human) must hear some of the new note before it can be recognized. Thus the green lines appear slighly after the notes sound.

Below the top row of green lines are a collection of red lines that describe the systems predictions of when orchestra events should be played. My system cares only about scheduling the pending orchestra note and will reschedule this note every time new information becomes avaialble. Thus, if you watch the rightmost orchestra note, you will see that its position wiggles around as more and more solo notes are detected. Eventually, the scheduled time will finally occur, at which point the note is "yesterday's news" and the system shifts it focus to the next orchestra note. Thus the time that an orchestra note is actually played remains in red on the image.

I chose the piece, the Strauss Oboe Concerto 1st movement, since it does an especially good job of illustrating this idea. The movement has a repeated motive consiting of a string of sixteenth notes for the oboe ending in a point of synchonization with the orchestra on a strong beat. If you watch carefully, you will see the orchestra's prediction of the goal point shifting subtley until it finally occurs. I hope this movie is illuminating.

Strauss Oboe Concerto Mvmt 1