In our "score following with skips" formulation the student plays from a score, but may move around freely, repeating
local sections as desired, or making distant jumps. The figure shows our state graph model for movement within the score.
Using a data model that computes the probability of a frame of audio (30 ms.) given a score state we can score the
probability of any path through the graph. The globally most likely path can be computed offline using the usual dynamic
programming algorithm. However, it is also possible to construct the most likely path in an online manner (as the
practice session evolves). When all paths descend from a unique node in the dynamic programming search tree (above)
the most likely path is known for all time before this unique node. The latency in delivering this optimal path
is usually manageable for giving feedback in a realistic practice scenario.
What does the analysis uncover?
Our analysis partitions the audio in a sequence of excerpts (praticed fragments of the score). The video
shows the results of our audio analysis on a practice session of a Rode etude. The current note
is highlighted in blue.
One can navigate through the practice session by clicking on notes to jump to the most recent iteration of that note in
the practice session. This facilitates reviewing the practice session by allowing the user to focus on important
moments.
Offline Summarization of the Pratice
The figure shows the degree of emphasis the player places on different sections of the score in the practice
session. Our color scale moves continuously from black to green to blue, with black representing the least number
of visits to a note, and blue denoting the most number of visits.
We can also construct a single "best" performance of the piece, splicing together portions of the identified excepts.
In doing so we optimize a criterion that favors steady rhythm and tempo while traversing the spliced portions,
including over boundaries between splices.
Online Feedback of Pratice
Online feedback on tuning. During a practice session if a note is detected as flat/sharp it can be
colored as red/blue until the next time the note is played.
