# Towards automated audio to score transcription — Audio-to-score alignment

This post is the first in a series describing the process of making a tool for transcribing guitar music. This is clearly a task for ML, and so we need a dataset for training/validation. In the data preparation stage, obtaining scores aligned with music is incredibly challenging. To the best of my knowledge, the approach I present here is original.

The Challenge

We need a dataset of onsets in audio format labelled with the notes that begin at the onset.

Since such a dataset doesn’t exist, we must make it ourselves. Yes, we could use synthesised audio, but real-world data should result in better generalisation.

Now, having acquired hundreds of fingerstyle pieces from various artists in both audio and score format, how do we match these two together? This problem is an active area of research, and in its more general form can also be found in speech recognition systems.

Here I will put forward a weakly formed proposition.

Proposition: If we had a perfect audio-score alignment algorithm, we would probably already have a perfect music transcription algorithm.

Though these two problems are not identical, they are obviously closely related. The point is, audio-score alignment isn’t easy.

Dynamic Time Warping and Hidden Markov Models

Most systems for audio-score alignment work using DTW or HMMs. Neither is particularly “better” than the other [*], and I would argue further that the metrics used to compare techniques are not particularly relevant, since we focused on onsets, not the entire piece—which has been the focus of most prior work.

DTW: A technique solving a 2D dynamic programming problem, minimising an objective function, $f$, along a path from $(0, 0)$ to $(m, n)$ where $m$ and $n$ are the length of the two sequences.

In our particular application, we have two audio sequences, and $f(t_1, t_2)$ is a measure of the dissimilarity between “frames” $t_1$ and $t_2$, then DTW will give a the warping path $s = ((0, 0), (t_{a1}, t_{b1}), (t_{a2}, t_{b2}), …, (m, n))$ minimising $\sum f(s_k)^2$. There is also a penalty term for deviating from the diagonal, and obviously other penalties can also be added.

Applying DTW to the alignment problem

It is obvious then how DTW is applied to alignment of temporal sequences. Directly comparing an onset from the score (i.e a collection of notes) to a “frame” of the audio is certainly not a great idea. Instead, we convert the score to an actual audio file, so yes we do end up synthesising after all!

For measuring disimilarity between the audio, we let $f$ be a measure of the cosine distance between chroma vectors from the audio. The cosine distance is a reasonable measure and its output is bounded (which turns out to be useful later).

Chroma vector: a 12-D vector relating to the pitch classes (C through B). There are many techniques for obtaining chromatograms from audio that I will not cover here. Getting chroma to best represent harmonic features is still an active area of research, and the state of art is moving from classical algorithms to deep learning based approaches [1].

In our application, since we are only concerned about times when onsets occur, the audio sequences should be restricted to the onset times. Obtaining onset times from the score is fairly trivial, but getting the potential onsets from the recorded audio is not. However, as long as most of the onsets are detected and the number of false positives is minimal, the DTW should be able to produce a good alignment. Thus, we can simply use packages that provide onset detection tools such as librosa and madmom. From my limited testing, madmom has the best performance, due to their CNN approach [2].

Wrapping it up

It should begin to make sense how the techniques I’ve presented above are relevant to solving the problem with DTW. However, not everything has been discussed. Firstly we must aske ourselves the fundamental question:

In obtaining an alignment/path, what are we trying to minimise?

Yes, we are trying to minimise the “distance” b.the diagonal. But, both the potential onsets from the audio and the score are imperfect! This means that the path representation needs to be able to inform us which onsets to skip. Here I propose the following interpretation of the warping path.

Given a warping path $s = ((0, 0), (t_{a1}, t_{b1}), (t_{a2}, t_{b2}), …, (m, n))$, for any step from $(t_{ak}, t_{bk})$ to $(t_{a(k+1)}, t_{b(k+1)})$, if $t_{ak} = t_{a(k+1)}$ then $t_{b(k+1)}$ is skipped. Likewise if $t_{bk} = t_{b(k+1)}$ then $t_{a(k+1)}$ is skipped. Since we are limited to steps from the set $\{(0, 1), (1, 0), (1, 1)\}$, the only other possible step is $(1, 1)$, where there is no skip.

Clearly, we do not want to be skipping too much but we also don’t want to skip onsets that actually exist. To achieve a balance, we apply a penalty term a constant penalty term to step types. For the $(1, 1)$ step, there is no penalty, but for the other steps, there’s a penalty term of 1. Note also that 1 is the maximum output of $f$. So in a sense, a single skip should improve the rest of the alignment by at least a value of “1”—for example bringing a perfectly disimilar match to a perfect match.

Results

Overall, results are promising from testing on a few songs. Songs with a lot of slapping and thumb slaps result in significantly worse performance—this is not surprising since slapping produces very different sounds depending on the guitar. But even without slaps, most songs have misaligned sections where one of the sequences is ahead by one onset.

Post processing

Intuitively, since we don’t actually need only a samples of onsets and their sounds, we could filter out matches that have high error (i.e. $f(t_1, t_2)$ is large). However from manual inspection, a lot of correctly aligned notes also report high error, and many incorrectly aligned notes also report low error. This is probably due to a combination of the following:

• Chroma are 12-D representations. An C3 and C4 played on the guitar will have very similar chroma.
• There is more to an acoustic sound than just the frequencies that are present.

Perhaps using chroma variants that span multiple octaves would be better?

[*] In fact a particular version of HMM ends up being identical to DTW.

[1] Filip Korzeniowski and Gerhard Widmer, “Feature Learning for Chord Recognition: The Deep Chroma Extractor”, Proceedings of the 17th International Society for Music Information Retrieval Conference (ISMIR), 2016.

[2] Jan Schlüter and Sebastian Böck, “Musical Onset Detection with Convolutional Neural Networks”, Proceedings of the 6th International Workshop on Machine Learning and Music, 2013.