Play music sheets with image segmentation

In my previous  blog, I have shown how morphological operations in images works. I have also shown how we can apply it on image segmentation (differentiate cancer cells from normal cells). In this chapter, we try to read and play simple music sheets using image segmentation.

We start by first knowing some basics of reading music sheets. I used some of the knowledge from this site and discuss it here. Below shows a basic representation of notes. The 5 horizontal lines were the symbols are written is collectively called as the staff.

To play a song right, one would need to know both the note and  its beat. The note refers to the pitch while the beat is the length of time. The beat is identified from the type of note symbol used. Below shows some types of notes with their corresponding note values. The pitch is determined by the position at which the notes are placed. The pitch values are shown at the left of the staff.

To play the song via image segmentation, we should obtain both the beat and the pitch or basically, the note symbol and its position relative to the staff. Identifying the pitch would be easy since I can just get the centroid (will discuss more later) of a note and find its position relative to he staff. However, identifying the beat would be difficult and I still lack some skills to identify it. So for this activity, I am only able to identify the pitch and play it in constant beats.

Choosing the song and preparation of images

I’ve chosen a familiar and classic song by Ludwig van Beethoven entitled Ode to Joy. Every time I here its tune, I always remember my elementary days where I’ll be walking on stage to receive recognition. Maybe I received so many recognition during those days (and none these years haha) that’s why it got stuck on my head. But aside from it, its catchy and simple to segment.

I have divided the whole score into four stanzas as shown below. I have ensured that there are 10 white row of pixels before the first line of staff for all stanzas. This is to have a common region for all stanzas where a note can be uniquely identified. The G-clef symbols were also cropped out to for easier segmentation. The color was also inverted in order to use morphological operation easier.

Segmentation of notes

There were 4 stages of morphological operations to completely segment the notes, each uses a different method and structuring element (SE). Note that this not the only method to segment them. This procedure was a result of many trials and errors.

a. Opening with a SE of a 4-pixel vertical line – this is to remove the horizontal lines of the staff

b. Closing with a SE circle of radius 4-pixels – this is to close the half-notes (notes with holes) and transform it to quarter-notes. This is okay since we are just interested in the positions.

c. Opening with a SE of a 5-pixel horizontal line – this is to remove all flags and stems (refer to sample music score image) of the note symbols.

d. Opening with a SE circle of radius 6-pixels – this is to remove other blobs that are not part of the note and to make the shape of the notes almost uniform after the previous operations.

Below shows the results of segmentation. We can see that the heads of the notes were segmented and all other things removed e.g. flags, staff, etc.

Identifying the pitch or simply the note

I have taken advantage of the vertical positions of the heads to identify their notes. We observe that the areas of the heads overlap with the neighbor heads. This means that we cannot use the maximum and minimum vertical position of the head to be the range of a note. Instead, we use its center to be its position.

A good way to get the center position is through its center of mass of essentially its centroid. The centroid can be calculated by summing all row positions of all pixels in the head and divide it by the total number of the pixels in the head (essentially, the are of the head). We can also do this for the column centroid but would not be more useful in this part.

centroid = sum(row positions of pixels)/ (number of pixels)

Below shows the range of rows that would identify the note uniquely. Row index starts with 1 from the top of the image. I have identified only C, D, E, F and G since these are the only notes present in the song.

C = rows 85-89

D = rows 77-81

E = rows 70-75

F = rows 63-67

G = rows 56-60

From this, the note of the head can be identified given its centroid.

Playing the notes with Scilab

Scilab has a built-in module to play notes given a wave function (usually sine wave) and the length of time (using soundsec function). I have used the sine wave function of the form sin(2*%pi*f*t) where f is the frequency of the note. The frequency of the notes can be determined from this site. I have the 5th tuning scale for my notes which have the following frequency values.

C = 261.63*2 Hz

D = 293.66*2 Hz

E = 329.63*2 Hz

F = 349.23*2 Hz

G = 392.99*2 Hz

For the length of time of each note, I used 0.45 seconds per note. A rest note was also added after each stanza using an input with zero frequency. You can listen or download the tune in this link ode-to-joy-tune.

I have successfully read the score sheet using image segmentation and played a tune out of it. Although the beats were not right, it still resembles for most of the tune.

The code can be viewed in this link song-play-code.