#morphological operations

[mashpotatos]

A9: Scilab Sings

Last activity, we used morphological operations and the IPD kit by Dr. Galda to identify abnormal “cells” in an image, but did you know we can use this kit in a variety of other ways? In this activity, we were made to let scilab read and play sheet music. It wasn’t an easy task, but it was pretty fun to work on.

The sheet music I used for this activity is a piano adaptation of the song ECHO by Crusher-P. The piano piece can be found here, while the original song can be found here. For this activity, I only included the melody following the treble clef up until the 64th measure of the piece.

First of all, we should know that scilab can produce sounds from sinusoid functions.

The frequency f of the wave here determines the pitch of the sound, and the time t defines how long the sound plays.

Now the product of this is called pure tone, a continuous sound that doesn’t quite sound like a note played by a piano. To fix that, we applied an sound envelope or an ADSR (attack, decay, sustain, release) envelope to emulate the sound of an instrument.

Here’s the note function I used, based on the one provided by our professor, Dr. Soriano:

//note function function n = note(f, t)

   n = sin (2*%pi*f*t);    //defines the envelope    env1 = linspace(0, 1, round(0.025*length(t))); //attack    env2 = linspace(1, 1, round(0.05*length(t))); //sustain    env3 = linspace(1, 0.9, round(0.05*length(t))); //drop before decay    env4 = linspace(0.9, 0.45, round(0.85*length(t))); //decay    env5 = linspace(0.45, 0, length(t)-length([env1,env2,env3,env4])); //release    env = [env1, env2, env3, env4, env5];    //applies the envelope    n = n.* env; endfunction;

So with that done I had two main objectives: identifying the kinds of notes (half, quarter, eighth, sixteenth), and their pitch, in addition to recognizing rests and dotted notes.

I stitched together the many lines of music to form one long continuous line from the first to 64th measure. This was my input image. 

As I said before, I used Dr. Galda’s IPD kit to analyze the notes here. After reading the image, I realized I had a problem. If I applied morphological operations to isolate the quarter notes, I’d be removing the half notes. And even so, how would I differentiate between quarter, eight, and sixteenth notes?!

I ended up using three segmentation methods for the notes:

one to detect the “filled” note heads that quarter, eight, and sixteenth notes shared,

another to detect the “hollow” half note heads, and

the last to detect the bars and flags of eighth and sixteenth notes

Here’s an example of how the first segmentation process worked. 

The other two used different thresholds and morphological operations.

With the notes indexed, I’d get the x and y-coordinates of the centroids of the note heads and used the x-coordinates to control the sequence at which the notes would be played, and the y-coordinates to determine the pitch of the note. Additionally, if the x-coordinate of a note head coincided with those included in a bar or flag, the note was identified as an eighth or sixteenth note.

For the rests and the dotted notes, I had to use two more segmentation processes. Fortunately, all rests are located in the same spots on the musical staff, so if a blob was found to occupy that spot, I could mark it as a corresponding quarter or eighth rest. For the dots, I had a bit of trouble since most of my notes were staccato notes. To remove the staccato marks, I only considered dots that didn’t line up vertically with other notes. After finding the notes closest to these dots, I applied the corresponding modification to their time (dotted notes are played at a total time equal to 1.5 times the time of the usual note). I... ignored the staccato notes for now, as modifying the envelope either produced no noticeable effect, or too much of an effect on the final notes in the wave file.

With all of that done, I have my final output. You can listen to it here. HEADPHONES WARNING!! It can get kinda loud.

That was a really tedious activity, but I can’t deny it was a lot of fun too. I met the requirements of the activity, plus adding in the sound envelope, together with recognizing rests and dotted notes. For that, I think I deserve a 12 out of 10.

Now I get to sing praises to the following as my acknowledgements:

Dr. Maricor Soriano, our professor who provided the note function

Joshua Abuel, for helping me with the process of detecting notes

Physics of Music Notes, for the note frequencies

Luman Magnum, for the info on how to  apply a sound envelope

Crusher-P, for creating the beautiful and addictive song ECHO

jazzermazzer99, for transcribing a piano adaptation (and making an easy version) of ECHO

[mashpotatos]

A8: Mighty Morphin’

Yes, bring back the Power Rangers references. In this activity, we’d learn about morphological operations, specifically erosion and dilation, and apply them to the segmentation process. The real challenge for me here was familiarizing myself with the Image Processing Design kit by Dr. Harald Galda.

Erosion and Dilation

So remember how operations with sets work? Morphological operations rely on those. In this part of the activity, we focus on two simple operations: Erosion and Dilation. Aside from our image, we need a structuring element to operate with. This can be almost any shape, as I’m about to show, and quantitatively dictates how an object changes. I’ll get right down to showing how this works using the following base images.

Okay, so to put it simply, erosion removes pixels, while dilation adds them in. Just how many pixels are removed or added is determined by the structuring element. Let’s start with dilation. We apply the structuring element to every pixel on the base image, noting the reference pixel of the element, and all the pixels not yet already part of the image are added in. In erosion, we again apply the structuring element to every image on the base, again noting the reference pixel. If the applied structure element can fit into the base image, that pixel is kept. Got it? No? Maybe demonstrating it would be better.

And that’s how it works for a simple 2x2 square element. But a structuring element can be almost any shape, right? So here are more examples.

For a 2x1 vertical line:

For a 1x2 horizontal line:

For a 3x3 cross shape:

For a 2-pixel diagonal line:

Segmentation

So how can this help in segmentation? Well, in cleaning mostly. We can use morphological operations to clean up images for an easier time in segmentation. We used the IPD kit by Dr. Harald Galda to perform morphological operations on an initial segmentation on the following image of “cells.”

That’s not all it can do, though. This kit allows us to identify and index continuous “blobs” in the image. We can then proceed to do things like measure the area of these blobs and use the histogram of these areas to find an acceptable range for an average “cell.”

So with this new range we can do things like filter out abnormal “cells.” Take this image for example. We were asked to highlight the “cancer cells,” using image segmentation.

Another life saved by Mighty Morphological Operations. 

Like I’d said in the introduction, I struggled a bit when it came to using the functions of the IPD kit. I still managed to produce the required output, but again with not much exploration. However, I like to think my presentation deserves a bit of credit, so I’m giving myself 10 out of 10. I’ve pretty much stopped attaching parts of my code, but that’s mostly because, besides the new functions of the IPD kit, it would mostly appear similar to at least one of the snippets I’d previously uploaded.

Time for some Mighty Morphin’ Acknowledgements:

Dr. Harald Galda, the legend himself who provided the IPD kit

Dr. Maricor Soriano, for guiding us in using the IPD kit to save the life of the cancerous paper

Joshua Abuel, my classmate who helped me realize the power of the IPD kit