bmcfee's music hacks

I do research. I hack music.

[functiontelechy]

The Keplerphone

Music Hack Day, Boston, 2014!

This time around, I teamed up with Jonathan Marmor to sonify data from Kepler as face-melting, shredding guitar rock from outerspace!  Behold, http://keplerphone.jonath.in/.

If you want to learn all about how it works, check out

IPython notebook of the core algorithm

Full source code of the final product

And just for fun, here's one of my favorite example outputs.

[functiontelechy]

Automating parody with the Yankomatic!

The Monthly Music Hackathon returns!  This time around, the theme was Automatic Music.  Most people interpreted this to mean automatic composition (in one form or another). But, since I know so little about composition, I instead went back to a project idea that came out of conversations with Matt McVicar about automatic lyric analysis.

I give you The Yankomatic!  Given a song title, it automatically generates a set of parody titles in the style of Weird Al Yankovic by substituting key words with food-related rhyming words.

The code for this hack is available as an IPython notebook in my music hacks github repostiory.  Here's a quick outline of how it works.

Step 1: Collect food terms

Initially, I had grand ambitions of crawling Wikipedia's lists of foods category.  There's great data in there, but the lack of consistent structure made it a bit too tricky to adequately crawl in a one-day hack.  Instead, I found a more concise glossary on Food.com, which turned out to be much simpler to crawl with a few lines of code using BeautifulSoup.  I discarded any multiple-word terms (eg, acidulated water), and lemmatized any plurals to include their singular forms as well (eg, strawberry from strawberries).

Step 2: Build a rhyme index

For each food term, I used NLTK and CMU Dict to extract its pronunciation(s) as a sequence of arpabet phonemes.  For example, the word huckleberry becomes the phoneme sequence HH AH1 K AH0 L B EH2 R IY0.  The letters indicate the phoneme, while the numbers indicate the syllabic stress: HUCKleBERRy.  Given a sequence of phonemes, I then used the P2TK syllabifier to partition it into syllables.

Jonathan Marmor helpfully pointed out that it's important to match the stress pattern when making word substitutions.  The first stage of the rhyme index maps each word down to its stress pattern: for example, the words huckleberry, avocado, and peanutbutter would all be encoded as 1010.  (Secondary and primary stress are treated equivalently in my implementation).  This makes it easy to find all the words with a matching stress pattern to a query.

After extracting the stress pattern, I then indexed each word by the phonemes in its final syllable.  There is minimal soft-matching here: if the final phoneme is a vowel and it has a consonant onset, the onset is discarded.  Slant rhymes and more general consonant substitutions are left as an exercise for the reader.

Step 3: Querying

When a query string comes in, say Welcome To The Jungle, we don't really want to replace any arbitrary word.  Instead, I tokenize the input, and then use the NLTK part-of-speech tagger to filter out nouns.  For each noun in the query, I extract its stress pattern and tail syllable as described above, and generate all possible substitutions (with the remaining words held fixed).

Aside from cobbling together a flask server and some bootstrap styling, that's it!

Examples

Here are a few of my favorite examples:

Welcome to the Jungle => Welcome to the Kugel

Sex Machine => Sex Sardine

Don't Fear the Reaper => Don't Fear the Caper

@pmrondon The Downward Spiral => The Custard Spiral 

@orlikeawhile The Beautiful People => The Beautiful Pickle

@cweichen You Never Give Me Your Money => You Never Give Me Your Chutney

@surlyhobbit Mr Tambourine Man => Mr Mangosteen Man

[functiontelechy]

Dynamic Beat Novelty and Exquisitely Salient Corpses

For the August installment of @musichackathon, we (the Music Hackathon regulars) attempted to implement an audio version of the exquisite corpse idea. My contribution to the corpse was an auto-remix script that dynamically stretches each beat of a song according to its (current) perceptual novelty. The basic concept is that the more frequently you hear a sound, the more rapidly it will be played: the listener must never be bored!

The concept and core guts of this hack (har har) grew out last month's project, the phase vocoder implementation from my Toronto project, and a heavy dose of inspiration from The Swinger.  The code is available as an IPython Notebook if you'd like to follow along.

As a first step, I implemented a slightly more generic version of Swinger, which allows the calling program to supply an arbitrary (and possibly indefinite, by way of python generators) sequence of stretch factors to be applied to each beat.

Next, the input song is broken up into beats, and a mean timbre descriptor is computed over each beat.  The beat-timbre descriptors are then clustered via k-means, so that each beat is assigned to one of the k clusters.  This results in a sequence of cluster labels, eg, "1, 1, 3, 1, 2, 2, ..."

We can compute then relative proportion of the i'th cluster within the first t beats as as r[i, t] = N[i]/t, where N[i] denotes the number of beats assigned to the i'th cluster at or before time t.  If we have T beats total, the quantity r[i, T] provides a global estimate of the frequency of each type of sound in the song.

In the example above, we obtain a sequence of frequency estimates r[1, t] as "1.0, 1.0, 0.66.., 0.75, 0.6, 0.5, ..."  Then, for each time t, the ratio r[i, t] / r[i, T] can be used as a measure of how (un)novel cluster i is: large values indicate that the sound has already occurred many times, small values indicate that it's relatively novel, but will appear again later.

Since the cluster frequency estimates tend to flap a bit (especially at the beginning of the song), an additional median filter can be applied to the sequence in order to smooth it out a bit.  Here's an example time-stretch sequence for Particle Man by They Might Be Giants:

Initially, the song is fairly homogenous, and therefore played quite rapidly.  The dip at beat 50 indicates the refrain, which differs significantly in timbre content from the verse, and gets stretched out accordingly. With each subsequent beat in the refrain, the frequency estimates approach the global estimate, as indicated by the steady upward slope in tempo.  Have a listen:

[musichackathon]

At the August 31st, 2013 Music Hackathon NYC, we’ll be attempting to collaboratively create about an hour of new music in just 7 hours. Anyone is welcome to join us in this experiment, but also feel free to do your own thing at this hackathon — there will be an opportunity for everyone to present...

[functiontelechy]

MHD Toronto and FRANKENMASHER 2000

What if Billie Holiday sang for Black Sabbath?  What if Kenny G and The Jesus Lizard formed a supergroup?  For Music Hack Day Toronto, I built the FRANKENMASHER 2000 to help answer these questions.

The hack

To keep the combined track sounding musical, a beat tracker is run on each song, and each beat of song A is time-stretched via phase vocoding to fit the corresponding beat in song B.

Finally, the beat-synchronized auto-mashup is pushed to Soundcloud.

Most of the pieces of this hack have been kicking around in librosa and my machine learning scraps repository, although it did require porting a phase vocoder into librosa-develop.

The full implementation is open-source and hiding out in my musichacks github repository.

Limitations

There are a few limitations to the current implementation:

Source separation is less than perfect.  Somebody please solve this already?

Beat-matching is rather sketchy when the two songs have drastically different numbers of beats. Right now, beats are truncated to allow 1-1 matching, though a more sophisticated dynamic programming scheme should be feasible.

Source separation is really slow.  Too slow to deploy as a usable web-app.  Sorry, everyone!

Examples

Here are a few of my favorite examples.  Search Soundcloud for frankenmasher2000 to see more.

[functiontelechy]

Handmadedness?

July's Monthly Music Hackathon was hosted at Etsy on Saturday, with the theme of "handmade music".  Not being much of a creative type myself, I instead opted to make a hack to measure the "handmadedness" of a recording.

To keep things simple, I wanted to focus on rhythmic and structural elements of music, and not get bogged down in quantifying production value, instrumentation, or general DIY aesthetic. Specifically, I looked at tempo and repetition.

Method

As a starting point, I took inspiration from Paul Lamere's blog post about click-track detection: the basic idea is to measure the consistency of the intervals between beats.  To cook this down to a single number, I defined the metronomicity of a song s as

,

where T is the global tempo (beats per minute), and σ is the standard deviation of the interval between consecutive beats (seconds).  A high score (close to 1) indicates constant tempo, possibly suggesting the use of a metronome or drum loop -- not very hand-made!

An alternative axis of handmadedness can be derived from repetition.  Machine-assisted music often makes heavy use of samples, which might be detectable by finding sequences of duplicated acoustic content.

To test this idea out, I implemented a method based on the structural repetition features used by Serrà, Müller, Grosche, and Arcos for song segmentation.  The basic idea is to first compute a self-similarity matrix by finding the most timbrally similar beats at each beat in the song.  As an example, here is the self-similarity matrix for 2pac - Trapped:

The blocks of diagonal lines (around beats 30-50, 80-100, 140-160) represent the repeating sample used in the chorus, and the more evenly distributed spots in between represent the verses.  The main diagonal (i=j) has been suppressed to eliminate trivial self-similarities.

In the next step, the self-similarity matrix is skewed so that the diagonals become horizontal.  It is then smoothed by a horizontal median filter to remove spurious links and fill in any gaps.  The resulting repetition feature matrix looks like this:

Summing over the rows of the matrix then gives a count of how often each particular beat is repeated throughout the song (as part of a longer sequence).  The curve in the bottom plot above clearly identifies the chorus as being highly repetitive.  Normalizing this curve by the maximum number of allowed repetitions (k, if each beat is linked to at most k other beats), and averaging over all beats results in a single number that quantifies the total amount of repetition in the song.  For want of a better term, lets call this number loopiness.

Results

I ran both of the above methods on the CAL500 data set, mapping each of the 500 songs to a point in (metronomicity, loopiness) space.

Here's a playlist of the top 10 most metronomic songs:

Not surprisingly, it's all electronic/dance music.

The 10 least metronomic songs are more of a mixed genre bag:

(Note: this list should also include John Lennon - Imagine, The Beatles - Strawberry Fields Forever, and Bob Seger - Turn the Page, but these songs were not in the Rdio catalog.)  While the top two tracks are certainly sample-based, there is actually a great deal of tempo variation at the level of individual beats.

The loopiest songs, again not surprisingly, mostly consist of electronic/dance music, but now with some alternative and new wave mixed in:

Finally, the unloopiest songs:

With the exception of John Lee Hooker - Boom Boom, these songs generally lack sequential repetitions. The inclusion of Boom Boom can be explained by looking at its repetition plot:

While the general riff of the song is definitely repetitive, there seems to be enough subtle variation within each repetition to prevent it from sounding like a sample.  Definitely hand-made!

Implementation notes

My implementation relies on the current development branch of librosa: use at your own risk!  Machinedness features are available as an IPython notebook.  Pre-computed machinedness features on CAL500 are available on gist.

[functiontelechy]

Mend-a-break

The Amen Break appears in all kinds of music, often surrounded by superfluous ornamentation. It's such a great beat, why would anyone bother using any other drum loop?

I got to thinking: could one make a song entirely out of the Amen break? Of course, that would be boring. What's interesting: make EVERY song out of the Amen break!

For Music Hack Day: Philadelphia, I built Mend-a-break: an application that reconstructs any song using only the samples from the Amen Break.

Method

The idea behind mend-a-break is to represent an audio signal y (the uploaded song) as sparse combination of convolutions against a fixed basis d (drum samples).  Specifically, it solves the convex optimization problem

for a sparse set of activation coefficients a.  The alpha parameter allows the user to specify the amount of "breakiness" in the reconstruction: high values of alpha activate fewer samples, leading to a cleaner drum sound and poorer reconstruction of the original song.

To make this work at (approximately) web-speed, a few hacks had to be put in place.  The uploaded signal is chopped into a sequence of half-overlapping, one-second frames.  The convolutional decoding is performed at an extremely low sampling rate of 1024Hz.  The activation functions are then up-sampled to 22KHz, and convolved with the basis (7 one-second clips from the Amen break) to reconstruct each one-second frame.  Finally, the overlapping frame-level reconstructions are Hann-windowed and summed to produce the full song reconstruction.  

To prevent my laptop from dying a fiery death mid-demo, Mend-a-break currently limits reconstructions to 30-seconds.

Implementation

The application is implemented entirely in python using the flask library, with a small amount of javascript on the front end.  All audio analysis was done using LibROSA, and the convolutional decomposition is part of a machine learning module I've been developing lately.  Once the uploaded track has been resynthesized, it is automatically uploaded to the user's Soundcloud page to annoyshare with the world.

Examples

Here are a few example tracks generated by Mend-a-break:

[musichackathon]

Jazz & Technology Forum, Saturday April 27th, 2013 at Ace Hotel

[functiontelechy]

Generating the non-standards

To coincide with International Jazz Day, this month's Monthly Music Hackathon focused entirely on jazz.

To build the model, we downloaded a collection of standards in HumDrum format, and extracted the chords from each song.  After collapsing out inversions, we ended up with an alphabet of around 600 unique chords, such as A-7#11 or F#o7.  We then trained a first-order hidden Markov model over the chord sequences using scikit-learn. (Yes, this model is completely absurd. Half the point was to troll challenge the performers with entropy!)  Once the model has been trained, it is simple to generate a new chord progression by drawing a sample from the model distribution.

Of course, a chord progression isn't enough: we need a melody too!  To generate the melody, we built a database of measures extracted from fake books, where each measure is identified by a chord.  Then, for each chord in our generated sequence, we select a random measure from the database matching that chord, and copy that fragment of the melody into the new composition.

The end result of this process is something like this piece, which we called MARKOVJAMMMMM.

There's still a good amount of cleanup work to do on this project.  Next month, we plan to make it web-enabled and fully automated.  In the meantime, the chord progression modeling code is available on github.

[musichackathon]

As part of UNESCO International Jazz Day, the April 2013 Music Hackathon NYC will focus on how new technology can be used to research and create Jazz. We’ll be holding it in a new location, the Ace Hotel on 29th St and Broadway, which has a great conference room for the hackathon and a nice bar...

[functiontelechy]

"Rage with the machine" OR "Le Badly Drawn Lyrics Visualizer"

Saturday was my second visit to the NYC Monthly Music Hackathon. The hackathon differs from Music Hackday, in that you have much less time to put something together: about 6-8 hours instead of 24. As a result, the hacks tend to be a bit smaller in scope, but it's a nice opportunity to play around with some APIs and get something working without having to sweat the details.

This time around, I teamed up with Matt McVicar, and pulled up a hack idea that's been sitting in my brain for quite some time: automatically generate an animated rage comic from the lyrics of a song.  The end result looks something like this:

The hack basically works as follows:

Receive a search query (artist/title text),

search Echo Nest for songs matching the query, limited to the Rdio-US and musixmatch-WW buckets,

retrieve lyrics from musixmatch and query musicmetric for line-by-line sentiment analysis,

map sentiment output (1: bad up to 5: good) onto rage face prototypes,

display lyrics and animate corresponding faces in time to the music.

The hack as presented basically works, despite failing mid-demo due to exceeding my API rate limits (sorry!), but we had to make several simplifications from the original design.

Some compromises

The original plan calls for time-stamped lyrics which can be displayed in sync with the music via Rdio's playback widget.  We had planned to use the LyricFind API, which provided this kind of data for several hacks at the Boston hackday, but were sadly thwarted by LyricFind's lack of open access.  

After spending a while sniffing around, we settled on MusiXMatch, which has a simple API, but does not provide timestamps. We hacked around it by guessing a simple linear mapping of line number to onset time within the song, but this is clearly wrong for just about every song. A more sophisticated approach would attempt to localize phonemes within the audio stream, but that's much more work than would be feasible in an 8-hour hack.

We had also originally planned to do a build a more elaborate sentiment model, using word-level valence and arousal scores to map out a larger emotional space of rage faces. We eventually had to abandon this idea due to poor word coverage in the dataset, and wound up using the musicmetric sentiment analyzer instead. The happy<->angry continuum provides enough meat for a few chuckles in the demo session, but lacks the nuance necessary to accurately include some of the more interesting rage faces.

Relying on five different web services definitely has its drawbacks: there's a large amount of lag between song selection and playback. Implementing a local sentiment analyzer would go a long way toward improving responsiveness, and might be a good project for some subsequent hackathon.

[functiontelechy]

AutoChipTune: Anatomy of a hack

So I finally made it out to my first Music Hack Day in Boston over the weekend, and good times were had by all!  While sitting on the bus from NYC the night before, I thought it would be fun to build a hack that automatically generates 8-bit chiptune files from recorded audio.

Here, I'm going to explain what exactly I did and how it works (or doesn't).

The code is on GitHub, if you want to follow along.

Step 1: Building a chiptune

For our purposes, a chiptune is essentially a program that runs on specialized sound-generating hardware. Because I'm a sucker for tradition, I wanted to make tunes for the NES APU. Programming these things directly in assembler is a bit of a pain, but fortunately there's a higher-level intermediate Music Macro Language (MML) that simplifies this task.

The overall process is then: Audio -> Analysis -> MML -> NSF -> Epic robot dance party.

The MML->NSF step is explained in great detail by nullsleep's tutorial, and formed the starting point of my hack. Getting the MCK toolchain to work in linux was a bit of a hassle: my embarrassing kludge solution used wine to run MCKC_E.EXE (MML->NES), and dosbox to run NESASM.EXE (NES->NSF). Probably a better solution exists, but this got the job done.

Now, onto the real hack...

Step 2: Generating MML

At this point, it helps to know what exactly our hardware can do. The NESAPU provides four basic channels: two variable-width pulses (A and B), a triangle wave (C) and a noise channel (D).  (I didn't use channel E, as I don't quite understand it yet.)

To a first approximation, the A&B channels are used for melody (and harmony), C is used for basslines, and the noise channel is used for percussion. If we had a symbolic representation of the input audio (say, a MIDI file), it would be relatively straightforward to map onto the hardware and get something reasonable out.

Of course, we don't usually have MIDI files of every song kicking around, so we'll have to improvise a bit. This is where the Echo Nest's analyze API comes in handy!

From Echo Nest, we can retrieve a JSON object containing a pretty detailed breakdown of the acoustic content of pretty much any song (including uploaded audio), containing tempo estimation, timbre, pitch, loudness, and more.

For a 24-hour hack, I wanted to keep things as simple as possible, but there are a few basic requirements to get output that sounds at all musical: rhythm, pitch, volume, and percussion.

Step 2a: Rhythm

The analyzer provides a sequence of variable-duration segments which are intended to be somewhat stationary, meaning that the sound doesn't change too much within a segment. These segments typically last a fraction of a second, and (hopefully) correspond to a single note or chord. Our strategy will be to extract the notes from each segment, and then play them back in sequence, but this will sound pretty awful without proper rhythm.

The first rhythmic component is the overall tempo of the song (in beats per minute), which is given by the 'tempo' field of the analyzer output, and for a 120bpm song, translates pretty easily into the MML directive "ABCDE t120".

Next, the segment durations (provided in fractional seconds) needs to be converted into musical time (eg, eighth note, quarter note). I first assume that each "beat" corresponds to an eighth note. Then, to estimate the approximate duration of an eighth note, I compute the median segment duration across the entire song, which gives a number like M=0.25.

Next, for each segment duration S, I  estimate the number of (fractional) eighth note durations (M) by minimizing  over values of alpha in (1, 0.5, 0.25, 2, 4, ...) for eight, quarter, half, sixteenth, and 32nd respectively. Dotted notes are also be done in a similar fashion.

Now we know how long to play each note, so the next step is to figure out which notes to play...

Step 2b: Pitch and melody

Analyze provides pitch information for each segment as a 12-dimensional pitch vector, which contains the amount of normalized spectral energy for each note in the chromatic scale (C, C#, D, etc).

We have three tonal channels to work with (A,B,C), but because octave information is discarded, it's not clear how to assign pitches to different voices (treble and bass).  I add a few simplifying constraints here:

Channel A (treble) always plays the loudest tone,

Channel B (treble) plays harmony against A, one octave higher,

Channel C (bass) always copies A, but one octave lower.

It's a fairly simple matter to find the loudest tone at each segment, but some care needs to be taken to keep it sounding musical. Instead of taking the raw pitch vector at each frame, I use the global key estimate (eg, Amin) to scale down pitches outside of the primary key by some given factor (say, 0.25). After this filtering, the loudest tone is assigned to channels A and C.

Next, I find the second-loudest tone, and if its energy relative to the loudest tone exceeds a set threshold (say, 0.5), it gets assigned to channel B.

Finally to put a bit of dynamics into the mix, I quantize the segment loudness (in decibels) and relative pitch energies, and adjust the output channel amplitudes accordingly using the v[0-15] MML directives.

Now we've almost got music, but it needs something else...

Step 2c: Percussion

Music ain't music without drums. NESAPU gives us a noise channel, which can be strobed to simulate something like a snare or a cymbal, and using it adds some rhythmical flair to the whole chiptune experience.

Percussion events are detected by counting up the total pitch energy (prior to key-filtering) in each segment. If the total energy is more than 4.5x that of the loudest tone, I assume that this is the result of broad-spectrum noise instead of a complex chord, and activate the noise channel (D). Whenever this event occurs, the secondary tone is not necessarily reliable as a harmony, so the B channel is suppressed. (Probably the A channel should be suppressed too, but that makes the output sound choppy.)

... And that's pretty much it!

Examples

Here are a few example outputs. As you can see, octave errors are a huge pain, but it does ... uh .. something.

Limitations and to-dos

This was just a 24-hour hack, so there are a lot of things that don't quite work right and could be improved:

Octave unfolding: Most melodies don't stay within a single octave range, so the results right now sound pretty weird. This could be resolved by analyzing the audio content directly, instead of (or in addition to) using Echo Nest's API.

Loop detection: the output files right now are huge, because the converter doesn't know anything about repeating patterns.

Better timbre modeling: it would be cool to use the timbre descriptors to shape the variable-width pulse channels, eg, to simulate voices or distorted guitars, etc.

Better harmony: the bassline especially could be improved by using a bit more harmonic smarts here.

Better handling of long songs: the current implementation just outputs the song as a single sequence, but this can easily overflow the NESAPU memory. There are solutions to this using bank switching (similar to how it works for the main NES cpu), but I didn't attempt it here.

EDIT: I should also note that the current implementation works on artist/title search queries, and not directly on audio. This can be fixed pretty easily with the Echo Nest API, I just didn't bother with it in the interest of skipping upload time.

So that's pretty much it. Also be sure to check out the other excellent hacks at the Hacker League site!