#gnu parallel

[vishers]

Using `mkvtoolnix` and `GNU Parallel` to Whip Some .ASS

I recently needed to work some mkvmerge=/=mkvpropedit magic for some anime. It's been a long time since the days when I started watching Bleach with my friends by sshing to one laptop from another and using mplayer in a terminal to play the show and the state of mkv support has come a very long way since then.

These days I prefer playing all my media through Plex which is just hard to describe in all its towering greatness. The show that I wanted to watch though had its ASS subtitles and accompanying fonts broken out from it's MKV files for reasons unknown to me. Because of this and the amazing malleability of MKV I decided to roll up my sleeves and fix the problem myself.

shopt -s extglob parallel -q --tagstring {/.} --line-buffer mkvmerge \ -o {.}_merged.mkv {} --language 0:eng {.}.en.ass \ --attachment-description '' \ --attachment-mime-type application/x-truetype-font \ --attach-file {//}/'fonts/A-OTF-FutoMinA101Pro-Bold.otf' \ … 147 similar lines … ::: Season*/!(*_merged).mkv

The main reasons I'm writing this up at all are several fold:

Often times what I'm doing with GNU Parallel is complex enough that it warrants an actual shell function or script or it's so simple that it more or less amounts to slapping one of the arguments onto the end. Because of this I've rarely explored parallel's native expansion support which this task gave me an opportunity to do.

The task was complex enough also that I learned about parallel's double expansion which I'm sure has been the source of many frustrating missteps in the past now that I know it's there. Essentially bash will process the arguments to parallel first obeying all the normal rules and then pass them to the exec of parallel, then parallel will pass them again to a subshell which will parse them again through all the normal rules. Keeping this all straight in your head is not easy but I think the essential rule is probably something like "If you actually notice the double expansion you should write a function/script for the behavior and otherwise you should probably be running with -q" or something similar but better worded than that.

It's fun to mess with mkv files. ¯\_(ツ)_/¯

To review the script:

We're setting extglob because we want to be able to process only mkv files that haven't been _merged yet.

We're invoking parallel with -q because we don't want the subshell to word split our carefully constructed arguments again.

We're using --tagstring {/.} --line-buffer because we want to see our job output live so we know it's working (it takes a bit to make these changes to the matroska files). It's not directly documented AFAICT that --tagstring supports GNU Parallel expansion but I took a chance on it and was pleasantly surprised. This particular expansion is the basename-sans-extension version which seemed like a sensical tag for the logged lines.

This feature alone is worth using parallel for when you're really doing complex parallel work. It's a really efficient way to provide active log lines that are still comprehensible later on. --line-buffer just makes sure that you always get a full line from a job rather than the lines from the various jobs mixing mid-line.

-o {.}_merged.mkv is a nice way to express the bash-ism of ${x%.*}_merged.mkv.

--language 0:eng {.}.en.ass is a bit speculative as it's what I think I should have run but I didn't initially since I was working off an example that didn't include it. Nevertheless I think I'm interpreting the docs correctly. Originally I just didn't include the --language 0:eng bit so the subtitles were attached as an unknown language and I had to then go back through and correct that with an mkvpropedit run.

Then I used a bit of Emacs Keyboard Macro magic to transform the Dired fonts buffer into a series of attachment arguments so that the ASS subtitle track could be properly rendered. The parallel expansion there of {//} was also something I hadn't seen before and is how I managed to get away with this from the root of the seasons directory rather than needing to process the seasons one at a time. That one specifically expands to the dirname of the input line or as a bash-ism ${d%/*}.

Finally we're taking arguments from the extglob that matches all the Seasons mkv files excluding the _merged files which you can see are what the parallel tasks are actually producing.

With all this I was able to completely saturate my wired connection to my Synology and efficiently process the entire show. I love the smell of burning silicon in the morning.

I hope this little foray into parallel/mkvtoolnix land teaches you something like it taught me.

Happy scripting!

[vishers]

Effective bash/jq: Displaying All The AZs

I wrote the following script the other day because A) text is always superior to a screenshot, B) C-r is a thing, and C) I can't seem to find where Amazon actually publicly documents their AZs.

I wanted to share because I think it shows a couple of techniques which I don't see very often in scripts as well as yet again proving that it's good to know the tools you use.

#!/usr/bin/env bash { cat availability-zones || { for region in $( cat regions || { aws ec2 describe-regions --output=text --query \ 'Regions[].RegionName' | tee regions } ) do aws --region="$region" ec2 describe-availability-zones | jq '.AvailabilityZones[]|{ZoneName,RegionName}' | tee -a availability-zones done } } | jq -Ss 'reduce .[] as $item ( {}; .[$item.RegionName].azs += [$item.ZoneName] )'

cat availability-zones || … | tee -a availability-zones

This is the first bit that I think is both clever and useful. I use it twice in this script. Once here and the second one with the regions output. The idea is to use a file as a cache with cat's exit code being used to figure out whether the cache needs to be populated or not. To re-query the regions you just rm availability-zones regions and rerun the command. If you're happy with the regions you found but you want to re-query the zones you can just rm availability-zones and be good to go from there.

The key is to use cat on the one hand and tee on the other. tee is so fantastically useful and everyone should be aware of it. I'm an especially huge fan lately of the tee <<<"charnock" stephen idiom to both populate a file with a string while also printing it my screen. Very useful in my CD pipelines.

The next bit that I think is cool is the use of --query on my aws commands.

… aws ec2 describe-regions --output=text --query 'Regions[].RegionName' | …

Unfortunately you then see me immediately turn around and fall back to jq like a coward because JMESPath just does not compute for me. And really more so because jq is general whereas JMESPath only helps me in awscli.

Nevertheless I'm sure it's more efficient (for some definition of the word) to encode my JSON response processing in --query if I can rather than starting up the separate process (although I don't think I can measure jq's startup time.).

I also really like the use of jq's -S and -s flags here.

… | jq -Ss 'reduce .[] as $item ( {}; .[$item.RegionName].azs += [$item.ZoneName] )' …

If you're unfamiliar with them -S means sort keys and -s means slurp all input into an array before processing it. -s is getting me the ability to suck all the responses together into a single array so I can reduce it and -S is making it so that the humans I'm pasting this text to can easily find what they're looking for.

Just for kicks here's a fully parallelized version of the same code:

#!/usr/bin/env bash { cat availability-zones || { parallel aws --region='{}' ec2 describe-availability-zones ::: $( cat regions || { aws ec2 describe-regions --output=text \ --query 'Regions[].RegionName' | tee regions } ) | jq '.AvailabilityZones[]|{ZoneName,RegionName}' | tee -a availability-zones } } | jq -Ss 'reduce .[] as $item ( {}; .[$item.RegionName].azs += [$item.ZoneName] )'

Just because, you know, 😍 parallel 😍.

Cheers.

[zshello3]

140605 parallel --noswap finally works on OS X

Today witnessed the final fix of --noswap bug of GNU Parallel on OS X. View details of the bug here on Savannah (submitted by me). The problem is simple: strangely enough, vmstat is vm_stat on OS X, and the output format is completely different (from most Linux flavors). You need to use vm_stat -c 2 1 on OS X to get something like vmstat 1 2 on (most flavors of) Linux.

[zshello3]

140530 GNU parallel feature suggestion

I just suggested a feature — extension replacement string — for GNU parallel (and unfortunately screwed up Savannah's markup). Let's see how it goes.

The feature suggestion text:

Feature suggestion: replacement string for extension We already have +verbatim {} Input line. {.} Input line without extension. {/} Basename of input line. {//} Dirname of input line. {/.} Basename of input line without extension. -verbatim That is to say, if we label dirname with 1, basename without extension with 2, extension with 3, then we already have 123, 12, 23, 1, 2. Therefore, it might be a good idea to add a replacement string, possibly {..}, for extension. To give you an idea of why this could be useful, I do batch image processing frequently, for which I rename the images yet preserve the extensions (.jpg, .png, .gif, .tiff, etc.). Extracting the extension is as simple as an awk, but I do it frequently enough so it will be nice to have a replacement string out of box.