Analog Labs

An opinionated systems engineer plays his hits

[analoglabs]

Public key crypto using keys you already have

I almost always have my work-mate's ssh-public keys. Sometimes I want to send them encrypted files, thus the motivation.

reformat the public key

ssh-rsa format doesn't work with openssl. We need pem format, but more specifically, we need PKCS8 PEM format. ssh-keygen to the rescue:

ssh-keygen -e -m PKCS8 -f friend_pubkey.pub > friend_pubkey.pem

draw the rest of the owl

With that out of the way we just need to make a 256 bit (32 byte) key file, encrypt that with the pubkey we have, then use the key file to encrypt the secret data:

make the encryption key

openssl rand -base64 32 > key.bin # mk key

encrypt the encryption key

openssl rsautl -encrypt -pubin -inkey friend_pubkey.pem -in key.bin -out key.bin.enc # encrypt key

encrypt your secret data

openssl enc -aes-256-cbc -salt -in secret_file -out secret_file.aes256 -pass file:./key.bin # encrypt the secret file with the key

delete the plain key

rm key.bin # seal your fate

Use shred/srm if you're not on an encrypted fs

And done!

troubleshooting

or, things you'll see if you did it wrong.

You can use openssl rsa -inform PEM -pubin -in friend_pubkey.pem -noout to validate that you've got the correct format. It's confusing because you don't call out PKCS8 directly in OpenSSL, just with ssh-keygen.

If you use ssh-keygen -e ... -m PEM instead of -m PKCS8, OpenSSL will choke on the file with this:

unable to load Public Key

and/or

62757:error:0906D06C:PEM routines:PEM_read_bio:no start line:/BuildRoot/Library/Caches/com.apple.xbs/Sources/OpenSSL098/OpenSSL098-64/src/crypto/pem/pem_lib.c:648:Expecting: PUBLIC KEY

if you realize that it's supposed to be BEGIN PUBLIC KEY instead of BEGIN RSA PUBLIC KEY and just edit it, you'll get these:

unable to load Public Key 62924:error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:/BuildRoot/Library/Caches/com.apple.xbs/Sources/OpenSSL098/OpenSSL098-64/src/crypto/asn1/tasn_dec.c:1344: 62924:error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:/BuildRoot/Library/Caches/com.apple.xbs/Sources/OpenSSL098/OpenSSL098-64/src/crypto/asn1/tasn_dec.c:387:Type=X509_ALGOR 62924:error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:/BuildRoot/Library/Caches/com.apple.xbs/Sources/OpenSSL098/OpenSSL098-64/src/crypto/asn1/tasn_dec.c:768:Field=algor, Type=X509_PUBKEY 62924:error:0906700D:PEM routines:PEM_ASN1_read_bio:ASN1 lib:/BuildRoot/Library/Caches/com.apple.xbs/Sources/OpenSSL098/OpenSSL098-64/src/crypto/pem/pem_oth.c:83:

misc

If you're going to the trouble of encrypting something, it's a good idea to setup your encyption workspace so it's vaguely secure.

umask 0077

mkdir ~/sec; cd ~/sec

there is no step 3.

credits

I found http://www.czeskis.com/random/openssl-encrypt-file.html in a google search, and it was SUPER helpful.

Cheers.

[analoglabs]

Make a bootable USB stick from ISO on macOS

It seems like I do this infrequently enough that I always have to re-lookup how to do it. Then, every tutorial I find does it slightly wrong. So here’s the right way to make a bootable USB stick from an ISO on macOS.

Convert the iso to img

hdiutil convert -format UDRW -o /destination-image-path.img \ /source-image-path.iso

Should look like this:

Find destination device

<pre>` diskutil list # find the device representing your usb stick # make sure you're right, or you'll lose data # # it will be something like /dev/disk9, but instead # of 9, it will be some other number. You can # disconnect/reconnect your usb drive to find it by # seeing which one disappears and comes back # # Or you can just go off the brand/model/capacity # of the device; that's what I do

dd with pv

sudo su -c "pv -pterb < /that-img-file.img | dd of=/dev/rdisk9 bs=1m" # note that I used /dev/rdisk9 instaed of /dev/disk9. I can't be # bothered to lookup what rdisk means, but it makes it go faster

And it should look like this:

safely eject

diskutil eject /dev/disk9

or if you're paranoid

sync;sync;sync; sleep 10; diskutil eject /dev/disk9

And done.

[analoglabs]

The Netflix Problem

I moved houses over the weekend. Well, from an apartment to a house, anyway. Rather than transfer my existing service with Comcast over, I decided to treat myself and get 1 Gbps fiber from Century Link.

It cost me a couple hundred up front, and will probably be $109/month continuing. Not bad considering I was paying $80 for 120mbps via Comcast. Plus, having gigabit to the home is exciting.

There’s a catch, though. Connections to certain companies seem to be selectively slowed. We’ve noticed major problems connecting to both Steam, and Netflix.

So naturally, I spent an entire day trying to track down why.

Speed tests

Let’s do some speed tests!

First, fast.com from Netflix. This test is performed against the same infrasturcture that Netflix uses to stream videos.

via Gfycat

And we get 10Mbps. Ten. Fun fact, 10Mbps is 1% of the speed I pay for through CenturyLink. I watched Tallulah last night on Netflix, and it was dead slow. I can definitely confirm from all the distortion and compression artifacts that 10Mbps is about what I was getting.

Now let’s head to speedtest.net.

502Mbps down and 268.72Mbps up. That’s about what I’d expect out of a residential 1Gbps link, so the speed itself isn’t surprising. What is surprising is the disparity between the Netflix test at 10Mbps and the Ookla test at 502Mbps. Why in the world is there an order of magnitude difference?

Down the rabbit hole

The next logical step is to see if there’s some network problem that exists between the router at my house and the servers at Netflix.

Since I already have Little Snitch handy, I’ll use that to grab the hostname of the fast.com server I’m connecting to.

ipv4_1-lagg0-c053.1.lax004.ix.nflxvideo.net. Neat. Let’s traceroute.

Nothing seems particularly out of line. No loss, no jitter; looks fine. So let’s ping all of the hosts mentioned in the trace to see if any of them feel overloaded. This won’t prove anything on its own, but it might give us a clue of where to go next.

A note about ping

Ping, traceroute, and mtr (the tool we’re about to break out) all use the Internet Control Message Protocol, or ICMP, messages to gather data. I could write a whole article on why dropped ICMPs aren’t a big deal, but here I’ll just say this: dropped pings are virtually meaningless on their own. The best they can do is point you towards your next troubleshooting step. Now that’s out of the way, let’s keep troubleshooting:

HERE is something to look at! dvr-brdr-02.inet.qwest.net looks angry. It’s consistently dropping quite a bit of icmp traffic. Let’s look at that guy a little closer. Am I always routed through that server, or is it just netflix traffic?

Looks like we’re not always routed through dvr-brdr-02, but a lot of the time we are. The most important trace is the one to speedtest.xmission.com.

See, we know that we can pull 502Mbps from speedtest.xmission.com as tested through speedtest.net. With that trace, we also know that we’re pulling that 502Mbps through our friend dvr-brdr-02. Since we care more about traffic through than to, we can rule out dvr-brdr-02 as a possible suspect. Though, that hop might be a tad overloaded.

So… What the hell? Netflix is still slow, and there doesn’t appear to be any real network issue between us. Well, let’s try a VPN.

The VPN test

Is our connection to Netflix still slow if it’s over a VPN? I subscribe to Goldenfrog’s VyprVPN, so let’s use that.

Wat. A four-fold speed increase. Not only that, but the speedtest.net results roughly match the fast.com results. This doesn’t conclusively prove anything, but it very strongly hints that CenturyLink is shaping based on destination. Bad bad bad.

Even more traces

Alright, let’s see what route we take to the vpn server.

There’s our old friend dvr-brdr-02 again. So we’re taking the same route out of Century Link’s network.

Now let’s see what route we take to the nflxvideo.net server when connected to VyprVPN and then compare it to our non-VPN’d trace.

It looks like either way we wind up on Cogent’s infrastructure with traffic destined to Netflix. If we were seeing destination based QoS/Throttling/Shaping from Cogent, we would probably see it over the VPN as well.

It’s not conclusive, but I doubt Cogent is throttling Netflix-destined traffic.

The result

Well, the result is kind of muddy.

I don’t think Cogent or Netflix are responsible for the slow down. I think CenturyLink is responsible. It could be a misconfiguration, or some overloaded piece of gear, or some disgruntled engineer that hates Netflix, or it could be on purpose. I honestly don’t know, and I honestly can’t know because I don’t have access to CenturyLink’s network gear.

But why? What’s the motive? To sell more TV subscriptions?

Maybe… My guess is that they don’t have the capacity in either engineering or infrastructure to serve their clients at full speed.

But it’s just a hunch.

It could also be the result of poor peering agreements. ¯\_(ツ)_/¯

The workaround

In reality, there’s no way I can win a fight against an internet service provider. It doesn’t matter how many times I call technical support, or how many blog posts I write, or how ways I demonstrate the issue. CenturyLink will never admit fault, and they probably won’t fix the actual issue either.

So instead, I’m going to route traffic destined for Netflix over a persistent VPN link. It’s not optimal, and it’s really really shitty that I have to pay for an extra vpn service just to use the internet service I already pay for. Extremely shitty.

But… I guess that’s life.

I’m a bit worried about how many more of these I’m going to find. Am I going to have to route dropbox or Apple connections over the VPN too? At what point does it just make more sense to switch ISPs? I don’t know. We’ll see!

UPDATE

We’re just going to cancel and resubscribe to comcast at 250mbps. Having a real quarter gig is better than a whole fake gig. Google help us.

[analoglabs]

Find your CenturyLink PPP credentials with this one weird trick

Preface: You can call CenturyLink, and they’ll be happy to give you these credentials. I just didn’t feel like calling.

I just got gigabit fiber from CenturyLink (YAY!) The tech that came out to set it up installed a Technicolor C2100T that I bought for $100. I also have an ASUS AC5300 that I'd prefer to use. Switching over is easy enough, you just need your PPP credentials from CenturyLink.

Luckily, these credentials are really easy to lift from the C2100T, which saves a support call.

Lifting the creds

Login to your C2100T (probably at http://192.168.0.1)

Click on Quick Setup

3. Right-click on the password field and choose Inspect or Inspect Element

4. Double-click the type="password" part of the selected element and change the word password to text and hit return.

5. Your PPP password is now visible.

[analoglabs]

Make centos seven great again

CentOS no longer comes configured as a server operating system. For example, NetworkManager and wpa_supplicant have no place on a server that is going to be connected to a single set of networks for its entire life. So let's make it great again.

Fix networking

Rename the network devices to something sane like eth0 and (if you have more) eth1, etc.

Update /etc/default/grub and add net.ifnames=0 to the kernel options. This tells the kernel to expose eth0 instead of eno2304923409 or whatever number it pulls out of its ass.

After you edit /etc/default/grub, rebuild the config with grub2-mkconfig -o /boot/grub2/grub.cfg. Without rebuilding the config, the kernel will never know it's not supposed to use the stupid ifnames. That makes the kernel very sad.

If you have a 70-persistent-net.rules file under /etc/udev/rules.d/, blow it away so udev doesn't shit itself on next boot.

Next, head over to /etc/sysconfig/network-scripts and move the ifcfg files to their new names (e.g. the first network device goes from ifcfg-enoXXXXXXX to ifcfg-eth0.) Once they're renamed crack those open with $editor and fix names in exactly the same way. Double check your changes, because if they're wrong, you won't have network on the next boot.

Package switcheroo

Remove some unnecessary packages with yum remove NetworkManager wpa_supplicant polkit and install some useful packages with yum install iptables-services epel-release.

Fix hostname

echo whateveryouwant.jpg > /etc/hostname then update your /etc/hosts file with your new hostname.

CentOS 7 is now usable

You should reboot so all the changes we made apply all at once. If you did something wrong, your system will probably be broken. If your system is broken, go rtfm.

Congratulations, CentOS is great again.

[analoglabs]

goleftpy-pad.js

Why do we shove the name of the language we're using into our product names?

[analoglabs]

Why is WiFi Calling dropping ass-loads of calls?

Or more accurately: *Why are my text messages failing to send, and some inbound calls dropping within 10 seconds of answering when using T-Mobile's WiFi Calling, and an iPhone Six Plus?

Short Answer

AiRadar (Asus) and/or beamforming (Netgear and everyone else) is breaking your iPhone's wifi-SMSs and wifi-calls.

Long Answer

I have an Asus rt-ac66u. Last week I was tinkering around and upgraded from the latest stable asus firmware to the latest beta asus firmware (3.0.0.4.378.9533, and 9.0.0.4.380.2695 respectively.) From that point forward, practically every inbound call (but pretty much zero outbound) and virtually all sms traffic over T-Mobile WiFi quit working.

Turns out that AiRadar, also known as beamforming is enabled by default in the beta firmware. Beamforming and Apple iPhone/iPads don't play well together. Turns out it really screws with your iPhone's networking stack and will cause a bunch of network weirdness including, but not limited to (ta da): wifi calling and wifi sms.

By the time I realized what was happening, I had bricked my router by trying to downgrade to asus' latest stable firmware and kicking it in the middle. Now I'm back on latest stable, sans beamforming, and wifi calls/sms are working perfectly again.

Why just inbound?

This is pure conjecture as I haven't taken any steps to verify this is actually correct.

I'm guessing the beta build uses beamforming for every wifi session unless it detects a failure (aka implicit beamforming), at which point it disables beamforming for the next session to the same device.

I'm also guessing my iPhone sleeps its radios when the screen isn't on, which would cause a new wifi session to come to life between the phone and the router with beamforming.

With those two assumptions, the sequence of events for an inbound call looks like this:

Inbound call causes iPhone wifi radios to wake and re-establish connection with the AP

AP uses beamforming to create the new connection

I answer the call, and then the beamforming connection drops, and subsequently drops the call

The router notices the drop and switches off beamforming to that device for the next session

New session gets established without beamforming, and voila wifi appears to work fine for the rest of the time I'm using my phone.

This also explains why SMSs sent via from my macbook would be broken.

Just a guess though. The 802.11 spec, and the router firmware code would know for sure.

my gear

Asus rt-ac66r (same as the rt-ac66u)

Asus Firmware 9.0.0.4.380.2695 (beta track)

iPhone Six Plus

iOS 9.3.2 (beta track)

T-Mobile WiFi Calling

If you're seeing the same behavior with different gear, feel free to leave a comment detailing what it is so people from google can compare/contrast.

[devopsreactions]

When someone doesn’t follow the coding style guidelines

by @uaiHebert

[devopsreactions]

The effect of GIL on multithreaded Python programs

by @uaiHebert

[analoglabs]

GIL needs to die in a fire.

[devopsreactions]

Checking shared hosting mail queues

by john

[analoglabs]

The ptsd is real. Here's looking at you, Bluehost mail queues.