Karl von Randow @karlvr - Tumblr Blog

jmap -heap fails on Ubuntu with OpenJDK

Whenever I try to use `jmap -heap`, which is usually when I’m in a hurry as things are going weird, it fails with:

java.lang.RuntimeException: unknown CollectedHeap type : class sun.jvm.hotspot.gc_interface.CollectedHeap

Easily fixed by installing the OpenJDK debug symbols.

I’m currently using the openjdk-8 package, so I install the openjdk-8-dbg package. I install without its recommended packages, as it wants to install all kinds of stuff!

apt-get install --no-install-recommends openjdk-8-dbg

Now my jmap -heap works perfectly.

#java #ubuntu #openjdk #jmap

We are excited to announce that Charles Proxy is now available on iOS!

With the iOS version of Charles you can capture and inspect network requests and responses on your iOS device. You can view metadata, headers and bodies in the app, so you can finally debug your app’s networking issues without a computer.

Charles for iOS currently supports the following features:

Capture HTTP and HTTPS network traffic on your device

SSL Proxying so you can view your SSL / TLS requests in plain text

View requests and responses in the app, or share each request or a whole session to Charles on your desktop

Running Charles on your iOS device means you no longer need to fiddle with WiFi network proxy settings. It also means that you can capture and measure network traffic that goes over the Mobile / Cellular data network.

Measuring networking performance over Mobile data is especially important for your mobile apps (as that is how a lot of users experience your app), and it can reveal large or slow requests, as well as opportunities to increase perceived performance by parallelising network calls.

We are thrilled to be able to finally bring Charles to the App Store. If you’d like to know a bit more about the history of Charles, and particularly Charles on iOS, I gave a presentation to try! Swift Tokyo in March 2018 that you can watch on YouTube.

We plan to bring more of the features that you know and love from Charles on the desktop to the iOS version over the coming months, so watch this space for more announcements.

We hope you enjoy using Charles on iOS!

#charles

Now that Apple has removed the ability to access the iOS App Store in iTunes 12.7 on Mac/Windows (See MacRumors article), I can't find any way to download an IPA file from the App Store. As a note,...

#ios

While using Virtualbox, one thing that I have found is that the size of the VM (a file with .VDI extension) expands over time. If you have selected the “Dynamically Expanding Storage” option when first creating the VM, you will find that the file size expands whenever you install new applications in the VM. That …

#virtualbox

Roadwarrior configuration for macOS 10.12, iOS 10 and Windows 10 using strongSwan and user certificates

#strongswan #vpn

GeoIP on PostgreSQL 9.5

This guide is for adding GeoIP functionality to PostgreSQL 9.5 (and earlier) using MaxMind’s GeoLite2 CSV files.

You can read about the MaxMind GeoIP2 CSV files, and download free versions. You’ll need the CSV files to continue.

I create two tables; geoip_blocks and geoip_locations to contain the data. I load the English country names, but you could choose the language you want when you populate the geoip_locations table. I have copied the relevant GeoLite2 CSV files into the /tmp directory in the example below, but you could read them from anywhere.

You can use the index support for cidr addresses built in to PostgreSQL 9.4 and later, but you get much better performance using ip4r instead.

I have examples of both approaches here: https://gist.github.com/karlvr/8ff1900bfc9cba36468640c26ae3b2bb

#postgres #postgresql #geoip

AWS CloudFormation UserData for Ubuntu 16.04 LTS

I use CloudFormation to create EC2 instances on AWS. I use this UserData script to install the CloudFormation scripts on the EC2 instance and then to run the init and signal steps.

"UserData": { "Fn::Base64": { "Fn::Join": [ "", [ "#!/bin/bash -x\n", "apt-get update\n", "apt-get install -y python-pip\n", "pip install https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz\n", "cfn-init -v --resource <RESOURCE NAME CONTAINING INIT CFG>", " --stack ", { "Ref": "AWS::StackName" }, " --region ", { "Ref": "AWS::Region" }, "\n", "cfn-signal -e $? --resource <RESOURCE NAME CONTAINING CREATION POLICY>", " --stack ", { "Ref": "AWS::StackName" }, " --region ", { "Ref": "AWS::Region" }, "\n" ] ] } } }

You then need to add the CloudFormation init configuration to the resource named above. I put it in the instance resource, but I don’t think it has to be.

For example:

"Metadata": { "AWS::CloudFormation::Init": { "config": { "packages": { "apt": { "awscli": [] } } } } }

You can also add a CreationPolicy onto a resource; usually the instance or the auto-scaling group. Creation policies have replaced WaitConditions.

"CreationPolicy": { "ResourceSignal": { "Timeout": "PT5M" } }

#ubuntu #aws #cloudformation

AWS VPC VPN Strongswan configuration

Create the VPN Connection in the VPC Management console on AWS, using static routing, then download the Generic configuration. The downloaded text file contains some values that you’ll need. There are two VPN configurations in it. I just hook up one on the server. Perhaps if you have two VPN servers you could set up one VPN on each.

These are the values of interest in the downloaded text file:

Pre-Shared Key

Outside IP Addresses

Customer Gateway

Virtual Private Gateway

Inside IP Addresses

Customer Gateway

Virtual Private Gateway

My server has an internal IP address, and sits behind a router, which has a public IP address. AWS VPC supports NAT-T so this is no problem. You just set “left” (below) to your internal IP and “leftid” (also below) to your public IP.

Here is the example /etc/ipsec.conf:

conn vpc mobike=no type=tunnel compress=no keyexchange=ikev1 ike=aes128-sha1-modp1024! ikelifetime=28800s esp=aes128-sha1-modp1024! lifetime=3600s rekeymargin=3m keyingtries=3 installpolicy=yes dpdaction=restart authby=psk left=<ip address of your server> leftid=<public ip address of your server>

conn vpc1 also=vpc auto=add right=<Outside IP Addresses: Virtual Private Gateway> leftsubnet=<your subnet> rightsubnet=<VPC subnet>

conn vpc1a also=vpc auto=add right=<Outside IP Addresses: Virtual Private Gateway> leftsubnet=<Inside IP Addresses: Customer Gateway> rightsubnet=<Inside IP Addresses: Virtual Private Gateway>

Here is the example /etc/ipsec.secrets:

<Outside IP Addresses: Virtual Private Gateway> : PSK "<Pre-Shared Key>"

Then in your AWS VPC configuration edit the route table and add a static route for your internal subnet to the Virtual Gateway device.

Check your security groups on your instances to make sure they allow connectivity from your internal subnet IPs. It can be useful to allow ICMP so you can test using ping.

Restart Strongswan:

service strongswan restart

Then try to bring up the VPN interface:

ipsec up vpc1

If all is going well you should see a successful connection result in a second or two. If not, something is wrong :-(

Try to connect to one of the servers in your VPC. If you can’t, check the security groups on them, or perhaps any firewall rules on your own machine.

Then bring up the vpc1a connection. This should result in the VPN showing as UP on the AWS VPC VPC Connection configuration page.

Once you’re happy, change “auto=add” to “auto=start” in /etc/ipsec.conf and restart Strongswan and the VPN would come up automatically.

#aws #strongswan #vpn #vpc

Cocoapods and Swift 3

Cocoapods supports Swift 3 as of 1.1.0. Cocoapods sets the SWIFT_VERSION of the pod targets to match the SWIFT_VERSION set on your project. This is a pretty clever solution to Swift evolution, and you can read more about it in the Cocoapods 1.1.0 announcement blog.

Note the bold text on project in the previous paragraph. You need to set the SWIFT_VERSION Build Setting on your project, not just on your project’s targets.

#cocoapods #swift #swift3

Setup exim4 and DKIM

I have setup exim4 to sign outgoing email using DKIM, with the goal of improving deliverability. I have setup exim4 to handle multiple outgoing domains with different keys.

The key to use is chosen based on the domain name of the From header, not the envelope mail from. This is because I believe DKIM verification uses the From header (https://dmarc.org/wiki/FAQ).

I based my approach on instructions from https://debian-administration.org/article/718/DKIM-signing_outgoing_mail_with_exim4, but they appeared to be either slightly incorrect or out of date.

Configure exim4

First you need to make up a DKIM selector. This is the part of the domain name that goes on the front of the ._domainkey.example.com DNS entry you’ll later make. I configure one DKIM selector for all of the domains, so I choose one that identifies the box or the entity that owns the box. If you set up DKIM for the same domain on another box with different keys, the selector is what identifies which keys to use.

Say you choose the selector “mydkim”.

If you have split configuration, edit /etc/exim4/conf.d/main/00_exim4-local_options and append the following to the end of that file. If you are not split, edit /etc/exim4/exim4.conf.template and insert the following after the MAIN CONFIGURATION title comment near the top of the file.

DKIM_CANON=relaxed DKIM_SELECTOR=mydkim DKIM_DOMAIN = ${sg{${lc:${domain:$h_from:}}}{^www\.}{}} DKIM_FILE = /etc/exim4/dkim/$dkim_domain.pem DKIM_PRIVATE_KEY = ${if exists{DKIM_FILE}{DKIM_FILE}{0}}

Run update-exim4.conf to update your configuration. Restart exim4 and check that you can still send email. I suggest sending email to another email account, not on your server. Check that /var/log/exim4/paniclog remains empty after successfully sending email.

Congratulations you’ve successfully setup exim4.

Exim4 will look in /etc/exim4/dkim/DOMAIN.pem for a private key to use to send email with a From address domain of DOMAIN.

Generate DKIM keys

Now you need to generate keys for exim4 to use. In the example below, replace “example.com” with your domain name.

cd /etc/exim4/dkim openssl genrsa -out example.com.pem 1024 -outform PEM openssl rsa -in example.com.pem -out example.com.pub -pubout -outform PEM

Don’t send any email yet. Exim4 will start using this private key immediately, and it will sign your email. But you haven’t setup your DNS yet.

Add DKIM public key to your DNS

You need to add your DKIM public key to your domain’s DNS. This is how a remote server can lookup the public key to check the signature your server has added to your email.

Open the example.com.pub file generated above.

Now if you send email to an email account not on your server, you should see a Dkim-Signature header in it. It will start with -----BEGIN PUBLIC KEY----- and end with -----END PUBLIC KEY-----. The line in between are your public key!

Create a new DNS record on your domain using the DKIM selector you chose above. In my example this is mydkim, so the record we add is for mydkim._domainkey.example.com.

The DNS record type is TXT. You should already have an SPF record of type TXT. If you don’t, google SPF records.

The value of the TXT record starts with:

v=DKIM1; k=rsa; p=

Directly after the last =, paste each line of your public key with no new lines. So it all ends up on one line.

e.g. v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDAPUE4hEV7wSh7v9s/NrhbJu7k1/Jqr3Mt7pspT6o7c/Q+GFP1Ko7VA0I12RYB+PeFUE+3i3yu1fsmzGn92GdhKd2nObZbs06Rynm48yVPmzXV2pEptebfOTfAdsJh3rryB0HQnPx+H1gKww1/nUagYlUjktBL7sRDGdjNqTIxYwIDAQAB

Save that DNS record.

If you want to verify that it worked, try this command-line tool:

dig txt mydkim._domainkey.example.com

Of course you probably need to wait for your DNS to update. If you know the primary DNS server for your domain, try querying that directly if you’re impatient.

Testing

Now exim4 is signing your email, and you’ve setup your domain, so it’s time to send a test email.

Send an email to [email protected] and in a few seconds you’ll receive an email back summarising the findings about your email. With luck you’ll have passed the DKIM check. If the DKIM check failed, scroll down to read why. If it is because the _domainkey subdomain we added doesn’t exist, perhaps you need to wait a little longer for the DNS to propagate. Check though that it reports the same subdomain that you added! This will confirm that you have set your DKIM selector correctly.

Now send an email to yourself on another email server, e.g. a gmail address. Check that the received email contains a Dkim-Signature header. That’s the magic bit.

Adding additional domains

When you want to add DKIM signing for additional domains, repeat the Generate DKIM keys, Add DKIM public keys to your DNS and Testing steps. You don’t need to configure exim4 anymore as it’s setup for as many domains as you like.

#ubuntu #linux #exim4 #DKIM

Because of bug #693758 I'd like to prevent apt-get upgrade and Update Manager from updating the "libgtk2.0-0" package. How can this be achieved?

Hold a package to prevent it from being upgraded unless you specifically upgrade it. I use this to hold postgresql packages, as installing upgrades results in a restart of the database server and a micro-outage that I want to better manage.

apt-mark hold postgresql-9.5

Then the postgresql and postgresql-contrib (and sometimes client) packages appear as “held back” when you run apt-get upgrade.

You can force them to upgrade when you’re ready by explicitly installing: apt-get install postgresql-9.5

#ubuntu #apt-get

A guide to setting up a Ruby on macOS without requiring sudo.

Setup ruby on macOS without sudo and with the latest version!

It’s clear that using sudo to install gems is a bad idea. And you’re stuck with an older version of ruby and gem. But is it a lot of work to change your ways? I thought so. But it isn’t.

First I nuke my gems from orbit (it’s the only way to be sure). Some of the gems are default gems so we can't uninstall them and you might see error messages for them.

for i in $(gem list | awk '{print $1}') ; do gem uninstall -a -I "$i" ; done

After this, gem list returns nothing, or just the default gems.

Then you can install rbenv, and a new version of ruby in its own little environment just for you, so no sudo:

brew install rbenv ruby-build # Add rbenv to bash so that it loads every time you open a terminal echo 'if which rbenv > /dev/null; then eval "$(rbenv init -)"; fi' >> ~/.bash_profile source ~/.bash_profile # Install Ruby rbenv install 2.3.1 rbenv global 2.3.1 ruby -v

Then you’re gem installing with no sudo 4eva. Of course you need to reinstall all the gems you’ve lost. I found it rather refreshing as I didn’t really know whether I still needed any of the ones I had installed!

#ruby #gem

Charles 4

At long last, Charles 4 has been released! Charles 1 was released in 2002, nearly 14 years ago. Since then we’ve been through Charles 2 and Charles 3, both big upgrades. But for the past 11 months we’ve been hard at work on the biggest upgrade we’ve ever made to Charles!

Charles 4 includes an upgrade to the latest and greatest HTTP standard, the latest and greatest IP standard, and the biggest UI update ever for Charles. It is also the first paid upgrade in Charles's long history.

HTTP 2 is all around you. It’s sneaky and you might not have noticed. Both the client and server need to support HTTP 2, otherwise they will downgrade automatically to HTTP 1.1. So with Charles 3 you would just have seen HTTP 1.1 everywhere. No more! With Charles 4 you can now see HTTP 2 working, and you can use all of your familiar tools; Repeat, Breakpoints, and so on. You’ll spot HTTP 2 hosts in Charles as they use a different icon—with a lightning bolt!

IPv6 is also all around you! If you’re lucky enough to have an IPv6 network and ISP you can turn on “Prefer IPv6 addresses when connecting to dual-stack hosts” on the Options tab in the Proxy Settings. Then you’ll see IPv6 addresses being used in Charles. Feel the modern IP networking (first published in 1998) flow through you!

With Charles 4 we’ve also taken the opportunity to spruce up the look of Charles, starting with the familiar, enigmatic and much-loved app icon.

I’ll let that sit by itself for a moment.

Fear not! It's still the Charles jug, we're not crazy! It's just had a big polish, and we think it's more beautiful than ever! We hope you do too.

We've also designed a completely new custom set of icons, giving Charles a more unified and refined appearance. Thank you to the wonderful Wolfgang Bartelme for all of his painstaking work on these icons and on our beloved Charles Jug 4.0.

Finally we've improved the native platform stylings on Mac OS X (soon to be macOS) and Windows; we've reduced border chrome and tweaked most of the dialog layouts in the app so that Charles looks more modern than ever.

On Mac OS X we’re using the brand new VAqua library by the excellent Alan Snyder. VAqua provides better graphical accuracy and higher performance, especially on retina screens. The keenest eyes amongst you may also notice that Charles 4 now uses sub-pixel antialiasing! Zoom in to check it out. And on the latest versions of Mac OS X, Charles 4 uses the new San Francisco typeface. Finally, for my Charlesitans with dark menu bars—there’s a little something for you in Charles 4 too.

On Windows we’re using the latest version of JGoodies, with support for Windows 8 and Windows 10. No more will your menus be odd colours! We’ve also fixed the egregious aliased text that previously appeared whenever you looked at XML responses.

There are lots of other little changes and improvements in Charles 4. We’ll be documenting these and updating the website over the coming weeks. Reach out to us on Twitter @charlesproxy if you have any questions or comments!

Don’t forget to download Charles 4 and check it out!

#charles

Using the command-line gem tool, how can I install a specific version of a gem?

Say you need to run the pre 1.0 Cocoapods?

sudo gem install cocoapods -v 0.39.0

pod _0.39.0_ install

#gem #cocoapods #ruby

I don't use the RI or RDoc output from the gems I install in my machine or in the servers I handle (I use other means of documentation). Every gem I install installs RI and RDoc documentation by d...

gem update is the slowest thing my computer does. This makes it not so slow. Put it in your ~/.gemrc and never look back:

gem: --no-document

#gem #ruby

letterboxd

App.

This week we launched our Letterboxd iPhone app as a free download through Apple’s App Store. We’re ecstatic to put this app in your pocket (18,000 of you have grabbed it already!), and tremendously excited about how it will evolve the way we use Letterboxd. This was a monumental achievement for the team at HQ (particular hat tips to Grant, Ryan and Karl); the scope of the API + app development turned out to be comparable to the amount of work we did in 2011 while preparing our initial public beta.

Download from the App Store

The launch was covered by TechCrunch, IndieWire, The Next Web and many others, and the response from those who’ve installed it has been overwhelmingly positive. Today I’d like to share a little insight into the process and tell you about what we’ve built.

Process

After committing to build an iPhone app for Letterboxd, we first needed to add an API layer to our server infrastructure, to handle the transfer of data between external clients and the site. We’d piloted a small portion of the required work for Film Fest Buzz, and over the course of the past year we’ve built out the remainder of the API, which today comprises 40 individual endpoints.

The app’s interface represents an iOS-friendly take on the site’s aesthetic, and some parts of it may influence the future direction of the site. The majority of the app’s screens were designed in late 2014 — our prototype extended to 63 individual layouts, a large proportion of which have made it into the app we shipped this week.

API and iOS development was done concurrently for much of last year, and in November we invited Patron members to install and test an alpha version of the app. As always, we’re indebted to everyone who helped us to test the various builds over the past four months. (If you’ve been testing the app, please install the App Store version and we’ll get in touch when there are future betas to look at.)

What can it do?

Version 1.0 of the iPhone app supports:

Browsing of popular, highly rated and most anticipated films

Film info including poster, trailer, cast, crew, studio and genre data, plus popular lists and reviews

Film actions: rate, like, mark as watched, add to watchlist, add/remove from a list

Logging of films with date, rating, review and tags (plus flags for spoilers and rewatches, and Facebook sharing if your account is connected via the site)

Viewing (and filtering) of your activity feed

Viewing and commenting on lists and reviews

Viewing of member profiles

Following and blocking of members

Searching for films, reviews, lists and people

Profile editing (including favorite films)

Reporting of content and members

What can’t it do (yet)?

Some items from our wishlist didn’t make it into the first version but will be added in future updates. These include:

More browsing options, including decade/year and genre

Sorting and visibility filters

List creation/deletion, editing and re-ordering

Availability on streaming services

Detail views for more screens (such as watched films and liked content)

Completion percentages for lists and cast/crew

And there’s some OS-level integrations we plan to add as well:

Notifications for follows, likes, comments, etc.

A URL scheme so the app is able to open relevant web links natively

3D Touch support

Reactions

We asked on Twitter and Facebook for your reactions to the app news, and boy did you come through with flying colors. The three recipients of a year’s Pro subscription are Jennifer MacDougall for this delightful depiction of app love, Pedro Strazza for Jonah Hill’s delayed fist pump, and poor Niall Murphy who expressed his disappointment on behalf of the entire Android community in a single word. (“[We]’ve never seen such devotion in a droid before.”)

SSL

The API requires all communication with our servers to be made over an SSL connection. Prior to the launch of the first alpha version last year, we also enabled SSL as an option for site traffic (try it). We intend this to become the only way to access the website as well.

What’s next

As alluded to above, we weren’t able to make this a dual release with an accompanying Android version. We’ll be putting some focus on the public API and Android next (we’ll let you know as soon as we have something to announce), along with more site and app improvements. And if you have any app feedback, we’d love to hear it through the usual channels.

#ios

varnishlog queries

I always have to look up how to do queries on varnishlog. This is a good link for brief docs on the query language: https://www.varnish-cache.org/docs/trunk/reference/vsl-query.html

varnishlog -q 'ReqURL ~ "^/about/" and ReqHeader:User-Agent ~ "Safari"'

#varnish

Trending Blogs

Recently Viewed Blogs

Karl von Randow