Cotap Engineering

[cotapengineering]

iOS Tech Talks at Cotap

We’ll be hosting two cool tech talks at our office on September 9th. Come by and check them out!

https://www.eventbrite.com/e/tech-talks-at-cotap-tickets-18236055533

[cotapengineering]

Perfect forward secrecy using AWS ELB and Cloudformation.

This post describes how to enable Perfect forward secrecy and disable deprecated ciphers on an Amazon Elastic Load Balancer (ELB)

You can see how your site compares using Qualys SSL Labs tool.

One thing to note is that the higher your rate, the lower your compatibility will be with all clients/browsers.

Surprisingly little attention is paid to how SSL is configured. SSL is relatively easy to use, but it does have its traps. SSLLabs provides a tool to make sure that you are aware of possible security issues with your SSL setup. A grade of A generally ensures that you have taken steps to reduce the number of existing vulnerabilities to secure your encrypted network communication.

The first step is to follow the instructions on the AWS docs page about generating and uploading a SSL cert. To achieve a grade of A you need a signed certificate. However you do not need an EV cert.

When configuring your ELB, you can create your own policies and apply them to your port 443. Take a look at AWS Policy Table to see the latest one available.

In order to ensure Perfect Forward Secrecy on the ELB you must enable the Elliptic Curve Diffie–Hellman ciphers (ECDHE), as well as enable Server Order Preference. You can find more details on Jeff's blog post.

To get a grade A, you also must disable RC4 and SSLv3 since they are proven to be weak.

All of the above is done using the following CloudFormation template.

https://gist.github.com/maraca/136c7c5880bdca07da86

This is the very minimum list of ciphers that you will need to achieve a grade A. Feel free to add others that are just as secure and will provide more compatibility with your client's browsers.

As always, if you have any questions you can reach me on twitter or ping me directly on cotap.me/martin!

[cotapengineering]

Debugging Android Keystore Signing

// <![CDATA[ function styleCode() { if (typeof disableStyleCode != 'undefined') { return; } var a = false; $('code').each(function() { if (!$(this).hasClass('prettyprint')) { $(this).addClass('prettyprint'); a = true; } }); if (a) { prettyPrint(); } } $(function() {styleCode();}); // ]]>

Recently, the Android developers at Cotap began seeing this error when installing alpha and beta versions of the app:

INSTALL_FAILED_DUPLICATE_PERMISSION perm=com.cotap.xxx.permission.C2D_MESSAGE pkg=com.cotap.xxxx

At first I chalked it up to some bug in Android 5.0, since it started showing up only in L and above. However, one of our team members, Lillian Michalek, pointed out that it was only happening in builds I shipped. It turns out this was due to a change in the way Android 5.0 handles custom permissions. See an explanation from CommonsWear. At this point I knew there had to be more going on. Clearly, I should not be getting that exception because we were all using the same keystore to sign the apps built in our particular machines.

As a sanity check, I bumped the version number and built a production version of the app. Lo and behold, I could not install it on my phone. Meaning that, if I pushed that to the Play Store, customers would not be able to update their app. I asked Lillian to run the same experiment, building on her machine, and installing it on her phone. She had no issues.

Problem:

Although I had the same keystore on my local machine as Lillian, the app was being signed by a different keystore, with the same alias and password. This happened because, as part of reinstalling my development environment, I innocently ran the following keytool command before dropping our shared keystore into my machine:

$ keytool -genkey -v -keystore keystore_name.keystore -alias keystore_alias -keyalg RSA -keysize 2048 -validity 365

You can learn more about the Java Keytool here. The command above created a key that had the same alias and password as the one in our project. Worse, even though I was telling it to use a keystore in our project via the Gradle build, this was being overridden by the one I had created above.

signingConfigs {     release {          storeFile file("keystore_name.keystore")

         keyAlias "keystore_alias"          storePassword some_password          keyPassword some_other_password     } }

To prove I was using another keystore to sign my version of the apps I needed to dig deeper into what was being signed by what. These two answers to the same question listed the steps needed to open an apk and retrieve the .RSA file in order to see what keys were being used to sign the app. As you can see in the image below, Lillian had a different certificate.

Another giveaway in hindsight: since we are using a shared certificate in our project, the keytool command should not be aware of it. So if I ran the following command, I should not see anything: 

$ keytool -list -keystore keystore_name.keystore

Instead there was a certificate with that name as a result of my previous “keytool -genkey” command.

Another way to see which flavor of the apps were signed with what certificate, you can use the following Gradle command, also from Stack Overflow: 

$ ./gradlew signingReport

This will give you an output showing you what keystore is being used for which app:

Solution:

To right this wrong I deleted the keystore that the keytool had created, and made sure to drop the right shared keystore in my local project. To delete a keystore you can use the following command:

$ keytool -delete -alias keystore_alias -keystore keystore_name.keystore

 I got that in Stack Ov… jk! This site lists common Keytool commands. After that, all was groovy.

Bonus!

I know no one checks their production keystore into git. Right?!?! Here is how we save our keystore password in the Mac Keychain.

Put your keystore in your favorite directory. Here is our signingConfigs in the build.gradle file:

signingConfigs {     release {         storeFile file("keystore_name.keystore")

        keyAlias "keystore_alias"         storePassword getPassword('ANDROID_KEYSTORE_PASSWORD')         keyPassword getPassword('ANDROID_KEY_PASSWORD')     } } 

Now use this method to pull the password from the Keychain. Notice it will prompt you in the command line if it can’t find it:

def getPassword(sysvar) {     String pw = System.getenv(sysvar)     if (pw == null) {         def command = """security find-generic-password -w -a whatever_you_want"""         def proc = command.execute()         proc.waitFor()

        if (proc.exitValue() == 0) {             pw = proc.in.text.trim()         } else if (System.console() != null) {             pw = new String(System.console().readPassword("\n\$ Enter keystore password: "))         }     }     return pw } 

Lastly, add your password to the Keychain. Below are the 4 steps: Now your password is automatically entered when building! Enjoy!

Now your password is automatically entered when building! Enjoy!

[cotapengineering]

Infrastructure as code, automation, monitoring, disaster recovery, security, scaling and cost tracking are all subjects that are easily accessible but too often overlooked until it is already too late.In this session Cotap will share what AWS offers to help them stay ahead of the curve.

By following 4 simple rules they will show how Cotap's Engineering team has been able to run for the past 12 months with over four nines of availability. We deploy 3 to 5 times a day, run in 2 regions/6 AZs and still manage to keep AWS costs below the monthly salary of an Engineer.

[cotapengineering]

Optimizing Android List Scroll Performance

After implementing our latest feature set for Cotap on Android, we noticed that scroll performance for long contact lists had taken a hit. How much of a hit are we talking?

Yikes.

The problem

The first step to fixing this was to identify what was going wrong. What were the most expensive operations during draw and were they necessary? By timing and logging each draw operation, we found them to average at 1.99 ms for each row in the contact list, but a whopping 94.94 ms for each section header in the alphabetized list. Some section draw operations took close to 200 ms.

In addition to the choppiness due to an expensive draw, we noticed there was a significant number of garbage collection notifications being posted to the log during scroll. For a scroll up and down the list, the system needed to stop to garbage collect 34 times, resulting in skipped frames each time.

It turned out that each section needed to execute some expensive logic in order to draw itself. Since we introduced a "hide contact" feature, each section needed to understand its own contents before drawing. If for example, the "J" section only had John Smith, but then he was hidden by the user, then the section header shouldn't appear. So, each section header should know whether it has any unhidden members before deciding what to draw.

We were carrying out this logic in a painfully slow way. First, we would query the local database for a list of all hidden contact IDs. Then, for each contact ID in that list, we would query again to get its section. This was flawed: as the user accumulated hidden contacts, each draw operation required more and more queries.

Round 1: A better query

Our first improvement involved condensing down all that logic into one query. Instead of requiring a query for each hidden contact, we pulled out a list of all hidden contacts in a single query and then looked at the section for each one. After this improvement, the average draw time for a section view went down to 34.6 ms and the garbage collector kicked in 19 times during one scroll through the list.

Better! But not good enough.

Round 2: A much better query

At this point, we probably should have given up on the idea of optimizing the query further, but we didn't because we thought of The Most Optimal Query Of All Time and had to give it a try. This time, we asked the database to look for unhidden contacts in a given section. And stop after the first one. By doing this, the section got the information it needed - does there exist an unhidden contact in me? - and we didn't need to loop through any lists. Plus, we added indexing by section.

Stats after this improvement: section draw took an average of 17.19 ms, and garbage collection kicked in one time.

Definitely better. But still not good enough.

Round 3: Okay, okay - forget the queries!

We finally stopped avoiding the heavy lifting involved with keeping track of things in memory. This involved keeping a set of visible headers around for constant-time lookup by the draw function. Any changes in contact status and sort order needed to update the set. This was not cheap, but it happened infrequently.

After removing queries and implementing this caching of visible sections, we finally started to see the kind of results we were looking for. Section drawing went down to an average of 1.85 ms, and we stopped to garbage collect one time, as before.

Round 4: Google says, "use a ViewHolder"

Finally, we thought we would polish things off by implementing the recommended ViewHolder pattern for drawing contact rows and sections. This pattern limits costly resource lookups by finding each child view one time, then storing it with the parent view for quick access when views are recycled.

With this final change, we averaged 1.42 ms to draw a contact row, 0.92 ms to draw a section, and one pass of the garbage collector.

Let's see it! What does our "Round 4" version look like compared to what we started with?

The data

For the curious, here's a more complete chart of the data cited for each change above. Disclaimer: this data's not super scientific. These numbers were collected by scrolling up and down the list one time. Things may have been going on in the background that slowed things down slightly, the garbage collector may have been just about ready to kick in for some test runs but on others, etc. Nevertheless, this data has reasonably good correlation with perceived performance of scrolling at each step and is at least an interesting metric.

Some Takeaways

You can forget about doing any database queries on the main thread while rendering a list if you care at all about performance.

We tried out some tools while debugging this, Systrace is worth mentioning - if you run it on your app, you can view an interactive chart of execution times of processes on your device.

This situation is a chronic optimizer's nightmare. We made things a lot better, but we still see room for improvement. Where do you draw the line?!

[cotapengineering]

Using the HTML5 ContentEditable attribute for an admin friendly UI

When we built the new Cotap.com, we switched to Wordpress as our CMS in order to give our business team the ability to manage content rather than taking up developer resources for simple copy changes.

While creating the necessary inputs required to manage the headers, paragraphs and bullets, we quickly realized the admin pages were turning into a long pages filled with inputs.

Aside from making an overwhelming form for our marketing team, that method also separated copy and design elements. When a page is reduced to a series of inputs it becomes difficult to understand what impact, if any, copy changes will have until they are saved and viewed on the front end.

We knew the best solution would be one where editors could simply click on an element, make the copy change, and immediately see the impact it would have on the page.

HTML5’s ContentEditable attribute provided the user-side functionality we wanted. We created a simple javascript snippet that would take any text changes in a ContentEditable element and pass it to a paired hidden input.

https://gist.github.com/d395ecf19842eeff0813

With the Javascript in place, we just needed the editable element and it’s companion input 

Element:

The ContentEditable attribute tells the browser how to engage the content.

The editable class is used by the javascript to target the element.

The data-editable-input-id points to the id of the input where the value is saved.

Input:

It needs an id to target

The type attribute should be set to hidden so users don’t see it.

Placeholder isn’t necessary unless you’re showing the input.

https://gist.github.com/33da60e9518e71898bf5

The end product is a robust yet simple interface that our marketing and business teams love. It also has freed up extra development time and removed the delay in getting copy changes deployed.

[cotapengineering]

Using Chef's search functionality with test-kitchen

At Cotap, we have been integrating test-kitchen as part of our workflow to continuously test our infrastructure.

When combined with chef-zero, one very useful but hidden feature of test-kitchen is the ability to search for nodes.

To do so, you simply have to drop files in your 'test/integration/nodes' folder. Each JSON file will then be a representation of a node in your environment.

How does it work?

The first thing to do is to make sure you are using the chef-zero provisioner in your .kitchen.yml.

https://gist.github.com/maraca/34fdff3a9c16e02eefac

Then add a statsd host to your environment by writing to  'test/integration/nodes/statsd.json'.

https://gist.github.com/maraca/ae8d41db5c98dd532324

You can now search for the statsd instance inside of a recipe:

https://gist.github.com/maraca/a9fea217e975ab681cbf

And that's it! Feel free to ask me questions on Twitter if you have any!

[cotapengineering]

Hello World! using Packer, Chef and Berkshelf on EC2.

We use Chef and Berkshelf to deploy and configure software on EC2.

In an attempt to decrease the time required to bootstrap an instance, we’ve decided to play with Packer a bit. The idea behind Packer is to build images that will work for different environments, being Vagrant, AWS, OpenStack etc.

Here is a Hello World with Packer, Chef and Berkshelf, to build an instance-store EC2 AMI. You need to have Packer installed to get started. Here is an excellent tutorial for it.

Generate x509 certificates and update your ~/.bashrc

The first step is to generate x509 certificates. Here is a script to simplify things:

https://gist.github.com/maraca/9400086

Once you have done so, upload the x509 certificate to your IAM user in the EC2 console.

Next, update your bashrc with the right information. You should have these five environment variables, with the correct information, in your `~.bashrc`

https://gist.github.com/maraca/9398024

Write a hello-world chef recipe

The code is fairly simple. It installs Nginx and drops an index.html file where nginx’s default site expects one to be.

https://gist.github.com/maraca/9397730

Note that `recipes_default.rb` should be located in `recipes/default.rb`, but Gist doesn’t allow ‘/’ in the filename.

The next step is to tell Packer to build an AMI with your chef recipe pre-installed so that you can launch instances that will say ‘Hello world!’ as soon as you bring them up.

To do this, you need to tell Berkshelf to put all the cookbooks in one location. Using the --path option, you can run `bundle exec berks install --path vendor/cookbooks`.

This command will download all the dependencies associated with your hello-world cookbook and place them in a vendor/cookbooks folder.

Next, create a packer.json configuration file. Again, the packer.io website is a very useful resource with complete documentation. Here is what we came up with:

https://gist.github.com/maraca/9397905

In order to bundle an instance-store EC2 instance, you need to have the ec2-ami-tools installed, so we placed that in a “shell” provider, while our “hello-world” recipe is located in a “chef-solo” provider.

When the chef-solo provider has run and completed, Packer will proceed to install the ec2-ami-tools. Once it’s done, it will pack the AMI, upload it to an s3 bucket of your choice, and register the AMI.

You can see that sensitive information is passed using variables, which is why it is important to set up your environment variables early on.

Time to build your first Packer AMI!

https://gist.github.com/maraca/9398172

Once Packer finishes building your new AMI, it will output its id at the bottom of the logs:

Using CloudFormation, you can quickly launch an instance with your new AMI:

https://gist.github.com/maraca/9398453

After launching an instance, you can access it directly and see your Hello World message.

Tada!

This is a very light introduction to Packer, but we hope it gets you started quickly!

[cotapengineering]

Animating the Keyboard with the iOS7 Interactive Pop Gesture

Each new version of iOS comes with exciting micro-interactions that are brilliant and intuitive, but not always well documented or understood on release. One such interaction is what Apple calls the "Interactive Pop Gesture" – the animated dismissal that occurs when swiping from the left edge of the screen across to the right.

By default, an instance of UINavigationController with at least two UIViewControllers on the stack will have this gesture enabled. With no extra code required, the navigation controller will animate the view dismissal interactively as the user drags.

Screenshot: From the demo application, the default behavior when swiping to go back

While exploring how the back gesture works in Cotap, we noticed a small problem: the keyboard is not a part of the ViewController hierarchy, and as such does not animate along with the current view. Initially, we were confused – Apple’s own iMessage app clearly animates the keyboard when performing an interactive dismissal, so why shouldn't our app get that for free as well? However, further exploration led us to discover that even Apple doesn't always have this interaction. In the Notes app, swiping back from editing a note to the list page immediately dismissed the keyboard, making for a rather jarring transition.

At Cotap, we take great pride in providing excellent user interactions in our apps, and we knew right away that we had to find a way to have a smooth interactive pop animation just like the Messages app. After several iterations, we've come up with a solution that we believe works in the best spirit of the new animation APIs in iOS7, and works with a minimal amount of custom animation.

Today, we're proud to present a new CocoaPod: TAPKeyboardPop. This new CocoaPod provides a light category for UIViewController that uses the new UIViewControllerTransitionCoordinator protocol to hook in to the default animation that Apple provides for the Interactive Pop Gesture.

Screenshot: The animated keyboard dismissal in action

Currently, we've only provided support for the most basic use case (animating the keyboard off the screen to the right, just like the normal dismiss animation) but we're accepting pull requests to enhance the pod.

It's been a surprisingly long and interesting road to this point, and we'd like to provide some insight into what we've learned along the way. When we initially attacked this problem, we went back and forth briefly about how to cleanly hook into what was happening during the transition between view controllers. We had a few ideas, but ultimately settled on listening to the gesture recognizer for the first pass. UINavigationController provides a method interactivePopGestureRecognizer that will provide a default gesture recognizer that can be listened for and used to animate the keyboard along with the user's dragging. This has the drawback of not being a part of the same animation stack as the real transition, and doesn't allow you to handle cancellation smoothly. While we were able to find an approximation of the animation curve that the default animation uses, it was never quite as smooth as the real one.

After the initial pass was done, we discovered that the functionality in the UIViewControllerTransitionCoordinator protocol should allow us to actually hook directly into the animation that Apple provides including their custom animation curves. This significantly simplified the code required to provide this animation, and allowed us to experiment with some of the new interactive transition APIs in iOS 7. Overall, we're very pleased with solution, but are eager to hear how it works for others in the community.

[cotapengineering]

Instance Deregistration with Chef and Sensu

Cotap's back end is hosted on EC2, and most of our hosts live in Auto Scaling Groups. This means that we have hosts spinning up or shutting down dynamically, based on the load we're getting.

We recently began using Sensu to monitor our hosts' uptime. Sensu alerts the on-call engineer whenever a host goes down, and that meant we needed a way to tell it not to send an alert when the host was being shut down by its Auto Scaling Group.

Fortunately, our hosts are all Ubuntu 12.04 AMIs and we automate all our configuration with Chef. The combination of these two gives us a clean, simple way of notifying Sensu during a shutdown.

When an Auto Scaling Group shuts down a host, it does it via a standard Unix shutdown, switching the host to runlevel 0. When switching to a runlevel N, the system executes all scripts in /etc/rcN.d. So we wrote a script to live in /etc/rc0.d which will deregister the host from Sensu monitoring.

To get the script deployed to each host, we set up a phantom Chef service with a shutdown script living in /etc/init.d. When Chef provisions a host, it copies the script over to /etc/rc0.d, which means it gets run every time the system switches to runlevel 0. This script sends Sensu an HTTP DELETE request to remove itself as a node from Sensu's list.

Thanks to our friends at Kwarter for their input on this project!

https://gist.github.com/maraca/646cbdeebdfc7b681cfa