zombees

[aesthma-blog]

Accelerated SSL

There are any number of ways to improve the performance of SSL, ranging in difficulty from a simple config change to installing dedicated hardware or desperately trying to get the protocol modified.

This article focuses on accelerating the performance of SSL by improving OpenSSL's hash, block and key algorithm performance. By increasing the data which you can transmit and the number of SSL handshakes that can be performed, you can reduce the overhead of SSL on your current server hardware.

Beyond configuration and implementation improvements, I will also present two additional solutions: adding an SSL accelerator card and modifying OpenSSL to utilize Intel's IPP library.

OpenSSL version

OpenSSL 1.0.0d (or newer) provides a few important performance improvements over the older, and more commonly deployed 0.9.8 series:

SSL_MODE_RELEASE_BUFFERS -- if your application supports it (or you patch it to do so), memory utilization per SSL connection is drastically reduced

RSA performance -- significantly increased, particularly on the 32-bit x86 arch

As a result, you should strongly consider upgrading if you haven't already done so. RHEL 6 already provides OpenSSL 1.0.0, older versions have 0.9.8. However, you will need to take some further steps outlined below to correct performance degradations found on Nehalem and newer processors with RC4 and AES.

Architecture

Many of the changes presented below are dependent upon the system architecture you use. Only 32-bit x86 and 64-bit x86_64 have been compared, other architectures (SPARC, MIPS, etc.) will have very different performance characteristics.

If possible, you should ensure that your OS, application and OpenSSL are all 64-bit. RSA handshake performance is 2x better on x86_64, and for any high traffic site you will need the additional memory footprint that a 64-bit process provides.

Because of the specialized AES-NI instructions added in Intel Xeon Westmere (56xx series) and newer processors, it is highly desirable to utilize these processors in your hosts serving SSL traffic.

Ciphersuite selection

Based on a previous article, two ciphersuites have been selected:

TLS_RSA_WITH_AES_128_CBC_SHA: RSA + AES-128-CBC + SHA1

TLS_RSA_WITH_RC4_128_SHA: RSA + RC4 + SHA1

You may also choose to permit TLS_RSA_WITH_AES_256_CBC_SHA and TLS_RSA_WITH_RC4_128_MD5. All of the other current TLS ciphersuites are inferior from a performance or security standpoint, or are not sufficiently widely supported to consider now.

These ciphersuites contain three algorithms, each of which needs to be accelerated to improve overall performance:

Hash function (MAC)

Block cipher (bulk encryption)

Key exchange

Hash function

The two ciphersuites we chose both use the same hash function: SHA-1. MD5 and SHA-256 are also available in other ciphersuites; MD5 only for RC4, and SHA-256 is not widely supported yet.

SHA-1

IPP provides an 18% performance improvement, approximately matching the SSL accelerator card with a single CPU core. A single core can hash 3.6 Gbps of data.

Block cipher

Two block ciphers were selected: AES and RC4. This should provide sufficient coverage for all legacy and modern clients to successfully negotiate a ciphersuite with the server.

AES

AES is the preferred block cipher from a security perspective. It is also preferable from a performance standpoint if you are using a Westmere or newer CPU.

Moving from OpenSSL 0.9.8o to 1.0.0d results in a 48% decrease in performance on Nehalem and newer processors due to an undiagnosed issue with the x86_64 ASM in OpenSSL that means it is no longer properly optimized for this architecture. The easy fix is to simply prevent the ASM from being built and used when compiling OpenSSL.

IPP is a better solution though, as it enables OpenSSL to utilize the AES-NI instruction set on Westmere processors and yields a 280% increase in performance (above the fixed 1.0.0, for a total of 647%!). Performance is equivalent to the AES-NI OpenSSL engine, with the added benefit of lesser performance gains on with older CPUs.

AES-256 has similar performance characteristics, although it is 25% slower.

Using AES-128 and IPP, a single core can encrypt 5.6 Gbps of data.

RC4

While not as secure as AES, RC4 is important for two reasons:

Older clients may not support AES, notably selected browsers on Windows XP

It offers higher performance than AES on pre-Westmere CPUs

You can easily obtain a 32% performance increase by applying this patch, if you are using a Nehalem or newer CPU in 64-bit mode.

If you don't want to apply this patch, another option is to prevent OpenSSL from building and using the ASM version of RC4 in Configure.

It would take 5 CPU cores to equal the performance of the SSL accelerator card, which makes this encryption function the only clear winner for the accelerator card. It is also the only algorithm tested here which I did not implement replacement IPP functions for the OpenSSL native logic. IPP does include Arcfour functions to accelerate this cipher.

A single core can encrypt 3.2 Gbps of data.

Key exchange

While there are a variety of available key exchange algorithms, only RSA is well supported at this time.

RSA

RSA keys come in various sizes, and the size selection is essential to determining the number of handshakes that can be performed per second. A 1024-bit RSA key is considered less secure, but provides the best performance currently available (sizes below 1024-bit should not be considered). Operations on a 2048-bit RSA key are approximately 5 times slower. Key sizes above 2048-bits are not necessary unless you have specialized needs; they will substantially impair performance.

The easiest way to improve RSA performance is to upgrade to OpenSSL 1.0.0, which increases performance by 25%.

The SSL accelerator card yields a 195% increase in performance over a single unaccelerated CPU core; IPP yields a 116% increase over an unaccelerated core. On two or more cores, the IPP version eclipses the SSL accelerator card.

Using RSA-1024 and IPP, a single core can process 4,238 handshakes per second.

Testing methodology

All of the benchmarks in this article were obtained by running openssl speed -elapsed on a quiet host with the scaling_governor set to performance. Unless noted, the results are for a single core's performance (specifically an Intel Xeon X5650, which is a Westmere CPU) using a 64-bit OpenSSL binary. Performance on multiple cores will scale linearly, provided that your application can utilize multiple cores efficiently.

Performance characteristics will be different for other CPUs (such as pre-Nehalem Xeons), as well as 32-bit OpenSSL.

SSL card results were obtained by running openssl speed -elapsed -multi 12. The card used is a Cavium Nitrox, model CN1620-400-NHB.

AES: openssl speed -elapsed -evp aes-128-cbc

RC4: openssl speed -elapsed -evp rc4

SHA-1: openssl speed -elapsed -evp sha1

RSA: openssl speed -elapsed rsa1024

Acceleration options

The SSL accelerator card used for this benchmark retails for approximately $500. There are a few downsides to using a hardware device:

Must maintain a spare pool for quick replacement

Not general purpose

Constrained supply, may be difficult to procure in some regions

Requires some work to integrate with OpenSSL

May not have a free PCIe x4 slot in some servers

Intel's IPP library is not free software, and requires a developer license at $200 per user. However, you may freely redistribute the product utilizing IPP (e.g. OpenSSL) without royalties, and there is no per-server cost. There are some downsides:

Not free or open source

Requires some work to integrate with OpenSSL

Utilizes the system CPU

Consider that upgrading the system CPU may be cheaper (and more beneficial for other purposes) than a dedicated accelerator card. A hex-core Intel Xeon X5650 is only $300 more than a quad-core Intel Xeon E5640, and the two additional cores will easily eclipse the performance of the SSL accelerator card in every algorithm except RC4.

IPP patch

The IPP cryptography samples include a patch against OpenSSL 0.9.8j, which I have modified to work with OpenSSL 1.0.0d. My changes are simply to make the patch apply cleanly, the code is otherwise identical to Intel's original patch. That patch is available here.

When building this patched version of OpenSSL, you'll need to keep a few things in mind:

It expects IPP to be installed in /opt/intel/composerxe-2011.3.174/ -- if your IPP is somewhere else, simply modify the path listed in Configure to point to the right place.

You must use the build targets linux-elf-ipp (32-bit) and linux-x86_64-ipp (64-bit), instead of linux-elf and linux-x86_64.

You should still apply the RC4 patch as well, unless you're using a version of OpenSSL that already includes it (none do as of this writing) -- while IPP does have functions to accelerate RC4, they are not utilized in Intel's patch.

Summary

Obtain IPP and the cryptography library

Upgrade to OpenSSL 1.0.0d, patched to utilize IPP

Modify (or upgrade to a patched version) of your application which sets SSL_MODE_RELEASE_BUFFERS

Only offer AES-128 and RC4 ciphersuites

Deploy on Intel Xeon Westmere and newer processors

[aesthma-blog]

Smart Inline Load Balancing

There are two common methods of placing servers behind a load balancer appliance (SLB), Inline and Direct Server Return (DSR).

DSR sends responses from your server directly back to the client, bypassing the load balancer entirely. This can be substantially better for web traffic which most commonly answers small requests with larger responses.

Inline allows you to do various things at the load balancer which may be important to you, but at the cost of routing all of your non-local traffic through the SLB. In this article, I present a way to easily implement smart inline load balancing which removes this requirement.

While DSR generally performs better and allows you to use the same load balancer pair for a lot more VIPs, there are some key reasons why you might want to choose inline load balancing:

SSL termination

Port translation

Syn flood protection 1

Impressing the load balancer into double duty as a simple firewall

DSR cannot support any of the above features because it doesn't see both sides of the data flow.

Standard inline load balancing

The standard configuration consists of:

Add a virtual IP to a pair of load balancers

Hot / cold redundancy

Only one load balancer receives traffic at any given time

Configure hosts behind the load balancer to use the high availability IP of the SLB pair as their default gateway

All of your production traffic (e.g. responses to HTTP requests) are sent through the load balancer and from there back to the requestor. However, that isn't all -- all of your host's traffic going outside the local subnet, including ssh, database queries, backup jobs are also sent through the load balancer.

This can negatively impact performance since the work the load balancer has to do is unnecessarily increased. More problematic for some environments is that you cannot have a single host behind multiple VIPs in this configuration -- there is only one default gateway on the host.

Multiple VIPs (hot / hot)

Why would you want to have hosts behind multiple VIPs? One good reason is if you want to run your load balancer pair hot / hot instead of the standard hot / cold method.

During a failure, only 50% of your traffic is impacted

You're better able to handle abnormal traffic spikes (e.g. abuse)

Your response times may improve, even if you aren't exceeding the load balancer's capacity

With both load balancers in production all the time you are less likely to have different configs that go unnoticed (until a failure)

Keep in mind that to maintain redundancy, you should not exceed the capacity of a single load balancer during your expected peak traffic. However, this can still be a good safety net if you find you are over capacity temporarily.

Load balancer configuration is beyond the scope of this article but the basic idea is to configure two VIPs:

VIP1: load balancer A active, load balancer B standby, port 80 -> 80 VIP2: load balancer A standy, load balancer B active, port 80 -> 81

and then respond to DNS lookups with both IP addresses. You will need to enable port translation on one of the VIPs because there must be a 1:1 mapping of server port to VIP for this to work.

Smart inline configuration

The alternative (smart inline load balancing) uses iptables to selectively alter the gateway on outbound TCP packets. This means we can avoid sending unnecessary traffic through the load balancer and have the same host behind an arbitrary number of VIPs.

Example config

This example works on RHEL 6, but is adaptable to any Linux system using iptables. You will need to adapt the iptables config to your particular environment, and then restart networking on the host (/etc/init.d/network restart).

Gateway: 10.0.0.1 VIP1: 10.0.0.2 VIP2: 10.0.0.3 Host: 10.0.0.50 Ports: 80, 443

$ ip route show

[...] default via 10.0.0.1 dev eth0

$ cat /etc/sysconfig/iptables

*mangle -A OUTPUT -s 10.0.0.5 -d ! 10.0.0.2 -p tcp -m multiport --sports 80,443 -j MARK --set-mark 1 -A OUTPUT -s 10.0.0.5 -d ! 10.0.0.3 -p tcp -m multiport --sports 81,444 -j MARK --set-mark 2

$ cat /etc/sysconfig/network-scripts/rule-eth0

pref 1 fwmark 1 lookup 1 pref 2 fwmark 2 lookup 2

$ cat /etc/sysconfig/network-scripts/route-eth0

default via 10.0.0.2 table 1 default via 10.0.0.3 table 2

RHEL 5 and older

There is an additional problem where RHEL4 cannot handle rule files at all and RHEL5 cannot handle IPv6 rule files. This script can be deployed to handle both cases. Place as follows:

/sbin/ifup-local /sbin/ifdown-local

What it does

For outgoing traffic on the host that originates on ports 80, 81, 443, 444 it will alter the gateway accordingly. Any other traffic gets passed through unaltered. In this example, you would have your web server listening on all four ports on the local machine and both VIPs listening on just 80 and 443.

If your hosts only have one IP, you could remove the -s 10.0.0.5 portion entirely or add -s ! 127.0.0.1 to prevent matching localhost traffic.

Similar config lines are accepted for IPv6 in /etc/sysconfig/ip6tables, /etc/sysconfig/rule6-eth0, /etc/sysconfig/route6-eth0. You need at least RHEL 5.6 for proper IPv6 functionality.

Caveat

There is one major caveat with this method that you should be mindful of. If the configuration is not in place on a host, but the load balancer's healthcheck object is served successfully (e.g. http://real_server/status.html) then the server will be in rotation but unable to serve production traffic.

To ensure this doesn't happen, always put the configuration in your system's config files so that restarting the host or network stack will reload the config. You may also want to implement a monitoring check to verify that the iptables config is present, which can be viewed with:

$ sudo iptables -L OUTPUT -t mangle -n $ ip route list table all $ ip rule list

Hardware load balancers are purported to have better syn flood protection because they include custom hardware to generate syncookies faster during an attack. ↩︎

[aesthma-blog]

OpenSSL: Cipher Selection

OpenSSL supports a number of different SSL ciphersuites, which can have a huge impact on the overhead that SSL imposes on HTTP traffic. Below I approach SSL from the performance side of things. Sites deploy HTTPS for one of two reasons: 1. Financial or highly sensitive data is involved, there are regulations mandating security policies, and it would be extremely advantageous to someone if they obtained the data you're transfering (even if it takes a year to crack the encryption). 2. To avoid [Firesheep](http://codebutler.com/firesheep) style attacks or for integration with SSL secured services (e.g. Twitter). Nefarious persons might like to acquire the data, but they are probably not going to spend millions of dollars to crack your encryption. The second use case is far more common. For this type of deployment performance is key, and security is a bit of a secondary concern (which isn't to say you should be insecure). If you fall into the first category, I presume you are already well aware of the procedures required of you. ### Ciphers The following ciphers are supported in TLSv1: * [3DES](http://en.wikipedia.org/wiki/Triple-DES) has historically been the preferred cipher for high security HTTPS transactions. It has been slow since the dawn of time, although it has ubiquitous browser support. * [RC4](http://en.wikipedia.org/wiki/RC4) is considered less secure than 3DES, but is similarly very well supported. * [AES](http://en.wikipedia.org/wiki/Advanced_Encryption_Standard) is a newer cipher, formerly Rijndael, which won a contest to become the standardized replacement for DES. * [Camellia](http://en.wikipedia.org/wiki/Camellia_(cipher)) * [SEED](http://en.wikipedia.org/wiki/SEED) Other ciphers exist, but I don't feel that they're suitable for various reasons. #### SSLv2 It hopefully goes without saying, but under no circumstances should you allow SSLv2 connections in 2011 -- SSLv3 was standardized in [1996](http://www.mozilla.org/projects/security/pki/nss/ssl/draft302.txt)! SSLv3 and TLSv1 are both acceptable. ## Cipher performance

This test was run with a single 64-bit instance of **openssl speed** on an Intel Xeon E5620. Assembly instructions were disabled for AES and RC4 ciphers per [OpenSSL: Outmoded Assembly](http://zombe.es/post/4059999783/openssl-outmoded-asm). #### Results 100,000 Kbyte/s is my threshold for acceptable performance. This represents 1 CPU core (of 8 in my case) running at 100% utilization to transfer 780Mbit/s of data (which is a reasonable saturation point for a gigabit Ethernet link). This eliminates Camellia-256, SEED, and 3DES. RC4 is the fastest cipher, if you are using a processor which does not support [AESNI](http://en.wikipedia.org/wiki/AES_instruction_set). AES-128 is the next fastest cipher, and much faster than RC4 if you have AESNI support. It's about 54% slower if you don't. AES-256 is slower still, and unless explicitly configured otherwise, any browser that supports AES-128 will also support AES-256. Camellia-128 is a tad slower still. It's quite unlikely that any browser would **only** support Camellia so it is currently not essential. **NOTE:** openssl-0.9.x has different cipher performance: Camellia-128 is 4x slower, and RC4 is slower in very old 0.9.x releases. For this and other reasons, you should be using openssl-1.0.x. ### Cipher negotiation By default, OpenSSL negotiates the cipher to use like this: 1. The user's browser sends its cipherlist. 2. This is compared with the server's cipherlist[^1]. 3. The first cipher in the user's list that the server supports is utilized for the connection. The downside here is that it places the end user in control. From a performance standpoint, we want to be calling the shots here. This behavior can be changed with flag **SSL_OP_CIPHER_SERVER_PREFERENCE** to [SSL_CTX_set_options](http://www.openssl.org/docs/ssl/SSL_CTX_set_options.html#NOTES) in your application. Apache's mod_ssl has support for it as option _SSLHonorCipherOrder_. This changes the behavior to prefer the first shared cipher using the order specified in the server's cipherlist. For example:

client: 3des, aes-256-cbc, aes-128-cbc, rc4 server: aes-128-cbc, aes-256-cbc, rc4, 3des

Without this flag, **3des** is selected. With it set, **aes-128-cbc** is selected. That client connection might use 97.5% less CPU to encrypt the data stream as a result. #### Cipher ordering For servers with AESNI (Intel Westmere, AMD Bulldozer, and some others), I recommend: >AES-128, RC4, AES-256, Camellia-128 For servers without AESNI (or without an OpenSSL patched to support AESNI): >RC4, AES-128, AES-256, Camellia-128 In order to do this, you'll need to set your cipher string in your application's configuration. You can see what ciphers will be matched by a string with **openssl ciphers -v 'STRING'**, e.g.: >openssl ciphers -v 'AES128-SHA:RC4:AES:CAMELLIA128-SHA:!ADH:!aNULL:!DH:!EDH:!eNULL:!LOW:!SSLv2:!EXP:!NULL' will select AES-128, RC4, AES-256, Camellia-128 in that order.

AES128-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(128) Mac=SHA1 RC4-SHA SSLv3 Kx=RSA Au=RSA Enc=RC4(128) Mac=SHA1 RC4-MD5 SSLv3 Kx=RSA Au=RSA Enc=RC4(128) Mac=MD5 AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 CAMELLIA128-SHA SSLv3 Kx=RSA Au=RSA Enc=Camellia(128) Mac=SHA1

This can be configured in Apache with _SSLCipherSuite_. ### How widely is AES supported? All modern browsers[^2] support AES, however there is one big caveat: any browser which uses the system SSL services on Windows XP or older does not support AES. This includes: * Chrome * Internet Explorer * Safari but not Firefox or Opera. Windows Vista and newer do support AES at the OS level. This is a big motivator to continue permitting RC4, because the only other cipher in common use that these browsers are likely to support is 3DES. ### Why disable 3DES? You may be wondering: why not just permit 3DES as a fallback cipher at the end of my cipher string? Consider the following potential DOS Attack: 1. A large botnet of (unknowingly) controlled PCs is configured to permit **only** 3DES 2. These users continue using your service as they normally would, any nothing appears out of the ordinary to them (and there is no compromise in over-the-wire security) 3. Your servers are now processing this traffic 95% slower than normal How would you handle a sudden need for 2x the processing power to serve the same amount of traffic? While this scenario is unlikely to occur, it does highlight the extreme potential for denial or degradation of service simply as a result of the processing overhead of 3DES vs. any other cipher. With that said, if you know that you have clients that cannot support any cipher but 3DES or if RC4 is not secure enough for your needs, then you should add it to your cipher list. ### SSL session resume If you have a VIP with Nehalem and Westmere servers behind it where any of the hosts might answer an HTTPS request to the same public IP, you must have a consistent SSL cipher list across all of them. This is somewhat mitigated by the standard practice of assigning SSL connections in a sticky fashion on load balancers, but ideally you would have eliminated stickiness by deploying a shared SSL session cache between all of the hosts. Clients may open a new HTTPS connection (SSL session resume) which re-uses the previously negotiated session options, including the cipher. If all of the servers can resume the session (because they hold the original session id and data in cache), then you run the risk of a client who negotiated on a Westmere host and thus agreed to AES-128 being sent to a Nehalem host for its next connection where it should have agreed to RC4. Either prefer RC4 across all hosts behind this VIP, or divide your hosts between multiple public facing IPs. ## Conclusion 1. Use equipment that supports AESNI (or prefer RC4) 2. Enforce server side cipher preference 3. Pare down your cipher list, eliminate extremely slow ciphers like 3DES and insecure ciphers like export versions 4. Use openssl-1.0.x, in 64-bit mode [^1]: openssl-1.0.0d default: AES-256, Camellia-256, 3DES, AES-128, SEED, Camellia-128, RC2, RC4, DES, various export ciphers [^2]: Check with https://www.fortify.net/cgi/ssl_2.pl

[aesthma-blog]

OpenSSL: Outmoded Assembly

When I see software that provides assembly instructions my architecture, my gut reaction is to enable them. They must be faster! Why else would someone have taken the time to descend to the lowest levels of coding hell? It's not a magic bullet though. Assembly is still written by a programmer, and it still suffers from bit rot just like any other code. Its platform specificity can magnify the effect; you may be running the code on a platform that has a compatible instruction set but which needs totally different instructions to properly optimize the same operation. ### The problem While running OpenSSL benchmarks, I observed some curious behavior. OpenSSL on newer Xeon processors is as much as **50% slower** with asm than without (for certain ciphers). For reasons addressed elsewhere, I'm mainly interested in just three algorithms: * aes-128-cbc * rc4 * rsa-1024 It turns out that this problem specifically only impacts RC4 and AES, which isn't so lucky.

I used openssl-1.0.0d for all of my testing, although I found similar results with openssl-0.9.8r. **openssl speed** is the source of the benchmark results, running on just a single core in all cases. ### Is it the compiler? At first I wondered if it was just a matter of a newer compiler toolchain producing better code than the pseudo-hand written asm that OpenSSL originally acquired years ago. Since I'm beholden to [RHEL](http://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux) at work, I ran some tests[^1] with versions of OpenSSL built on RHEL 4 and 5[^2], which have very different versions of the common toolchain.

##### RHEL 4

asm: aes-128-cbc 85,513.56k no-asm: aes-128-cbc 146,107.05k

##### RHEL 5

asm: aes-128-cbc 85,284.18k no-asm: aes-128-cbc 167,597.40k

A 15% improvement isn't bad, but since I would normally build openssl with asm enabled I would never see that gain for AES. asm performance remains the same between the two, and is far lower than the C version. Something else is at issue here. ### Architectural changes x86_64 processors keep the same instruction set (and occasionally add new ones like [AESNI](http://en.wikipedia.org/wiki/AES_instruction_set)), but that doesn't mean they keep the same performance characteristics forever. Optimized code for an Opteron 270 will probably still run on a Xeon X5560 five years later, but it isn't necessarily optimized anymore. It can be a bit of an arms race to keep up with the latest hardware quirks. I ran **openssl speed**[^3] across three hosts with different Xeon processors: Harpertown (5400 series), Nehalem (5500 series), Westmere (5600 series). This revealed that some change in the CPU architecture from Nehalem on caused OpenSSL's assembly code to become less efficient than the GCC compiled C code, but only for AES and RC4 (which coincidentally are the ciphers I care about). Every other cipher continues to perform at least as well with asm than without, and many of them (such as RSA) are substantially faster. ### Results ##### openssl-1.0.0d with asm on Westmere

type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes md5 25733.55k 85951.91k 234419.20k 410698.07k 522136.23k hmac(md5) 40607.96k 126173.42k 297999.27k 451871.74k 531436.89k sha1 24997.02k 77700.48k 184074.50k 283976.43k 335233.02k rmd160 20760.18k 58180.05k 124640.68k 173745.83k 196400.47k rc4 283818.19k 308370.67k 261646.51k 263007.91k 262647.50k des cbc 44095.51k 45725.21k 46270.38k 46344.53k 46388.57k des ede3 17276.04k 17488.11k 17485.82k 17500.50k 17484.46k seed cbc 45346.65k 45829.25k 45666.74k 45812.39k 45809.66k blowfish cbc 89570.52k 93845.12k 95046.06k 95244.63k 94516.57k cast cbc 77486.21k 80917.33k 81672.87k 81765.03k 81862.66k aes-128 cbc 78122.06k 83396.93k 84696.03k 84403.54k 85325.14k aes-192 cbc 66005.65k 69767.59k 70664.70k 70910.29k 71065.60k aes-256 cbc 57154.74k 60040.38k 60761.86k 60856.32k 60959.40k camellia-128 cbc 76557.16k 106521.11k 120180.99k 124169.90k 125476.86k camellia-192 cbc 63078.86k 83880.51k 91553.11k 93668.01k 94090.97k camellia-256 cbc 63201.99k 83897.11k 91556.78k 93672.79k 94401.88k sha256 32503.04k 69011.39k 114596.78k 136929.28k 145926.83k sha512 24876.78k 99736.85k 145548.37k 200672.83k 226355.88k whirlpool 19835.40k 41370.62k 69102.45k 80712.29k 75937.11k sign verify sign/s verify/s rsa 512 bits 0.000114s 0.000009s 8804.9 105697.7 rsa 1024 bits 0.000572s 0.000029s 1749.3 34239.9 rsa 2048 bits 0.003495s 0.000101s 286.1 9882.1 rsa 4096 bits 0.024595s 0.000386s 40.7 2590.2

##### openssl-1.0.0d without asm on Westmere

type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes md5 18629.85k 69272.04k 192093.38k 350936.41k 459634.22k hmac(md5) 36383.35k 113622.20k 266558.29k 401562.97k 468322.99k sha1 19238.83k 61410.09k 152286.46k 242361.34k 293481.13k rmd160 17462.79k 51741.31k 116493.57k 170108.25k 196741.80k rc4 324468.11k 349412.22k 357622.95k 360358.19k 366261.59k des cbc 44055.78k 46050.52k 46879.49k 46962.35k 47125.85k des ede3 17255.43k 17489.71k 17535.40k 17550.34k 17502.58k seed cbc 45675.97k 45870.89k 45715.14k 45889.54k 45886.12k blowfish cbc 89574.70k 93861.93k 95049.98k 94363.13k 94524.76k cast cbc 78034.80k 80866.50k 81670.31k 81606.31k 81920.00k aes-128 cbc 158038.23k 165169.24k 167165.95k 167066.62k 167766.70k aes-192 cbc 137546.14k 142435.09k 143960.83k 143626.92k 144209.24k aes-256 cbc 122027.35k 125287.85k 126534.74k 126574.93k 126872.23k camellia-128 cbc 93205.31k 95191.42k 95708.93k 95844.69k 95690.75k camellia-192 cbc 72242.07k 73267.86k 73551.62k 73626.97k 73385.08k camellia-256 cbc 72224.74k 73108.46k 73555.54k 73559.04k 73594.20k sha256 19389.61k 45023.25k 79790.42k 98912.94k 106834.60k sha512 14506.04k 57937.00k 98487.38k 144708.55k 169426.94k whirlpool 13411.79k 27530.37k 44540.43k 52997.12k 56243.54k sign verify sign/s verify/s rsa 512 bits 0.000271s 0.000020s 3688.9 48818.2 rsa 1024 bits 0.001408s 0.000064s 710.1 15543.2 rsa 2048 bits 0.008497s 0.000223s 117.7 4476.0 rsa 4096 bits 0.054863s 0.000788s 18.2 1269.7

#### AESNI AESNI is a partial solution, since it supersedes the AES asm code on supported CPUs (Westmere and newer). But that doesn't help out poor Nehalem, who arrived before that instruction set. AESNI is also a huge performance gain that makes AES easily the preferred SSL cipher if you have hardware that supports the instruction set. #### 32-bit I didn't test as thoroughly with a 32-bit openssl, but asm vs no-asm on a Westmere produced:

RC4: 32% slower AES-128: 5.6% slower

32-bit OpenSSL is much slower for most operations than 64-bit OpenSSL, so if at all possible you should be using a 64-bit OS and application. ### Conclusion Since just wholesale disabling asm hurts us badly[^4], the ideal solution is to either fine tune OpenSSL's asm for these processors or to selectively disable it when the C solution produces better results (such as by consulting the cpuid level). One of the problem cases (RC4 on x86_64) has been solved in OpenSSL already and a [patch](http://cvs.openssl.org/filediff?f=openssl/crypto/rc4/asm/rc4-x86_64.pl&v1=1.12&v2=1.13) is available[^5]. The other three problem cases (RC4 on 586, AES on x86_64 and AES on 586) have no solutions at present. The best option right now is to disable asm for these ciphers using this [patch](http://dl.dropbox.com/u/24257718/openssl-1.0.0d-noasm_aes_rc4.patch) to Configure. This has a relatively modest negative performance impact on pre-Nehalem Xeons, but significantly improves it on Nehalem and later Xeons. You can also disable asm entirely by configuring a build with: > $ ./config no-asm although as mentioned, this is a bit overkill since there are other algorithms which still benefit from asm optimizations. You can tell how openssl was built with: > $ openssl version -f -p

platform: linux-x86_64 compiler: gcc -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -m64 -DL_ENDIAN -DTERMIO -O3 -Wall -DMD32_REG_T=int -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DWHIRLPOOL_ASM

This output suggests that asm was enabled when this build was made. [^1]: $ *openssl speed -evp aes-128-cbc* [^2]: gcc 3.4.6 and 4.1.2 respectively [^3]: $ *openssl speed* [^4]: RSA performance is 58% slower in my tests without asm [^5]: If you don't apply this patch, you should also remove 'rc4-x86_64.o' from line 130 in Configure in addition to the other changes in openssl-1.0.0d-noasm_aes_rc4.patch