JavaScript and Maps (in that order)

[cwmma]

Mistaken Assumptions in JavaScript

I help maintain the process.nextTick implementation used in browserify. A simplified explanation of how it works is as follows

var timeout = setTimeout(cleanupFunc); var i = -1; while (++i < tasks.length) { tasks[i].run(); } clearTimeout(timeout);

basically we set a timeout for a cleanup function which we clear after we run all the tasks, the idea is that the only way for this to be able to run is if there is an exception thrown in the task that is run, otherwise nothing can stop the loop from completing synchronously and clearing the timeout.

This is actually a mistaken assumption, can anyone identify why?

Simple, if somebody calls window.open inside of a task then firefox will pause execution of function while it waits for the window to open and spin the event loop while it waits causing the cleanup function to be called in the middle of the other function.

This would seem to be a bug in firefox but I wonder how many other async libraries you can break in firefox by opening a new window in the callback.

[cwmma]

Porting Node.js Crypto to the Browser part 3:

RSA and DSA Public Key Signatures and RSA Encryption.

Previously

Part 1: Ciphers

Part 2: Diffie-Hellman.

Public key encryption

Public key encryption, also known as asymmetric encryption, is encryption where instead of using one key which both encrypts and decrypts a message, there are two keys: one which encrypts and the other which decrypts, which are powerful tools that allow you to do a couple things.

Encryption

If you want to send somebody a message, you encrypt it with a standard symmetric key algorithm like AES or ChaCha20 using a randomly-generated key. Then they use the recipients public key to encrypt the symmetric key and send it. They can decrypt that key with their private key and use it to decrypt the message.

Sign messages

There are two main scheme for signing messages, which work slightly differently.

RSA is the simpler of the two and works like this: take a message, get its hash, then use your private key to decrypt said hash. Then you send the message and the signature, so the recipient can hash the message and then encrypt the signature with your public key. At this point, the decrypted signature should match the hash.

The key to this is that it should be infeasible to create a message that decrypts to a specific message without access to the private key. (Yes, this sounds backwards, but it’s just terminology; the private key always decrypts and the public key always encrypts. But with all good ciphers, choosing which direction is decryption and which is encryption is somewhat arbitrary).

DSA is slightly harder to describe, as it doesn’t do something easy to explain like encrypt and decrypt a message, but can be thought of as generating a message authentication code that can be checked with the public key.

RSA

First you need to collocate p, q, d (which is just p*q), n, and e. n and e are the public key and p, q, and d are the private key. I didn’t have to implement this, but if you are curious as to how to create them then check out Wikipedia.

To encrypt a message, you raise it to the power of e mod n. To decrypt it, you raise it to the power of d mod n. n is referred to as the modulus, and it is important as it governs the size the message you encrypt has to be. The length is called k.

There is a shortcut for decrypting the message called the Chinese remainder algorithm. The private key will in addition to d also have p, q, dP (which is d mod (p - 1)), dQ (d mod (q - 1)) and qinv (q ^ -1 mod p). This allows you to instead of calculating c ^ mod n (aka c ^ mod (p*q)), you can calculate m1 (c^dP mod p) m2 (c^dQ mod q) and h (qinv * (m1 - m2) mod p), and then m2 + h * q is the decrypted message. While this sounds more complex, all the operations are on smaller numbers making, it much faster. (I confirmed this when implementing; it was more than twice as fast).

One last thing is that the amount of time it takes to calculate the above is very closely related to the values of the message and the key, meaning if an attacker can measure how long it takes to decrypt something several times, they could figure out what your private key is. The solution is called “blinding,” and it involves adding another layer of randomness to the process. First you need to pick a random number r which is less than the modulus (n). Also r needs to be coprime with n, meaning that the greatest common divisor (largest number that divides both equally) is 1. Luckily we know both devisors of n (p and q), so all we have to do is generate a random number with a byte length the same as n and check that r is less then n, that r mod p !== 0, and that r mod q !== 0. To calculate the blind value with r pow e mod n (with e being the public exponent), then you multiply the message by that (mod n) and calculate the encrypted value. Once you are done, you need the value to unblind it, which is r ^ -1 mod n, then multiply the decrypted value by that (mod n) to get the unblinded value.

Originally I calculated the binding with code based on end-to-end, but the way they implemented modular inverse was very slow. When I replaced that code with one built into bn.js, my tests started completing in 45 seconds instead of 2 minutes 45 seconds. I’m not saying this to pick on end-to-end but to underscore how important for performance a good big number library is. That single unoptimized inverse took more then twice as long as all other functions involved in RSA decryption combined.

DSA

DSA has five parameters: p, q, g, x, and y. p, q, and g are public and shareable because they are both difficult to compute and don’t represent a security risk if multiple parties use them. To compute them, you first pick two lengths L and N; these are your key length, with N being the length of the hash function (e.g. sha1 would be 160, and sha2 be the same as their name, e.g. sha256) with L being the size of the secret key. A common choice is: pick a prime q of bit length N. Then pick a prime p such that p-1 is a multiple of q (or put another way p-1 mod q = 0). G is a number where g pow q mod p = 1; this can be found by picking a number h and calculating h pow (p - 1)/q mod p. If the result isn’t 1, then that’s your g. This doesn’t need to be random, so you can start with h being 2 and work your way up. As you might recall from my Diffie-Hellman post, finding pairs of prime is not easy, so reusing them makes sense especially if you need to generate a bunch of keys.

The private key (x) is a random number less then q. The private key (y) is g pow x mod p.

A DSA signature consists of two output values: r and s. First, generate a random number k less than q. Now calculate r as (g pow k) mod q. In theory, r could be zero and you’d have to start over, but to quote RFC 6979, “this is an utterly improbable occurrence”. You may notice that r requires neither the message nor the private key, thus this can be pre-calculated (which is helpful, as it is the most computationally expensive part). You calculate s by taking the hash of the message h, adding the private key multiplied by r to it, and multiplying the result by k pow -1 (kinv), all of this mod q, or in other words (h+x*r) * kinv mod q. If s is 0, pick a new k and start over.

It is vital that k be random because an attacker knows r, s, h, and q, so in the equation s = (h+x*r) * kinv mod q only k and x are unknown, so if the same k was used twice then you know the signer’s public key.

This is not theoretical – see this article for how a bug in the Debian random number generator meant that there were only 32,767 possible random numbers (k values). Testing all of them for a given signature is not hard.

The solution to this is to use the technique outlined in RFC 6979 to use an HMAC of the private key and message to calculate k. This deterministically calculates k in an unpredictable way that is guaranteed (for a secure hash algorithm) to be unpredictable for a given private key and message.

To verify the signature, you calculate h = the hash of the message, then

w = s pow -1 mod q u1 = h * w mod q u2 = r * w mod q v = (g pow u1) * (y pow u2) mod p mod q

v should equal r.

Actually implementing it

If there has been a theme to these posts, then it is implementing the actual crypto algorithms are the easy part. The minutia you need to actually get it working is the hard part and public key signing/encryption is no different as it requires messages to be padded and for you to be able to read key files.

Padding

For RSA to work, the message has to be just the right size. It needs to be the similar in byte length to the modulus, but it can’t be bigger then the modulus (when treated as an integer). Additionally, for signatures you need to be explicit about what hash algorithm you used, else somebody could theoretically do something sneaky by substituting a weaker hash value. For encryption, you need to inject an element of randomness into the encryption process to prevent the same message from encrypting into the same thing (or two very similar messages, letting you figure out the key which encrypted it).

Signature padding

For signatures to pad the message up to the byte length of the modulus the method is

0x0, 0x1, 0xff ... 0xff, 0x0, digest identifier, hash

or in English, two bytes with the value zero and 1, modulus - (hash length + digest identifier + 3) bytes of value 255, a byte with value 0, the digest identifier and then the hash. The digest identifiers are given in section 9.2 of this paper but not for every digest Node supports, so for ripemd160 I had to just encrypt the signature that node produced to figure out the value that OpenSSL uses. I believe they are binary representations of the asn1 OID values of the hash algorithms, but it was far easier to reverse engineer them out of OpenSSL than figure out which where the ripemd160 one is defined.

Encryption padding

Padding for encryption has two different schemes: PKCS1 v1.5 and OAEP. PKCS1 is simpler, but

It is possible to generate valid RSAES-PKCS1-v1_5 ciphertexts without knowing the corresponding plaintexts, with a reasonable probability of success.

which can lead to attacks. So TL;DR: use OAEP.

PKCS1 v1.5

Take you your message M which you need to get up to the length of the modulus. You first generate message length - (modules length + 3) bytes of random numbers, none of the bytes being zero. The way I did this was by generating random numbers, discarding any bytes that were zero until I had enough. There are plenty of other ways to do this, but most of them are likely going to create non-uniform distributions of random numbers. The message is [0, 2] + non zero random bytes + [0] + message.

To remove the padding when decrypting, you just look for the second zero byte and the message starts in the next position.

OAEP

OAEP, which stands for Optical Assyrian Enlisted Pandas (or something equally irrelevant to how it works), is a padding scheme which is non-deterministic over its entire length (as opposed to PKCS1, where only the leftmost portion is random, and the right part is the message).

OAEP has a function it calls MGF (mask generation function). MGF can be defined in Node.js as:

https://gist.github.com/calvinmetcalf/bc298830342b7de9efc3

You take a hash function and hash an empty message. Call the result iHash

Then create a buffer filled with zeroes of length modulus 0 (message length + hash length * 2 + 2). Call it ps.

Concat iHash, ps, a byte with the value of 0, and the message. Call that db.

Generate random bytes the same length as the db and call that seed.

Call mgf(seed, db length). Call the result dbMask.

XOR dbMask and db to create maskedDb.

Call mgf(maskedDB, hash length) and call that seedMask.

XOR seedMask to seed to create maskedSeed.

Concat a zero byte, maskedSeed and maskedDB to create the message

Note: Node doesn’t give any way to specify a digest algorithm here besides sha1.

To decrypt:

Calculate the hash of an empty string and call it iHash.

Call the length of iHash hashLen.

The first byte of the padded message is zero, so get a slice starting at the next byte of hashLen. This is maskedSeed.

The rest of the message is maskedDB.

XOR maskedDB with mgf(maskedDB, hashLen). This is seed.

XOR maskedDB with mgf(seed, maskedDB length). This is db.

The first hashLen bytes of db should be equal to iHash. (You know your message has been tampered with if it isn’t.)

iHash is followed by some number of zero bytes (possibly none) and then a byte with value of 1. (If it doesn’t, your message has been tampered with.)

The rest of db is your message.

Motherfraking PEM files

Now comes the hard part, getting the public or private key out of the file that the user provides. The file in question is likely going to be a .pem file. PEM files are the sole surviving part of a failed IETF proposal for encrypted email. They are files which look like:

-----BEGIN RSA PUBLIC KEY----- MIGGAn87CzBsWj+7ILyW0Z//IDUD6BXkgZ2cCA9tRIjcbNscID7H5Msb+0u9tHDe vWyamlj+OSSmJVbUStIy43S6LGnmBvvxn2sfVelZvlZaCndZpj/0QcyMx06RD/0t Vm9G+X8z8WLqjA/6r5qYkjUESMQJh9uEYveuaVV2ripdzjRDAgMBAAE= -----END RSA PUBLIC KEY-----

and consist of a header and footer around a bunch of base64 encoded data. The header and footer are of the form -----(BEGIN|END) :tag-----. The base64-encoded data is DER-encoded data.

What is DER encoding you ask? It is the Distinguished Encoding Rules for encoding data in the Abstract Syntax Notation One (ASN1) format. More details can be found in this article, but what you need to know is that DER/ASN1 encoding is a binary method for encoding data. The key above decodes as:

sequence of 2 elements

integer value 161960552770399697736583593721290722658968249097500897481864131194167084781582467842371267641339824876964119703500297546233667881250293993668028296697189038155281236041428666387090163250593165185711140282087579809282559115211132751074088235101133730773838767046174614028835929718065895239952789579051709507

integer value 65537

You will note that this just has values, not the keys, meaning you have to find the scheme which defines how a particular key type is packed in.

The types we need to care about are:

integers

sequences

object identifier

bit strings

octet strings

There are 3 algorithms we can use to make keys:

RSA, which we covered earlier

DSA, which we didn’t cover, but what you need to know is that both public and private keys have p, q, and g parameters, and the public key has a y parameter and the private key an x.

Elliptical Curve DSA. Didn’t cover that one here, but what you need to know is both public and private keys need to know the curve ID (which is an object identifier) and there are private and public keys which are bit strings and octet strings respectively.

There are 5 types of keys that someone can give you (that I have found):

Algorithm specific private keys. These are keys that start with BEGIN (algorithm) PRIVATE KEY, e.g. -----BEGIN RSA PRIVATE KEY-----. The use of these keys looks to be a legacy feature and they aren’t recommended (especially for encrypted private keys, as they use their own half-assed encryption setup).

Generic private keys that start with -----BEGIN PRIVATE KEY-----. This is a generic wrapper that gives you information on what algorithm and specific parameters for it and then the private key data as an octet string that you then decode to get the private key which is private key-specific data (as opposed to algorithm parameters).

Encrypted private keys that start with -----BEGIN ENCRYPTED PRIVATE KEY-----. These contain an encrypted octet string and information to allow it to be decrypted if you supply a password. When decrypted, it is a generic private key.

Algorithm-specific public keys. As far as I can tell, OPENSSL will only make them for RSA, so THEY all start with -----BEGIN RSA PUBLIC KEY-----. Deprecated.

Generic public keys, which start with -----BEGIN PUBLIC KEY-----. These are similar to the generic private key files: they have an algorithm ID, parameters for the algorithm and then a bit string (not an octet string like private keys), which decodes into public key data.

Parts of these are defined in a variety of RFCs, with only part of any given RFC still being used. The rest of this isn’t defined anywhere, as far as I can tell.

Algorithm-specific private keys

While officially being deprecated, you can still see these keys being used in the wild. These keys are generally simpler in structure than the generic ones, but the encrypted ones are much less secure.

——-BEGIN RSA PRIVATE KEY——-

This is an RSA private key, which you can generate in OpenSSL with the genrsa command. The parameters it contains are all ints and are (with symbols which represent them in parentheses):

version (as in version of the file format)

modulus (n)

public exponent (e)

private exponent (d)

prime 1 (p)

prime 2 (q)

exponent 1 (dp)

exponent 2 (dq)

coefficient (qinv)

Strictly speaking, all you need is n and d, but p, q, dp, dq, and qinv are used for the Chinese remainder theorem, and e is used for blinding, so you can create a public key out of the private one.

——-BEGIN EC PRIVATE KEY——-

This is an elliptical curve DSA private key:

version (an integer)

private key (an oct string)

named curve (an object id, but in some sort of over complicated arguments object as argument 0)

public key (a bit string, but also in the arguments object as argument 1).

The named curve and key parameters are part of a more complex structure defined in the ASN1 spec, but the EC private key is the only one which uses it, so if we took the time to learn it, that would be time we would never get back. We just need to thank Fedor for figuring it out for us.

——-BEGIN DSA PRIVATE KEY——-

For regular DSA keys, they have the following attributes (all ints):

version

p

q

g

y (public key)

x (private key)

Legacy encrypted private keys

I have alluded to how these keys do encryption badly. Here is how it works (most of which I learned from this Stack Exchange answer). If you want to protect your key with a password, then instead of

-----BEGIN RSA PRIVATE KEY----- MIICVAIBAAJ/OwswbFo/uyC8ltGf/yA1A+gV5IGdnAgPbUSI3GzbHCA+x+TLG/tL

your key will start with

-----BEGIN RSA PRIVATE KEY----- Proc-Type: 4,ENCRYPTED DEK-Info: AES-192-CBC,04D2D7882E0C474E07E542FE997D2A49 vfB5Gtm34n3SeI6JELjWiGw6O+j+tGR6Wbi3SNeAZkfSA8PTjei6PVHr+dGK5zMd

The first part of the DEK-Info is pretty easy to figure out: it’s the encryption algorithm and block mode. As for the second part, as you might remember from the post on ciphers, you can’t just use an English string as the key to a cipher, but it needs to be a binary sequence of a specific bit length and that to prevent the encryption from being deterministic you use a salt. (That is what the hex string that makes up the second part of the DEK-Info is: a salt.)

But what kind of salt is it – a salt for the key derivation function, or an initialization vector for the cipher? Trick question: it’s both. The string itself is the initialization vector for the cipher, and the first 8 characters are the salt for the key derivation function. For the record, don’t do this at home; a very nice attribute of key derivation functions is that they can make output of any length, so there is no reason not to use all of the string as the salt for the key derivation function and then use it to generate the key and the iv for the cipher.

Remember from part 1 that really really bad key derivation function EVP_BytesToKey which Node.js uses to create keys for ciphers? Guess what, that’s the key derivation function used here, and much like Node, it is used with a broken hashing algorithm (MD5 hard coded in, I might add), and a single iteration. On the one hand, it is nice that it’s using a salt, but on the other hand, the way it’s using the salt breaks a important rule of cryptography: Don’t reuse random numbers.

Long story short, if you had access to an encrypted key of somebody else’s, you could quite efficiently do a dictionary to decrypt it, so if you needed yet another reason to avoid these keys here it is.

Generic private keys

These are the keys that start with -----BEGIN PRIVATE KEY----- and define a wrapper around a private key from one of the algorithms mentioned above. The generic format is:

version (int)

algorithm info (sequence)

algorithm id (object id)

parameter (varies)

private key (oct string)

The algorithm parameters will vary depending on what it is and the private key is DER-encoded data as well, which must be decoded.

RSA

The algorithm ID is 1.2.840.113549.1.1.1, the parameter is a null value, and the private key decodes to exactly the same 9 elements as the RSA specific key.

ECDSA

Note: I was never actually able to generate a generic elliptical curve key using the OpenSSL pkey command, but I was able to generate an encrypted one, which should decrypt to be the same thing.

The algorithm ID is 1.2.840.10045.2.1, the parameter is an object id that lists the curve to use, and the private key decodes to the same 4 values as the EC specific key.

The only curve I dealt with was secp256k1, and its ID is 1.3.132.0.10.

DSA

The algorithm id is 1.2.840.10040.4.1 the parameter is a sequence of 3 integers which are p, q, and g, and the private key decodes to a sequence of a single integer which is x (the public key). Note that DSA is the only one where the generic version is not the same as the algorithm-specific one. This is because the p, q and g parameters are costly to generate and reusable, multiple parties in the same organization might use the same p, q, and g parameters, so they are more akin to the curve name in ECDSA, but unlike in ECDSA they are rather long (the size of p, g and y is the key length, g and x are about 1/10th of the key length), so including them twice would double the size of the PEM file. The public key (y) can be calculated with g pow x mod p, so omitting cuts the size down 30%.

Encrypted private keys

These keys are wrappers around encrypted generic private keys. They have the header -----BEGIN ENCRYPTED PRIVATE KEY----- and have the following format:

encryption info (sequence)

encryption standard id (object id)

parameters (sequence)

key derivation info (sequence)

key derivation id (object id)

key derivation parameters

salt (integer)

iterations

cipher information (sequence)

cipher id (object id)

initialization vector (oct string)

encrypted data (oct string)

The encryption standard is always going to be PBES2 (PBES1 is for working with older ciphers with key lengths of 60 bits and lower), so the encryption standard ID will always be 1.2.840.113549.1.5.13, as far as I can tell the only key derivation algorithm that works with this is PBKDF2 with SHA1 with an object ID of 1.2.840.113549.1.5.12, PBKDF2 is an order of magnitude safer then EVP_BytesToKey, especially as by default more then 2000 iterations are specified by OpenSSL. SHA1 is likely safe for this now, but I’m not sure if that will continue to be the case. For ciphers, OpenSSL seems to support AES (in both 128, 192, and 256 variants) with ECB, CBC, OFB, and CFB modes (CRT mode isn’t supported and GCM doesn’t work as far as I can tell). You can view the IDs of the 12 working modes here.

Algorithm-specific public keys

Like their private cousins, these are depreciated, and as far as I can tell you can only make them for RSA keys in OpenSSL. They start with a header of -----BEGIN RSA PUBLIC KEY----- and contain a sequence of two integers: modulus (n) and public exponent (e). Don’t use this.

Generic public keys

These are like the generic private keys. In fact, they are almost exactly the same, except that they don’t include a version number as the first parameter, and the third parameter is a public key instead of a private key, which apparently means it needs to be a bit string instead of an octet string. Otherwise, the algorithm identifiers are the same and the algorithm parameters are also the same.

RSA

The public key bit string decodes to the same as the RSA-specific public key, a sequence of 2 integers n and e.

DSA

The public key bit string decodes to be an integer (not a sequence of a single integers like you’d expect, based on how every other thing here has done it). This is y (the public key).

EC

The public key bit string doesn’t contain anything DER-encoded, but is an encoded elliptical curve point (an uncompressed one to be specific).

Conclusions

When you serialize data, please choose a format that includes keys in addition to values. Nothing is worse than having to dig through the Internet tying to figure out what the values in a field represent and what order they are in. Also, those “well-known” OIDs for representing algorithms are never as well-known as you’d hope, so use human-readable ones. ASN1 has joined WKT-formatted projection info as one of my most-hated formats.

Package Size

On a more practical note, the pubic key aspects of the crypto module have caused it to start getting rather large, to deal with that we are breaking out the methods as much as possible into stand alone modules that like the inherits package uses the native crypto module on the server and the pure JavaScript version in the browser. So far we’ve got

create-hash which is identical to require('crypto').createHash

randombytes which is identical to require('crypto').randomBytes

create-hmac which is identical to require('crypto').createHmac

browserify-aes which provides an object with all 5 cipher related functions.

diffie-hellman which provides an object with getDiffieHellman and createDiffieHellman functions.

The rest should be done in the near future.

Thanks

I’d like to thank Nolan Lawson for editing this and all the other members of the crypto-browserify team Daniel Cousens, Dominic Tarr, Fedor Indutny, and JP Richardson.

[cwmma]

Porting node.js crypto to the browser, part 2: Diffie-Hellman

Part 1: Ciphers

In the previous post, I talked briefly about how passwords are usually a bad way to create keys for encryption. One alternative method is Diffie-Hellman key exchange, which I ported to Browserify in this library.

Basic Idea

Two people can calculate a key, known only to them, communicating publicly. Here's how it works, using some absurdly small numbers you shouldn't even think about using in reality, as my cats could crack them:

You agree on a base and a prime, for example 2 for the base and 35963 for the prime.

Then you calculate a random number, say 26971. This is your private key.

Then calculate your public key, which is the base, raised to the power of your private key mod the prime (mod meaning divide by the prime and take the remainder). In JavaScript, this would be:

Math.pow(2, 26971) % 35963

This is your public key and is 8182.

You send your public key to your friend, then they send you their public key 32789. You compute their public key, raised to the power of your private key, mod the prime or:

Math.pow(32789, 26971) % 35963

Meanwhile they do the same thing, assuming their private key is 17226:

Math.pow(8182, 17226) % 35963

Both of you calculate that to be 28001 which should be impossible for somebody else to calculate based just on 2, 32789, 8182 and 35963. (Although in this example they could, because the numbers are so small.)

Advantages

Diffie-hellman is somewhat lesser known then other asymmetrical cryptography methods like RSA, so what advantages does it have? Its main advantage is that it allows for forward security, with the disadvantage being that it requires a input from both parties before encryption happens.

We will cover RSA in more details in a later post, but what you need to know is just that RSA is based on encryption where there are two keys: one for encrypting and one for decrypting. To send an encrypted message, one party encrypts the key with the public key of the other party and then sends the encrypted message along with the encrypted key to the other one.

With Diffie-Hellman, the recipient would calculate a temporary private key and use that to create a public key, and give that to the sender. The sender would calculate a temporary private key and use that to create a public key and shared secret, then use the secret to encrypt the message. Then they would forward their public key and the encrypted message to the recipient and throw out the temporary keys. (Both parties would also have more permanent public keys they'd use for signing their messages, we will cover that in a later post.)

You should be able to see why RSA is popular with encrypted email; you can fire off an encrypted email without the recipient needing to do anything except post their public key somewhere. But it doesn't have "forward security," because if the recipient's private key is ever compromised, somebody could use that to decrypt the message, since they key is right there.

With Diffie-Hellman, on the other hand, since only permanent keys are used to authenticate the message, if a key is compromised, you can't use that to decrypt old messages. But the drawback is that you need interactive communication to establish the shared secrets, which makes it unwieldy for things like email. There is also an overhead of having to come up with new keys for every message.

Big numbers

If you tried running this example in your console, you would have noticed that Math.pow(2, 26971) % 35963 returns NAN. This is because, in JavaScript, Math.pow(2, 26971) returns Infinity. Thus some sort of "big number" library for dealing with math operations on integers of arbitrary giganticness is mandatory for all JavaScript encryption libraries. I looked at a bunch and bn.js by Fedor Indutny was the only standalone big number library that had the optimizations necessary for this kind of crypto.

Pow mod

Hopefully you have realized that doing

var powered = Math.pow(2, 26971) var result = powered % 35963;

is not the most efficient way to calculate a pow b mod c as the intermediate result is

https://gist.github.com/calvinmetcalf/cecdf494a0d6a2b5e9ef

(Calculated with this Erlang script.)

Bearing in mind that the huge number above is for the ridiculously small keys that we are using as examples, how can you calculate it in a realistic manner?

The simplest way is:

https://gist.github.com/calvinmetcalf/1c3ad6247c8c6a98e38e

This breaks down when b a decimal number 200+ digits long, which is the case in a more realistic scenario of a 768-bit prime number. (35963 in our example is 16 bits long.) There are various ways of speeding this up, and I've spent two days trying to come up with some examples that I can understand well enough to explain to a non-math person but can't.

What you need to know is that bn.js is fast due to the Montgomery reduction, which is a technique which speeds up the operation a x b mod c a lot, but only for large numbers. If you want more details, feel free to check out Fedor's code.

This ended up not being the tricky part of bringing it to the browser.

Finding Primes

Note Diffie-Hellman objects can be created in one of three ways:

1. Specify a MODP group

This is the easiest way: specify one of the MODP groups, which are predefined primes based on digits of Pi. The ones relevent to Diffie-Hellman are modp1 and modp2 defined here and modp5, modp14, modp15, modp16, modp17, modp18 defined here.

RFC 3526 includes a table of key lengths, and you'll notice that the strength estimate does not scale with modulus bits. We will discuss this and alternatives in a later post on elliptical curves.

2. Specify a prime (and maybe a generator)

This method is pretty much what it says on the tin. OpenSSL checks that the prime is "safe," and if it isn't, in Node version 0.10 it throws an error. However, in later versions it just silently notes this in a property of the Diffie-Hellman object. The OpenSSL definition of a safe prime is not actually met by the MODP primes, meaning you can't set them manually in Node 0.10.

3. Specify a prime length (and maybe a generator)

In this last method, we need to generate a new prime, which the user will need to send to the other party. This is where things get tricky, and is what took the longest. Our task is to:

generate a random number

of the specified bit length

that is prime

and is "safe."

What does "safe" mean?

To be a safe prime means that, for a prime p, then (p-1)/2 must also be prime. Note that since all primes besides 2 are odd, (p-1)/2 is the same as p >> 1 (p right shifted by 1 bit). Then if the generator is 2, then p mod 24 must be 11 or if the generator is 5 then p mod 10 must be 3 or 7. At some point in the past, if the generator was 3 then p mod 12 needed to be 5, but this is no longer the case.

The algorithm I came up with (with help from Fedor) was as follows, to generate a safe prime of b bits:

Generate random buffer of Math.ceil(bits / 8) bytes.

Make a big number p out of it.

Then, while the bit length of p is larger than b, we shift it right a bit.

So that both p and p>>1 are odd, we make sure the rightmost 2 bits are set.

If the generator is 2, then we add 4 to it until p mod 24 is 11, or if the generator is 5, we add 4 until p mod 10 is 3.

Calculate p>>1 and call it p2.

Test if p2 is prime. If not, go to 9.

Test if p is prime. If so, return p.

If the generator is 2, then we add 24 to p and 12 to p2. If the generator is 5, then we alternate adding 4 to p and 2 to p2 while adding 16 to p and 8 to p2. For all other generators, we add 4 to p and 2 to p2.

If the bits of p are greater then b, go back to 1. Otherwise, go back to 7.

You'll note that we test p2 before we test p. This is because p2 is smaller than p, and we need to test a lot of candidates to find a safe prime, so we want to eliminate bad candidates as quickly as possible. This influences our prime testing methods, where we use three different tests, starting with one that is fast but prone to false positives and ending with a slower one that is less likely to incorrectly name something as prime.

Edit: after I wrote this, I realized it was more efficient to test p right after testing p2 for each test, not do all 3 tests for p2 then all 3 for p.

Method 1: Simple Sieve

This and the next method were taken from self-signed, and just involves generating (and for the love of God, caching) a list of small primes using the Sieve of Eratosthenes (I think). I just used Fedor's default of all primes less then 0x100000. (You can see the code to generate them here.) Then we go through this list and, for each prime pr, we see if our prime mod pr is 0.

The only modification I made to this for Diffie-Hellman was to also check if pr and the prime we were checking were the same number. If that's the case, then our number is prime. Fedor could get away without checking, because all of his primes would need to be large, but I have to reproduce the OpenSSL API, which will give you a prime with as few as 15 bits.

Method 2: Fermat primality test

This isn't too complex. Basically, for a prime p, choose a number a such that a is larger then 1 but smaller than p. a pow p-1 mod p should equal 1, unless p is quote "a Fermat liar." We just check for a = 2; thats good enough because if any fall through the cracks...

Method 3: Miller Rabin

The Miller-Rabin primality test is similar to Fermat's, but more complete. You can read the details on Wikipedia, but basically it is a heuristic that lets us say we are pretty sure that this number is prime, and the benefits of pretty sure outweigh the time it would take to calculate. For the size of primes that it is reasonable to be able to find in a browser, Miller Rabin is overkill most likely. (I have yet to see it knock out a prime that Fermat didn't catch.)

Performance

The performance for generating primes is not great, based on some benchmarking.

The cost of testing a prime candidate scales at least linearly (possibly worse).

The number of candidate numbers to test to find a prime scales linearly.

The number of candidate primes to test to find a safe prime scales linearly too but...

The number of candidate numbers to test to find a safe prime scales exponentially.

So in other words, it gets real slow real fast. So don't generate any primes in the browser in production.

Thanks to Nolan Lawson for proofreading this and Fedor Indutny for doing all the hard work. Join me next time for RSA signing.

Questions, comments and complaints can be directed at me on twitter.

[cwmma]

Porting Node.js crypto to the browser, part 1: all about ciphers

Part one in a series on porting the Node.js crypto library to the browser, this one focuses on porting the functionality of crypto.createCipher, crypto.createCipheriv, crypto.createDecipher, and crypto.createDecipheriv, which was done in the library browserify-aes.

Some background on the Node.js crypto library

crypto is one of the first Node.js standard libraries to have been written, so it is not quite idiomatic with the others. The ciphers are streams, but since they predate the stream library, they also have synchronous update and final methods that work similarly to write and end, but simply return any data they produce instead of passing it into the stream.

Almost all crypto functions are CPU-bound instead of IO-bound, so simply making them non-blocking wouldn't be much help on a single core machine. That, and making the operations async would add latency.

The fact that the cipher operations are all synchronous also rules out any possibility of using the WebCrypto API. I will leave it as an exercise to the reader to write a crypto library with its own API which efficiently makes use of Node.js and WebCrypto in an isomorphic API.

Lastly, the Node.js crypto library is really a thin wrapper around OpenSSL. Yes, this is the same OpenSSL that has been in the news a lot lately, and I don't want to really pile onto that train. But let me just say that OpenSSL is a typical crypto library, and crypto libraries as a universal rule (with a few exceptions) have terrible documentation. So I had to go and spelunk through the OpenSSL code much more then I would have liked, in order to figure out how things worked.

XOR

In case you aren't familiar with the term, XOR or xoring refers to the eXclusive OR operation to combine two pieces of data. Represented sometimes by a caret (^), it means that for a ^ b, if both a and b are false, then the result is false. However, if both a and b are true, then the result is also false. Only in the case where exactly one of a and b are true, but the other is false, does XOR evaluate to true.

We can construct a truth table as follows:

|t|f| t|f|t| f|t|f|

We can do this with numbers by turning them into binary representations and doing the XOR operation individually on the bits. So to XOR 240 and 170, we'd get 70 because:

1111 0000 (240) ^ 1010 1010 (170) = 0101 1010 (70)

This is very helpful in cryptography, because when you take a piece of binary data a, and XOR it with another piece b, you go through b, and anywhere there is a 1, you flip the bit in a (and don't if there is a zero in b). You can think of b as being a mask over a (though masking refers to something else), which hides the contents of a behind b by mixing them together.

AES

Implementing the AES block cipher was actually really easy. There are quite a few open source JavaScript AES implementations out there, but if there is one thing that all JavaScript crypto libraries have in common, it is that they all have their own hand-rolled buffer implementation. I ended up basing my version on the one from triple sec, which is based on the one from crypto-js.

AES is a block cipher which takes blocks of 128 bits (16 bytes) and a key with a length of 128/192/256 bits (16/24/32 bytes) and outputs 16 bytes of encrypted data. And it can also do the opposite, i.e. take a key with 16 bytes of encrypted data and output plaintext. If you want more information, check out out the Wikipedia article on AES.

Modes of operation

A block cipher by itself is actually rather useless, because it is determanistic. In other words, the same block of 16 bytes and the same key will always output the same cipher text. Why this is bad is illustrated quite well on the Wikipedia page on modes of operation. Basically if you just encrypt with AES (the technical term for which is encrypting using the "Electronic Code Book," or ECB, mode of operation), it leaks information about the structure of the data, which can be used to make powerful inferences about the message.

To do this, you need a mode of operation and a second piece of data, besides a key, called either a nonce or an initialization vector. Both nonces and IVs are used in the same way, but they differ in that the security of an IV depends on it being random, while the security of a nonce only depends on it being never reused with the same key.

Before we get to modes of operation, a brief digression into the first big roadblock I hit.

Key Derivation

Disclaimer: human readable passwords are generally an a very bad way of creating keys for ciphers, you almost certainly want to use some other way to create a key such as something based on public key encryption, I will discuss some of those in other installments.

So AES takes a key of exactly 16, 24, or 32 bytes and an IV of 12-16 bytes (or zero for Electronic Code Book). But crypto.createCipher takes a string password of arbitrary length, so how does Node.js create the key it needs? Simple: it uses the OpenSSL function EVP_BytesToKey, and it does so with MD5 as the digest algorithm, no salt, and an iteration count of 1. First off, even without knowing how EVP_BytesToKey works internally, just by knowing the parameters that Node sends to it, you can take a guess that the keys it derives are very weak.

What do I mean by weak keys? Basically, if somebody wanted to crack data stored with keys derived this way, a few things would make it very easy.

First off, the keys are not derived with a salt. A salt is a random piece of data that is not secret, but used to prevent the derived key from being deterministic. Essentially, you generate a random number which you use, in addition to the password, to derive the key.

The salt doesn't need to be secret, and you can send it to the recipient in the clear. The benefit of this is that, without the salt, an adversary can simply hash a dictionary of common passwords ahead of time and then just go through them one at a time and try to decrypt your text. This is called a rainbow table, and constructing one wouldn't be too much of a problem for an attacker, unless you also had the foresight to increase the number of iterations.

The iterations means that, instead of hashing a piece of data once, you do it 10 times, or 100, or 1000 times. This technique, combined with the salt, means that suddenly going through a dictionary of common passwords requires much more work.

So after some trial and error and looking at the code, it turned out the correct formula is basically:

https://gist.github.com/calvinmetcalf/c259f12686aa7de31732

This is simplified; the full version is here. If you're interested, PBKDF2 (which is much much more secure) works like this:

https://gist.github.com/calvinmetcalf/91e8e84dc63c75f2aa53

This is available in Node as crypto.pbkdf2. So if you ever encrypt things in Node with a password, you should never use createCipher; instead you should use createCipheriv and use PBKDF2 (or a similar algorithm like scrypt) to generate the key and IV. The correct IV length for all the modes of operation is 16 bytes, except for Galois/Counter Mode, or GCM, which is 12, and ECB, which is 0. (We'll discuss these in the next section.) If you don't want to roll your own, I have a library that does it for you.

Modes Of Operation, continued

The modes of operation can be divided broadly into three groups:

Block cipher

Self-synchronizing

Stream cipher

Let's go through them one by one.

Block cipher modes of operations

These modes only work on plaintext, where the length is an exact multiple of 16 bytes. In order to deal with messages that are not, the end of the message is padded up to a multiple of 16 bytes using a padding standard known as PKCS7, which is defined here and is very simple.

PKCS7 just requires you to take the number of bytes of padding you would need to add in order to get a multiple of 16 (and if it is already a multiple of 16, you add a full 16 bytes of padding) and then you add that number of bytes each containing that number. For instance, if you have 15 bytes of data, you'd add a single byte containing the number 1, if you had 25 bytes of data you'd add 7 bytes each containing the number 7, and if you had 32 bytes of data you'd add 16 more each containing the number 16. Block cipher modes that Node supports are:

Electronic Code Book (ECB)

Cipher Block Chaining (CBC)

Self-synchronizing stream cipher modes of operations

Stream ciphers are ciphers that can work on arbitrarily-sized inputs without padding. So stream cipher modes of operation are actually modes that transform AES into a stream cipher. The self-synchronizing ones can lose a certain amount of the encrypted message and still decrypt (most of) the rest of it. The only one that Node supports is called Cipher Feedback (FB), but it has three variants:

Cipher Feedback (CFB)

Cipher Feedback 8 bit (CFB8)

Cipher Feedback 1 bit (CFB1)

Stream cipher modes of operation

These transform AES into a normal (synchronous) stream cipher, which is basically just a deterministic random number generator which can be XORed with the plaintext to create the cipher text. Node has three:

Output feedback (OFB)

Counter (CTR)

Galois/Counter Mode (GCM)

Electronic Code Book (ECB)

We've discussed this before, but ECB is less of a mode of operation then a lack of mode of operation. It is good for three things:

testing

getting yourself hacked

implementing other modes

For all intents and purposes, you should just pretend this mode doesn't exist and move along.

Cipher Block Chaining

This is a very simple mode of operation which can be implemented entirely from the Wikipedia entry – heck, entirely from the picture. It takes the previous cipher text, XORs it to the plaintext and then encrypts that. For the first block, which doesn't have a previous piece of cipher text, it is XORed with the IV.

This method has some advantages:

It is really simple.

If you lose a block of encrypted text or a block of cipher text gets curruped, the subsequent block will decrypt wrong but the the rest will be fine.

You can parallelize decryption of the message.

This method has a couple downsides:

You can't parallelize encryption.

It is vulnerable to a known plaintext attack in certain circumstances. When used in a streaming context, where an attacker can know the cipher text of a block before the plaintext for the next block is sent to be decrypted, then the attacker can figure out if a specific plaintext encrypts into a specific cipher text. In other words, an attacker can ask, "Does this block decrypt to this?" That might not sound like much, but this formed the basis of the BEAST attack, so you should avoid this mode if possible.

Cipher feedback (CFB)

This mode is very similar to CBC, but instead of XORing the plaintext to the previous cipher text and then encrypting, it encrypts the previous ciphertext and then XORs it to plaintext to create the next cipher text.

The first benefit of this over CBC is that you don't need to have a full block of plaintext to encrypt it. You have the 16 bytes of data you got from encrypting the last block or IV. If you just get 10 bytes of plaintext, you XOR that with the first 10 bytes of the encrypted data (and save it). If you then get 10 more, you XOR the first 6 bytes with the last 6 bytes of the encrypted data, concat that to the end of the 10 bytes of cipher text you already made, encrypt that and XOR the first 4 bytes with the last 4 bytes of the previous data.

Cipher feedback 8 bit (CFB8)

Regular Cipher feedback is like CBC in that you can lose a block of cipher text and, while the next block would decrypt incorrectly, the one after wouldn't. CFB8 allows you to lose any number of bytes of cipher text and still eventually decipher things correctly.

The way it works is that you encrypt the IV and create a 16-byte buffer, which we will call your "pad," then for every byte of data you want to:

XOR the byte to the first byte of the pad,

take that output byte and shift it into the last position of the pad, shifting all the other bytes over one position, shifting the first byte out,

encrypt that to make the new pad.

So if you have a pad:

data: a, b, c, d, e, f, g position: 1, 2, 3, 5, 6, 7, 8

And you get a piece of data:

data: h, e, l, l, o position: 1, 2, 3, 5, 6

You XOR h and a to get z. Then you shift the buffer to the left and shift in z:

before: a, b, c, d, e, f, g <---a <---z after: b, c, d, e, f, g, z position: 1, 2, 3, 5, 6, 7, 8

Then encrypt that to get the next pad. The upside is that for mangled data, you are much more resilient. But the downside is that you have to encrypt once per byte instead of once per block – 16 times more often.

Cipher feedback 1 bit (CFB1)

This is like CFB8 except instead of per-byte, it is per-bit. You XOR the first bit, shift the result in on the right and the old one out on the left, and then encrypt again. While this is obviously even more resilient, it means you are encrypting 128 times more often than most other ciphers, which has obvious performance implications.

Output feedback (OFB)

This is a simple stream cipher that works by encrypting the nonce, XORing that with the first 16 bytes of plaintext, then reencrypting the nonce and XORing that with the second block. This is another one you can implement entirly from the picture on Wikipedia.

OFB tends not to to be used a lot compared to other streaming modes, because unlike the next two streaming modes, you can't do random access. If you know some data is 16 * 500 bytes in, you have to encrypt the nonce 500 times to then decrypt that data.

Note that, for this mode and the rest, security depends on the nonce not being reused, but it doesn't matter on it being predictable. So if you have to encrypt a series of messages with OFB, you can just increment the nonces for each message and this would be more secure for a large number of messages because you'd avoid accidentally randomly getting the same nonce twice (see birthday problem).

Counter (CTR)

There are multiple ways of doing a couter mode of operation (and the next one, Galois/Counter Mode, does it differently). I was able to figure out how OpenSSL did it with the following code:

https://gist.github.com/calvinmetcalf/2bcb9d7e552a2fbdba08

Basically it takes the IV and encrypts it, XORs that with the first 16 bytes of the plaintext. Then it adds 1 (treating the IV as an unsigned 128-bit number and wrapping around) and does it again. It's dead simple, and allows random access to decrypt data. Although, you can't use the trick of adding one to the IV for each message to avoid accidentally repeating IVs. That being said, if all of your messages are less then 256 GB, then you can just get away with treating the IV as a list of 4 32-bit numbers and increment the 3rd one (instead of the 4th).

Galois/Counter Mode (GCM)

As suggested by the slash, GCM is actually two different things: it is a mode of operation in counter mode and a Galois message authenticator.

The counter mode in GCM is slightly different from how it is in CTR mode. Instead of just taking a 16-byte IV and incrementing it like a 128-bit number, it takes a 12-byte IV and then adds a 32-bit counter (starting at 2). In other words, it acts like CTR if all the IVS had to end in 0002. This is so you can encrypt subsequent messages with incrementing IVs.

GCM is not just a regular mode of operation, but an "authenticated" mode. This means that not only does it make sure that nobody can read your message, but it also makes sure that nobody has tampered with your message.

It does this by creating a Message Authentication Code (MAC). A MAC is similar to a hash, but requires a key to calculate it. You can also think of it as an encryption scheme that can't decrypt the cipher text it produces, which is always 16 bytes.

In addition to authenticating the text it encrypted, it also authenticates other data, such as metadata or the destination in a network request. For instance, authenticating the time a message was sent would be useful, as it would prevent somebody from taking that message and later resending it.

So GCM first takes a 16-byte block that is all zeroes, and then encrypts this and uses the result as the key for the MAC it calculates using the GHASH algorithm. It then appends 0002 to the end of the IV and encrypts the message just like CTR mode, except after it encrypts it it updates the GHASH with the encrypted data.

When the message is done, if it wasn't an even multiple of 16, the GHASH is updated with zeroes to get it up to an even multiple (although it doesn't pad with a full block if it was already an even multiple). Then the GHASH is updated with a block containing the length of the additional authenticated data and the length of the encrypted data as 64-bit numbers, and then the result of the GHASH is retrieved.

Remember how we started with the counter set to 2? We take the original IV and append 0001, encrypt that and XOR it with the result of the GHASH. This is our "tag" that somebody can send along with the encrypted message to verify that it has not been tampered with.

The GHASH algorithm involves taking that initial encrypted block of zeroes and calling it our key, then creating a 16-byte buffer filled with zeroes and calling it our state. When we update with a block, we XOR that block to the state and then create a new state by multiplying the current state to our key over a Galois field. This is kind of complicated; I don't 100% understand it myself, so I based mine on the nicely BSD-licensed version in the Stanford Javascript Crypto Library (note the homebrewed buffer object). If you really want to know how it works, check out section 6.3 of the spec.

The GHASH function used for message authentication is both the best and worst part of GCM. Apparently Galois field multiplication is very fast when implemented in hardware (supported by Intel chips) but not so fast in software.

Side note: I have a JavaScript library for Chacha20/poly1305, which means that if you are encrypting and decrypting on a computer powered by an Intel chip (i.e. not your phone) and you are using a low-level language, then it will be very fast. In all high-level languages (e.g. decrypting it in your browser) or computers with non-Intel chips (e.g. your ARM-powered phone), it will be much slower.

There are two other downsides of the GCM mode, the first being that it only works in Node 0.11 or higher. Lower version of Node can use HMACs, which are message authentication codes made out of a hash algorithm.

The last issue is that authenticated encryption doesn't really work with the Node streaming model of ciphers. When using GCM to decrypt, you have access to the decrypted data immediately, but don't know if it's valid until you get all of the ciphertext.

To work around those issues, here is a simple authenticated encryption mode that runs on multiple versions of Node and prevents you from using invalid data:

https://gist.github.com/calvinmetcalf/03516ed7ff7d7ff2572c

As for stream-friendly authenticated encryption, that is a whole different discussion for another day, but it seems to be an outstanding problem, I've been working on ideas in this library.

Other Modes

There is actually one more cipher that I didn't include called XTS. I didn't include it because I had trouble finding documentation on it, and what I could find about it mainly discussed how it wasn't very good. Feel free to open an issue if you can find documentation (specifically about how the tweaking works), or even better, send a pull request.

Wrapping Up

Overall if there is one thing I got out of this is that a block cipher is only about 10% of what you need to successfully build an encryption system. In order to take plaintext and a password/secret, turns it into cipher text, and allow it to be round-tripped, it also needs:

A method to come up with a key of the appropriate length and an IV. In addition to using algorithms like PBKDF2 and a salt to derive a key from a password, there are other methods of securely coming up with a key and IV. For instance, public key techniques like Diffie-Hellman and RSA avoid the need for a password entirely. Getting this wrong will cause a crypto system to be hackable; see the Debian key bug and WEP for systems where predictable keys led to hacking.

A mode of operation to make it actually useful for encrypting data. Picking the wrong one can lead to hacks like BEAST. Stream ciphers like RC4 (broken for other reasons), as well as ChaCha20 and Salsa20, avoid this problem entirely.

An authentication method for making sure the data that was received was also the data sent. Getting this wrong, e.g. by using or decrypting data before you authenticate it, usually does not cause breaches by itself, but instead opens up other breaches that would have been avoided if the validity of the data was checked earlier. For instance, the POODLE attack, which strictly speaking was caused by a bug with bad padding, wouldn't have been possible if the validity of the data had been checked.

I'd like to thank Nolan Lawson for proof reading this and Dominic Tarr for starting crypto-browserify (and putting up with all my pulls). If you've managed to stick around this long, feel free to ping me on Twitter if you find any errors. Also if you find any problems with browserify-aes, open an issue. Join me next time for Diffie-Hellman!

[cwmma]

No I didn't use the Web Crypto Api

Recently I've done a deep dive into Node crypto, including implementing a large portion of it in the browser for Browserify.

In future blog posts, I'll likely cover the intricacies involved in implementing getCipher, createDiffieHellmann, getSign/getVerify and getECDH. But to start out, I'll cover the question I get most often.

Did you use the Web Crypto API?

The answer is not really, only the random number generator. And I don't see myself using much more.

The number one reason is just that it is totally async, while the Node one is mostly synchronous. I did look into it for PBKDF2, which is async in Node and doesn't seem ready for production.

First off, when people talk about Web Crypto, they are talking about a couple APIs defined in this spec.

The random number generator (crypto.getRandomValues) is pretty widely implemented and easy to use; the rest of the API is namespaced under subtle because the methods it contains have "subtle" intricacies to avoid them being footguns. Or to quote the spec:

It is named SubtleCrypto to reflect the fact that many of these algorithms have subtle usage requirements in order to provide the required algorithmic security guarantees.

The SubtleCrypto API has very patchy support browser-wise. Firefox and Chrome both implement parts of it, but different parts, and it's hard to tell which ones. Internet Explorer implements it without using promises for the async aspects (bear in mind it is all async), so for all intents and purposes they may as well have implemented some other spec. As for Safari, who knows that the hell they are up to.

The last and biggest issue is that the API does so much, but also somehow very little. Take a look at the succinct syntax for PBKDF2 in Node:

https://gist.github.com/calvinmetcalf/37fe1d4fefe28c8ce98c

(In Node v0.11, the digest algorithm is optional and defaults to SHA1. In previous versions it couldn't be specified).

Now compare that to SubtleCrypto:

https://gist.github.com/calvinmetcalf/7bc5dd44770eb163e018

(Note that this only works on Firefox, and if you specify a different digest algorithm it explodes with a vague error.)

What is SubtleCrypto actually doing?

First we have to import our "key"; the arguments this takes are: what type of key we are importing (in this case raw, which means not actually a key), followed by the "raw key" (a.k.a. what we want to use with PBKDF2), followed by an "algorithm object" specifying what algorithm we want to use with the key once we import it. This is followed by a boolean specifying whether the key is exportable and a listing of methods that are approved to use the key, so if we wanted to use the key with another SubtleCrypto method we'd need to specify deriveKey instead of deriveBits for the allowed algorithm. It's async and returns a promise, and all arguments are required.

All this, to take a string and turn it into a form appropriate for PBKDF2, an algorithm that takes a string as input.

Then deriveBits takes a PBKDF2 object like the one you gave to importKey, except with a bunch more fields including iterations and salt, as well as digest identifier (not an algorithm object, but an algorithm identifier for some reason). In addition to the algorithm object, it then takes the same key from before and the output key length.

Fundamentally the API is built around key objects, which in theory can be managed by the user agent. The idea is that when somebody types a password into a form, the password can immediately get wrapped into a key object that is non-extractable and can only be used to derive a key or bits, or a plugin like LastPass could store passwords and provide them to the browser in that format. This will be all well and good when that actually happens, but in the meantime we have an absurdly complex spec with complex features, implemented with the hope that venders will implement stuff using it.

This feels like the IndexedDB API (and be aware that is not a compliment). IndexedDB has a maddeningly hard to pin down level of abstraction that is higher-level than something like LevelDB and lower-level then something like SQLite3, and in the end manages to be both extremely complex and extremely low-powered with a massive number of ways to use it wrong.

Things that would make the API much better

Here are a few improvements I would propose for SubtleCrypto:

A getAlgorithms method and an algorithmInfo method. The API is intentionally neutral about what it's going to support, and if I can't know what my browser should do from reading the spec, then that means I need to be able to figure it out from inside the browser. This includes other info, like which hash functions PBKDF2 and HMAC take.

While the key object idea is fantastic, it isn't actually useful now, and in theory may never be useful (think of that second parameter to history.pushState). Designing the API in a way that accepts complex key objects doesn't prevent the API from also accepting buffers or strings and doing sensible things with them.

There is no streaming interface for encryption. Even in the absence of implementing the actual stream spec, the ability to incrementally encrypt data is extremely useful and a feature of most encryption libraries.

The whole Heartbleed fiasco underscores the problems with a "subtle" approach to crypto. The mythical crypto experts that know what they are doing and can correctly configure an algorithm to be unexploitable are not the people setting up servers and running web apps. If there is a way to use a crypto library wrong, people will use it wrong. A strong argument could be made that what the web needs is something more along the lines of NaCL, a library that requires effort to use incorrectly.

Notes:

A big thanks to my company AppGeo; much of this research grew out of an effort we've been making to secure our apps.

Lastly if you're curious about my lack of posts recently, it's because I couldn't bring myself to bump the blog post about Kublai off the front. It's still not really ready, but I've got too many opinions to keep inside.

Edit: Spelling and grammer help from @nolanlawson.

[cwmma]

A Thousand Days of Kublai

We picked you because you licked our iced coffee. I said 'she likes what we like' and that's how we picked between you and your sister. Somehow we missed the (physical) thing that truly set you apart from your sister (who is named Zelda and eventually ended up in the care of Rachel's family), which is your thumbs thumbs.

I remember being so worried on the drive home with your carrier in my lap. I couldn't help thinking that we would get into a car accident and I would blow it, this responsibility. We made it home without incident but that drive remains one of if not the most nerve wracking drives of my life.

Rachel and I fell into roles that may or may not be an echo of things to come, where I am the one who spoiled you which Rachel was the one who gave you tough love.

I remember when you brought in your first bird, there were feathers everywhere, I didn't know whether to scold you or high five you.

The days when I got home before Rachel were always my favorite because you were always very affectionate to the first one home. You would jump into my lap and rub your head against my face. Sometimes you then curl up in my lap and take a nap. On weeks where Rachel worked late coming home to you was the highlight of my day. Some mornings I'd wake up and find that you had gotten between Rachel and I you would be cuddled up in my arms.

We got you a couple weeks after Rachel and I moved in together. You were always there for me when I needed someone to hug, and those first couple of months after we moved in together I needed a lot of hugs. I'm not sure we would have been able to make living together work if you hadn't been there, and no that isn't me just talking you up, I've been saying that for years. You can also take some of the credit of helping me quit smoking.

Rachel and I loved you so much, On your first birthday we threw a party for you and served crab cakes, you didn't like them, but had a fun time anyway.

You always liked to go outside and then come right back inside an hour or so later when it rained that could be a problem as you'd want to get in bed with us sopping wet. I'd often let you out about an hour before we got up and then let you back in when we got up. The joke was that you are our weather cat, because if you were wet when you came back in, it was raining, if you was cold, it was cold.

One time when it was snowing you were out for 14 hours, that was by far the longest you'd ever been out and when you came back you were warm and dry. We've always speculated you were cheating on us with another family that day.

On the night of the 5th I let her you at 3am, it was raining so I expected you to be back fairly soon. But when we woke up you hadn't come home. As the day progressed and you didn't come home things got very nerve wracking. When I finally went to bed that evening it suddenly hit me and I burst into tears for the first time in 18 years.

The next day we put up signs and that's when a neighbor told us that the morning of the 5th a tri-color cat was killed by a coyote very close to our house. Rachel went home and I talked to the neighbor a little longer. When I got home I could hear Rachel crying on the back steps from the street out front.

It wasn't definitely you, but it might be, the only way we'll ever know for sure one way or the other, is if we learn it isn't you by you coming home. We keep hope alive but everyday it becomes a little less likely that you are alive. It's been a month and you haven't come back, while for a part of me you will always be my Schrodinger's cat, I've had to accept that it is more likely then not that a coyote took you from me.

Accepting this has been one of the hardest things I have ever done in my life. I just have so much trouble accepting that you are no more, but even if it's true, you will always have a place in my heart until the day I die.

You were only on this world for a little over 1000 days. But I wouldn't trade that time for anything because while there is a piece missing from my heart right now, for 1000 days our apartment was a home because you were in it.

Goodbye Kublai Linso Metcalf 8/9/11-7/5/14 I miss you so much.

[cwmma]

Module: The Nitty Gritty

My last post covered some of the general import syntax of modules and today I'm going to go deeper into some of the features that tend not to be covered when people talk about modules. Similar to my last post I'm going to try to stick to facts in the first part and super biased opinions in the second part.

Bindings and what not

In JavaScript numbers and strings are references to specific values and are passed around by reference, in other words if you do:

var a = 3; var b = a; a++;

a is now equal to 4 and b is equal to 3, because a and b were both just references to the number 3, when you incremented a all that did was set a equal to 4, it didn't effect b as that was a separate reference to the number 3.

On the other hand objects (and thus arrays and functions) do not work that way so:

var a = {num:3} var b = a; a.num++;

now both a.num and b.num are both equal to 4. That is because a and b do not both refer to objects that have the value {num:3} but instead they refer to the same object that originally had a value of {num:3}.

In the ES6 module syntax if you want to export an object you can do it via

// lib.js export default { num: 3, increment: function () { this.num++; } }

If you then import it into multiple files they are just references to the same object so

// a.js import a from './lib.js' console.log(a.num); // 3 setTimeout(function (){ a.increment(); }, 1000);

and

// b.js import b from './lib.js' console.log(b.num); // 3 setTimeout(function (){ console.log(b.num); // 4 }, 2000);

The other thing to note is that if there was no increment method on the exported object, you would only get an error when you tried to use the method, a second into the script being run.

In ES6 you can also write the script as follows

// lib.js export let num = 3; export function increment () { num++; }

then

// a.js import {num, increment} from './lib.js' console.log(num); // 3 setTimeout(function (){ increment(); }, 1000);

and then

// b.js import {num as b} from './lib.js' console.log(b); // 3 setTimeout(function (){ console.log(b); // <-- what does this print }, 2000);

It prints 4, this is because what modules export are bindings. Put another way, you could think of 'num' and 'b' as both being references to an object, but the object has a value of 3 (and latter 4) instead of it being a set of key value pairs.

Static Analysis

The other thing to note is that if there was no increment function exported from lib.js then instead of getting an error 2 seconds into running module a.js you would get an error before a.js was run due to modules being statically analyzed before they are evaluated and then imports fetched before hand, in other words somewhat how browserify works (browserify actually works more like the first example as it would only throw a static error if you had an import for a module that didn't exist, but this is contrasted to node.js and unbundled AMD modules which only bother to look up whether a module exists when the code that imports it is run).

Circular dependencies

A tricky thing in any module system is how to deal with situations where 2 modules require each other. After some discussion with Guy Bedford I somewhat understand how ES6 deals with them. Basically function declerations hoist accross modules so this example (from Guy's es6-module-loader) would work:

even.js

import { odd } from './odd' export var counter = 0; export function even(n) { counter++; return n == 0 || odd(n - 1); }

odd.js

import { even } from './even'; export function odd(n) { return n != 0 && even(n - 1); }

but on the other hand

a.js;

export var obj = {}; import {something} from './b'; something();

b.js

import {obj} from './a'; obj.val = 'asdf'; export function something() {}

would throw an error because 'obj' wouldn't be a object yet when b runs because ... I'm just going to quote Guy here

The point here is that obj is bound by the time b executes, but the assignment has not been made as we still have to execute sources in some underlying order.

In other words you can think of them as being like regular JavaScript variables where the variable to be used is initialized at the top of the scope, but doesn't have a value until later on.

Commentary

Static Analysis

Lets look at the static analysis first, this has the benefit of preventing a whole class of errors from ever happening in the first place, namely importing something that doesn't exist but that you don't use until later. For example in node you could have a file that has

//lib.js exports.foo = function () {/*someting*/};

and then another with a typo

//app.js var lib = require('./lib'); function rarelyRun(){ lib.fooo(); }

You would have to actually run the rarelyRun function to spot that error in node.js, but in ES6 modules you this would cause an error the first time you tried to run it.

The downside of static analysis isn't actually anything to do with the static analysis but what is done in the name of static analysis, namely you can't put import (or export) statements anywhere except top level code, i.e. no

var bar, baz; if (foo) { import {_bar, _baz} from 'bat'; bar = _bar; baz = _baz; } else { bar = 'bar'; baz = 'baz'; }

Putting statically analyzed imports inside conditionals is less stupid that it seems at first glance as it makes writing polyglot programs that can run in different environments possible. For instance, it's common to see something like the following in the wild:

var myThing = something; if (typeof define === 'function' && define.amd) { define(function () { return myThing; }); } else if (typeof exports === 'object') { module.exports = myThing; } else { window.myThing = myThing; }

this exports your module as an AMD one if that exists, a Common JS one if that exists and otherwise assumes you are in a regular browser and attaches it to the window. Creating things like this is impossible in ES6 modules.

Circular References

This is a step back compared to Node.js, remember this

a.js;

export var obj = {}; import {something} from './b'; something();

b.js

import {obj} from './a'; obj.val = 'asdf'; export function something() {}

well the node.js version of this

a.js

exports.obj = {}; var b = require(./'b'); b.something();

b.js

var a = require('a'); a.obj.val = 'asdf'; exports.something = function() {};

would work fine. Why this works is best stated in the require.js documentations

[...] use exports to create an empty object for the module that is available immediately for reference by other modules. By doing this on both sides of a circular dependency, you can then safely hold on to the the other module.

The ES6 version seems like a step back compared to CJS/AMD as it doesn't cover as many of the edge cases.

Mutable Binding

The ability to import a name, from a module, and have it look like a global variable while at the same time changing like an object property is the killer ES6 module feature nobody asked for, nobody needs, and nobody wants.

[cwmma]

Module Drama Recap

A recent proposal to change ES6 modules slightly has unleashed a torrent of drama. What follows is a recap of the ES6 module system, followed by some background, followed by a parable, followed by my opinions.

ES6 Modules

The original module system for ES6 involved modules being able to export names via either putting export in front of declarations

export var/let/const foo = bar; export function foo(){}

or with export statments

function foo(){} export {foo} export {foo as bar}

these names could then be imported into another module

import {foo, bar as baz} from 'path'

if you needed to import an object with all of the names you could use

module name from 'path'

much work was put into making sure that circular dependencies and other edge cases were worked out leading to the restriction that import, module, and export statements must be at the top level so that they will always run, so no

if (something) { export let foo = bar }

There was also originally an ability to create inline modules via

Module name { // stuff }

but this was removed on the grounds that it was too complex for now and can be added later.

A major issue with this setup was that there are no anonymous exports, it is very common in node.js and require.js systems to have a module be a single function or an instance of a class, but with the original setup you would have to give a name to that single thing. You see this in python with super ugly stuff like "the foo module which exports a foo member" but just for fun also stuff like "the bar module which exports a Bar member" and leads to usage where people just use foo.foo and bar.Bar.

The last major change to modules before recently was the addition of 'default exports' which added the ability to export something without giving it a name via

export default function (){} //or function foo() {} export default foo

which is actually a more flexible syntax then the export from syntax which can only take a name

// good function foo(){} export {foo as bar} // bad export {function (){} as bar}

on that note the other thing you can't do is

// bad function foo(){} export foo

but I digress these default exports could be imported with

import foo from 'path'

this was done about a year ago and always seemed to be as a bit of a stop gap as it left the module system in a somewhat inconsistent place where different ways of exporting functions meant they had to be imported differently, for instance with underscore which is a collection of methods, to get it's map method you can either do

module _ from 'underscore' _.map();

or

import {map} from 'underscore' import {map as _map} from 'underscore'

though you can't do

//BAD let _ = {}; import {map as _.map} from 'underscore'

now on the other hand for jquery which is also a collection of methods, but are attached to a function you would have to do

import $ from 'jquery' $.map();

because if you tried to do

module $ from 'jquery'

map would actually be at

$['default'].map()

the solution proposed by TC39 is to remove the module/from syntax so that you either use a default import or you import specific methods and if you need all the full module object do something like

import 'foo' let foo = this.get('foo')

if you can't tell the idea seems to be that you would need this very often.

This change has spawned one of the most contentious threads on the es-discuss mailing list since somebody pointed out that Promise.resolve and Promise.cast were functionally identical. To understand why you have to remember that these aren't the only JavaScript modules.

A lengthy digression on AMD and Common JS modules.

The JavaScript community currently uses 2 module systems, AMD and Common JS (CJS), while some people describe them as 'competing module systems' those people are wrong. The CJS and AMD are for all intents and purposes the same module system used by 2 different groups. The major difference between the 2 systems is that CJS modules are individual files while AMD are JavaScript objects, a CJS module wrapped in

define(function (require, exports, module) { ... });

is an AMD module and while AMD modules do have some ways of importing and exporting code that CJS modules does not have, the trend has been (in my biased experience) away from using them.

These days when people talk about a AMD/CJS split they are usually meaning browserify/webpack/npm vs r.js/cujo.js/bower split.

The important thing to remember is that since inline modules were removed from the spec, the things that the ES6 spec is covering are the things that the AMD/CJS communities agree on, namely syntax and that in code a module is just an object.

Another Tangent

A coder spends his free time writing a module to download cat pictures, he publishes it to NPM. A user opens an issue about the possibility of using it to download dog pictures or maybe even generic mammal pictures. The coder feels that the user does not understand the purpose of this module and instead of adding the feature, or even just dismissively pointing out that pull requests are accepted, the coder argues with the user about whether the user really wants this.

This is a bad habit that many of us occasionally suffer from. The reason for it is that when you put a lot of work into writing something, its easy to to take people trying to offer suggests to your project as people attacking your code baby. If you publish your project then you're no longer the only user of it and other people have different needs than you and need different features then you, this is natural.

My opinions

With modules I feel like TC39 is suffering from this syndrome as well. Instead of taking criticism as suggesting a need to work towards adapting the system based on the experience learned from actual JavaScript developers using modules in the wild, they seem to be taking criticism as something they need to defend against and this makes the node.js community feel like TC39 doesn't care about them, because when they point out that they've learned a thing or 2 about modules in the past several years, TC39 seems to say 'we don't care'.

People put a lot of hard work into making the module system, but having an inflexible take it or leave it approach does nobody any good. Modules have yet to be implemented by any engine so if comprehension, which firefox already implemented, can be taken out of the spec, then there is no reason that anything should be off the table regarding modules.

[cwmma]

Making Promises Fast

As alluded to in my last post there were some performance killing bugs I had to fix in my promise library lie, through the majority of the issues were actually with immediate, the scheduling library lie uses.

Lie

The only perf oriented change I made to Lie was to make Promise.all speed up by adding the following line to Promise.resolve

if (value instanceof Promise) { return value; }

basically i forgot about step 2 of the spec which allows Promise.resolve to simply return the promise if it was created by this constructor, with that not there I was treating all promises as untrusted thenables, which meant I was going through the somewhat time consuming unwrapping procedure an extra time. This sped up Promise.all because all values that are passed to Promise.all go through Promise.resolve so I was doubling my work unwrapping them twice.

Immediate

The idea behind immediate is that every time you call immediate(func) if this is the only thing in the queue we schedule a drain to happen async but very soon (sooner then setTimeout which has a min of 4ms). When the drain happens it goes through and calls each of the functions in the queue, the tricky thing is that if immediate is called by function while the queue is draining, it needs to be appended to the current queue not scheduled for the next one.

e.g. if you have

function a() { immediate(b); } function b() { setTimeout(c, 5); }

then it would run go

async wait

a

b

5ms wait

async wait

c

So the first reason it was slow was because I implemented a very naive double ended queue via .shift

function drainQueue() { var task; while ((task = handlerQueue.shift())) { task(); } }

so this is an awful way to do this because every time you call shift it moves all of the items every single time. This is not efficient, I replaced it with

function drainQueue() { var task; var i = -1; while ((task = handlerQueue[++i])) { task(); } handlerQueue = []; }

making this change meant that Lie could actually finish the parallel tests, doing so in 9548ms this is pretty slow, one of the reasons being that drainQueue will always try to access an item past the end of the array which can trigger various deoptimizations, switching the code to

function drainQueue() { var i = -1; while (++i < handlerQueue.length); { handlerQueue[i](); } handlerQueue = []; }

fixes that getting the speed to somewhere between 2829 and 8465, basically there is a race condition that sometimes super slows down the code. The issue was that v8 optimizes functions that are 'hot' by compiling them with type info to native code. If a type changes then the function gets deoptimized and the compiled code is discarded. Since compiling the function is not free we want to avoid having to throw out the result. The problem with the previous queue was that the array could grow so long that the i variable can grow big enough that it is no longer a small integer and deoptimizing the function, changing the code to the following fixes that.

function drainQueue() { draining = true; var i, newQueue; var len = handlerQueue.length; while (len) { newQueue = handlerQueue; handlerQueue = []; i = -1; while (++i < len) { newQueue[i](); } len = handlerQueue.length; } draining = false; }

this brings the time down to 2490ms, making lie the 3rd fastest of the 6 we looked at yesterday. Likely at some point in the future I'm going to want to do what asap, bluebird, and when do and implement a circular buffer, but some tests with the implementation that when uses suggest that that isn't a chock point at the moment.

Back to Lie

Right before I was going to publish this I wondered, what if we double the amount of parallel things we do?

By default the tests do an array of 25 promises resolved by all, but what happens if we double it to 50. At worst the amount of time should double right?

No, for lie the amount of time increased 10 fold ~20,000ms, through it could have been worse, RSVP runs out of memory and crashes.

the problem turned out to be the function I used to make sure the handler's were only called once

function once() { var called = 0; return function wrapper(wrappedFunction) { return function () { if (called++) { return; } wrappedFunction.apply(this, arguments); }; }; }

true to it's name once was only used in one place and I was able to replace

var onceWrapper = once(); var onError = onceWrapper(function (value) { handlers.reject(self, value); }); var onSuccess = onceWrapper(function (value) { handlers.resolve(self, value); });

with

var called = false; function onError(value) { if (called) { return; } called = true; handlers.reject(self, value); } function onSuccess(value) { if (called) { return; } called = true; handlers.resolve(self, value); }

this causes lie to go from taking 20,000 ms to taking 3,000ms and bringing the original version with 25 parallel things down to 1825ms. This is faster then when.js.

Conclusion

Use version 2.7.4 of lie, it's super fast.

[cwmma]

Comparing promise libraries

Today we're going to compare promise libraries, this isn't about finding the best library as such a thing does not exist, all of the libraries make different trade offs to try to be good for certain things.

So disclosure: I'm going to try to be as unbiased as possible but I did write one of them (lie) so if nothing else, that library is going to be optimized for the same trade offs I find important.

There are a lot of spec compliant promise implementations, my criteria for picking these are

I must have heard of it.

It must be used in the wild.

It must have ~ES6 syntax (take a resolver function to create a promise at the minimum).

Provide microtasks in a reasonable manner (aka avoid setTimeout in modern browsers).

Be aplus spec compliant.

The libraries I chose were

bluebird

lie

Q

RSVP

then/promise

when

And these broadly fall into 2 categories:

bluebird, Q, and when are tool kits that in addition to providing promises also provide a plethora of helper functions for dealing with promises in the footsteps of the venerable async library.

lie, RSVP and then/promise are basic promise libraries which stick close to what the ES6 promise object provides, namely they have static resolve, reject, and all methods and their prototype has .then and .catch methods.

I had a bunch of comparisons I was going to do but then I found a nasty bug in Lie and spent a bunch of time on that so I'll just cut the my findings.

My benchmark results are here If you want to do the benchmarks yourself just download the bluebird repo and run them there, the size comparisons are here.

Bluebird

Bluebird is the fastest by far, through the speed should be taken with a grain of salt as the bench marking tests were written by the guy who wrote bluebird written by somebody else originally but the live in the bluebird repo, that being said they are the best promise speed tests (as opposed to latency) that I can find so that's what we got.

Bluebird is approximately equal in size to the other 5 libraries combined (15k minified and gzipped) , so my recommendation would be to without a doubt use it in Node and if you don't have to be hyper concerned about download size use it on your client size App. Client size libraries are where I tend to be most concerned about size and I would avoid using bluebird there, if your library works in node and the browser you can use browserify to set up an automatic substitution that replaces bluebird with something smaller on the client side while using it in node, it's what I do in a couple libraries.

When

When was pretty consistently the second fastest of the libraries but at less then a third of the size of bluebird when is by far the best choice for promise tool belts in size contentious situations (i.e. client side libraries). Plus a couple of the guys who write it are Boston based like me so extra credit.

Q

Q is an order of magnitude slower then all the other libraries. I really can't recommend using it in production, which is sad because it is the most popular of all of these (I've seen articles using Q and promises interchangeably). Q is the only one of these libraries to that does not implement version 1.1 of the A+ promise spec, implementing the 1.0 version. The 1.1 test suite is deviously thorough and the lack of conformity at the moment is another reason to avoid Q.

RSVP

RSVP is a bit of a strange case, unlike the previous 3 libraries the only extra method it has is a 'hash' method which is like all but for objects, (edit: as pointed out on twitter, it also has finally) but at the same time it's more similar in size to Q and When (~4k minified and gzipped) and has some other features like uncaught promise catching and long stack traces that tend to be in the tool kit libraries, its performance is a a bit strange in that it is very bad in version 0.10.28 of node but in 0.11.13 it (and then-promise) are both significantly faster making me suspect some new optimization in v8 that is helping it. Though even in 0.11.13 it's performance isn't that good and if I was picking a 4k library I'd go for when which is faster and has more features. That being said it's bundled with ember so if you're in an ember app or something else that already includes it it's a fine choice (and now works in web workers)

Then-Promise

This library is the first of smaller ones aiming to be drop in replacements for ES6 spec promises in order to make it easy for libraries to return promises. That being said it does offer a few nonstandard methods. Size and performance wise it's pretty similar to Lie so while I'd recommend my own library this is an equally good choice.

Lie

My promise library which like then-promise is a minimal one, it's performance is now the same as then-promise but when doing this article I had to fix a performance killing bug (which will probably be the subject of another blog post). Overall it's size is pretty similar to then-promise (200 bytes larger then then-promise when gzipped and minified), its performance is similar (faster in certain situations slower in others) and while it doesn't offer any helper functions it does use low latency async techniques on a wider set of platforms via immediate.

Honestly I use this one for my stuff and we use it on pouchdb for the browser but you'd do well with either this or then-promise.

Edit: see my next post, I ended up making lie considerably faster and can now say that it is faster then then-promise and you should use it.

Wrap up

I'm somewhat concerned about how badly Q did, if there is an issue with the benchmarks I would be more then happy to rerun them after a fix goes through. Also in the interest of fairness as I did wait until after I fixed a bug in my own library if any of the other libraries put through speed improvements I can rerun these tests.

[cwmma]

NPM as a build system

EDIT: In case I wasn't clear enough, when you use npm to run a script with npm run foo or npm test it makes sure all of the local modules are in your path, so if you only have uglify and mocha installed locally, you can still write "test": "uglifyjs ./lib && mocha test/*.js"

The reasons that you often don't need grunt have been done to death, so I don't feel any need to lend my voice except to agree. I've also seen Make thrown around as another alternative. Make is a fine program on *nix systems but for node.js it's missing a copule things,

it is global, you have to specify ./node_modules/uglify-js/bin/uglifyjs instead of just uglifyjs.

it is a pain to use on windows.

Now before we turn this into a windows/*nix flame war let me point out, I do not develop on windows, I despise developing on windows, the only time I ever voluntarily touch windows machines is when I'm helping my fiance fix her laptop. But that being said I work with people who use windows and I'd like to continue working with them as I have no interest in doing the heroic work they do with our legacy .NET apps.

Many people starting out with node are also on windows. While we may later be able to seduce them over to the *nix side, for now they are learning on the machine they have.

So back to your build step needing to work on windows, keep in mind not every node project on windows is simply going to get deployed to a linux box so simply using vagrant is not the answer. I've used IIS node to replace pieces of old asp apps, and I've used node to compile static assets that are then pushed somewhere.

And yes it is theoretically possible to get make installed on windows, but not easy, most of the suggestions I've seen either contain sourceforge links or references to Cygwin. These are not things that sane people want to do.

What instead?

For the 99% of projects that are a build step and some tests, you can use NPM as a build system, just put them in your package.json, when you run the script that way, npm makes sure your path has your node_module .bin folder is in the paths so if you use mocha to test you can just add

"test": "mocha -R nyan test/test.js"

to the scripts object in your package.json, and it will run tests on all your files that start with test. and are in the test folder, if you like tape you can do

"test": "node test/test.js | tspec"

jshinting can be done with

"lint": "jshint lib/*",

or you can combine it with your testing setup

"test": "jshint lib/* && node test/test.js | tspec"

both of these will look to .jshintrc for the rules, no need to specify them in a config.

karma can be run this way easily and testling can as well with browserify

"test": "browserify test/tests.js | testling"

For building browserify excels at using a package.json script, but if you must use [require.js] instead of putting the overly verbose and totally unnecessary 30 lines of config options to do something simple in with the build stuff, you can put it in a build.js file and then go

"build": "r.js -o build.js"

but if you don't like having to specify:

https://gist.github.com/calvinmetcalf/9837692

you can just have a package.json with:

https://gist.github.com/calvinmetcalf/9837700

(I'll get to the syntax in a minute) though to be fair since I often have a couple steps here I usually just separate it out into a couple steps:

https://gist.github.com/calvinmetcalf/9837726

with browserify you can set specify other things in the package.json, for instance the "browser" key defines a map where you can exclude or substitute things from the build, when you specify the substitutions here instead of in the grunt file then other people browserifing a library that uses your lib as a dep also get the substitution applied, in other words, you need to be doing this anyway, why do it twice?

browserify-shim is a very helpful library in this respect (so helpful, grunt-browserify includes it but fails to mention it causing a confusing blending of the options in the docs).

If things get too busy we can just launch a batch script with

"build": "./build.sh"

over in build.sh same rules apply, it will work in windows if you stick to node commands (and git) you work cross platform, you can also call node scripts aka

"build": "node build.js"

in the some the examples I used | and && to separate commands on a single line and used > to out put a file. In shell commands && means the same thing as in JavaScript, it's a lazy and operator so command1 && command2 means run command1 and if it doesn't fail, run command2.

The other commands related to pipes/stdin/stdout. Commands can read from stdin and write to stdout and in node those are both streams, available at process.stdin and process.stdout, though another thing that outputs to stdout in node is console.log. By default unless it is piped anywhere your console will print stdout and thats why programs like browserify will print your bundle on the screen if you forget the -o option.

The | command (pipe) takes the output from one commands stdin and pipes it to the next commands stdin, so when you want to browserify a file and give it to testling, you can do

browserify . | testling

which takes the output of browserifing the current directory (the dot) and pipes it to testling.

or if you want to take the ugly tap output from tape and make it look nice, use

node test/test.js | tspec

which pipes it to tap-spec, or you could use

node test/test.js | tnyan

to pipe it to tap-nyan, because testing should be fun.

the > command means write stdout to a file, so you'll sometimes see people write

browserify . > bundle.js

which means write the bundle to bundle.js (yes I know you could also write -o)

then can be combined with pipes as well so to browserify and minify you can write

browserify . | uglifyjs -mc > dist/script.js

which browserifies it and then pipes the output to uglifyjs to minify it with the mangle and compress option.

N.B. uglifyjs is the command line program, but it's in NPM as uglify-js so you need to do npm install --save-dev uglify-js

The < operator reads the specified file as stdin so the following, while stupid, would work

browserify . > temp.js && uglifyjs -mc < temp.js > dist/script.js

in which we write our browserify output to a tempfile and then read it into stdin for uglify, a non contrived example would involve something like istanbul which always writes code coverage to a file and prints test results to stdout, so if you wanted to run coverage on a mocha script and then pipe it to coveralls then

istanbul cover ./node_modules/mocha/bin/_mocha --report lcovonly -- tests/*.mocha.js -R spec -t 5000 && coveralls < ./coverage/lcov.info

would do it

N.B. ./node_modules/mocha/bin/_mocha is the path to the real mocha binary, the mocha in your path (that we would get if we ran mocha) is a command line parser that will prevent istanbul from working right.

You will sometimes see people writing the above as

istanbul ... spec -t 5000 && cat ./coverage/lcov.info || coveralls

those people are wrong, cat is for concatenating files, not for reading them into stdout for you because cat isn't going to be available on windows.

You want to watch your bundle and have it rebuild when you change anything, try watchify

"watch": "watchify . -o ./dist/bundle"

note you need to specify the output folder, you can't use > because ... it throws an error for some reason.

WINDOWS!

There is no rm -rf so instead use rimraf which will remove a directory and all the stuff in it recursively so from our previous example

rimraf ./foo

will remove ./foo, and ./foo/bar, and ./foo/bar/baz and anything that might be in them.

N.B. Don't be an idiot with rimraf, if your careless you can delete everything, especially on windows.

To do batch copies ala grunt-copy you can use copyfiles which I wrote expressly as a replacement for grunt-copy, so to move all the JavaScript files from the vendor and src folder to the dist

copyfiles ./vendor/*.js ./src/*.js dist

but if you want them with out the outer vendor and src folder

copyfiles -f ./vendor/*.js ./src/*.js dist

if if you want to do a globstar

copyfiles -f './vendor/**/*.js' './src/**/*.js' dist

for operating system consistency you need the quotes around globstars, and for once that's not windows that is the problem, it's OSX with its out of date bash.

Conclusion

You probably don't need to use any build system for many node projects because npm can act as a build system and for the love of god don't use make because that's practically holding up a sign telling .NET programers to continue writing .NET program. We want them publishing all the things to NPM.

discuss on reddit

Other articles on the subject

Running Scripts with npm

Task automation with npm run

[cwmma]

JavaScript style mistakes I see a lot

Edit: removed recursive reference, it was incorrect (my bad) but you should probably avoid recursive references in general.

Edit2: discuss on reddit

Edit3: forgot hoisting was such a hot button issue, switching to the hopefully less controversial statement that declarations should be your default

A quick run down of a few very common JavaScript style issues, that while aren't outright errors you should probably avoid.

Function expressions, not on objects

This is when you write

var name = function () {};

instead of

function name(){}

the bottom version should be your default which you should use unless you have a reason to do it the other way. Some of the benefits include

hoisting (though you might passionately disagree)

reference to the functions name inside the function can't be changed.

because you have to pick one for consistency and I've never seen project choose the top version.

using module.exports, to export an object

That is when you write

module.exports = {foo: function(){}, bar: function(){}}

instead of

exports.foo = function (){};

exports.bar = function (){};

the difference is that the bottom syntax can handle circular dependencies, and the top can't, they are otherwise identical.

making me guess your exports key.

that is if you only export one thing from your module, doing

exports.someRandomName = thing;

as opposed to

module.exports = thing

unless you have circular dependency you have to deal with, making me write

var yourThing = require('yourModule').someRandomName;

is going to cause me to judge you.

[cwmma]

PouchDB 2.0.0

PouchDB 2.0.0 is out and has a lot of great stuff, first off you'll notice breaking changes they are:

allDbs has died in a fire. This optional disabled by default feature complicated the db creation process to no end and was the source of many race conditions in testing. If you were one of the 3 people using it fear not, while it was previously impossible to get this behavior otherwise it now is due to another new feature.

PouchDB now uses prototype for inheritance, no more of that copying methods from object to object. This does mean that var get = db.get will blow up, so write var get = db.get.bind(db) instead.

plugin now take advantage of prototype, meaning you can add them after you create a database.

For indexedDB and webSQL adapters the schema has been updated to give correct doc counts, your db should migrate fine,

for the leveldb adapter we switched from being a folder with 4 leveldb and an annoying '.uuid' file it is a single leveldb with 4 sublevels and no '.uuid' file. This also will transfer your data over the first time you open an old style one in the new.

these last 2 mean that v1 PouchDBs will open in v2 but not the other way around. Also v1 ones will become v2 ones as soon as they are opened in the new release so don't switch back and forth.

Non breaking changes:

leveldb based PouchDB creation is now totally async with no fs.

all the methods return promises (except the replication ones). This is in addition to taking callbacks. If you hate promises then the only change you'll notice is that we more consistently return errors instead of throwing. We use the superfast bluebird in node and in the browser we default to native promises and fallback to the supersmall lie.

it works in web worker now, including in chrome where you have access to indexedDB inside a worker.

we prevent overly enthusiastic caches in IE from preventing requests from reaching CouchDB

we no longer screw around with your options object so you can re use it for multiple calls.

if you pass emit as the second argument to your temp query function, map reduce no longer evals it.

a whole host of things to make replication much faster

a whole lot of tweaks to the webSQL adapter thanks to Nolan.

the PouchDB constructor is now an event emitter and you can listen for create and destroy events.

If somebody has shat all over your JavaScript environment by adding enumerable properties to Object.prototype, PouchDB has you covered.

Put sugar you can specify _id and _rev as the 2nd and 3rd arguments to put, i..e. instead of db.put({some: 'stuff', _id: 'myId', _rev: '1-rev'}), you can do, db.put({some: 'stuff', 'myId', '1-rev'), when you are creating a document you can omit the rev (though if you omit the _id and do include the rev you'll create a document with that rev).

Some fixes that make developing much easier include

qUnit has been totally replaced with mocha and chai, if you've ever had the misfortune of using qUnit in node or with async tests or both you'll know why this is so awesome.

all the tmp test dbs are in their own folder which is deleted afterwards, no more mess after failed tests.

Grab it from npm, component or bower, or download the regular or minified version.

[cwmma]

Dear TC-39, can you please take another look at your module syntax

Dear TC-39,

You've been doing some fantastic work recently, all the new features slated for ES6 look amazing. Arrow functions and the functional parameters sugar do such a good job at making functions just that much more fun to use. Set, Map, WeakMap great and really going to help for data centric apps (like all the GeoSpatial stuff I do). Generators, well I'm already using them and they are fantastic. Promises are already in browsers, that's so fast it's unbelievable. All these great things you've done make this just so much harder. But the module syntax you folks are proposing for ES6, I think you should take a fresh look at it.

Most of the discussion we've had on the module system has been framed poorly, it has been people like me who it rubs the wrong way attacking it and people like you who spent a lot of time developing it defending it. But really we're all on the same side, we want the best possible module system that we are able to actually get into browsers in a realistic time frame. More specifically a module system that does static analysis (finds the dependencies without running the code), so that we can avoid the significantly added layer of complexity that comes with dynamically loading modules.

I originally was going to write a while thing about the flaws in the system you guys are proposing, I even made this flow chart to show how complicated the export syntax is and a whole list of links to code that wouldn't be writable in the new syntax. But then it occurred to me that it is hard to tell if you guys writing the spec really even think this syntax is any better then what is currently out there, because if you do your doing a very bad job of selling it to us in the node community. Static analysis of dependencies and imports with browserify or the CommonJS wrapper in require.js (define(function(require, exports, module){...});) show that in practice there is no need for restrictions on the location of imports to successfully find them in static analysis and people trying to conditionally load modules in a situation where they can't isn't a problem in the wild.

There are sources which describe ES6 modules as a compromise between CommonJS and AMD modules, but again that isn't really the case any more as using the CommonJS wrapper has become more prevalent in AMD code to the point that most debates about CommonJS vs AMD have much more to do with tooling and deployment strategy. In many ways ES6 modules are redefining the one thing about modules that the AMD and CommonJS communities agree on, syntax.

Which brings us to the root issue, TC-39 hasn't sold the node and AMD community on their new modules system. Coming from node and AMD systems this one is more complicated with significantly less functionality, specifically

Lack of import statements (as opposed to declarations).

Syntax that is still geared towards exporting a bag of values as opposed to a thing.

Generally high level of complexity of import export syntax.

The people in the node community giving you grief about this syntax aren't just bikeshedding but are people building production apps with a module system that they like and you haven't convinced them that you have something better. And that's just sad because the node community is your biggest fans. We love generators, we are using block scoping in the wild, and have some very strong feelings about Promises, much of which is positive.

TC-39 that's what I'm asking you, take a hard look at your module syntax and ask yourself

Are we taking advantage of the experience that the community has gained building production systems with CommonJS and AMD?

Are we taking advantage of the examples offered by more modern build tools like browserify and require.js?

Are there real and non-hypothetical advantages gained from restricting imports to declarations?

Have we taken into account other features of ES6 like destructuring in designing these modules?

If you weren't involved in the spec writing process, would the export syntax make as much sense?

From an outside perspective it feels like TC-39 is suffering from Not Invented Here syndrome ,if committee members don't think that this is the case then I think a lot of people would appreciate it if you guys tried to sell us on the advantages of this syntax as compared to current tools if members don't feel that way I don't think I'm the only one who would prefer modules done right in ES7 to settling in ES6.

Edit:

Comments on reddit

Also in case it's not clear, I'm not advocating to just drop in node modules into the JavaScript spec, just it would be nice to have a similar level of features.

[cwmma]

Allow me to reintroduce you to Proj4js

tl;dr Proj4js used to be really bad, now it's better and you should use it and help build it.

Background

A while ago I decided the write a program to parse shapefiles in the browser into geojson. Despite the shapefile's reputation as as perversely bizarre format (it's binary and mixed endian) the part that ended up being the trickiest had nothing to do with the file itself. The problem was with the communities that produce shapefiles and consume geojson. Geojson almost exclusively records coordinates in unprojected latitude, longitude notation (WGS84 to be exact, though somebody did give me NAD83 once). Shapefiles tend to store coordinates in a projected coordinate references system.

The problem I faced was how could I get shape files in any of the thousands of projections out there to just work. There was a program called Proj4js, but it was in a sorry state, the math was all there, but the interface was cumbersome to say the least.

I (and quite a few others) have spent quite a bit of time fixing it up and I want to reintroduce you to this very helpful library.

Getting it

You can install it with npm, bower, component, jam or just stick the script in your browser. It's build in browserify and exports a UMD that should work in any module system (Dojo might give you some trouble still, but that is an upstream problem fixed in browserify which will be fixed next time you build).

Usage

The global it exports if you drop in the script is proj4 (as is the name in npm, bower, and jam package but component is proj4js/proj4js as it goes by repo) and proj4 is a function which will be all you need 90% of the time. You use it like so

proj4(fromProj, toProj, coords);

It's that simple, a few notes:

From and to proj can be either wkt strings or proj strings, note the lack of need to to create a new Proj object or anything.

Ihe coordinates can be in 2 formats, [x,y] or {x:x, y:y} if you pass in an array you get one back, if you pass in an object, you get that back instead. No point class needed.

If you only give one projection then it's assumed you want to project from wgs84

If you don't pass in a coordinate you can an object with forward and inverse methods

You can't just pass in an EPGS code, there is no realistic way to store all of them in a easy way. Most of craziness of the old version related to trying to let people use arbitrary epgs codes.

So if you want to convert a latlng to a projected coordinate,

var newCoords = proj4(wkt/projString, latlng);

If you want to avoid having to reparse the string each time

var trans = proj4(wkt/projString);

var newCoords = trans.forward(latlng);

var newLatLng = trans.inverse(xy);

Bottom line

Proj4 works for every projection I've been able to test against, find a problem, open an issue, or even better, open a pull (the code is still ugly, but not as bad as it used to be thanks to browserify).

[cwmma]

Guide to arguing about promises

If your using JavaScript you may have noticed that opinions about promises are much like hands, most people have one, many people have several. Here is your cheat sheet so you can pretend to care while just using the version that has shipped in Chrome and Firefox.

Promises vs Callbacks

The original argument, centering around what the default API in node should be. This is actually a rehash of an old argument and promises never had a chance. They weren’t performant the first time and by the time they were the ship had sailed on the API.

Bottom Line, always accept a callback, though feel free to return a promise too.

errors, foot guns , and jQuery

Using throw in a synchronous program when you have a problem is like sending up a flare, you get a stack trace, you can pause your script in dev tools, it’s great.

Using throw in async code is like sending up a grenade, you have no idea where it’s going to hit and it’s probably going to be unhelpful at best.

The mainstream promise implementation catches all errors and passes them to the second function you give to then. It also has a catch method which is sugar for then(null, something). Many people believe that this is bad because it means that unless you explicitly handle an error it gets silenced. Especially if the error is in your error handling code.

JQuery famously subscribes to this and insists on waiting until 2 browsers ship the other kind of promises before even starting to change their promises.

A compromise suggestion thrown around is a done method that rethrows errors. But you’re still requiring explicit action for errors to not be swallowed and we are still throwing asynchronously.

Most importantly most of this ignores the fact that in async code it’s often OK to ignore errors because an error can handle itself often by the call back not being called and while the current setup is probably not the best it is likely the least worst.

Bottom Line: you can’t remove all the foot guns from programing because sometimes you need to rocket jump.

monad and chain

Promises have had resolve and reject methods to create already fulfilled promises for quite a while. The spec added a cast method which I always assumed was meant to coerce jQuery promises to real promises.

Somebody on es-discuss pointed out that resolve and cast do essentially the same thing. Like 50 comments latter I bothered to read the thread and what emerges us a group unhappy with the current spec.

Currently then is never called with a promise, always a value. In other words in the current model async is like pregnancy, all of nothing. This is a benefit of promises because it let's you deal with nested async code in a much simpler manner.

There is another promise method proposed called chain which unwraps a promise but only one of one layer. In other words if you fulfilled a promise with a promise then would give you the value but chain would give you the inner promise.

An argued use case for this (bear in mind I'm writing this on my cell phone while waiting for laundry so this is entirely from memory) is for when you want to act differently based on whether something contains a promise or a value, like a storage object that returns promises but can store values or promises. You might want to act differently if what you get back (inside the promise) is a promise also, which you can't do with just then.

The main issue with this is its trying to treat promises as a little bit async. Which misses the point the current implementation. Where you would treat all promises you got back from the storage thing as similarly async and get in with your life.

The reason that cast/resolve relates is that while they act the same for then they would act differently for chain. But in all honesty this argument relates more to the long running monad argument which boils down to whether it is a good idea for promises to be implemented as a concept that nobody who understands is able to explain or is a simpler model for them better in the long run.

Bottom Line: do you care that promises aren't monads?

[cwmma]

Block Scoping JavaScript: no Automatic Curly Bracket Insertion

Riddle me this JavaScript user, when you write

if(something) doStuff();

what is that equivalent to? If you're like me you guess that much like how semicolons are inserted by the parser, curly brackets are inserted as well, meaning that the above statement would be equivalent to

if(something){doStuff();}

But no, from this recent thread on esdiscuss it would seem that the above statement is equivalent to

void(something && doStuff());

Now currently these 2 methods of parsing the statement (for the purposes of our example) have identical results, but with ES6 and the block scoped variables there will be a difference,

if(something){let x = 5;}

is not the same as

void(something && let x = 5);//not valid

If for no other reason that the lower statement isn't valid as let x = 5; is a declaration not of a statement, but hopefully you get my point that

if(something) let x = 5;

Would declare x in the outer scope and not do nothing like one might expect.

While this might be a bit confusing at first what it really means is that, with a few exceptions, block scope and curly braces are synonymous and will always do exactly what they look like they are doing, HOORAY!!!

*the exception is the top part of a for loop (but not a while loop as you can't declare anything in the head) in that for(let x in y){} x is part of the lower scope.