Don’t always trust your libraries to do what you expect.
I recently implemented chunked or “multi-part” uploading to a video uploader. As part of that I made use of dropzone (this was already there from work I had done earlier) and resumable.js (the glue to make support chunked uploading). This was a great win, users with spotty internet connections could now upload large files without worry of the upload failing halfway through. As part of my testing for this feature, I would start an upload, disconnect from the internet, wait, and reconnect and ensure the upload finished. Hooray!
This made it out to production and everything seemed okay. However, a few weeks after it was rolled out we got a few reports of “This file just wasn’t working when we uploaded it”. Initially it seemed a lot of those reports were due to encoding and our transcoder not playing nice with it. Eventually a report came in that one of the videos that was uploaded, when played back, was missing the last few seconds of video! Uh oh.
This immediately smelled of something going foul with either the chunking, or the rebuilding of the file server-side. We reached out to our transcoding service and learned we were missing a few parameters in the request we were making prior to the actual upload. Well, cool, I went about starting to fix that when I noticed that the exact number of chunks I was uploading was off by one.
It turns out that resumable.js will, by default, will set the last chunk to upload’s size as in between n to 2n-1 (n being the chunk size). https://github.com/23/resumable.js/issues/51 Is the issue where some unfortunate soul discovered this odd behavior. Anywho, the company behind resumable.js decided that because of their specific use-case around the last chunk’s size, they wanted this to be the default behavior. This caused me some headache, but at least they stuck in the forcechunksize option in there to override their silly default.
So, lesson learned, always check to be sure that library you’re trusting to work, actually works the way you are expecting it to.
Fault-tolerant HTML5 uploads with Resumable.js [Open JavaScript]
With [23 Video](http://www.23video.com) we (at [23](http://www.23company.com)) receive a large number of really, really large files every day. At times, we get files of several gigabytes transferred over a simple HTTP connection. Usually, this works out alright, but we've wanted to make better use of some new features in modern browsers to make uploads more stable -- specifically, to allow users to pause their uploads and have the uploads automatically resume if the internet connection fails (either locally or to our servers). This approach is fairly new and is only available in Chrome (11+) and Firefox (4+), so no one has packaged the feature-set we wanted nicely yet. This calls for open sourcing a few hundred lines of code and a victory lap: **We bring you [Resumable.js](http://github.com/23/resumable.js) as our answer.** Resumable.js is a JavaScript library built on top of the HTML5 File API to: * Allow selection or dropping of multiple files. * Handle queuing and progress indications during upload. * Uploads files in small chunks, rather than in a single big batch. * Automatically resume uploads if something fails, for example if the network connection drops. The first two features have been available through Flash and [related](http://plupload.com) [tools](http://www.swfupload.org) for quite a while, but I'm pretty excited about the latter two. Chunking and auto-resuming will allow for a degree of stability in HTTP uploading we haven't seen before: Resumable.js will simply keep uploading until all the bits of a file has been sent. And it all happens transparently to the user.
The [Resumable.js](http://github.com/23/resumable.js)-based uploader will make its appearance in 23 Video later this summer, but we will also add support for this method in our [API for direct browser uploads](http://www.23developer.com/api/browser-based-uploads) -- in fact the prototype uses this API method behind the scenes.
