Now, when a user clicks the download button, the server:
Enumerates all the requested images in that album.
Immediately writes to the client’s http response the http headers to say it’s a download and the file name is.zip.
zipstream writes the header bytes of zip container.
Creates an http request to the first image in S3.
Pipes that into zipstream (we don’t actually need to run deflate as the images are already compressed).
Pipes that into the client’s http response.
Repeats for each image, with zipstream correctly writing envelopes for each file.
zipstream writes the footer bytes for the zip container
This is so much better than before, as:
The download is now immediate, with only a second or two of latency.
The pipeline ensures that the whole process only runs as fast as the slowest bottleneck, which is usually the client download speed. It’s auto throttling.
Everything is in memory, and nothing ever touches disk. Only as much work as needed is ever done. eg Aborting a 1 GB download at 1MB, will only waste 1MB of CPU processing and IO bandwidth.
We can run many thousands of downloads concurrently on one server, as at each point in time a download only takes minimal resources: 2 http requests and a few JS stream objects.
Alternatively, we can run smaller cheaper servers, and get the same experience.
The code is significantly simpler, no need to manually throttle, no need to clean up temp directories after, no work queues.
No need for a cache, as we only stream what we need. Again less code to maintain.
gamechangershow
s-u-f-j-a-n-s-t-e-v-e-n-s
jimmyeatworldofficial