Stream your response with node.js
Did you know that http response doesn't have to be one big chunk of data? In this article I will talk a bit about http streams.
Http streams allows you to send chunked data as a response. Why I need it? It simply allows you to not wait for whole response and send just what is ready to be send.
var http = require('http'); http.createServer(function (req, res) { res.writeHead(200, { 'Content-Type': 'text/html', 'Transfer-Encoding': 'chunked' }); res.write('Hello') setTimeout(function () { res.end(' world'); },3000); }).listen(3000, '127.0.0.1');
Run this code with your node.js and then curl localhost:3000 --raw as you can see Hello was send immediately and world waited 3 seconds to be sent.
This is probably the most simple use case. But why we used curl over a browser? Problem with browsers (my chrome) is that it buffers the response so even it has the Hello text it does render only whole response so you will see that your page is loading for 3 seconds.
It doesn't mean that something is wrong with the code above. Imagine that as a response we send a file. You would read a file content and then you would send it as a normal response. Nothing wrong with that unless you get tons of requests.
The file content must be saved in a buffer in order to send it. So as number of simultaneous requests would increase your server could go out of memory.
That's where become handy to use a stream pipe.
In other words the data from the file are saved in buffer as chunks, so the buffer doesn't hold whole file. You should then send those chunks as response as it would have NO advantage if you wouldn't.
var http = require('http'); var fs = require('fs'); var server = http.createServer(function (req, res) { var stream = fs.createReadStream('a_file.txt'); stream.pipe(res); }); server.listen(3000, '127.0.0.1');
We just created a file read stream and we created a pipe to response. So now your memory should be just fine and everyone won.
In Ruby on Rails you can use streams from version 4. Read this blog post.