Musings of a hobby programmer

I code a lot, bots especially. Sometimes I find something interesting and write about it here. Or sometimes I find something that some people might not know and also write about it here. I am always open for messages, lets chat about some interesting coding topics someday! Enjoy your stay and I wish you happy coding and reading!

[lines-of-code]

When falsy values betray you....

Things like this can quickly lead to your NodeJS memory being filled with multiple GB of empty arrays and crashing your Node Application, so watch out!

ytvids is an array that has an amount of x entries. subytids is an empty array.

Basically what I wanted to do here is take ytvids and split it into multiple arrays with the defined length of 50 entries each. .splice is destructive, meaning it removes elements from the array permanently compared to it's sister method .slice, which only gives back the entries but doesn't take them out of the array

var array = ["A", "B", "C"]; console.log(array.slice(0, 2)); // => ["A", "B"] console.log(array); // => ["A", "B", "C"] console.log(array.splice(0, 2)); // => ["A", "B"] console.log(array); // => ["C"]

So what this loop is supposed to do is splice out up to 50 entries every time the code in the loop runs and add the new array to the subytids array. And the code runs because an array is considered a truthy value in JS. So it runs and it splices and it runs and it splices. But someday, the array will be empty and splicing only returns []. What I did not consider is that an empty array actually STILL counts as a truthy value. Meaning the loop does not stop. And it keeps pushing and pushing empty arrays coming from .splice to subytids. Up to the point that there is just too much memory in the heap and it crashes.

What I ACTUALLY should have done in the first place, is use while (ytvids.length). Length is the number of items in an array and that counts down after each splicing. 0 in JS is considered a falsy value and breaks the loop when no more entries are remaining in the array.

There is only a handful of things that are considered falsy in JS, so you might want to remember those:

false (boolean)

null

undefined

0

NaN

'' or "" (empty strings)

document.all (although this is rarely used apart from detecting IE)

[lines-of-code]

The logistics that is Node Streams

Just a few weeks ago, streams in Node were an absolute mystery to me. Piping? Flow control? What the heck is all this about? I was lost. But sometimes, an understandable real life example can come up anywhere, even at work.

Due to the code being a little lengthy, I’ll put it under a read more!

But may you be my witness, I had an idea to write a YouTube player plugin for my discord bot and there was nothing that could stop me. Apart from the dreading weight that were streams.

Long story short, It took me around 2 weeks to get it working properly (apart from the crashes the thing is actually super fun by the way, but more about that in another post if people are interested.)

Then, it was time to go to work. Well, actual work that is, I do have a job that is not programming (unfortunately, how does make monies?!?). I work in logistics, meaning I send and receive packages and products. That day, we got a huge shipment of products, 3 trucks full of 33 palettes each. Usually I am not responsible for it, but that day they strapped me in to help unload. And the first truck we unloaded actually reminded me a lot of how streams work.

We had a truck. And we had 33 palettes to unload and store. And it was 3 people that did the unloading.

So the first person, the truck driver started unloading palette after palette, placing them right outside of his truck. I came over, picked up one of the palettes and took it with me through half the storage halls and placed it down near the place where it was supposed to be stored. A colleague of mine then picked them up from where I placed them down and neatly organized them at their final position.

While transporting the palettes around, the truck driver had to take a phone call. Fortunately, he was fast enough to place enough palettes outside before so I could just keep coming and pick up the next one. By the time he was done, there was still one palette left but he immediately started taking out more.

My colleague wasn’t as fast as me either so a few palettes accumulated where I placed them down. But he diligently took them and stored them regardless.

Now what does this have to do with streams? A lot actually.

We have the truck which we will consider a source of data. Each one of the 33 palettes inside is a little bit of data. We were 3 people that worked on unloading the truck and each of us I will consider a single stream.

The truck driver (Stream A) takes a little bit of data (one of the palettes) and places it down for me (Stream B). I pick it up. That’s what piping is. One stream takes data from a source (either a data source or another stream, see stream as source next), places it down for another stream to pick it up and leaves it there. The 2nd stream then goes to the pick up spot, takes a data packages and does something with it. In the case of streams, the pick up location is a queue of data. An example for that is that even if the source stream is not giving anymore data to the receiving stream (see the trucker having to take a phone call) the other stream still has data it has to work through. Had to learn this the hard way when I tried to pause stream playback, the source stream was faster with data pushing than the destination stream could work through the data. Same principal goes for my colleague and me, or Stream B and Stream C. I was placing down the palettes at another pick up spot and my colleague took them and placed them in storage. It's another piping of data from Stream B to Stream C.

Below I have a little bit of code that represents the example above.

// What stream piping looks like in general var srcStream = new ReadableStream(); var destStream = new WritableStream(); srcStream.pipe(destStream); // Stream piping in the example above var streamA = new ReadableStream(); var streamB = new WriteableStream(); var streamC = new WriteableStream(); streamA.pipe(streamB); streamB.pipe(streamC); // Stream C uses the data in a meaningful away

So the lesson I want you to remember here is simple: Next time you work with streams, imagine me carrying around a lot of palettes from pickup point A to pickup point B. And have another person in your computer then use the data that it picks up from pickup point B.

In case you got any questions, send me a message, an ask or reply to the post and I’ll see if I can’t answer. Or maybe there is another topic that you’d like me to tackle, I’d be happy to see what I can do!

Obligatory ?