Automatically kill an application if it takes up too much memory on Linux / Debian / Raspberry Pi
I use a Raspberry Pi 3B to do video encoding — see this post.
However, under certain circumstances, ffmpeg might end up ballooning in memory, and this can get serious very fast given that a Raspberry board has a relatively small amount of RAM.
In my case, this can happen if my local network goes down in a specific way... I think ffmpeg ends up not getting an instant refusal / timeout on connecting to the server and buffers forever instead. This could also potentially happen if whatever your camera is pointed at suddenly becomes much more complex to encode, and your options cause ffmpeg to buffer the input instead of dropping frames.
So I figured, maybe I could look into running a script that kills ffmpeg as soon as it takes up too much memory. I found a base of a script somewhere on the Internet, but it didn’t work; here’s a fixed version.
TOTAL=`cat /proc/meminfo | grep MemTotal: | awk '{print $2}'`
USEDMEM=`cat /proc/meminfo | grep MemAvailable: | awk '{print $2}'`
if [ "$USEDMEM" -gt 0 ]
then
USEDMEMPER=$(( (TOTAL-USEDMEM) * 100 / $TOTAL))
echo "Current used memory = $USEDMEMPER %"
if [ "$USEDMEMPER" -gt 80 ]; then
killall -9 ffmpeg
fi
fi
To be honest, this makes the title of this post inaccurate: this script evaluates the total active usage of the entire system, not how much one application takes up by itself. But generally, since you’re trying to avoid a “one application makes everything else go OoM” disaster scenario, this is for most intents and purposes identical. And you could change it to use a different memory type, maybe, but MemActive is the one that would be relevant to me, since I leave a tmpfs (ram disk) worth a couple hundred megs running on the board.
And, of course, this script explicitly runs a killall command on ffmpeg, not “the application that is taking up too much memory”. It’s not a smart script; it assumes you already know which application(s) is susceptible to misbehave.
So, we’ve got our watchdog script. Let’s save it at /home/pi/watchdog.sh
We could add a loop right in the script but that’s probably not the most elegant solution. We could also use “cron”, but the instant I learned it was possible to wipe all of its data simply by calling it with no parameters, I figured I would stay away from that thing as much as possible.
Let’s use the services system instead! (It’s systemd & systemctl.)
We have to create the service definition and its timer, separately.
First, the service: let’s sudo nano a new file into /etc/systemd/system/ffmpeg_watchdog.service
[Unit]
Description=FFMpeg Watchdog Service
After=network.target
[Service]
Type=simple
User=pi
WorkingDirectory=/home/pi/
ExecStart=/bin/bash /home/pi/watchdog.sh
Restart=on-failure
[Install]
WantedBy=multi-user.target
In case you’re wondering, ExecStart has “/bin/bash” in front of the path to your .sh file, because it doesn’t magically know that this is supposed to involve the shell / bash.
Now, let’s create the timer, which will define when and how often our newly-created service will run. Create the same file but replace .service by .timer!
[Unit]
Description=Run FFMpeg watchdog
[Timer]
AccuracySec=1s
OnBootSec=1min
OnUnitActiveSec=10s
Unit=ffmpeg_watchdog.service
[Install]
WantedBy=multi-user.target
It’s fairly straightforward once you’ve figured out what to do—something that describes at least 90% of Linux-related tasks. Now you just need to enable all of this with two systemctl commands:
sudo systemctl enable ffmpeg_watchdog.service
sudo systemctl enable ffmpeg_watchdog.timer
And now everything should be active! Make sure to test it by forcing the application to devour all your RAM; your new protection should kick in. 10 seconds and 80% is a good interval for me, since the bloat happens slow and steady under my circumstances, but depending on your application, you might need to adjust the .timer to check slightly more often.
Anyway, the usual caveats of my “tech solutions” posts apply: I don’t pretend to know all the answers, I don’t know if this is the proper way to do something, and I don’t know if this will work for you, but I do hope it will help you, or at least guide you towards what you’re looking for! Cheers.
