Cloud Foundry: Push Fails, Cannot Allocate memory!
We have an in-house Cloud Foundry setup on Openstack for our developments teams. Last week on the our environment we had application pushes failing strangely with the following errors.
Done uploading FAILED Error uploading application. An unknown error occurred.
And nothing else to start our investigation with, but luckily we looked at the Cloud Controller logs first, and found this.
Job VCAP::CloudController::Jobs::ExceptionCatchingJob (id=346) FAILED (6 prior attempts) with Errno::ENOMEM: Cannot allocate memory - zip -q -r --symlinks /var/vcap/data/cloud_controller_ng/tmp/safezipper20150724-19425-ub32ql/package.zip
Cannot allocate memory is a Ruby error to denote there is not enough RAM. A quick at the free showed us this.
(root)$ free total used free shared buffers cached Mem: 2056468 1960548 95920 0 1148 89396 -/+ buffers/cache: 1870004 186464 Swap: 0 0 0
Now that was not looking good with close 95% of the system memory occupied. Usually the following commands should work to free up some RAM.
(root)$ sync (root)$ # To free pagecache (root)$ echo 1 > /proc/sys/vm/drop_caches (root)$ # To free dentries and inodes (root)$ echo 2 > /proc/sys/vm/drop_caches (root)$ # To free pagecache, dentries and inodes (root)$ echo 3 > /proc/sys/vm/drop_caches
Now running free for some happy news, did not show any significant change.
(root)$ free total used free shared buffers cached Mem: 2056468 1877436 179032 0 128 7564 -/+ buffers/cache: 1869744 186724 Swap: 0 0 0
Though there was a drop in usage, it was not significant enough to run away with.
Swap to the Rescue
Noticed that instance Flavors in the Openstack setup had no Swap provide to the Flavor type. Though this is the norm, it would have saved a lot work if it had been setup during flavor definition.
So now, we had to create the Swap manually. First create the file to be used as swap.
(root)$ dd if=/dev/zero of=/swapfile bs=1024 count=256k 262144+0 records in 262144+0 records out 268435456 bytes (268 MB) copied, 4.74066 s, 56.6 MB/s
This creates a NULL, /dev/zero, ridden file, /swapfile, of size approximately 256MB. Not much, but enough. Then we formatted it as swap.
(root)$ mkswap /swapfile mkswap: /swapfile: warning: don't erase bootbits sectors on whole disk. Use -f to force. Setting up swapspace version 1, size = 262140 KiB no label, UUID=xxxxx-xxxx-...
Forget about the warning. To turn on the swap, issued the following the following command.
(root)$ swapon /swapfile
For the protection of this location, we make it write-only-by-root.
(root)$ chown root:root /swapfile (root)$ chmod 0600 /swapfile
It would be better to define the swapiness to enhance performance, we did this with following commands.
(root)$ echo 10 | tee /proc/sys/vm/swappiness 10 (root)$ echo vm.swappiness = 10 | tee -a /etc/sysctl.conf vm.swappiness = 10
Adding swap did help with successful push commands, and having our applications successfully deployed.
Takeaways
The Cloud Controller Machine should be given enough memory.
Add swap to every Flavor of the Openstack instances, so even if the RAM overflows, Swap is there to rescue.