Upgrading Ubuntu Server 14.04 to 16.04.
A couple of days ago I got it into my head to upgrade one of my Exocortex servers from Ubuntu Server 14.04 LTS to 16.04 LTS, the latest stable release. While Ubuntu long-term support releases are good for a couple of years (14.04 LTS would be supported until at least 2020) I had some concerns about the packages themselves being too stale to run the later releases of much of my software. To be more specific, I could continue to hope that the Ruby and Python interpreters I have installed could be upgraded as necessary but at some point the core system libraries would be too old and they'd no longer compile. Not good for long-term planning.
First off, whenver you're about to do a major upgrade of anything, read the release notes so you know what you're getting yourself into. You'll also usually find some notes about all the new goodies you'll be able to play with.
In the past I've had nothing but trouble using the documented Ubuntu release upgrade process, so much so that I've had clients sign "I told you so," documents when they pressured me to do so because the procedure could reliably be expected to leave the system completely trashed, and a full rebuild was the only recourse. This time I set up a testbed in Virtualbox which consisted of a fully patched Ubuntu Server 14.04.5 LTS install. I ran through the documented upgrade process, and much to my surprise it went smoothly, leaving me with a functional virtual machine at the end of a 45 minute procedure (most of which was automatic, I only had to answer a few questions along the way). The process consisted of logging in as the root user (sudo -s) and running the updater (do-release-upgrade).
So, if it's so easy, why am I writing a blog post about it? Why worry?
Why worry, indeed. Read on.
First off, to replicate the upgrade experience of my testbed I used my hosting provider's web console instead of SSH'ing into the virtual machine, on the hypothesis that I was much less likely to lose remote access to the machine that way. I did, incidentally, quite a few times, but all I had to do was close the web console window and re-open it for everything to be usable again. The web console is slow, though... I'm sure the process chugged along nicely but the output wasn't speedy at all. Second, and this was a huge problem that took a couple of hours to resolve, there was a bug which resulted in the UUID of the root partition listed in the file /boot/grub/grub.cfg not matching the one listed for the root partition in /etc/fstab. This caused the systemd-remount-fs.service system service to (more or less) silently fail and the root partition was mounted in read-only mode to protect data integrity. This is all well and good, but when you identify the problem and need to fix it... and you can't edit anything because the system's in read-only mode...
Long story short, I e-mailed tech support at my hosting provider and asked them politely if they'd mount a recovery ISO on my virtual machine, so that the next time I booted it it would load a live CD instead. They did so, and once it loaded I was able to mount the root partition as a writeable device, edit the /etc/fstab file so that it reflected the correct identifier for the root partition, and cleanly shut down. Another comment on the support ticket, and the recovery ISO was removed from the back-end, so my virtual machine came back up more or less as expected.
Now the rest of the situation...
It is a known and documented bug that the MySQL database server will break when you upgrade to Ubuntu Server 16.04. I found that the workarounds in the ticket do not work. To fix things I had to completely purge MySQL from my system (please note that this does NOT delete the databases - your data will be safe!) with the commands sudo apt-get remove mysql-common, sudo apt-get autoremove (to uninstall the no-longer-needed dependencies), and sudo apt-get purge mysql-common to wipe out the configuration files which were ever so slightly incompatible in ways that MySQL doesn't even bother to tell you that they're not valid anymore. I then re-installed MySQL (sudo apt-get install mysql-server libmysqlclient-dev) and I was able to interact with my databases once again. The MySQL system service had to be enabled (sudo systemctl enable mysql) so that it will start up automatically.
Etherpad-Lite seems to work with the new versions of node.js and npm installed in the new release of Ubuntu Server. I just had to re-enable the startup service for it (sudo systemctl enable etherpad) and it passed all of my functionality tests. Good to go.
There are one or two API services I was running on the same VM that just don't seem to be compatible with the new system libraries. After messing with them for an hour or two with no success, I've decided that I'm going to have to use somebody else's instances instead. That's a project for a later time.
Because it was overdue for an upgrade, I wiped out the existing Ruby installation (rvm implode), installed the latest stable release of the language interpreter (rvm install ruby-2.3.1) and package manager (gem install bundle), and re-installed all of Huginn's dependencies (cd ~/huginn && bundle).
One last thing: I had to destroy and rebuild the virtualenvs for each of the constructs running on that virtual machine. This wasn't a big deal at all because destroying one means deleting a single directory (rm -rf env/) and rebuilding it requires three on the outside (virtualenv env && . env/bin/activate && pip install -r requirements.txt). A little use of the up arrow to repeat previous commands and I had everything done inside of five minutes.
Final analysis: If I hadn't had to perform live CD surgery on that virtual machine, I'd have had the upgrade done in about two hours, most of which was spent waiting for packages to download and unpack. Having to troubleshoot the messed up boot process and fix the /etc/fstab file roughly tripled the amount of time required. I hope that if anybody out there runs into the same situation, this blog post will help them speed up the process by describing all of the gotchas you'll run into.
Good luck. And don't forget to back up first!