Daily Thoughts - VMware Edition
You know whenever VMware forces you to use the CLI tools you are in for a rough patch. Today was certainly no exception to the rule. I had updated the vCenter server and some of our hosts in the Production cluster at work to vSphere 5. VUM said that the rest of the hosts were incompatible. Why you ask? Because the hosts had OEM software modules that were not supported with ESXi 5. Simple enough, I think, I'll just remove those software modules.
Well, guess what? You can't remove the software modules. There is no esxcli or PowerCLI command to do it. There is no function from within the VI Client either. You can see the software modules, but right-click to your heart's content. It will not avail you. The closest command I could find involved the vicfg-advcfg command. Be warned though, here there be dragons. I may shoot from the hip on occasion, but I was not feeling enough of a cowboy to disable the OEM CIM service to see what would happen. I have this vision of the blade server exploding and raining shrapnel across the datacenter.
The offending module was a Dell OEM ESXi VIB that probably came from a custom ESXi 4.1 installer supplied by Dell. You might question why some hosts had this custom version, and not others? Well, there is no good explanation. As far as I can tell, we moved several hosts from Dev to Prod a while back and my associate used a Dell installer to get the OpenManage VIB. Unfortunately, we did not use this image in Dev, so it never came up when I was testing the upgrade. What does VMware suggest? That you should build your own custom installer using their PowerCLI Update Manager suite and then add your own VIBs to the image.
Dell has their OpenManage VIBs on their site. Just don't use their search tools, b/c that will only find the ESXi 4.1 compatible VIB. Another search FAIL for Dell, it's a good thing that Google indexes their site better than they do. I found the VIB, installed the PowerCLI Update Manager, and read through their cryptic documentation. After 30 minutes, I had read between the lines and actually managed to build a custom ISO including the Dell VIB. I added it to VUM, attached the baseline, ran a scan and ... Incompatible.
What. The. Fuck.
Before you ask, yes I did tick the little box in VUM that removes third-party software that is incompatible. The upgrade still failed. So it looks like although Dell updated their VIB, it does not upgrade the older version. Dell FAIL number two for the day.
I finally broke down and installed ESXi 5 manually using the ISO from VMware. There is a Force Migrate option that apparently is much more forceful than VUM in this regard. It tells you so several times (lots of big red warnings and hitting special accept buttons). After convincing the installer that I did, in fact, want to force the migration, all proceeded normally. This is good news, in that I do not have to do a clean install of ESXi on each server. Still, it is a manual process and I have to repeat it six more times. It would have been much nicer to use VUM. I suppose if I had a datacenter with 100 hosts in it, I would find a scripted way to do this.
Now I have to wait until the next maintenance window to complete the upgrades, unless I can get special dispensation to do the upgrades at night. But since it is a manual process, I don't really want to do that. Maybe in the morning before anyone else comes in. I'm up at 5:30 every morning anyways.
Time for a clementine, oh my darling.