Proxmox VE 3.1 with ZFS
Spent the majority of my weekend trying to get Proxmox to work with ZFS. I'm using the following hardware:
Intel Core i5-3570K
ASUS P8Z77-V LK Motherboard
16GB Corsair Vengeance DDR3 1600
Supermicro AOC-USAS2-L8i RAID Adapter (flashed to IT mode)
6 x 3TB Seagate Barracude 7200RPM
22GB Intel SLC SSD
Coursair Builder Series CX600 600W PSU
Antec P280 Black Super Mid Tower
Step 1: The first major task was to flash the Supermicro card to "IT mode" and is an entire blog post in and of itself. If anyone is interested I'll write it up.
Step 2: Create Proxmox bootable USB My system doesn't have an optical drive so I downloaded an ISO of Proxmox VE 3.1 from http://www.proxmox.com/. Just dd it over:
dd if=pve-cd.iso of=/dev/XYZ bs=1M
Step 3: Install PVE 3.1 At "boot:" type "linux ext4" (or just press enter if you prefer ext3). For some reason the Proxmox installation medium uses hd0 to identify the boot disk no matter what drive you select for installation. To get the OS to install on my system I had to disconnect the storage drives from my HBA.
Step 4: Update the system If you don't have a Proxmox subscription you will need to replace the enterprise repository in /etc/apt/sources.list.d with
deb http://download.proxmox.com/debian wheezy pve-no-subscription
then
apt-get update apt-get upgrade
At this point I shut the system down and re-connected my storage drives.
Step 5: Install ZFS Dependencies
apt-get install pve-headers-$(uname -r) build-essential linux-source-2.6 ln -s /lib/modules/$(uname -r)/build /lib/modules/$(uname -r)/source
Step 6: Install ZFS
wget http://archive.zfsonlinux.org/debian/pool/main/z/zfsonlinux/zfsonlinux_2%7Ewheezy_all.deb dpkg -i zfsonlinux_2~wheezy_all.deb apt-get update apt-get install debian-zfs
Step 7: Configure ZFS Probably best to install the drives by id rather then sdxy so
ls /dev/disk/by-id/ata-*
Create a partition table for each disk:
parted /dev/disk/by-id/ata-XYZ
Again, do this for each disk in your pool. Then create your storage pool, I'm clever when it comes to names so I called my "storage"
zpool create -m none -o ashift=12 storage raidz2 /dev/disk/by-id/ata-XYZA /dev/disk/by-id/ata-XYZB /dev/disk/by-id/ata-XYZC
and so on. Note: I chose to use raidz2 because I have 6 drives. Since I'm using all consumer hardware and non-ECC ram, I got paranoid and turned off deduplication
zfs set dedup=off storage
Step 8: Create a filesystem for vz
I want Proxmox to use my ZFS pool for vm's and containers so it was necessary to mount /var/lib/vzas ZFS, that ended up being trickier that I thought.
PVE will automatically recreate important folders in vz but to be safe, I copied them out
mkdir ~/temp cp -R /var/lib/vz/* ~/temp
Then I created my first filesystem on the storage pool
zfs create storage/vz zfs set mountpoint=/var/lib/vz
Edit /etc/default/zfs and change the following lines:
ZFS_MOUNT='yes' ZFS_UNMOUNT='yes'
Edit /etc/fstab and comment out the mount for /var/lib/vz
# /dev/pve/data /var/lib/vz ext4 defaults 0
Reboot the system
cp -R ~/temp /var/lib/vz
Conclusion
At this point, my PVE system is ready to start adding VM's and containers. Here is the output of df -h / and df -h /var/lib/vz
Filesystem Size Used Avail Use% Mounted on /dev/mapper/pve-root 5.5G 1.4G 3.9G 26% /
Filesystem Size Used Avail Use% Mounted on storage/vz 11T 384K 11T 1% /var/lib/vz
Edit: Forgot to put my references
http://zfsonlinux.org/index.html http://www.andybotting.com/zfs-on-linux http://askubuntu.com/questions/201896/zfs-package-zfs-ubuntu-does-not-create-zfs-module http://blog.rebelit.net/526
















