ZFS degraded? Talk about a loss of productivity!
I had plans for tonight.
I was going to
Critical Alerttweak the rc script in the MediaBrowser FreeBSD port,
update my GitHub pull request to plexhometheater to upstream my FreeBSD patches, and
work on getting OpenWrt installed on my new TP-Link Archer D9 wireless modem router.
But then this happened:
Hmmm... that doesn't look good! Let's see what we've got here...
freenas% zpool status pool: zstore state: DEGRADED status: One or more devices could not be opened. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Attach the missing device and online it using 'zpool online'. see: http://illumos.org/msg/ZFS-8000-2Q scan: scrub repaired 0 in 1h53m with 0 errors on Tue Nov 11 17:10:55 2014 config:
NAME STATE READ WRITE CKSUM zstore DEGRADED 0 0 0 raidz2-0 DEGRADED 0 0 0 5730871870871882368 UNAVAIL 0 0 0 was /dev/gptid/280d84d2-b8ce-11e3-ab3c-bc5ff4fb5e9e.eli 11385700896761651914 UNAVAIL 0 0 0 was /dev/gptid/28f3eb96-b8ce-11e3-ab3c-bc5ff4fb5e9e.eli gptid/29dd9498-b8ce-11e3-ab3c-bc5ff4fb5e9e.eli ONLINE 0 0 0 gptid/2ac661a8-b8ce-11e3-ab3c-bc5ff4fb5e9e.eli ONLINE 0 0 0
errors: No known data errors
freenas% ls -lah /dev/gptid total 1 dr-xr-xr-x 2 root wheel 512B Nov 12 20:39 . dr-xr-xr-x 11 root wheel 512B Nov 12 20:39 .. crw-r----- 1 root operator 0x77 Nov 12 20:39 28e66586-b8ce-11e3-ab3c-bc5ff4fb5e9e crw-r----- 1 root operator 0x78 Nov 12 20:39 28f3eb96-b8ce-11e3-ab3c-bc5ff4fb5e9e crw-r----- 1 root operator 0x82 Nov 12 20:39 29dd9498-b8ce-11e3-ab3c-bc5ff4fb5e9e crw-r----- 1 root operator 0xb4 Nov 12 20:48 29dd9498-b8ce-11e3-ab3c-bc5ff4fb5e9e.eli crw-r----- 1 root operator 0x84 Nov 12 20:39 2ac661a8-b8ce-11e3-ab3c-bc5ff4fb5e9e crw-r----- 1 root operator 0xb5 Nov 12 20:48 2ac661a8-b8ce-11e3-ab3c-bc5ff4fb5e9e.eli
Since I am running raidz2 with 4 disks, having 2 failures means I am on my last legs here; if either of the remaining drives fails I lose ALL my data! Damn I wish I had that backup I've been talking about...
To cut a long story short, I shutdown, checked all the wiring connections in the FreeNAS mini, and after rebooting I had 4 drives showing again. I have read that resilvering can stress the remaining good disks, so I was nervous. But I charged forth and used the FreeNAS GUI replace button, and the first drive has just now finished resilvering (3 hours later for 1.7Tb of data).
The heat is off... at least I have some redundancy back. I'm off now to get that 4th disk back.
Thank god I chose raidz2. Guess I should think a bit more about those backups...