mirrorshades

the street finds its own uses for things

[mirrorshades]

A Return to Form

Configs and dotfiles

Before we get into anything else, here's a repo with my configs and dotfiles for running FreeBSD as a workstation.

A lot of the good stuff was cribbed from Jessie Frazelle's dotfiles repo, or from her post on running Linux on a MacBook. Check it out.

Background

I've been using Mac OS X since the Public Beta, though not as a daily driver for a few years after it was initially released.

Pismo, PowerMac, PowerBook (12" albook!), MacBook, iMac... It's been a long road, really, and until the last few years I've been very happy.

Before OS X, I used Linux or OpenBSD as a desktop (and before that Windows 3.1), and was generally pretty happy for many years, but I did spend a lot of time screwing around with X11 configs and dist-upgrade breaking my world, and all the usual stuff you might remember from 2003 or 2004.

Around then OS X started getting pretty usable, especially given my relatively simple professional needs:

A terminal

OpenSSH

Really, that's it, for work.

Obviously a web browser, and a media player were nice to have, but as a sysadmin most of my day is spent at a shell prompt on some other machine. Very little of the development work I do is on my local machine, but is instead housed in a zone on some compute node in some datacenter somewhere.

So for a long time OS X was good. Each release was faster on my existing hardware (no, really!) and less buggy, and sometimes had useful features I cared about.

Mostly I didn't use the new features, though, and as time went on the number of features and things it was doing I would prefer it not skyrocketed.

I used Terminal (and later iTerm2) and ssh. Safari got slow so I switched to Firefox, and then Chrome. All my media is in iTunes now, and yeah, it's kind of awful, but I don't spend a lot of time actually looking at it.

Circa OS X 10.6, I was really happy with OS X. It was fast, stable, and never got in my way.

Around OS X 10.9, though, things started going wrong. 10.10 improved a few of these things, but overall it just kept degrading. It's slower, there are a lot of really distracting "features" I can't seem to actually reliably disable: It's tied into my phone, and my wife's phone, so when she adds events I get duplicate notifications (deliver once being a fallacy, I suppose), disrupting me from my work. I disable this, but ...

It harasses me every day to upgrade. It desperately wants to just upgrade whenever it wants. More and more it acts like the Windows machines I've had to support over the last 20 years, which is deeply frustrating.

It regularly does things in the background without asking, consuming all my bandwidth (again: most of my work is remote, so I'm particularly sensitive to latency.)

And yeah: I've disabled all these things. They keep getting re-enabled, and so it's not hard to take the hint.

Periodically when I try to log into OS X, it will just hang on me, which is sort of beyond the pale.

Upgrades have gone sideways because there are files in /usr/local of all things. And now SIP in 10.11 breaks all sorts of stuff.

I've just been fed up for a while. I recognize that mostly it's because the thing that was working so well for me has now moved on -- it's this digital life hub thing, instead of a nice UX that let me run ssh reliably. So maybe it's not you, it's not me, it's just us. And that's ok.

Alternatively:

OS X has become an obnoxious puppy ("I WANTED TO LOVE YOU SO MUCH I PEED ALL OVER THE FLOOR"), and as I've grown older, I've become more of a cat person ("...you do your thing, I'll do mine. Cool? Cool.")

It was more about the constant distractions than the cost of maintenance.

So What Then

I spent some time defining the problem and how I might solve it.

The primary issue was I spent too much time telling OS X to shut up and leave me alone. Some of this is me and how poorly my brain operates these days -- I've had 3 kids in the last 5 years. I don't like to play the twins card, but having twins does something to your brain. The doubled up sleep deprivation and long, long periods of stress altered me in non-trivial ways.

Time is precious and my mental state is constantly fragile. Focus is incredibly difficult to achieve and impossible to maintain if something distracts me.

I needed something that wouldn't bother me, or consume my time pointlessly. One way or another I was going to have to build this thing. Something off the shelf like what OS X 10.6 was for me was unlikely to simply exist.

I tried running various Linux on my MacBook, but discovered everything I hate about managing Linux on a server platform is in fact amplified in a desktop context. It was less bad than I remembered from 10 years ago, but it was still a poor comparison to when OS X was good -- again, for my requirements.

I tried to think of an extreme that wasn't too extreme -- something that was minimalistic, but not egregiously so (like just rocking a console and screen and no GUI at all.) Something that had features and technologies I cared about, without wandering off into the weeds periodically to bring me back a decapitated bird.

OpenBSD was my first choice. I've used it for many many years in firewalling or routing contexts, I used to use it as a desktop -- but the upgrade process sort of killed it for me. I didn't want to deal with patching and recompiling my OS, or remembering to look at a web page for errata (or writing a script to do that for me.)

I've used FreeBSD for projects in the past (circa FreeBSD 5.x) but not in years. However, they've integrated ZFS and DTrace from illumos (which is where I spend the vast majority of my time, logged into things), pf from OpenBSD, and they have both binary OS updates and packages.

If you poke around the FreeBSD site you'll notice two things which I also found to be extremely commendable: The FreeBSD Handbook is an amazing piece of documentation. It's far shorter than the Solaris System Admin Guides I knew and loved when I was getting started with Solaris in 2005, but it's a really well crafted piece of work.

The other is the FreeBSD Code of Conduct. CoCs are (rightfully!) becoming more popular with projects and conferences, and I don't know when they added theirs -- but it's well thought out, and I appreciate the effort there.

The Workstation

I didn't want to install FreeBSD on my iMac. My wife uses it for photo and video editing, my music is all tied to OS X, sometimes I have an hour to play the latest Shadowrun game. Dual booting would just be a pain.

I decided to build a computer for home use. I hadn't done this in at least ten years -- I've built plenty of servers in that time, but at home I'd been happy to just have my Macs.

I was careful to find hardware that was slightly older, and well supported. I didn't want to end up with some device that was generally okay but periodically got flaky. I spent a lot of time reading forums, driver man pages, and so on. In retrospect I probably didn't need to do this much due diligence, but I was being paranoid about the time I might have to spend later fighting with the fallout of a poor purchasing choice.

I ended up purchasing:

Monitor: Dell S2415H 24" LED

Motherboard: Asus Z97

CPU: Intel Core i7 Quad Core 3.5GHz

CPU fan: Cool Master Hyper 212

RAM: Crucial Ballistix Elite 8GB DDR3 (x4)

Storage: Intel 730 480GB SSD

Video: EVGA GeForce GTX 750Ti 2GB

Case: Corsair Obsidian 750D

So... for what I need, a quad-core box with 32GB RAM is pretty ridiculous. But I'm hoping I can not do this again for quite for a years, so. The case has plenty of room for disks, if that becomes a thing I need over time, as well. (Because, ZFS.)

(I include the CPU fan because it made me laugh while I was installing it. So ridiculous.)

My 4 (and a half she'd be quick to point out at the time) year old and I put it together one night after her sisters went to bed. She sat with me through the installation and I set up her first user account for her. For posterity, kid:

12:37:43 gaea:~$ uname FreeBSD 12:37:47 gaea:~$ id nora uid=1002(nora) gid=1002(nora) groups=1002(nora)

FreeBSD Installation

I had done testing in VMs before I got the hardware, so I knew what to expect. And honestly -- there's not much to say here. I hit enter a bunch of times and then I had a system with ZFS on root, running FreeBSD. I was really pleased with how simple this was.

Both FreeBSD and OpenBSD have made huge strides in installation.

FreeBSD Packaging

pkg(8) is a nice tool. The ports tree is massive, and I haven't found any software I needed that didn't exist there.

In a server context I'd have the same problems I do with any other distro packaging setup: Missing compile-time options one some things, etc, but for home use it's been very smooth sailing.

FreeBSD Upgrading

The freebsd-update(8) tool just works. Sometimes it works so well I wonder if it actually did anything. I walked releases from 10.1 RELEASE, through security patches, up to 10.2's most recent patch level. Two commands each time, and zero problems with any of it.

freebsd-update fetch freebsd-update install

As A Desktop

There is a distribution of FreeBSD called PC-BSD that's targeted at desktops. However, if my goal was this minimal, stay out of my way, sort of environment, I didn't want a bunch of user-friendly add-ons trying to help me out. I wanted a simple environment that would help me focus.

This guide at cooltrainer.org was absolutely perfect for me. I had zero problems following it, and it got me a working GUI in under a half hour. The big kicker with X11 for me has always been fonts: They've historically been a pain to manage and setup. Either something has changed, or my threshold has moved, because with the exception of Google in Firefox I'm pretty happy with how things look.

The next decision I had to make required some experimentation: Which Desktop Environment to use? After poking at KDE, GNOME, LXCE, WindowMaker (yup), Enlightenment 17 (less said the better, and I ran E16 for ages back in the day) someone on Twitter (I can't remember who, now!) mentioned i3.

Back around 2003 I had co-workers and friends who were using ion and ratpoision, but I never really got tiling window manages for whatever reason.

i3, it clicked immediately. Loved it.

I stole most of Jessie Frazelle's dotfiles to make i3 and urxvt nice to look at.

(Initially I was using a terminal emulator called sakura, and it was nice enough. After I pulled in Jessie's dotfiles I figured I may as well just use urxvt like everyone else.)

I spent a few evenings on getting things setup how I want. With a new OS X install there was very little I'd do beyond customize my shell environment and change the window highlight from Aqua to Grey. (And enable FileVault, firewall, etc.) Spending time on configuring i3 didn't feel like a waste of time, though. It was iterative (as you can reload the configuration live) and the effects on my workflow felt immediate.

I really like i3.

This was back in April, and I've been happily using this setup for the last 7 months. I switch between the FreeBSD box and my iMac, depending on what I'm doing, but when I need to focus I'm in i3wm.

On a Laptop

(My notes.)

After several months, I decided to try it on a laptop. I love the MacBook chassis, but the OS is making me nuts. FreeBSD doesn't support the wireless NICs in mine yet, and of course I can't swap it out.

A few hours of research later (looking at modern netbooks that appeared to be compatible) I decided to get a Thinkpad X220 based on a thread on misc@openbsd.

(Someone on that thread coincidentally mentioned that X220s could take 16GB RAM, regardless of the documented max being 8GB.)

Looking at used gear on eBay, I decided to get something with as little RAM, but a faster CPU, as possible. I ordered the laptop eBay, and a 120GB Intel SSD, and two sticks of RAM from Amazon.

I haven't used a non-Apple laptop in over a decade. Simply speaking from a physical, mechanical perspective, they're amazing machines. Incredibly well-engineered. They don't feel flimsy. I'd grown to like the chiclet keyboard, and never felt like the touchpad got confused or would activate incorrectly.

The touchpad on the X220 is pretty crap. The nub mouse is going to take some time to get used to. They keyboard is nice, though I keep hitting the nub instead of the B key.

xset b off and hw.syscons.bell=0 were something that took me a few minutes to remember, but oof. So necessary. The bell on this thing is like someone taking a ball peen hammer to a piece of tin inside your skull.

Install

I couldn't get ZFS on root to work the X220's buggy UEFI firmware. I tried various things (legacy mode, MBR, manually writing zfsboot after install, FreeBSD 11 snapshot -- which has a fix for this, potentially, but the snap I tried was a bit buggy. I also don't want to be running HEAD. It goes away the "don't break" requirement.)

Finally, I split the SSD into 40GB UFS for the OS and another partition for a zpool.

In reality this doesn't effect me much. ZFS on root is nice to have, but I've been living without it on OS X, and I can live without on this laptop for a while longer.

Beyond that, the install worked just fine.

Wireless

Wifi support was the big reason I chose the X220, so I was a bit frustrated when the system get reassociating with the network. After an hour or so of debugging and googling, I found a post that described the problem. The workaround was changing ifconfig_wlan0 in /etc/rc.conf to

ifconfig_wlan0="-ht WPA DHCP"

-ht disables 802.11n (see ifconfig(8).)

Once that was in place, everything worked great.

Sleeping and Locking

FreeBSD won't sleep the laptop if you close the lid. Kind of a deal.

I felt sort of silly while I was writing it -- because I've been using Macs for so long -- but I wrote a i3 keybind to call i3lock and then acpiconf -s 3. It's similar to my keybind for just calling i3lock, so I imagine I'll sometimes screw it up, but it's easy for me to remember either way:

bindsym Control+Mod1+l exec i3lock -c 111111 bindsym Shift+Mod1+l exec i3lock -c 111111 && sudo acpiconf -s 3

(Requires sudo be configured for that command with NOPASSWD.)

Hardware Upgrade

Swapping out the disk and RAM in the machine was trivial. You can add an mSATA drive if you want to mirror or get extra storage but for my use cases (basically a dumb terminal with all data I care about elsewhere) I decided not to bother.

Performance

Both on the laptop and desktop, FreeBSD swims. To be fair, both machines have a ridiculous amount of RAM for what I'm doing (16GB and 32GB respectively; I mean c'mon), and an SSD -- and...

13:17:22 gaea:~$ ps -U bdha | grep -v ssh | grep -v bash PID TT STAT TIME COMMAND 986 - Ss 0:00.39 /usr/local/bin/i3 993 - S 0:00.88 i3bar --bar_id=bar-0 --socket=/tmp/i3-bdha.TvyJg3/ipc-socket.986 995 - S 0:00.06 i3status --config /home/bdha/.i3/status.conf 1000 - S 0:00.77 urxvt -ls 1061 - S 0:00.63 urxvt -ls 2847 - S 0:07.91 urxvt -ls 2896 - S 0:00.32 urxvt -ls 2964 - I 0:00.00 /usr/local/bin/dbus-launch --autolaunch 2adb3a5ba60595820f094822554267df --binary-syntax --close-stderr 2965 - Ss 0:00.00 /usr/local/bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session 3053 - S 0:00.45 urxvt -ls 3350 - S 0:00.00 /usr/local/bin/xclip -in -selection clipboard 3356 - S 0:00.68 firefox 3022 4 I+ 0:07.98 vim freebsd_osx_migration.txt 3359 6 R+ 0:00.00 ps -U bdha

So it's hardly fair to compare it to anything else... but realtalk. It's fast.

It'll remain fast.

The Problems

The biggest one I have is password sharing. All of my personal passwords are in 1Password. I've copied a bunch of them over to KeepPassX (which is what we use at work and on illumos infra), but syncing is definitely a pain point. I could stop using 1p, but honestly -- it's so convenient I'm used to not having to jump through an extra hoop to log into something.

Copy and paste is still a bit frustrating. Sometimes urxvt will get confused; it won't select things properly. I'm not happy with the hotkey I have set up for it. I imagine I'll put a bit more time into figuring out what's up here, because copying chunks of text shouldn't be something I ever have to think about.

On the laptop, I sometimes forget that I have actual pgup/pgdown keys. Will have to get used to that. :-)

Conclusion

This was a long post, given my requirements were "doesn't break" but also "doesn't waste my time." It seems a bit odd that I spent so much time on customizing configuration here, like I would have done in (forgive me) my youth.

However, for me, this was a one-time cost and I'm getting a lot of out it.

I am less frustrated, and more focused working on this setup. A big chunk of that is even outside the constant popups in OS X, there's simply less to be distracted by.

I've gone so far as to have to literally switch a cable to move between machines (as opposed to a KVM), to help me train my brain into a different context.

Overall I'm quite happy with the choices I made here.

FreeBSD and i3wm are simple (in the best ways), fast, reliable, and most importantly for me -- non-invasive.

[mirrorshades]

Installing Chef on Joyent's SmartOS

This is effectively the same procedure as installing Chef on Solaris with pkgsrc or OpenSolaris/OpenIndiana with IPS. We'll be using Joyent's provided pkgsrc setup and pkgin installer.

If you're using SmartOS on the Joyent Cloud, you'll have pkgin already available. If you're running SmartOS yourself, you'll need to install it.

@benjaminws threw up a bootstrap template for the following as well!

The bits:

pkgin install gcc-compiler gcc-runtime gcc-tools-0 ruby19 scmgit-base scmgit-docs gmake sun-jdk6 wget http://production.cf.rubygems.org/rubygems/rubygems-1.8.10.tgz tar -xzf rubygems-1.8.10.tgz cd rubygems-1.8.10 ruby setup.rb --no-format-executable gem install --no-ri --no-rdoc chef mkdir /etc/chef cat <<EOF>> /etc/chef/client.rb log_level :info log_location STDOUT chef_server_url "http://chefserver.example.com:4000" validation_client_name "chef-validator" node_name "host.domain.com" EOF

Drop your validation.pem in /etc/chef, and then run chef-client.

[mirrorshades]

Booting OpenIndiana on Amazon EC2

Since OpenSolaris was axed, we haven't had an option for running a Solaris-based distribution on EC2. Thanks to the hard work of Andrzej Szeszo, commissioned by Nexenta, now we do.

This should be considered a proof of concept, and is perhaps not ready for production. Use at your own risk!

Spin up ami-4a0df023 as m1.large or bigger. This is an install of OpenIndiana oi_147.

Authentication

The image doesn't currently import your EC2 keypairs, so you'll need to log in as a user. root is a role in this image, so you'll need to log in as the oi user.

The oi user's password is "oi".

# ssh [email protected] The authenticity of host '1.2.3.4 (1.2.3.4)' can't be established. RSA key fingerprint is da:b9:0e:73:20:81:4f:a2:a7:91:0d:7d:3c:4b:cb:80. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '1.2.3.4' (RSA) to the list of known hosts. Password: Last login: Sun Apr 17 06:54:24 2011 from domu-3-4-5-6 OpenIndiana SunOS 5.11 oi_147 September 2010 oi@openindiana:~$ pfexec su - Password: OpenIndiana SunOS 5.11 oi_147 September 2010 root@openindiana:~#

To make root a real user again, use the following commands:

$ pfexec rolemod -K type=normal root $ pfexec perl -pi -e 's/PermitRootLogin no/PermitRootLogin yes/' /etc/ssh/sshd_config $ pfexec svcadm restart ssh

You can now log in as root as you'd expect. This behavior is changing all over the place (including Solaris 11 proper), but I don't mind being a dinosaur.

Caveats

There are some limitations, however.

Boot Environments

You have no console access

If an upgrade fails, you can't choose a working BE from grub

For the same reason, you won't be able to boot failsafe

Zones

You won't be able to pull an IP from EC2 for your zones

Only one Elastic IP can be assigned to an instance, so you won't be able to forward a public IP to an internal zone IP

You'll be able to forward ports from the global zone to zones, of course, but this is less useful than zones having unique IPs associated with them

EBS

There is a bug in devfsadmd which doesn't like device IDs over 23. I describe how to deal with this below.

Boot Volumes

There are two EBS volumes assigned with this AMI. The 8GB one is the root pool device. The 1GB is where the boot loader (pv-grub) lives.

Triskelios joked earlier that your instance got into a hosed state, you could mount the 1GB volume elsewhere, munge your grub config, then assign it back to your the busted instance. This theoretically gets around not having console access. It's also hilarious. But could work.

Upgrading to oi_148

oi_147 has some known bugs, so we want to get up to oi_148. You could also update to the OpenIndiana illumos build (dev-il), but we'll stick with 148 for now.

The old opensolaris.org publisher is still available, as there is software on it not available in OpenIndiana's repo. However, we need to set that publisher non-sticky so it doesn't hold back package upgrades. If you don't set the repo non-sticky, you won't get a complete upgrade. You'll be running a 148 kernel, but lots of 147 packages. One symptom of this is zones won't install.

root@openindiana:~# pkg publisher PUBLISHER TYPE STATUS URI openindiana.org (preferred) origin online http://pkg.openindiana.org/dev/ opensolaris.org origin online http://pkg.openindiana.org/legacy/ root@openindiana:~# pkg set-publisher --non-sticky opensolaris.org root@openindiana:~# pkg publisher PUBLISHER TYPE STATUS URI openindiana.org (preferred) origin online http://pkg.openindiana.org/dev/ opensolaris.org (non-sticky) origin online http://pkg.openindiana.org/legacy/

Once that's done, we update the current image.

root@openindiana:~# pkg image-update Packages to remove: 4 Packages to install: 7 Packages to update: 531 Create boot environment: Yes DOWNLOAD PKGS FILES XFER (MB) Completed 542/542 11254/11254 225.5/225.5 PHASE ACTIONS Removal Phase 1882/1882 Install Phase 2382/2382 Update Phase 19953/19953 PHASE ITEMS Package State Update Phase 1073/1073 Package Cache Update Phase 535/535 Image State Update Phase 2/2 A clone of openindiana exists and has been updated and activated. On the next boot the Boot Environment openindiana-1 will be mounted on '/'. Reboot when ready to switch to this updated BE. --------------------------------------------------------------------------- NOTE: Please review release notes posted at: http://docs.sun.com/doc/821-1479 ---------------------------------------------------------------------------

A new BE has been created for us, and is slated to be active on reboot.

root@openindiana:~# beadm list BE Active Mountpoint Space Policy Created -- ------ ---------- ----- ------ ------- openindiana N / 94.0K static 2011-04-04 23:00 openindiana-1 R - 3.90G static 2011-04-17 07:20 root@openindiana:~# reboot

Once the instance comes back up, log in:

# ssh [email protected] Password: Last login: Sun Apr 17 06:54:25 2011 from domu-3.4.5.6 OpenIndiana SunOS 5.11 oi_148 November 2010 oi@openindiana:~$ pfexec su - Password: OpenIndiana SunOS 5.11 oi_148 November 2010

As you can see, we're running oi_148.

To be sure the upgrade is happy and we don't have any sticky 147 bits left:

root@openindiana:~# pkg list | grep 147 root@openindiana:~#

Awesome.

ZFS on EBS

Create some EBS volumes and attach them to the instance. You'll need to specify a Linux-style device path for the volume. The older OpenSolaris AMI required a numeric device, as Solaris expects device IDs to be 0..23. This either seems to have been broken at some point in the last year, or doesn't work with pv-grub. Regardless, we can work around it.

$ ./aws create-volume --size 128 --zone us-east-1b $ ./aws create-volume --size 128 --zone us-east-1b $ ./aws create-volume --size 128 --zone us-east-1b $ ./aws create-volume --size 128 --zone us-east-1b $ ./aws attach-volume vol-vvvvvvvv -i i-iiiiiiii -d /dev/sdc $ ./aws attach-volume vol-vvvvvvvv -i i-iiiiiiii -d /dev/sdd $ ./aws attach-volume vol-vvvvvvvv -i i-iiiiiiii -d /dev/sde $ ./aws attach-volume vol-vvvvvvvv -i i-iiiiiiii -d /dev/sdf

Once the volumes are available, you'll see messages like this in dmesg:

failed to lookup dev name for /xpvd/xdf@2128 disk_link: invalid disk device number (2128)

Which is the devfsadmd bug I mentioned above. Solaris expects device IDs to be 0..23, and devfsadm doesn't know how to deal with anything higher.

There's very likely a way to automate this, but I just wrote a stupid script that creates links in /dev/dsk and /dev/rdsk for the devices we've attached to the instance. Until the devices have the proper links, you won't see them in format or iostat. And cfgadm doesn't work in a Xen guest, so.

The device IDs are consistent, however. The first two disks in the system (the rpool and the pv-grub volumes) are 2048 and 2064. The device IDs increment by 16:

root@openindiana:~# format < /dev/null Searching for disks... Failed to inquiry this logical diskdone AVAILABLE DISK SELECTIONS: 0. c0t0d0 <drive type unknown> /xpvd/xdf@2048 1. c0t1d0 <??????HH???X?[??????? cyl 4095 alt 0 hd 128 sec 32> /xpvd/xdf@2064 Specify disk (enter its number):

So now we link in the new devices:

root@openindiana:~# ./links.sh c0t2d0 2080 root@openindiana:~# ./links.sh c0t3d0 2096 root@openindiana:~# ./links.sh c0t4d0 2112 root@openindiana:~# ./links.sh c0t5d0 2128 root@openindiana:~# format < /dev/null Searching for disks... Failed to inquiry this logical diskFailed to inquiry this logical diskFailed to inquiry this logical diskFailed to inquiry this logical diskFailed to inquiry this logical diskdone AVAILABLE DISK SELECTIONS: 0. c0t0d0 <drive type unknown> /xpvd/xdf@2048 1. c0t1d0 <??????HH???X?[??????? cyl 4095 alt 0 hd 128 sec 32> /xpvd/xdf@2064 2. c0t2d0 <??????HH???X?[??????? cyl 16709 alt 0 hd 255 sec 63> /xpvd/xdf@2080 3. c0t3d0 <??????HH???X?[??????? cyl 16709 alt 0 hd 255 sec 63> /xpvd/xdf@2096 4. c0t4d0 <??????HH???X?[??????? cyl 16709 alt 0 hd 255 sec 63> /xpvd/xdf@2112 5. c0t5d0 <??????HH???X?[??????? cyl 16709 alt 0 hd 255 sec 63> /xpvd/xdf@2128 Specify disk (enter its number):

Create our ZFS pool:

root@openindiana:~# zpool create tank mirror c0t2d0 c0t3d0 mirror c0t4d0 c0t5d0 root@openindiana:~# zpool list tank NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT tank 254G 80K 254G 0% 1.00x ONLINE -

Once the EFI labels have been written to the disks, format stops throwing errors on them, as well:

root@openindiana:~# format < /dev/null Searching for disks... Failed to inquiry this logical diskdone AVAILABLE DISK SELECTIONS: 0. c0t0d0 <drive type unknown> /xpvd/xdf@2048 1. c0t1d0 <??????HH???X?[??????? cyl 4095 alt 0 hd 128 sec 32> /xpvd/xdf@2064 2. c0t2d0 <Unknown-Unknown-0001-128.00GB> /xpvd/xdf@2080 3. c0t3d0 <Unknown-Unknown-0001-128.00GB> /xpvd/xdf@2096 4. c0t4d0 <Unknown-Unknown-0001-128.00GB> /xpvd/xdf@2112 5. c0t5d0 <Unknown-Unknown-0001-128.00GB> /xpvd/xdf@2128 Specify disk (enter its number):

And, just for fun:

root@openindiana:~# dd if=/dev/urandom of=/tank/random bs=1024 count=204800 204800+0 records in 204800+0 records out 209715200 bytes (210 MB) copied, 9.14382 s, 22.9 MB/s

So now you have ZFS on EBS, with the ability to do anything ZFS can. Snapshots will be much, much faster than EBS snapshots (though not complete copies, and will obviously be lost if your pool is lost, whereas EBS snapshots are complete copies of the volume and can be cloned and mounted out of band), enable compression, dedup (though this would probably be terrifyingly slow on EC2), and so on.

links.sh is available here.

Virtual Networking

This is fodder for another post, but something I've done elsewhere is to use Crossbow to create a virtual network with zones and VirtualBox VMs. The global zone runs OpenVPN, giving clients access to these private resources. This model seems perfectly suited to EC2, given the IP assignment limitations noted above. Unfortunately I don't' imagine VirtualBox is an option here, but even just a private network of zones would be extremely useful.

And perhaps someday EC2 will let you assign multiple Elastic IPs to an instance.

Conclusion

While there are still a few bugs to be worked out, this proof of concept AMI does work and is ready to have its tires kicked.

I'm pretty stoked to have Solaris available on EC2. Thanks Andrzej and Nexenta both!

[mirrorshades]

Building python26@pkgsrc on Solaris

Python fails to build its socket and ssl modules. This is fixed in 2.7, but won't be backported to 2.6.

This recent post to pkg@netbsd details the problem and links to the patch:

http://mail-index.netbsd.org/tech-pkg/2011/04/11/msg006984.html

[mirrorshades]

Building DBD::mysql with SUNWspro and pkgsrc on Solaris 11 Express

I use resmon, which is a pretty nice system metric aggregator. It relies on the system Perl specifically for Solaris::Kstat, and so you don't have to install pieces of the CPAN to get it running. Earlier tonight I decided to point its default MySqlStatus module at our MySQL master and ran into a few annoyances.

Historically, getting DBD::mysql installed with the system Perl has proven somewhat painful.

We use the packages from mysql.com, whose libmysqlclient is not built shared, so you can't build DBD::mysql against them. I could have installed pkg:/database/mysql-5? but I already have a MySQL install via pkgsrc.

So, simply:

Make sure /opt/SUNWspro/bin/cc is first in your path, and:

# /bin/perl Makefile.PL --libs="-L/usr/pkg/lib/mysql -R/usr/pkg/lib/mysql -lmysqlclient -lz" --cflags="-I/usr/pkg/include/mysql -I/usr/include -m32" # make && make install

And huzzah. DBD::mysql.

[mirrorshades]

Perl 5.12.2, Solaris, Sun Studio, -m64, -Dvendorprefix woes.

****UPDATE**** Nick Clark dug into this and determined it's a bug in Sun Studio 12.1. Use 12.2 to build Perl. If anyone wants at Oracle wants to buy him some beers, send someone from Sun with them. ****UPDATE****

I spent a fair chunk of yesterday afternoon (between diaper changing, swaddling, swinging, singing, and so forth) debugging a weird problem with Perl 5.12.2 on Solaris.

I had been deploying an updated pkgsrc build with ABI=64 and Sun^WSolaris Studio 12.1 for a new project, and ran into perl@pkgsrc segfaulting on certain modules. Extremely weird. I pulled the source and built that without issue, adding only -Dcc=cc -Accflags='m64' Aldflags='-m64' to build it 64bit with Studio.

This particular project requires deploying Perl modules in tiers, and I thought I would use vendor_perl for stuff I want installed by default that may not necessarily need to live in site_perl. As soon as I rebuilt Perl with -Dvendorprefix the same modules started throwing segv at me.

About five hours of rebuilds later (works fine with gcc, Studio and 32bit, 64bit on Linux with gcc, etc), and here's the bug report.

Narrowing it down to that I just decided to use APPLLIB_EXT and site_perl.

Very weird.

[mirrorshades]

Building nginx@pkgsrc on Solaris/sspro w

Hosed by default. See this post.

For amd64, you'll want to use

CONFIGURE_ENV+= NGX_AUX=" src/os/unix/ngx_sunpro_amd64.il"

You can also just add that to the Makefile.

[mirrorshades]

The First Law of Systems Administration

This post details an outage I caused this week by making several poor decisions. Each point contains lessons I have learned over the past 10 years, and in this instance studiously ignored. Things I am typically very careful to avoid doing. My record for not breaking things is actually pretty decent, but when I do break things it tends to occur under the same set of circumstances (I'm tired and in a hurry). Even with a decade of experience and a process that mitigates failures, I managed to do something really, really dumb. A couple months ago I attended Surge in Baltimore, a conference whose focus is on scalability and dealing with failures. The best talks came down to "this is how we broke stuff, and this is how we recovered." Hopefully illuminating this particular failure will not just help someone else recover from something similar, but remind my fellow sysadmins that sometimes you just need to take a nap. ## The First Law Backups. Never do anything unless you have backups. ### Stupidity the First A few weeks ago I added an OCZ Vertex 2 SSD to a ZFS pool as a write cache. These are low-end devices, with not a great MTBF, but my research suggested they would fit our needs. The pool in question is configured as an array of mirrors. The system was running Solaris 10 U7, which does not have support for either import recovery (-F), import with a missing log device (-m), or removal of log devices. I had tested the SSD for about a week, burning it in. The SSD was added without a mirror. I was quite pleased with myself: The performance increase was obvious and immediate. Good job, me, for making stuff better. A week after being added to the pool, the SSD died. The exact error from the Solaris SCSI driver was "Device is gone." The zpool hung, necessitating a hard system reset. The system came back up, with the SSD being seen as UNAVAIL. We lost whatever writes were in-flight through the ZIL, but given the workload, that was going to be either minor or recoverable. I made myself a bug to RMA the SSD and order a pair of new ones, and stopped thinking about it, annoyed that a brand new device died after less than a month. The stupid: Adding a single point of failure to a redundant system. Bonus stupid: Not more than a month ago I argued on a mailing list that you should always have a mirrored ZIL, regardless of whether or not your system supported import -F or -m. Yup. I ignored my own advice, because I wanted an immediate performance increase. Extra bonus stupid: Not fixing a problem relating to storage immediately. Sysadmins wrangle data. It's what we do and when we do it well, it's why people love us. Leaving a storage system in a hosed, if working, state, is just asking for pain later. Begging for it. ## The Second Law You are not a computer. Sometimes you are just too tired to work. Never do anything when your judgement is impaired. In particular, never make major decisions without confirmation when you are overtired (and had, perhaps, just gotten a flu shot). It leads to calamities. As sysadmins we often have to work on little sleep in non-optimal situations or environments. We sometimes take it as a point of pride that we can do incredibly complex things when we're barely functional. At some point you are going to screw yourself, though. One thing I know about myself: I get really stupid when I'm too tired. If I get woken up at 0300 by a page, I can muscle-memory and squint my way to a fix. If I've been up for 14-16 hours and I've been getting say, maybe, four hours of sleep a night for the past two months? I'm going to do something dumb. ### Stupidity the Second I have been upgrading systems to U9 over the last few weeks. The system with the UNAVAIL SSD came up on the rotation. With U9 I'd be able to remove the dead log device. We announced a 30m outage. And here is where impaired judgement comes in. If the following two thoughts are in your head: * I am exhausted * I just want to get this done Stop whatever it is you're doing. Go take a nap. Wait until a co-worker is around so they can tell you "holy crap, why are you eating live scorpions covered in glass? Stop that stupid thing you are doing!" My wife is well aware that I do stupid things when I'm tired and tells me "do that later. Go to bed." Listen to my wife. I decided to go ahead and upgrade the system with the DEGRADED pool. I have rolling backups for everything on the system except the dataset containing our spam indexes (which are required so customers can view spam we have discarded for them, and release false positives). Rather than wait to sync that dataset off-system (3-4 hours, and why hadn't I just started a rolling sync earlier that day? Or had one for the last two years?) I decided to go ahead and upgrade the system. The stupid: Why would you ever put unique data at risk like this? Bonus stupid: Why is the data unique? There is no reason for it to be so. Replicating ZFS is trivial. Oversights happen, but this is still dumb. (My systems all live in pairs. With very few exceptions there are no snowflake services. I take snapshots of my MySQL master. I replicate them, so I can clone and boot them to restore data quickly. I have MySQL replication set up so I can do hot failovers. I have zones replicated via ZFS, I have backups of /etc and /usr/pkg/etc even though the configs are all in git. I replicate all other big datasets to cross-site failover systems with standby zones. I do backups. So why, in my big table of datasets, does this one thing have a big TODO in the replicate column?) Postpone the maintenance window. It's ok. Sometimes scheduling conflicts come up. Sometimes you aren't as prepared as you thought you were. Your customers don't care that they weren't supposed to be able to access something for 30 minutes tonight, but instead can't tomorrow night. Really. Get some sleep. Wake up tomorrow and feel lucky you didn't totally break something and potentially lose unrecoverable data. ## The Third Law Don't make a problem worse. Especially if you caused it. Never do anything to disks which contain data you need, even if that data is currently inaccessible. Move the workload somewhere else. Hope you think of something. You are already eating live scorpions covered in glass, don't go setting them on fire too. ### Stupidity the Third I exported the pool and restarted the system. It Jumpstarted happily. I logged in and... # zpool import pool: tank id: 17954631541182524316 state: UNAVAIL status: One or more devices are missing from the system. action: The pool cannot be imported. Attach the missing devices and try again. see: http://www.sun.com/msg/ZFS-8000-6X config: tank UNAVAIL missing device mirror-0 ONLINE c0t2d0 ONLINE c0t3d0 ONLINE mirror-1 ONLINE c0t4d0 ONLINE c0t5d0 ONLINE Additional devices are known to be part of this pool, though their exact configuration cannot be determined. # zpool import -F tank cannot import 'tank': one or more devices is currently unavailable Destroy and re-create the pool from a backup source. At this point there was a fair amount of cursing. The thing is, I knew the pool was fragile. I knew that reinstalling the system was going to blow away /etc/zfs/zpool.cache, which is likely the only reason U7 was happy to import the pool after the SSD died initially and it got rebooted. But my judgement was impaired: I was making really stupid decisions. The stupid: Doing something irrevocably destructive to a fragile, unique system. ## Regretful Morning At this point I was screwed. I couldn't import the pool. I had no backups. I got critical zones back up on other systems (using data that had been replicating off the now hosed box), so services would not be unduly affected. Everything was back up, but customers couldn't see messages we had discarded for them, and as such not release important mail that had been improperly discarded. After an hour of trying various things (like logfix, and booting newer instances of Solaris) I gave up. At 0430, I woke up my co-worker Rik, and explained I had totally screwed us. "That does sound pretty bad." I stood up another zone so we could start importing the last seven days of messages from the message queue (which we keep as a hedge in case something just like this happens, though I doubt anyone expected me to be the cause). In the process of this, he rewrote the reindexing system to make it an order of magnitude faster. We went from the refill taking 2 days to 6 hours. ## The Road to Recovery Once the refill was running my body shut down for five hours. My brain working slightly better, I started thinking: I had a copy of the old zpool.cache, which contained configuration about the now-defunct tank pool. But how could I turn that into something useful? Keep in mind: My data was on the disk. No corruption has occurred. It was just my version of ZFS that didn't want to import the pool with a missing log device. How could I force it to? I had thought about several things before crashing: The logfix tool basically replaces a missing log device with another by walking the ZFS metadata tree, replacing the device path and GUID with another device or a file. Okay, I could try something like that, right? But the code needs Nevada headers or Nevada. I came back up to James McPherson having built a logfix binary for Solaris 10. Unfortunately it didn't work (but also didn't eat anything, so props to James). So if logfix wasn't going to work, I was going to have to do something really complicated. Digging around with zdb. Terrifying. James got me in touch with George Wilson, who had written the zpool import recovery code in the first place. He suggested some things, including: # zpool import -V -c /etc/zfs/zpool.cache.log tank cannot open 'tank': no such pool Well, that's not good. zpool import by itself can see the pool, but can't import. Specifying the secret recovery flag (-V) doesn't help, using the alternative cache file that has configuration for the log device claims to not even see the pool! However: # zpool import -V -c /etc/zfs/zpool.cache.log pool: tank id: 17954631541182524316 state: DEGRADED status: One or more devices are missing from the system. action: The pool can be imported despite missing or damaged devices. The fault tolerance of the pool may be compromised if imported. see: http://www.sun.com/msg/ZFS-8000-2Q config: tank DEGRADED mirror-0 ONLINE c0t2d0 ONLINE c0t3d0 ONLINE mirror-1 ONLINE c0t4d0 ONLINE c0t5d0 ONLINE logs c0t6d0p1 UNAVAIL cannot open Okay, so I can see the pool using the old configuration data, but I can't import it. And it's seen as DEGRADED, not UNAVAIL. It's importable. That suggests that I don't need to go digging around with zdb or a hex editor. George is also starting with the import command, not a hex editor. That seems to imply he thinks it's recoverable. (I get that sinking feeling that something you thought was going to be really complicated and dangerous is, in fact, trivial. And you've realized long, long after you should have.) So: -V is the old import switch. I bet that would work on U7. U9 has an actual recovery mechanism now. Maybe... # zpool import -F -c /etc/zfs/zpool.cache.log tank Pool tank returned to its state as of Thu Nov 04 01:25:50 2010. # zpool list NAME SIZE ALLOC FREE CAP HEALTH ALTROOT rpool 136G 2.05G 134G 1% ONLINE - tank 272G 132G 140G 48% DEGRADED - Twelve hours later, there is much more cursing. ### Ghost of the Arcane A lot of UNIX comes down to reading documentation and determine which switches are going to solve your immediate problem. Here, it's two: -F and -c. That's it. Let's assume that twelve hours previous I was well-rested but still astoundingly dumb, and had managed to get myself into the situation where my pool was UNAVAIL. Because I was well-rested, I would have read the docs, understood them, and recovered the pool within a few minutes. Instead, I had to recharge my brain, created a lot of work for my co-workers, and annoyed my customers. Good job! Ok. Now I want to get rid of the busted log device. The newly imported degraded pool is on ZFS v10. I need to get it to at least v19, which is when log device removal was added. Thankfully U9 supports v22. # zpool upgrade tank This system is currently running ZFS pool version 22. Successfully upgraded 'tank' from version 10 to version 22 And get rid of the dead log device: # zpool remove tank c0t6d0p1 # zpool status -v tank pool: tank state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 c0t2d0 ONLINE 0 0 0 c0t3d0 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 c0t4d0 ONLINE 0 0 0 c0t5d0 ONLINE 0 0 0 errors: No known data errors And the pool is back online in a usable state. Before we make any changes to the newly recovered pool I take a snapshot and send it to another system. This takes a few hours. This means that if the new indexer has a bug that doesn't interact with the existing index, we'll be able to go back to the pristine data. I also start up the rolling replication script on the dataset. The first send takes a few hours; incrementals 20-30 minutes. Both of those things should have already been in place. ## That How They Do Shortly before I got the pool back online, the 7 day import had finished and we had announced to customers they could get back to seeing their discarded messages. Well, now I had the last 30 days of spam, and all the metadata that went with it. Rebuilding the other 23 days on the new index was going to be both non-trivial and slow. We would have to pull the information off disk for each message (around 2TB of spam), and some data was only stored in the index. The decision was made to revert to the original dataset. I pointed the new index refiller at it, and 9 minutes later we had the last 12 hours of spam indexed. We swapped around, merged the metadata from the temporary dataset into the original one, and we were back online. We made the announcement, wrote a blog post, and everything was good again. Almost as if I had never done anything incredibly stupid. ## Coda ### Postmortem Maybe you are the lone SA at a small company, but you still have resources to ask for advice. There are certainly people on IRC whose opinion I value. Your boss and co-workers may not know as much about systems as you do, but they can probably recognize a three-legged chair when it's in front of them. It is easy to do stupid shit in a vacuum. Talking to people about it is probably enough for you to recognize if it's a bad idea. I'll have another post coming up (with pretty graphs, hopefully) abut hybrid storage pools and their impact on performance. Two SSDs just came in to act as a mirror for this host, so it should be interesting. ### Your Co-workers You have broken something. You feel dumb and defensive, and pissed off at yourself. Don't take it out on the people who are helping you get the system back online. When you break something and can't fix it, you create work for other people. Make sure you thank them and apologize. Act like a professional, or even just a regular human being. I can think of a few instances where Ricardo Signes has had to save my bacon in the last few years, but probably nothing so major as this case. I had to wake him up at 0430 to give me a hand, and while he's paid to do it, it's unfortunate how rare it is to find people as pleasant and professional as he is. Over the years I've worked with lots of smart people, but few as smart and even-tempered as rjbs. Manhug! ### Wheaton's Law A brief tangent. Sysadmins are admittedly used to other people breaking things and wanting us to fix it. Treat your co-workers, customers, and users with respect. Do not call them lusers, do not make them feel bad. It is extremely aggravating at times, but they are not a puppy who just had an accident on your new carpet. They are adults, and your colleagues. At some point you may find yourself on the other side of the table: You have done something and now they can't get any work done. Hopefully they will recall that when they screwed up, you did not berate them, and will afford you the same courtesy. Educate them after you have solved their problem. Don't be a dick. ### Thanks Special thanks to James McPherson of Oracle/Sun and George Wilson of Delphix (previously of Oracle/Sun) for giving me a hand. George pointed me to -V and -c which finally helped me realize just how dumb I was being and got my pool back online. ### Vendor Support Once I realized I was screwed and got the immediate booms out of the way, I opened a case with Oracle. P1, at 0700. A rep got back to me around 1900. Nearly 12 hours later. For a "system down" event, affecting many customers, on a paid support contract. Andre van Eyssen says: If you have a P1 problem, call it in. Don't use SunSolve. Make the call. A support contract is not a panacea. Design your systems to be redundant and resilient. ### And don't do stupid shit when you're tired.

[mirrorshades]

textproc/libxslt on pkgsrc/solaris

Requires [this patch](http://mail-index.netbsd.org/tech-pkg/2010/08/23/msg005905.html) or you get symbol errors on compile. Need this for devel/hg-git. Working on getting illumos-gate pushed into github ([#105](http://www.illumos.org/issues/105)).

[mirrorshades]

apr bug on pkgsrc/Solaris x86

While working on the illumos infrastructure roll-out I ran into an issue with apache22 segfaulting, but .. mostly working, when SSL was enabled. Disabling SSL seems to fixed the issue. Other SSL-enabled programs were not affected. Turns out it's an apr bug, and kind of deeply weird. Described [here](http://mail-index.netbsd.org/pkgsrc-users/2010/02/25/msg011788.html).

[mirrorshades]

On software.

(grumpy face) < bdha> I see its value. < bdha> I just hate it. < bdha> As a sysadmin I feel I am allowed to feel that way.

[mirrorshades]

Segfaults with SSL-enabled and pkgsrc2010qN/solaris

Starting with 2010q1 I noticed that anything built with openssl, including, well, openssl, would segfault. To fix this, compile the openssl package with SSPRO. Don't forget to modify your mk.conf: #PKGSRC_COMPILER= gcc PKGSRC_COMPILER= sunpro # For sunpro CC= cc CXX= CC CPP= cc -E CXXCPP= CC -E I previously tried to compile everything with pkgsrc's gcc, but perhaps I'll change that policy now. Fix defined [here](http://wtf.hijacked.us/wiki/index.php/Pkgsrc_on_solaris#security.2Fopenssl).

[mirrorshades]

nginx on pkgsrc2010q2/solaris.

To compile the nginx package, you need to remove the patch-aa patch from distfiles. You also need to create hacks.mk and add the following to it: .if (${PKGSRC_COMPILER} == sunpro) .if (${MACHINE_ARCH} == i386) CONFIGURE_ENV+= NGX_AUX=" src/os/unix/ngx_sunpro_x86.il" .endif .endif Problem report and fix [here](http://mail-index.netbsd.org/pkgsrc-users/2010/03/20/msg011950.html).

[mirrorshades]

Building Postfix on OpenSolaris >=b130

NIS was finally sent into the cornfield around ONNV b130. Postfix does not have a definition for OpenSolaris (arguably 5.11), just Solaris 5.10. When building, it attempts to compile dict_nis and can't, unsurprisingly. To build, remove "#define HAS_NIS" from src/util/sys_def.h in the "#ifdef SUNOS5" section. With pkgsrc 2010q1, apply this diff to pkgsrc/mail/postfix/patches/patch-ag. The checksum is 3ea7ecaec06b0ff30fe1a1b2f5197def0219bd6b.

[mirrorshades]

Elsechan... < bda> I guess the X2270 is coming in soon. The power cables are here. < e^ipi> unless it's the computer columbian drug lords and the power cable is a warning < e^ipi> like a toe

[mirrorshades]

ZFS and iSCSI

I was asked to share out the pool on the X4500 via NFS and iSCIS. NFS I was familiar with, and have used a fair amount. iSCSI for all its new hotness factor, I've never touched. I was unsurprised, but pleased, with how trivial it is to set up. On the server (the target): x4500# zfs create tank/iscsi x4500# zfs set shareiscsi=on tank/iscsi x4500# zfs create -s -V 25g tank/iscsi/vol001 x4500# zfs create -s -V 25g tank/iscsi/vol002 x4500# zfs create -s -V 25g tank/iscsi/vol003 x4500# zfs create -s -V 25g tank/iscsi/vol004 x4500# zfs create -s -V 25g tank/iscsi/vol005 x4500# zfs create -s -V 25g tank/iscsi/vol006 x4500# zfs list tank tank 1.39G 13.3T 53.3K /tank tank/iscsi 1.54M 13.3T 44.8K /tank/iscsi tank/iscsi/vol001 246K 13.3T 246K - tank/iscsi/vol002 246K 13.3T 246K - tank/iscsi/vol003 247K 13.3T 247K - tank/iscsi/vol004 262K 13.3T 262K - tank/iscsi/vol005 263K 13.3T 263K - tank/iscsi/vol006 264K 13.3T 264K - tank/nfs 1.39G 13.3T 1.39G /tank/nfs This will start the iSCSI service (`iscsigtd`) and share not only the parent volume (`tank/iscsi`) but all the children as well. Accessing and using the disks on the client (the initiator) is just as easy: client# iscsiadm modify discovery --sendtargets enable client# iscsiadm add discovery-address 10.0.100.40 client# svcadm enable initiator client# iscsiadm list target Target: iqn.1986-03.com.sun:02:7ea6450a-4a26-cfe5-d679-fe0dbabe66b9 Alias: tank/iscsi/vol001 TPGT: 1 ISID: 4000002a0000 Connections: 1 Target: iqn.1986-03.com.sun:02:61d314ce-f4b3-ed1f-9891-e0c6c52f5601 Alias: tank/iscsi/vol002 TPGT: 1 ISID: 4000002a0000 Connections: 1 Target: iqn.1986-03.com.sun:02:eaea5a32-f54a-6d04-a453-888a580504c2 Alias: tank/iscsi/vol003 TPGT: 1 ISID: 4000002a0000 Connections: 1 Target: iqn.1986-03.com.sun:02:05272769-bd4a-6b54-8d6f-f525af20ad08 Alias: tank/iscsi/vol004 TPGT: 1 ISID: 4000002a0000 Connections: 1 Target: iqn.1986-03.com.sun:02:fb296689-109e-cb6d-9515-a07f581a81ce Alias: tank/iscsi/vol005 TPGT: 1 ISID: 4000002a0000 Connections: 1 Target: iqn.1986-03.com.sun:02:ec71ac8e-e417-ce3f-891c-ee1febdf9120 Alias: tank/iscsi/vol006 TPGT: 1 ISID: 4000002a0000 Connections: 1 client# format < /dev/null Searching for disks...done AVAILABLE DISK SELECTIONS: 0. c0d0 /pci@0,0/pci-ide@1,1/ide@0/cmdk@0,0 1. c1t600144F04B66BAA30000144F21056400d0 /scsi_vhci/disk@g600144f04b66baa30000144f21056400 2. c1t600144F04B66BAA40000144F21056400d0 /scsi_vhci/disk@g600144f04b66baa40000144f21056400 3. c1t600144F04B66BAA50000144F21056400d0 /scsi_vhci/disk@g600144f04b66baa50000144f21056400 4. c1t600144F04B66BAA60000144F21056400d0 /scsi_vhci/disk@g600144f04b66baa60000144f21056400 5. c1t600144F04B66BAA80000144F21056400d0 /scsi_vhci/disk@g600144f04b66baa80000144f21056400 6. c1t600144F04B66BAA90000144F21056400d0 /scsi_vhci/disk@g600144f04b66baa90000144f21056400 client# zpool create tank \ raidz c1t600144F04B66BAA30000144F21056400d0 c1t600144F04B66BAA40000144F21056400d0 c1t600144F04B66BAA50000144F21056400d0 \ raidz c1t600144F04B66BAA60000144F21056400d0 c1t600144F04B66BAA80000144F21056400d0 c1t600144F04B66BAA90000144F21056400d0 client# zpool status -v tank pool: tank state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 raidz1 ONLINE 0 0 0 c1t600144F04B66BAA30000144F21056400d0 ONLINE 0 0 0 c1t600144F04B66BAA40000144F21056400d0 ONLINE 0 0 0 c1t600144F04B66BAA50000144F21056400d0 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 c1t600144F04B66BAA60000144F21056400d0 ONLINE 0 0 0 c1t600144F04B66BAA80000144F21056400d0 ONLINE 0 0 0 c1t600144F04B66BAA90000144F21056400d0 ONLINE 0 0 0 errors: No known data errors Very nice.

[mirrorshades]

X4500 and ZFS pool configuration

---- **UPDATED: 2010-01-31 2245** RAS guru [Richard Elling][] notes a couple bad assumptions I've made in this post: * That disks are faster than the controller, they are not, so you gain nothing by spreading across controllers * That controller failure is a worry. It is not: [ZFS RAID recommendations: space vs U_MTBSI][] Much thanks to Richard for correcting me! [Richard Elling]: http://blog.richardelling.com/ [ZFS RAID recommendations: space vs U_MTBSI]: http://blogs.sun.com/relling/entry/zfs_raid_recommendations_space_vs ---- I was recently asked to help install and configure a Sun X4500 ("Thumper"). The system has dual Opterons, 16GB RAM, and 48 500GB SATA disks. The largest pool I'd configured before this project was a Sun J4200: 24 disks. The Thumper controller/disk setup looks like this:  That's six controllers, with 46 disks available for data. The mirrored ZFS rpool are on `c5t0` and `c4t0`. Placing the mirror halves across controllers allows the operating system to survive a controller failure. ZFS supports two basic redundancy types: Mirroring (RAID1) and RAIDZ (akin to RAID5, but More Gooder). RAIDZ1 is single parity, and RAIDZ2 double. I decided to go with RAIDZ2 as the added redundancy is worth more than capacity: The 500GB disks can trivially be swapped out for 1TB or 2TB disks, but the pool cannot be easily reconfigured after creation. From the ZFS Best Practices and ZFS Configuration guides, the suggested RAIDZ2 pool configurations are: * 4x(9+2), 2 hot spares, 18.0 TB * 5x(7+2), 1 hot spare, 17.5 TB * 6x(5+2), 4 hot spares, 15.0 TB * 7x(4+2), 4 hot spares, 12.5 TB ZFS pools consist of virtual devices (vdev), which can then be configured in various ways. In the first configuration you are making 4 RAIDZ vdevs of 11 disks each, leaving 2 spares. (ZFS pools are quite flexible: You could set up mirrors of RAIDZs, three-way mirrors, etc. In addition to single and dual parity RAIDZ, RAIDZ3 was recently released: Triple parity!) Distributing load across the controllers is an important performance consideration but limits possible pool configurations. Preferably you want each vdev to have the same number of members. Surviving a single controller failure is also required. RAIDZ2 is double parity, so you lose the "+2" disks noted above, but the vdev can sustain two disk losses. The entire pool can then survive the loss of `vdevs*2` disks. This is pretty important given the size of the suggested vdevs (6-11 disks). The ZFS man page recommends that RAIDZ vdevs not be more than 9 disks because you start losing reliability: More disks and not enough parity to go around. The likelihood of losing more than two disks in a vdev with 30 members, instance. The goal is to balance number of vdev members with parity. A vdev can be grown by replacing every member disk. Once all disks have been replaced, the vdev will grow, and the pools total capacity is increased. If the goal is to incrementally increase space by replacing individual vdevs, it can be something of a hassle if you have too many (say, 11) disks to replace before you get any benefit. The process for growing a vdev is: Replace a disk, wait for new disk to resilver, replace a disk, wait for new to resilver, replace a disk... it is somewhat time consuming; not something you want to do very often. So the tradeoff here is really: * More initial space (18TB) but less trivial upgrade (11 disks), and ok performance * Less initial space (15TB) but more trivial upgrade (7 disks), good performance * Less initial space (12.5TB) but more trivial upgrade (6 disks), best performance With the 6x(7) or 7x(6) configurations we have four free disks. One or two can be assigned to the pool as a hot spare. The other two or three disks can be used for: * A mirror for tasks requiring dedicated I/O * Replaced with 3.5" SAS or SSDs for cache devices I'll discuss Hybrid Storage Pools (ZFS pools with cache devices consisting of SSD or SAS drives) in another post. They greatly affect pool behavior and performance. Major game-changers. Unsurprisingly the 12.5TB configuration has the highest RAS, and best performance. It loads disk across controllers evenly, has the best write throughput, is easiest to upgrade, etc. Sacrificing 6TB of capacity for better redundancy and performance may not be in line with your vision of the systems purpose. The 15TB configuration seems like a good compromise. High RAS: Tolerance to failure, good performance, good flexibility, not an incredibly painful upgrade path, and 15TB isn't anything to sneer at. (Note: After parity and metadata, 13.3TB is actually useable.) The full system configuration looks like this: pool: rpool state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM rpool ONLINE 0 0 0 mirror ONLINE 0 0 0 c5t0d0s0 ONLINE 0 0 0 c4t0d0s0 ONLINE 0 0 0 errors: No known data errors pool: tank state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 raidz2 ONLINE 0 0 0 c0t7d0 ONLINE 0 0 0 c1t7d0 ONLINE 0 0 0 c6t7d0 ONLINE 0 0 0 c7t7d0 ONLINE 0 0 0 c4t7d0 ONLINE 0 0 0 c5t7d0 ONLINE 0 0 0 c0t4d0 ONLINE 0 0 0 raidz2 ONLINE 0 0 0 c0t3d0 ONLINE 0 0 0 c1t3d0 ONLINE 0 0 0 c6t3d0 ONLINE 0 0 0 c7t3d0 ONLINE 0 0 0 c4t3d0 ONLINE 0 0 0 c5t3d0 ONLINE 0 0 0 c1t4d0 ONLINE 0 0 0 raidz2 ONLINE 0 0 0 c0t6d0 ONLINE 0 0 0 c1t6d0 ONLINE 0 0 0 c6t6d0 ONLINE 0 0 0 c7t6d0 ONLINE 0 0 0 c4t6d0 ONLINE 0 0 0 c5t6d0 ONLINE 0 0 0 c6t4d0 ONLINE 0 0 0 raidz2 ONLINE 0 0 0 c0t2d0 ONLINE 0 0 0 c1t2d0 ONLINE 0 0 0 c6t2d0 ONLINE 0 0 0 c7t2d0 ONLINE 0 0 0 c4t2d0 ONLINE 0 0 0 c5t2d0 ONLINE 0 0 0 c7t4d0 ONLINE 0 0 0 raidz2 ONLINE 0 0 0 c0t5d0 ONLINE 0 0 0 c1t5d0 ONLINE 0 0 0 c6t5d0 ONLINE 0 0 0 c7t5d0 ONLINE 0 0 0 c4t5d0 ONLINE 0 0 0 c5t5d0 ONLINE 0 0 0 c4t4d0 ONLINE 0 0 0 raidz2 ONLINE 0 0 0 c0t1d0 ONLINE 0 0 0 c1t1d0 ONLINE 0 0 0 c6t1d0 ONLINE 0 0 0 c7t1d0 ONLINE 0 0 0 c4t1d0 ONLINE 0 0 0 c5t1d0 ONLINE 0 0 0 c5t4d0 ONLINE 0 0 0 spares c0t0d0 AVAIL c1t0d0 AVAIL errors: No known data errors The command to create the tank pool: zpool create tank \ raidz2 c0t7d0 c1t7d0 c6t7d0 c7t7d0 c4t7d0 c5t7d0 c0t4d0 \ raidz2 c0t3d0 c1t3d0 c6t3d0 c7t3d0 c4t3d0 c5t3d0 c1t4d0 \ raidz2 c0t6d0 c1t6d0 c6t6d0 c7t6d0 c4t6d0 c5t6d0 c6t4d0 \ raidz2 c0t2d0 c1t2d0 c6t2d0 c7t2d0 c4t2d0 c5t2d0 c7t4d0 \ raidz2 c0t5d0 c1t5d0 c6t5d0 c7t5d0 c4t5d0 c5t5d0 c4t4d0 \ raidz2 c0t1d0 c1t1d0 c6t1d0 c7t1d0 c4t1d0 c5t1d0 c5t4d0 \ spare c0t0d0 c1t0d0 This leaves c6t0d0 and c7t0d0 available for use as more spares, for another pool or as cache devices. I feel the configuration makes for a good compromise. If it doesn't prove successful or we've misjudged the workload for the machine, we have the ability to add cache devices without compromising the pool's redundancy. That said, I'll be quite interested in seeing how it performs!