thesprocket.org

The collected works of Chris Wood: Bash-speaking Linux server wrangler, Data Center spanner monkey, Photographer, Enthusiast

[sprockers]

#linuxfu: Fixing your network interfaces after a motherboard change

I've loved Linux since I first spent a long day downloading 150 or more floppy disks-worth of Slackware in my first autumn term at University in 1992.*

There was something magical about it even then in those days before web. There was so much it could do, and you could tinker beneath the hood and add go faster stripes if you wanted.

I remember my first linux kernel compile (version 1.1.16), and how I spent hours researching how X11 rendered the desktop, so I could squeeze the maximum number of pixels out of my cheap 15" CRT monitor without the refresh rate dropping below 60Hz (I think I ended up with some very odd resolution such as 886x664).

Times have changed, but my love of Linux hasn't.

I wanted to share a story today of how wonderful Debian is, but how at the same time how utterly perplexing it can be, if you don't know what is going on.

The story starts with a server that had a faulty motherboard. Specifically, the IPMI interface (if you don't know what that is, find out - it is a sys admin's best friend) had stopped working, and no amount of rebooting or reflashing would make it work.

A replacement motherboard was found, and sent out to the Data Centre, where I got remote hands engineers to fit it. In no time at all, the engineer popped up on chat to tell me the job was done, and that the server was up and at the login prompt on the console.

But I couldn't ping it. Why?

As always, a methodical and logical approach is the best way to deal with an issue like this.

First I realised that I didn't have IPMI access to the server, and there was a very logical reason for this - it was a new motherboard, so the IPMI on this board had not been configured. I got the engineer to reboot the server and configure the IPMI via the BIOS. That worked a treat - now I had control over the power and also a console onto the server through which I could properly debug the issue.

I logged in and su'd to root via the IPMI console (isn't this brilliant - the server itself has no working network interfaces, and yet I can still log into it from a different continent!), and tried pinging the intranet gateway. Nothing.

So, what is my network config looking like? I used 'ip addr list' to tell me what IPs I had configured. I didn't have any! Why would that be? The server was booting from the same hard drive it used before the motherboard was changed. It was the same server. So surely it should have the same running config as before?

It was then I made a breakthrough. Looking closely at the output of the 'ip addr list' command, I noticed something that wasn't right. The two network interfaces on my servers are universally designated eth0 and eth1 - it is one of those constants that helps you to keep your configs uniform when you have a lot of machines to deal with. However, what I saw was eth2 and eth3!

Neither eth0 and eth1 were there at all. Now the lack of pinging made sense - the interface configs in /etc/network/interfaces are for eth0 and eth1 - there is nothing in there for eth2 or eth3. At boot time the init scripts had not found any interfaces to configure.

What I didn't immediately understand was why it had happened.

Google proved to be my friend here, and ultimately the answer is pretty straight forward, although it took a while to pin down. I'm here to save you the leg-work if you are stuck with a similar issue now.

At boot time, the udev subsystem in Debian takes a look at your hardware, and applies a bunch of rules that help determine how those devices get configured for use. This means that every time you boot your server, the devices it finds are configured in the same consistent way.

One set of rules concerns network interfaces. These are enumerated by MAC address, which is of course unique for each interface. The rules match a MAC address to a network interface name.

When the server was first built, the interfaces were added to the rules file and assigned eth0 and eth1.

When the motherboard was replaced and the server booted, udev spotted that there were new two NICs present with MAC addresses it hadn't seen before. It built rules for those interfaces, and assigned them the next available interface names - eth2 and eth3.

Makes sense, doesn't it? But it is one of those scenarios that until you've experienced it and scratched your head a bit, it is difficult to get to the bottom of.

There is nowhere obvious to look that might explain why the names of your network interfaces have changed - but that's because you rarely ever have to tinker with udev.

To fix the issue, you have to modify /etc/udev/rules.d/70-persistent-net.rules.

In that file you'll find an entry for every NIC that the udev has ever seen. That means there will be entries for your original eth0 and eth1, along with the newer entries for eth2 and eth3.

The fix is simple. Comment out the entries for the original eth0 and eth1, and then modify the eth2 and eth3 entries so that they assign the NICs as eth0 and eth1. Then save and close the file, and reboot.

Problem solved.

I love Linux. I love that it uses logic and rules to ensure that devices get enumerated in the same way every time you boot. I love it that when things break you can still look under the hood and fix them.

-- 

*That included at least 25 floppies worth of LaTeX. And nope, before you ask, I didn't install them. Do you know anyone who ever did? Oh, and yes - I really was at University in 1992.

[sprockers]

#datacenterfu: Tying things together

If you have a cabinet of servers to rack up, then you are going to have a lot of cables.

There's power, and probably several network cables going into each and every server. The other end of the cables have to go somewhere, so you're going to have cables coming back together in bunches at switches and power distribution bars.

You could have a couple of hundred cables in total in a full cabinet - and that's a lot. If you take a look below your work desk you'll almost certainly find a mess of twisted cables that are all intertangled, and that's a scenario where you've got no more than a dozen or so cables to deal with.

When you're commissioning a new cabinet of servers you are going to need a strategy to avoid a twisted mess of cabling. If you don't tackle this then you leave yourself open to several problems. Firstly, you (or more likely a remote-hands engineer) will find it difficult to trace a cable in an emergency, and secondly, you're going to impede the smooth flow of air out the back of your cabinet. Heat is your enemy in a DC, and you should do everything you can to prevent it building up - so you need to tie all your cables away to the sides.

You're probably by now thinking that you know what the answer is to the problem. You're thinking plastic cable ties, aren't you?

Well, those will do the job, but they are not the best solution. Plastic cable ties are great at holding bundles of cables together, but they are a pain if you later realise that you missed a cable from the bundle, or worse that you need to remove a cable from the bundle. They also create mess - if you chop off the end of the tie once it's in place, you'll soon find you have a lot of plastic bits to dispose of, and most likely they are scattered inside the base of your cabinet where they are hard to reach.

A better solution is to buy a long reel of double sided velcro tape. This is easy to get ahold of - just Google for something like "velcro cable roll".

Some providers sell specially pre-cut products that are the velcro version of a traditional plastic cable tie. With these you pull the end of the velcro tie through a hole in the other end of the tie to create a loop, and then stick the cable to itself. I suggest you don't buy this product. It is a faf, and you'll waste time trying to use it.

Instead, I suggest you splash out on a 10m continuous reel that has a cross-section that is around 18mm wide, along with a good pair of scissors which you'll use to cut the roll into suitable lengths.

Other widths are available, but I find 18mm to be about right - it is still possible to thread it through the square rack-mount holes in your cabinet to provide an anchor (although it is a tight fit). I also find that if you go for a tape with a smaller width - e.g. 10mm, you'll find that your anchored cables sag as the velcro fabric will bend, and also that it is more difficult to obtain a secure fit as the fabric has to be closely aligned to wrap around itself. Buy 18mm wide fabric, and none of these are an issue any longer.

The great thing about buying a 10m roll of double-sided velcro is that you are free to cut it to just the right length for each bundle of cables that you want to tie together. 

In addition there is no mess, and if you need to extract a cable from a tied bundle, you just unfasten the velcro from itself, and then reattach afterwards.

I find that when I'm commissioning a new cabinet, I'll typically use somewhere between 5m-10m of velcro cable tie, to make a good job of things. So buy a 10m roll - if you have some left you can use it to sort out that mess under your work desk!

Oh, and finally, remember that those scissors you've bought to cut the velcro to length cannot be carried in hand luggage on a plane. Think ahead!

[sprockers]

#41photos 16/41 Butterley Reservoir My favourite photo from our day out walking at Butterley reservoir. Taken 26th October 2014

[sprockers]

#41photos 15/41 Autumn in Yorkshire We'd not been here before, although it was only 30 minutes from home by car. Marcia had torn the details for this walk from a magazine months ago, but we finally got around to trying it out. The walk was a circular around Butterley reservoir, which is just above the town Marsden. Despite only being around 5kms in length, it proved to be quite challenging, thanks mostly to the steep sides to the valleys that enclose the reservoir. The views were fabulous, helped along by the autumnal colours of the bracken, which was everywhere, and the moody cloud cover. Photographed on 26th October 2014, on a Samsung NX100 with 18-55mm lens.

[sprockers]

#41photos 14/41 Chalk Outline Silhouette I love the outline effect in this photo, which was a happy accident of the position of the sun and was enhanced by various filters. Shot at Nostell Priory on 5th October 2014.

[sprockers]

#41photos 13/41 Autumnal Carpet Taken in the Pleasure Gardens at Nostell Priory on 5th October 2014. I think the photo catches the early autumnal feel quite well - there are still green leaves on the trees, but enough have fallen to cover the ground.

[sprockers]

#41photos 12/41 I ♥ The Sky At Night

I work in Sheffield. It is a wonderfully vibrant city, and we are very lucky to have a large number of talented street artists who brighten up a lot of the back streets. This mural is one of many in the city by Rocket01. I think it is fantastic!

This photo was taken on 2nd October 2014, and has a high percentage HDRScape filter applied. HDRScape seems to do especially well with complex textures, such as the brickwork seen here. 

[sprockers]

#41photos 11/41 The Pie Village Trail

This was taken on a Sunday afternoon stroll. The sunlight light had that lazy autumnal feel to it, and was making long shadows through the trees in Wither Wood as we walked. Denby Dale is home, and famous for its out-sized pies.

Photo taken 28th September 2014

[sprockers]

#41photos 10/41 Rays of sun over Thunderbridge and beyond

Taken on 21st September 2014 on another of our family walks. Tried hard to capture the amazing rays of sunlight that we could also see, then spent quite a while trying various different filters to help make the rays more obvious. I quite like the result.

[sprockers]

#41photos 9/41 Ripples and turbines

This photo was taken on 7th September 2014 at Ingbirchworth Reservoir, near where I live.It was a lovely late summer day, and we were out for a family amble, to burn off a little energy.

The ripple wasn't intentional. My son threw some stones into the water just as I pressed the shutter, and this was the result.

[sprockers]

#41photos 8/41 Aubrey, Phyllis and a kite

Taken in Norfolk during our second family holiday this summer. This little church was all that was left of the village of West Harling - just a pretty church and its graveyard,surrounded by a low wall in the middle of a field, and surrounded by forest.

I imagined the kite would look bigger on the final shot than it did.

[sprockers]

#41photos 7/41 Sunset in Normandy I took this photo from the edge of the wheat field next door to the gite in Normandy that we recently holidayed in. The Calvados countryside has a familiarity to it - it isn't so different to the UK.

[sprockers]

#41photos 6/41 Grand Depart This year the Tour de France cycle race started in the UK, with two days of stages in my home county of Yorkshire. We were lucky enough to have the route on our doorstep, so we spent last Sunday sitting in the sunshine, soaking up the atmosphere whilst we waited for the riders to flash past. ‘Grand Depart’ can be roughly translated into the Yorkshire dialect as ‘Grand Day Out’, and it was! This is my favourite photo from the day. Camera: Samsung NX100, with various filters applied

[sprockers]

#datacenterfu: Spend a little extra on your PSUs, and save a lot

This is a posting about computer power supplies.

Yes! PSUs!

Ok, so PSUs are dull as dishwater, but please allow me a couple of minutes of your time to explain how getting PSUs right in your Data Center can earn you #datacenterfu points and save your employer money.

The problem with PSUs isn't their reliability. I don't recommend buying servers with dual PSUs, but that is a story for another day. These days if you buy a server, you can be reasonably confident that you won't see the PSU fail during the whole lifetime of the hardware. No - the problem with PSUs is heat.

Put your hand over the grille at the back of a server PSU. Chances are you'll feel a fair amount of heat being pushed out through the little fan. Some of that heat might be coming from your disks or your CPUs - that depends on the physical shape of your server, but very often the heat is coming from the PSU itself.

PSUs are not very efficient in their use of electricity. A run-of-the-mill low-end server will come with a cheap PSU, which may only be 70% efficient.

Where does the other 30% go?

Heat.

This is bad for lots of reasons.

For starters, a rack full of 70% efficiency PSUs produce a lot of waste heat. 30% of the electricity they consume is going straight out the back of your servers as heat. That might be useful if you are a bedroom hoster who could do with a little help to reduce their heating bills, but in a DC heat is your enemy.

A cabinet full of inefficient PSUs could easily raise the exit temperature of your cabinet by 10C. That's a lot. The heat will start to soak through everything in your cabinet, and from my own experience, hot servers suffer more hardware failures than cool ones. Broken hardware is typically expensive to replace, particularly once you factor in paying for your visit to fit it and any revenue lost due to your outage.

Then you have to think about all that wasted electricity. 30% is a big number, and if you pay your DC for the actual kWh or kVA that you use, then you may be paying up to a third more than you have to. That could be a lot of money over the life of your servers.

There is another hidden gem in there that you might not have noticed. If you can somehow eliminate that 30% wasted electricity then you suddenly have spare electrical capacity in your cabinets that you can make use of! And remember - it is usually electrical capacity you are short of in a cabinet, not physical capacity. Eliminate the wasted electricity, and you can run additional servers in the same cabinet space reducing the total number of cabinets you need, and saving you even more money.

So the big question: Can you eliminate the extra heat and wasted electricity that your inefficient PSUs are causing you?

Not entirely, however you can make a huge difference.

I use Supermicro for my servers, and they offer a number of different PSU options for each of their chassis. Supermicro have had all of their recent PSUs tested, and given an efficiency rating. You can see at a glance which PSU is going to produce the least heat.

Of course, you can't make a device like this 100% efficient, but Supermicro do a great job, as they've got a number of units that are around 94 or even 95% efficient. 

Here is a useful #datacenterfu tip for you - sometimes it pays to purchase a PSU that can output significantly more watts of power than you need, to get the most efficient PSU for your chassis.

Consider this: Let's say you need a 300W PSU, but the 300W model is only 80% efficient. There is also a 600W PSU, and that model has 95% efficiency. The 600W PSU costs you an extra $50 to buy. Your server won't consume any extra electricity with the more powerful PSU (The excess wattage capability is all kept in reserve), in fact your PSU consumes less electricity, as it is much more efficient. Over 5 years you save much more than the $50 additional upfront cost on your electricity bills.

So - if your vendor has options on PSUs - and most will, as they like to cater for different wattage outputs - then take a look and see which the most efficient model is.

Buy that model, and your cabinet will run cooler, fewer servers will suffer hardware failures, and if you pay for the electricity you use, your bills will be considerably lower over the lifetime of your cabinet.

That has to be worth doing.

[sprockers]

#41photos 5/41 Sheepdog Trials There have been sheepdog trials at Harden Moss above Holmfirth in West Yorkshire every summer for over a hundred years. Despite growing up near here I'd never been. It was my wife's idea to go this year, and what a treat it was! The scenery and colours were amazing, as were many of the dogs. Camera: Nexus 4, with various filters, including Drama

[sprockers]

#41photos 4/41

Manila’s American Cemetery from 38 floors up.

I was on a business trip to the Philippines, and was really lucky to be staying on the 38th floor in a brand new apartment building overlooking the huge American Cemetery in Manila. I looked out from the balcony most mornings before I headed into work, taking a few snaps on my smartphone if the weather looked promising. This was the best of the bunch.  

Camera: Nexus 4, with various filters applied including HDRScape

[sprockers]

#41photos 3/41

Over the moors

This photos was taken on a long hike with my wife, on the moors beyond Langsett Reservoir in South Yorkshire. We'd left our kids with grandparents, so took the opportunity to get out and do a longer walk, despite a persistent drizzle. After a few hours the rain dried up, and the sun made brief appearances between the clouds. This photo was taken as we were heading back towards the reservoir, and captures the edge of the moor as it gives way to wooded areas and farmland along with a heavily lichened dry stone wall, which is very common in this part of the world. I love the colours.

Camera: Nexus 4, various filters applied, including HDRScape