May 2, 2022
Remastered work
seen from United States
seen from France
seen from United States
seen from United States
seen from United States
seen from United Kingdom
seen from Germany
seen from United States
seen from Singapore
seen from United States
seen from Austria
seen from China
seen from United States
seen from United States
seen from Saudi Arabia
seen from United Kingdom
seen from China
seen from Germany
seen from Russia
seen from China
Remastered work
Testing Cisco Hyperflex Auto Healing
Let’s break the Hyperflex system !
In the lab there is a 5 node Hyperflex cluster up and running. Everything is healthy and the system is almost for 60% filled, so that is closer to a real test then a lab test. There are several ways to break the cluster and I choose the easiest one. Just power down one random server and see what happens.
If you want to know more about the Auto Healing feature of Hyperflex :
Summary :
Auto healing kicks in after 2 hours when a node is missing of the Cluster. The cluster is Online, but got a status “Unhealty”. The time of Auto Healing depends of the size and amount of data on your cluster. In my case with a cluster filled of about 50% the repair time is about 3 hours. During these hours the system can still be used without any problems !
Proof that the system is Healthy :
Before I am going to break it, let me show you that the system is a healthy cluster. There are about 200 VM’s running on it with HDParm. See other post.
Although the https://<HX Cluster IP Address>/ui is still under Tech Preview, I am using it to show some things in an easy way.
Hyperflex is Online AND Healthy !
I can see all 5 servers are not having any problems.
So that you see that the system is working like a normal cluster the next step will be :
And now we break the Hyperflex Cluster :
This can be done is various ways. Nice on, or just power it off. Not gracefully of course otherwise it isn’t a test.
Unhealthy isn’t it ?
Shutting the server down without any grace will make the HX Cluster unhealthy right away.
In vCenter you can also see that the cluster is unhealthy.
Also on the Dashboard of the UI you see a big red cross which indicates that something isn’t right… right ?
The unhealthy system is also noticed in the Alarms.
And we can’t get any details of the HX5 server, because it’s powered off.
In the events we see the time when the cluster became Unhealthy.
Waiting for 2 hours :
The system is still online, but unhealthy. Autohealing is turned on, but it will kick in after 2 hours of node missing. It could be possible that the node is having maintenance or something. So we will patiently.
Autohealing is in progress :
Yeah. You will see it’s in progress. The user won’t notice a thing. The system is up and running all the time with all the data on it.
If you click the blue “I” icon you will see :
You can see how far the process is. “Time remaining before current healing operations finished” is a wrong sentence ! It’s the time that the system is busy with the Auto Healing Process.
With UCS Performance Manager I can see all the bandwidth that’s being used at the links from and to the HX Systems. When Auto Healing is busy, yeah, there is some more traffic, because we copy all the blocks instead of regenerate it.
Cluster is healthy again :
Just sit back and be patient. The Hyperflex system can take care of itself and after some hours, the system is Healthy again. You will notice in my example that we still have 1 node failure tolerance left !
In the Events you will see the time when the system is healthy again.
Although we do have an Online and Healthy system again, because the system is pretty full, there is no room for Auto Healing when another node is failing again.
The best way to solve this is to expand the cluster again with a node.
Pretty Cool, huh ? If you have any comments or questions, please ask.
Testing Cisco Hyperflex Auto Healing was originally published on blog.joostvandermade.com
Top Feature of Cisco Hyperflex
Auto Healing !
Hyperflex is cool and I really like this feature. Not only on Powerpoint, but also how it’s working.
You have to ask yourself : How can a system heal itself ? Doesn’t a RAID controller do that for storage ? Why change that into something completely different?
How does it work ?
I assume you know how RAID is working. There are different RAID variants and you can see them in the wiki. Well. It isn’t RAID that is used for the data integrity.
First let’s explain how the data is written to the nodes.
The VM is basically a FILE.VMDK. Let’s split this file into blocks. This time I won’t go in detail. Each block will be placed onto a node. Where the block is written depends on an algorithm.
Because the system doesn’t use RAID and we want to have resilience, the Hyperflex system copies every written block to 1 or 2 other nodes. This depends on the Replication Factor (RF). For production it’s better to have a RF of 3, so that every block is written 3 times on the system. The copied blocks won’t be written on a same node where there is already the same block, otherwise you still could have a Single Point Of Failure. As you can see there is no data-locality. That’s okay, because every node is connected to a Fabric Interconnect and this devices does local switching. So even though the data is spread across the cluster, the data is still very fast. Huge bandwidth and no penalties of the RAID.
If a node is shutdown or something happens with it, we’re missing some blocks on the system.
After 2 hours the Auto Healing will kick in and the system will COPY the missing block. Because it’s a COPY and not a rebuild of a RAID set, it’s very fast. The blocks are written with an algorithm that each node only got a maximum of 1 block of the 3 same blocks.
The principle of Auto Healing is easy and fast without any interruption of the business !
So far all the writtens. Do you want prove that it is working ? See my Post :
Top Feature of Cisco Hyperflex was originally published on blog.joostvandermade.com