postheadericon vSphere 4.0 Fault Tolerance (Architecture Diagram, Video and Use Cases)

This is a response to the new vSphere Blogging contest that was announced in the middle of this month. I truly think that it's a cool idea, and I believe that regardless of winning or losing, the excitement and fun a blogger would have during his/her participation is something awesome by itself.

The rules say that my post need to be compact and straight to the point, so I won't be able to cover all the aspects about something huge like FT. If by any chance I failed to write such a short post, then here are some tips to avoid wasting your time:
1 – If you are one of those people who wear a tie at work, you should jump straight to the "Use Cases" section.
2 – If you are one of those people who are using the words: 10GbE, VMkernel and %RDY, then you probably don't have time for this, but take my advice and have a look on the next two sections.

Fault Tolerance Architecture Diagram:

From the newly launched vSphere Blog on VMTN I quote this part: "[..] they do say a picture is worth a thousand words [..]", well, I believe I've said that also once before on a previous post, and in fact this is the whole concept my blog is built on. So here is a blueprint for the FT with my own tweaks to save you (and me) a thousand words describing how FT works and how it's architectured.. (BTW, this is the first of vSphere blueprints to come):

 

Fault Tolerance Video Demonstration:

Thank god the contest rules mentioned the possibility of republishing an old content; otherwise I would have been rerecording and video editing this from the scratch, which is a kind of nightmare. I published this video back in April this year when the vSphere was just announced by VMware (the bits were not even available for download at that time), this means it's one of the very first videos ever published about this cool new feature.

Before you hit the play button, let me tell you why this video is deferent from many of the other ones that I've seen later on:
1 – In my scenario, I have three ESX hosts in a cluster rather than two as you may see in most of the FT demonstrations. What is so special about that? Well, it clearly shows the true concept of the "continues availability", where in case of a complete ESX host failure, the FT will not just failover to another host, but will also automatically assign a third host in the cluster to protect itself in case of another host failure until the SysAdmin attend to the incident.

2 – I'm using in this video a continues file copy to the protected VM throughout the host failure process. This is to show you a "real-life" scenario where your VM is busy doing something critical (backup for example). You really don't play movies in your mission critical VMs (I think Microsoft is the one who invented this idea in their Hyper-V live-migration demonstration, kind of weird!)

 

My Real-life FT use cases:

I'm taking off now my "VMware Evangelist" hat, and putting on the "VMware Customer" hat. What you'll read here is my real-life use cases for the FT, no marketing talk, no political debates. This "is" the real deal:

1 – Blackberry Enterprise Server & RoveIT Mobile Admin:
BES is one of our most business critical applications because it's being used by our higher management in their day-to-day communications. Initially we were depending on HA since we didn't think that our luck would be that bad to have an ESX host failure while one of the executives sending an email.

This continued to be the case until we deployed the RoveIT Mobile Admin & vCenter Mobile Access (with BES/MDS in the backend). We basically wanted to have a 24/7 access for our SysAdmins to our entire IT environment (including VMware) while they are on the go, using their Blackberry smart-phones (given by the corp for this specific purpose). This was mainly to improve our response time for emergency situations, and of course this service makes no sense unless it can tolerate the most severe situations of hardware failures. Enabling FT on both the BES and the Mobile Admin VMs allow us, from one hand, to ensure that our executives will never complain that they can't use their Blackberry whenever they need, and that "IT Suck". From the other hand, we, the IT suckers..er..i mean SysAdmins & consultants, can have a piece of mind that we will always be able to get to our backend systems wherever there is a problem that requires an immediate attention.

2 – ManageEngine Application Manager:
We heavily depend on the ManageEngine Application Manager in our environment, where we get real-time emails and SMS notifications for any issues happening either in the OS layer (e.g. disk usage, service status ..etc) or the applications (e.g. Exchange high local queue, MS SQL DB issues ..etc). In order to maintain this level of real-time notifications, we had to put this application in a very high availability. Although the application comes with optional cluster capabilities, the VMware HA really was doing this trick without paying extra money. In both cases (the cluster option or the ESX HA) if an ESX host fails, we will have to wait approximately 10min for the application to be powered and operational on another host in the cluster. This is not realistic for an application that is supposed to tell us that the ESX has failed at the first place. With FT we are able to have the application up & running all the time with no interruption whatsoever, and consequently send us the notifications of any Host/OS/Application issues no matter what happens across the underlying infrastructure.

3 – Custom Application – Online payment gateway:
We have an online customer payment service consisting of a custom written application integrated with the IBM Websphere MQ and a backend Oracle DB. Everything is in high availability as you would expect, except for the custom application! I must add also that it is poorly written that it needs human intervention every time the VM needs to be rebooted in order to bring it up again. That being said, HA is not even an option in case of host failures. Unfortunately the application developer does not know how to address these issues in his application, and we are stuck with that fact since he's working with the same backend payment gateway provider. We came up with two solutions for that:
a) The Long run: gradually migrate the online service to a new system with a new backend payment gateway. We are around 30% now on this new service.
b) The short run: put the custom application on FT enabled VM where we don't have to suffer from any unplanned downtime associated with the VM and/or the host.

 

The conclusion:

FT is a "must have" not a "nice to have" feature in any environment. I don't really understand the big debate around it from the so-called "experts" who have been flooding us on twitter or the blogosphere about reasons why it's "not enterprise ready yet". Most of these debates are coming really from people who have not seen enough of these enterprise environments they are talking about and the challenges we have every day with scenarios like the ones I've listed above. Surely enough, FT has a quite long list of limitations that you can find on any of these blog posts (or on the VMware website itself), but you should also know that VMware is working on most of these limitations in future releases. The number one limitation that you will always hear about is the (1 vCPU) restriction for the FT enabled VM, well, let me tell you two things about that to finish up my article:

1 – The vast majority of the applications running in any datacenter do not need, or even make use of SMP. My three use cases above are examples for that.
2 – VMware has published recently this blog post showing how a 1vCPU VM based on the revolutionary Intel Nehalem processor, can perform better than 2vCPUs using older generations.

P.S. This is probably one of my largest blog posts. I'm disqualified from the contest.

  • Share/Bookmark
  • MHG
    In designing a disaster recovery hotsite with VMware FT what would be our most likely single point of failure and what are the best practice recommendations for such a task? Thanks.
  • FT is not a disaster recovery solution nor should it ever be considered for that. You may want to check vCenter Site Recovery Manager (SRM) from VMware. It's a very powerful product that can automate your DR solution for VI3.5 & vSphere (check out my tutorials here as well)
  • MHG
    So, does FT actually require 2X the SAN / NAS storage for each VM?

    Excerpt ….

    VMware Fault Tolerance provides continuous availability for virtual machines by creating and maintaining a Secondary VM that is identical to, and continuously available to replace, the Primary VM in the event of a failover situation.
  • No, you just need one copy of the VM on the SAN/NAS. The second instance of the VM lives inside the memory, not the storage.
  • Amazing site! It's almost midnight, I've got to go to bed, and I'm going to lie there thinking about vSphere in a Box!

    Three things:
    1. I can't get the sound to work on your videos-- tried in FF v3.5.x and IE v8.x... I must be doing something wrong.
    2. In your diagram, you name the ESX hosts in a way that implies they are vESX, then talk about FT in a production environment. All your production hosts are pESX, right? I didn't actually watch the video on this-- perhaps I missed something...
    3. Your SAN is still the single point of failure, right? Perhaps you have some active/active replication going on that you don't mention.

    Thanks again!
  • Hi Ted,

    Thanks for your comment, and I’m glad you liked the site!

    1 – I’m not sure why you can’t get the sound working, I just injected a normal mp3 file into the video.

    2 – All my production ESX hosts are definitely physical. The vESX (or the while vSphere In A Box) are not meant for production what so ever. The diagram also is just an illustration/blueprint of how the FT works. My use-cases of course are the ones that I have in my production environment.

    3 – All the SANs these days can tolerate the most severe hardware failures (hardisks through RAID, redundant controllers ..etc), so I wouldn’t call it a single point of failure. If you are talking about DR, then this is another story, and this is why you have the VMware Site Recovery Manager (SRM), where you do have replication between two SANs/Sites and the SRM takes care of the failover/failback process for you. You may want to check my tutorials on SRM as well, I think you’ll like this incredible product and how it can add great value when it comes to DR.

    Regards
  • Sure thing ;)
  • One other issue that FT will not protect against is OS coruption, but that is what your Backups are for ;D
  • Hi Didier,

    First of all, thank you for the kind words, I really don’t deserve them !
    I agree with you 100% on the “application crash” part..but I must say also that it’s not VMware’s job to protect us from a lazy developer!…besides, our management will never blame us back for a software vendor’s bug that caused the application to crash, but they will definitely do so in case of hardware failures, reminding us with how much budget they assigned to us, and how it’s really our responsibility to design our systems properly to tolerate such scenarios…

    Thanks again :)
  • Superb post Hany! IMO you've already won the first round at the vSphere Blogging Contest.
    Now let's talk about FT... I desire it, I want it, I need it in my production environment and i will make sure my 'deep pocket' managers feel the same lool

    We have seen all the pro's, unfortunately there is a con; FT doesn't save you from buggy application crashing randomly... It could be your BES or your custom online payment gateway...

    The big advantage of FT is the ability to fail over in a snap with no single bit lost, that's awesome, isn't it :)

    Cheers,
    Didier
  • Suresh
    Hi

    really gr8 effort.... amazing diagram..

    Awesome.
  • Thank you guys for your kind words, you made me blush like a 6 years old girl.
  • It's good to see some testimonials from a VMware Customer which is:

    1. Real Technical Geek
    2. Real Fan of VMware
    3. Real Graphical Designer
    4. Good Writer
    5... -:)

    Cheers
    Vladan
  • Nice diagram !
blog comments powered by Disqus

My name is Hany Michael and I’m a Senior Consultant at VMware. I blog about various topics ranging from the core vSphere technologies all the way to the vCloud based products. (Read more)
Disclaimer
Any views or opinions expressed on this blog are strictly my own and not the opinions and views of my employer.