Posts Tagged ‘vSphere’

Yes, finally! It’s been like what? Five months?! Well, the delay in publishing this part was mainly because of the delay in certifying the new IBM HX5 blades on vSphere. It’s a quite long process that you can read about it here, but the good news is that the hardware is finally on VMware’s HCL, and that I can comfortably blog about the subject now without causing any confusions to the readers.

Before we dig deep into the new designs, I’d like to mention some minor changes in the diagram.

Updated Diagram

I’ve included the old configurations along with the new ones in one updated PDF. The main difference now is that I’m using normal pages for showing each configuration. In the old version I used the layers to show and hide the configurations as you select them. I thought now that using separate pages for different diagrams would ease the process of browsing through the configurations, and to tell you the truth, to reduce also the high complexity of designing the diagram. It’s a crazy process to keep track of all these layers in Visio especially when we are talking about more than 7,000 shapes floating on the same design area!

Now let’s get down to business.

Configuration (5) – HX5:

This is the Big Blue’s latest two-node blade technology. I emphasized on the “two-node” here because it’s the only configuration certified to run with vSphere as of the time of writing these lines. Please note that you can use up to 4 nodes with the HX5 but this won’t be supported by VMware. When we talk about two nodes here we mean the following:

• Having the base blade (CPU + Mem + HDD) (+Plus+) the MAX5 expansion try to scale up more memory for the blade.
• Having the base blade (again CPU + Mem + HDD) (+Plus+) another similar expansion board to scale all the blade components, that’s 4 x CPUs + 2 x Memory modules + 2 x IO expansion cards.

As you will see in the diagram, I chose the second option to talk about.

Now, what do we have here? it’s simply the redundancy at its best! We can place our networks here freely with full redundancy as you see in the layout of the vNICs. For example, if we have a failure in the CFFh expansion card on any of the two nodes, we will still be able to flow the traffic without any issues on the other CFFh card. Same thing holds true for the on-board ports, if for any reason one of these posts fail, the traffic will flow on the other node’s board.

Apart from that, I’m introducing here the DMZ networks for the first time. Most of the enterprises prefer to separate the DMZ networks/servers on different chassis for security reasons. While this is a valid decision, we can have with this blade configuration a workaround for organizations that are less paranoid about the DMZ security, yet with good isolation. Let’s see how this is done in details:

• For the networks, we have two dedicated blade switches that will be uplinking *only* to the corp DMZ switches (in this case Bay 9 & 10). This means we will have no traffic following from either the internal networks or the VMkernel networks. Same thing for the blade ports, you will always have the NICs 4, 5, 10 and 11 dedicated for the DMZ networks and running in full performance and redundancy.
• For the SAN, we can also ensure that we have a dedicated HBAs as well as an isolation. The uplinks to the SAN switches will be segmented across the two bays 3 and 4, and connected directly/physically to the appropriate SAN fabrics.

Configuration (6) – Virtual Fabric:

Before we start with this configuration, I would like to state that I am not quite sure whether these Emulex Virtual Fabric Adapters (VFA) are supported by VMware or not. While I can’t see them clearly on the HCL with the name VFA, I can see some Emulex documents saying that they are. Of course the reference here should be always the VMware HCL itlself, not anything else, but I will double check on that and update this post later. With that said, please refer to this configuration carefully and make sure to confirm this point before engaging with any vSphere design around it.

Now let’s dig deep into this cool technology. IBM simply has this Virtual Fabric concept of slicing your CFFh expansion card into 8 different ports. This doesn’t only mean that you have the flexibility to adjust the speed, but also the protocol. For example, you can choose to use either Ethernet, Fibrechannel, FCoE or even iSCSI with hardware initiators.

In our case here I used only Ethernet as the protocol for these ports, and then sliced them into 8 different vNICs with various link speeds. Perhaps a screenshot from the diagram would make things more clear.

As you see, we set the bandwidth for the SC to 1GB since we normally don’t require high BW for management, while we set 3GB and 5GB link speed for the Fault Tolerance and VM Networks respectively. By default these ports are set to 2.5GB ( 4 x 2.5GB = 10GbE into two ports), but you have the full flexibility to change that as you see.

Configuration (7) – CNA:

A very simple design to wrap up this series with. It’s the traditional CNA (oh yeah, it’s a common and traditional technology now!). As you see in the diagram, we have here a CFFh expansion card, and it has got four ports:

• Ethernet ports: that’s 2 x 10GbE Ethernet ports for the networking traffic. We will treat them here normally as we treat any 10GbE port. We will slice them via the vNetwork traffic shaping in vSphere to achieve the bandwidth that we want.
• FibreChannel ports: that’s 2 x HBA ports for SAN traffic. Instead of going into the traditional Bay 3 & 4 as we’ve see across the whole series and configurations, this time the traffic is multiplexed and pushed to the Nexus 4000 blade switches.

Did I just say Nexus 4000?! yep, that’s a specially developed Nexus switches by Cisco to be used only/currently with the IBM BladeCenter H/HT. But here is the catch, you will still need to have the Nexus 5000 switches to segregate the FCoE traffic coming from the Nexus 4000 and then forward the network and FC traffic to the existing LAN and SAN respectively. Of course we should have redundancy here at all layers. In the BCH we have two Nexus 4000 sitting in bays 7 and 9, while we have two Nexus 5000 switches in the back end.

Now what?

Well, as much as I worked really hard in this series to come up with different kind of configurations and design scenarios, as much as I enjoyed it! Now I need to move on to another vendor, but without all these mad options. I initially was planning to jump straight to the HP realm, however, i found myself involved in two different Cisco UCS vSphere designs lately, so it would make much sense to me if I blogged about this platform now. Don’t take my word for it though, I might surprise with a Dell or Fujitsu series, who knows?!

Motivation

I was having a discussion with one of the large enterprises here in Qatar lately, and I was quite surprised to know from them that they are hesitated to migrate their VI3.5 environment to vSphere because of the associated downtime. What surprised me was not the fact that they can’t afford a downtime, I’ve spent 6 years of my career working in the Telecom sector and I know for a fact that 1 second of downtime could mean a disaster, or even translate to a loss of thousand of $$. What surprised me was that they didn’t know that it is possible to do this migration without any downtime!

In this blog post, I will not only show you (and them) how I was able to perform my upgrade without even this single second of downtime, but I will also show how we were able to migrate our storage from one array to another without any service interruption whatsoever in our equally critical environment. To make things even more exciting, what I’m about to show you here is completely achievable using vSphere’s built-in features like VMware Converter, EVC, vMotion and Storage vMotion. There was no third-party tools used in this entire migration.

A brief environment overview

There is nothing better than diagramming this for easier follow-up. In the diagram below I’m illustrating a small portion of the environment showing the main components of the old ESX 3.5 hosts as well as the ESX 4.0 hosts. In our case, we decided not to go with in-place upgrade, and preferred to have a fresh install for the ESX hosts in the new vSphere environment.

You might have noticed that I included a video inside the diagram, and probably wondering why on earth would someone do something like that? The answer is simple: I’m showing-off! No seriously, I know many people (from VMware and specific storage vendors) who use my diagrams in their internal meetings with customers (really I’m not showing-off), and I thought it would be nice to have such small clip in the diagram that shows both the vMotion & SvMotion easy point-and-click approach.

Note: This is just an illustration not an S/vMotion architecture diagram! Wait for my A3 if you are interested to see the technology behind this…magic!

The Process

Step 1: We are running here vCenter on a physical server, and we want to utilize the same hardware for the new upgrade. The easiest way to achieve that is to P2V the existing vCenter 2.5 to another standalone ESX host in our environment. After the VM is migrated successfully and all the clean-up is done, the switch over from the physical to virtual can happen in a matter of seconds by disconnecting the physical server from the network, and connecting the VM (which has the same IP address of course) to the same subnet.

Step 2: Now that we have the vCenter 2.5 migrated, the next step is to perform a clean install on the freed physical server. Starting with the OS deployment, all the way to the vCenter 4.0 installation, initial configuration and licensing.

Step 3: The third step is to connect the new vCenter 4.0 to the old vCenter 2.5 licensing server. This part is important because the ESX 3.5 hosts do not leverage the new and improved licensing model that was introduced in the 4.0 release. This step is quite easy: you go to the “Administration” menu on your vSphere client, select the “vCenter Server Settings”, and then enter your old vCenter 2.5 hostname into the field as shown in the example below.

Step 4: Now we are ready to create a new cluster for the existing ESX 3.5 hosts on the left side of the diagram. The thing to note here is to create the cluster with the EVC mode enabled as shown below because we will be migrating the VMs between two deferent hardware/CPU generations:

Step 5: We create here a second cluster (EVC enabled as well) and add the new ESX 4.0 hosts to it as shown in the right side of the diagram.

Step6: Now, the trick here is to have one ESX 4.0 host in this cluster connected to both arrays in the environment – the EVA and the V-Max. We achieve that by connecting one HBA to the HP SAN fabric, and the second HBA to the EMC SAN fabric. Once this is done, and all the associated zoning and masking is configured, we can scan the HBAs and have all the datastores/LUNs available on this server that we will call it “Gateway”.

Step7: The fun begins. Since the gateway server is having the same shared storage with the ESX 3.5 hosts, all what you need to do here is to drag and drop your VMs from the old cluster to the new one. The vMotion will kick-in and do it’s magic to live migrate the VMs to the new gateway server. That’s right! We are live migrating virtual machines from ESX 3.5 to ESX 4.0 on the fly.

Step 8: Now to my favorite part in the whole migration process. Here we get to experience one of the most amazing features in vSphere – the Storage vMotion. It has been actually re-written with significant performance improvements that made it one of the most powerful tools for any VMware administrator in my opinion, and the best part is that it’s done now with a few mouse clicks through the GUI (checkout the diagram video, or this detailed post). As I mentioned above, we were migrating our workloads from the HP EVA to the EMC V-Max, and we felt quite confident (after intensively testing this in the lab for a week) that the SvMotion would be the best choice for our storage migration. The other reason for using SvMotion was the ability to thin-provision VMs on the fly. I’m not talking here about everything of course, but rather the development VMs that are hardly ever touched. We had so many VMs for our development department with quite huge space requirements, while in fact they are neither actively used all the time, nor they consume the disk space allocated to them. The thin-provisioning for these VMs saved us literally TBs of storage on the new expensive V-Max SAN.

Things to note:

• After you complete this migration you are not quite done yet. You should typically have your VM tools updated, and also the VM hardware upgraded from v4 to v7. While you will still run fine without these upgrades, it’s always recommended to be up-to-date in that regard, and to also leverage many of the new vSphere featuers like for example memory hot-add (my personal favorite!). The trick here is that you will need a VM reboot to perform that. In our case, for the less critical VMs we scheduled a planned reboots on weekly basis for the upgrades, and for the high-critical VMs, we just wait for the first possible OS reboot and we perform our upgrades along with it.
• Any storage vendor will tell you to do the thin-provisioning on the array directly, and I kinda agree with them on that, but this is not an option to everyone. Not all arrays come with this feature, or even if they do, not everyone can afford the licensing part. In our case, I simply couldn’t rely on the SAN admins for monitoring and maintaining these thin-provisioned LUNs on the array side, and from the other hand, there were some technical limitations associated with that in terms of SRDF replication or FAST v1 (but that is something specific to EMC, and relevant only to the time of writing this post).

Conclusion:

I will finish this post from where I started. The VMware vSphere is a very powerful and a true enterprise class virtualization platform. You’ve seen here how I was able to migrate the entire VI3.5 environment without one single second of downtime, and also how it was an extremely easy process to migrate our complete storage from one array vendor to another without any interruption in the servers/services whatsoever. There is nothing extraordinary in this scenario (except maybe the embedded video in the diagram), and you’ve seen how easy the steps are, and how everything we’ve done here is built in vSphere itself. Just know your requirement, plan your migration ahead, and you will be just fine!

I updated the diagram (v1.2) to fix a small typo and adjust also a couple of shapes. Thanks to Joshua Liebster & Bert Bouwhuis for driving my attention to this.

I know everybody skips to the diagram so I’ll save you the introduction, just make sure to quickly go through the notes that follow it:

• This is not an introduction to the VMware HA, and it’s not a very advanced diagram for it either. I assume here that you have a general idea on the topic before looking into it to appreciate this incredible technology. If you are a VMware professional you may also find this useful to keep your information sharp and present about the topic at any given time. You really don’t have to re-read the documentation every time you’d like to remember a small detail about the subject.
• I’m introducing in this diagram the “Layers” feature in Visio for the first time. The diagram may look somewhat confusing at the first glance, so I thought that it might be a good idea to use these layers for you to hide/show the topics that you are going through in the diagram. I can see some other use cases for the Layers in future diagrams, so I hope you will like it.
• This is an A3 diagram, sorry I know most of you just love the traditional A4 from the feedback I get, but seriously, it’s just TMI to fit in A4.
• Everything you see in this diagram, and specifically for the admission control, is *not* fictitious. This is a real cluster I built specifically before designing this diagram. I wanted everything to be 100% accurate and more importantly: realistic. If you zoom into the middle of the vCenter shape, you will be able to see the actual screenshot of the vCenter interface showing the HA cluster I used, and its runtime information window as well.
• It’s worth mentioning that this is not all the “advanced options” that you can use for VMware HA. I just selected the ones I thought that might be more frequently used. You can always get back to the official VMware documentation for the complete list.
• The Admission Control was probably the hardest part not just to visualize it, but also to understand it in the first place! That being said, I do not expect anyone with no prior reading on this specific topic to just get it from the first glance when he/she looks into the diagram. Duncan Epping has an excellent article that I think everyone already knows about it, but it’s worth mentioning that it’s the best place you will ever find for VMware HA in general. The diagram should help you though to understand it faster and easier. You can see all the numbers/calculations in front of you in one shot, and how all these numbers are related to each other.
• This HA lab was built in nearly 5 minuets and is 100% virtual. Long live Lab Manager 4.0 ! (more details here)

That’s all folks! I hope you will find it useful!

A Boring Introduction:

It’s been a crazy week! A lot of stuff is happening right now at my work, personal life, and my career. For example, I’m building our much-awaited “Private Cloud” at work, using both the ultimate vSphere Cloud OS and the rock-solid IBM hardware that was finally delivered this week. But wait, this is not ‘the’ exciting thing that was happing this week for me, it’s most definitely the news that I’ve received last Thursday about winning the first round of the vSphere blogging contest. I will not thank John Troyer & Mike Adams for their great idea and their incredible efforts for organizing this contest, and I will not thank Deepak Narain, the man behind this blog existence who kept pushing me to lunch it a year back (more on this soon), I will, instead, thank everyone for their kind words and encouragement (including the names I’ve just mentioned), I was literally thrilled by the emails, blog posts and “tweets” that were thrown at me since the news was out. THANK YOU!

Now, enough of this boring talk about me, myself and I, and let’s get started with this new round of the blogging contest. A heads up first: I was not supposed to participate this week since I’ve been so busy as you see, but I had a 24 hours after canceling some plans that I had at the last minute. That said, what you see here is not quite final, I believe I need to work more on the diagram especially the IO plane layout in the hidden vSS, and probably add a couple of more configurations to the video to show some cool stuff like the consistent network stats of a mobile VM jumping from an ESX to another. I’ll be updating all these stuff hopefully during next week.

The Configuration Video:

The Architecture Diagram:

Another vSphere diagram! I told you, you are going to see these blueprints more than any time before. Quick notes:

• This is an A3 scale diagram in case you want to print it.
• The diagram reflects the exact configuration on the video. I’ve done this intentionally to make it easier and faster for any one new to the vDS to understand the concept and the various configuration aspects.
• As I mentioned above, due to the very short period of time that I had, I will most probably modify small parts in the diagram to achieve better results. You can come back and check the version number of the diagram to download the latest updates.

MASTER IT!

I love this part at the end of any book/chapter published by SYBEX. It gets down and dirty with all the theoretic parts covered, and guide you through a practical path to try what you’ve learned. This is what I want to do here as well. The vDS is quite confusing as a concept and configuration for the first time, and I personally didn’t get it except when I started getting my hands on it and playing around with the configurations. The challenge here is that you probably won’t have the required lab to do this, especially that you need large number of NICs to test all the configurations. If you are one of my regular blog readers, you’ve probably guessed what I’m getting to. It’s the “vSphere in a box”!

Around three month back, I published a series of posts talking about building a vSphere configurations using ESX inside itself. Instead of rewriting the whole story again, here is the links for your consideration. One last thing to note here: the entire lab you’ve seen in the video was built using Lab Manager 4.0 as you will read in the following posts.

• vSphere in A Box: A “Virtual Private Cloud” Blueprint
• vSphere In A Box: Part (2): Putting the pieces all together
• vSphere In A Box: Part (3): The Lab Manager 4.0 Automation

Special Thanks:

I’d like to thank Duncan Epping for reviewing part of the contents here. I was having some doubts about few points and due to the time constrain, I didn’t have the time to research more on them. I asked for Duncan’s help and he was very kind to do so.

Additional Recourses:

These are the best resources that I’ve found so far for the vDS:

• WitePapers: VMware vNetwork Distributed Switch: Migration and Configuration
• VMworld 2009 Sessions: TA2525, TA2105
• Blog Posts: Eric Sloof, Barry Combs, Luc Dekens, ICT-Freak

This is a response to the new vSphere Blogging contest that was announced in the middle of this month. I truly think that it’s a cool idea, and I believe that regardless of winning or losing, the excitement and fun a blogger would have during his/her participation is something awesome by itself.

The rules say that my post need to be compact and straight to the point, so I won’t be able to cover all the aspects about something huge like FT. If by any chance I failed to write such a short post, then here are some tips to avoid wasting your time:
1 – If you are one of those people who wear a tie at work, you should jump straight to the “Use Cases” section.
2 – If you are one of those people who are using the words: 10GbE, VMkernel and %RDY, then you probably don’t have time for this, but take my advice and have a look on the next two sections.

Fault Tolerance Architecture Diagram:

From the newly launched vSphere Blog on VMTN I quote this part: “[..] they do say a picture is worth a thousand words [..]“, well, I believe I’ve said that also once before on a previous post, and in fact this is the whole concept my blog is built on. So here is a blueprint for the FT with my own tweaks to save you (and me) a thousand words describing how FT works and how it’s architectured.. (BTW, this is the first of vSphere blueprints to come):

Fault Tolerance Video Demonstration:

Thank god the contest rules mentioned the possibility of republishing an old content; otherwise I would have been rerecording and video editing this from the scratch, which is a kind of nightmare. I published this video back in April this year when the vSphere was just announced by VMware (the bits were not even available for download at that time), this means it’s one of the very first videos ever published about this cool new feature.

Before you hit the play button, let me tell you why this video is deferent from many of the other ones that I’ve seen later on:
1 – In my scenario, I have three ESX hosts in a cluster rather than two as you may see in most of the FT demonstrations. What is so special about that? Well, it clearly shows the true concept of the “continues availability”, where in case of a complete ESX host failure, the FT will not just failover to another host, but will also automatically assign a third host in the cluster to protect itself in case of another host failure until the SysAdmin attend to the incident.

2 – I’m using in this video a continues file copy to the protected VM throughout the host failure process. This is to show you a “real-life” scenario where your VM is busy doing something critical (backup for example). You really don’t play movies in your mission critical VMs (I think Microsoft is the one who invented this idea in their Hyper-V live-migration demonstration, kind of weird!)

My Real-life FT use cases:

I’m taking off now my “VMware Evangelist” hat, and putting on the “VMware Customer” hat. What you’ll read here is my real-life use cases for the FT, no marketing talk, no political debates. This “is” the real deal:

1 – Blackberry Enterprise Server & RoveIT Mobile Admin:
BES is one of our most business critical applications because it’s being used by our higher management in their day-to-day communications. Initially we were depending on HA since we didn’t think that our luck would be that bad to have an ESX host failure while one of the executives sending an email.

This continued to be the case until we deployed the RoveIT Mobile Admin & vCenter Mobile Access (with BES/MDS in the backend). We basically wanted to have a 24/7 access for our SysAdmins to our entire IT environment (including VMware) while they are on the go, using their Blackberry smart-phones (given by the corp for this specific purpose). This was mainly to improve our response time for emergency situations, and of course this service makes no sense unless it can tolerate the most severe situations of hardware failures. Enabling FT on both the BES and the Mobile Admin VMs allow us, from one hand, to ensure that our executives will never complain that they can’t use their Blackberry whenever they need, and that “IT Suck”. From the other hand, we, the IT suckers..er..i mean SysAdmins & consultants, can have a piece of mind that we will always be able to get to our backend systems wherever there is a problem that requires an immediate attention.

2 – ManageEngine Application Manager:
We heavily depend on the ManageEngine Application Manager in our environment, where we get real-time emails and SMS notifications for any issues happening either in the OS layer (e.g. disk usage, service status ..etc) or the applications (e.g. Exchange high local queue, MS SQL DB issues ..etc). In order to maintain this level of real-time notifications, we had to put this application in a very high availability. Although the application comes with optional cluster capabilities, the VMware HA really was doing this trick without paying extra money. In both cases (the cluster option or the ESX HA) if an ESX host fails, we will have to wait approximately 10min for the application to be powered and operational on another host in the cluster. This is not realistic for an application that is supposed to tell us that the ESX has failed at the first place. With FT we are able to have the application up & running all the time with no interruption whatsoever, and consequently send us the notifications of any Host/OS/Application issues no matter what happens across the underlying infrastructure.

3 – Custom Application – Online payment gateway:
We have an online customer payment service consisting of a custom written application integrated with the IBM Websphere MQ and a backend Oracle DB. Everything is in high availability as you would expect, except for the custom application! I must add also that it is poorly written that it needs human intervention every time the VM needs to be rebooted in order to bring it up again. That being said, HA is not even an option in case of host failures. Unfortunately the application developer does not know how to address these issues in his application, and we are stuck with that fact since he’s working with the same backend payment gateway provider. We came up with two solutions for that:
a) The Long run: gradually migrate the online service to a new system with a new backend payment gateway. We are around 30% now on this new service.
b) The short run: put the custom application on FT enabled VM where we don’t have to suffer from any unplanned downtime associated with the VM and/or the host.

The conclusion:

FT is a “must have” not a “nice to have” feature in any environment. I don’t really understand the big debate around it from the so-called “experts” who have been flooding us on twitter or the blogosphere about reasons why it’s “not enterprise ready yet”. Most of these debates are coming really from people who have not seen enough of these enterprise environments they are talking about and the challenges we have every day with scenarios like the ones I’ve listed above. Surely enough, FT has a quite long list of limitations that you can find on any of these blog posts (or on the VMware website itself), but you should also know that VMware is working on most of these limitations in future releases. The number one limitation that you will always hear about is the (1 vCPU) restriction for the FT enabled VM, well, let me tell you two things about that to finish up my article:

1 – The vast majority of the applications running in any datacenter do not need, or even make use of SMP. My three use cases above are examples for that.
2 – VMware has published recently this blog post showing how a 1vCPU VM based on the revolutionary Intel Nehalem processor, can perform better than 2vCPUs using older generations.

P.S. This is probably one of my largest blog posts. I’m disqualified from the contest.