Important: In case you haven't done that already, please take a moment to read the first post of this series.

Due to the insane number of expansion modules/options available in the IBM BladeCenter H, I had to split this post into two parts. In fact, I was initially planning to have around 12 different designs for vSphere on BladeCenter H (yes twelve) but I then I started to shrink and skip some designs to fit as many scenarios as possible in a reasonable two-part article. With that said, the following is by no mean a list of all the possible design scenarios you can achieve with this hardware platform. If you started the "mix and match" game, you may literally end-up with uncountable possibilities!

The Diagram

Here is some important notes before using the diagram:

• You will see different configurations in this post and the relevant architecture of each configuration in the diagram. This is done through the PDF layers, which basically means than you should *not* activate more that more layer in the same time.
• By default, "Configuration 1" is the first active layer when you open the PDF file. You can show/hide the other layers by simply clicking on them. Again, you should only show one layer/configuration at a time.
• You will always see two boxes on the right side of the diagram, the upper one will show you the current vSphere configuration, and the lower one will show you the relevant hardware configuration. You should typically start looking at those two boxes before scanning through the diagram to understand the "ingredients" of the design.
• At the time of writing this post, you will see four configurations only in this diagram, however, when I publish the second part, there will be additional configurations that I will add to the existing ones. In other words, the diagram will be updated later on to have those additional configurations so keep that also in mind.

The common design and configurations

You will find in most of the configurations a common design, unless I explicitly state otherwise. I will list them here in details:

The Clusters:

You will see two type of clusters:

• Management Cluster: it is typically a two node cluster running the management and infrastructure services. For example, if you want to virtualize the vCenter Server, the VM should be running on this cluster rather than the actual production clusters. Same thing holds true for other vCenter products like: AppSpeed, CapacityIQ, SRM and so forth. There are two reasons for doing that: the first, we don't want to run into the problem where vCenter Server is not accessible (there are some examples published in the community but my favorites are Jason Boche's Catch22s!). The second reason, we don't want to either affect our workloads' performance with our management virtual appliances or vice versa.
• Production Clusters: You can see here two production clusters (Cluster A and Cluster B). The take away from that is the following:
  • You don't have to stick with that number of hosts per cluster, it depends on what you want to achieve, and also on some configuration maximums that may or may not limit you.
  • The nodes have to be spanned across the two chassis as numbered and illustrated in (Config 1). There are two reasons for that: Firstly, you don't want your whole cluster to fail in an unluckily event when a whole chassis fails. Secondly, you have to keep in mind that VMware HA selects the first 5 hosts in the cluster and promote them as a "Primary" nodes, if they fail, your HA cluster fails.

The Blades:

You will see two consistent blades throughout the first four configurations, the HS22 and the HS22V. Both blade servers share the same IO expansion capabilities, however, there are a some differences between them. For example, the HS22V has no hot swappable HDD but it is superior in the memory capacity (144GB compared to 96GB in the HS22). In part-two of this article, I'll talk in details about the new HX5 and what it can bring to the table in terms of scalability.

The Expansion cards:

Every HSxx blade comes with two onboard 1Gbps Ethernet ports for basic networking. They will always show in vSphere as vmnic0, and vmnic1. These ports are in turn mapped to Bay 1 and Bay 2 in the chassis. Of course no one recommends implementing vSphere using 2 x 1GbE ports in an enterprise environment (although it will technically work), so we will use here what we call: expansion cards. There are two slots for expansion cards in any HS22/V blade, the first one is called CIOv (for vertical expansion modules) and CFFh (for the horizontal fast IO modules). The CIOv is usually used with the FC HBAs (although we will see later how we will utilize it for iSCSI connectivity), and they are mapped to Bay 3 and Bay 4 in the chassis. The CFFh on the other hand is mapped to four fast expansion modules (7, 8, 9 and 10). I say fast because this is the only card that can leverage the 10GbE connectivity (or Infinibad but it's not relevant to our series). Depending on the configuration, you will see how we will use different cards to support our designs, however, the onboard 2 x 1GbE port will be always common, and always there.

Now that we've talked about the common stuff, let's start talking about the unique configurations. Oh yes, we were just warming up!

CONFIGURATION (1):

We have in this configuration 6 x 1GbE pNICs per blade to support our MGMT, VMkernel and Virtual Machine networks. We teamed three pNICs here in a vNetwork Standard Switch (vSS) to serve the SC, vMotion and FT. The other three pNICs are teamed in a vNetwork Distributed Switch (vDS) to serve the VM networks. Let's dig litter deeper on how this is done.

As mentioned earlier, we have three type of IO ports on the blades: the onboard ports, the CIOv, and the CFFh. In order to achieve the maximum availability, we teamed one onboard port with a couple of ports from the CFFh card. In this case, if we had a failure in any IO port (on board or expansion card) we will be able to tolerate that failure.

The second consideration here is to distribute the load and bandwidth for our networks. For example, the SC network will be active on vmnic0 and standby on vmnic1. The vMotion will be active on vmnic1 and standby on vmnic0 and so forth.

You may have noticed also that we grouped the SC + VMkernel network on a vSS, while we grouped the VM networks on a vDS. The reason behind that is to ensure that you would still be able to control your SC network even if your vCenter fails. For the VM networks, you would still leverage the great enhancements and features of the vDS. This is *not* a best practice from VMware, and as far as I know there is no documentation recommending that. It is up to you whether you would go with that setup or simply have everything on a single vDS.

CONFIGURATION (2):

This is nearly identical configuration except for the IP SAN. In Config1 we were running on a FibreChannel SAN, while in this configuration we have an iSCSI. The thing to note here is that you will need to install your Ethernet expansion modules in Bay 3 & 4. We will swap also the CIOv card from being a FC HBAs to a traditional 2 x 1GbE card. Of course you will use in this case the vSphere iSCSI initiator for doing your storage networking. This is fine in nearly most cases, except the one where you will actually need to boot your ESX server from SAN.

Please also note there that you can use NFS with the same layout. Your 2 x 1GbE blade ports + the 2 x expansion modules (bay 3 & 4) will all serve your NFS requirement in a high availability design.

CONFIGURATION (3):

What you will see in this configuration is something a bit different. We are using here a 2 x 10GbE ports through the CFFh expansion card to serve "all" our networks. This card is mapped to two 10GbE expansion modules sitting in Bay 7 and Bay 9.

The trick here is this: how can you have a proper network segmentation if you are using two pNICs only? The answer, of course, is VLANS. As you see in the diagram, we have two production networks and one lab network. All these networks are tagged with a VLAN ID to flow the traffic through the vmnics to pNICS all the way to your enterprise/core switches. The ports on your core switches need to be of course in trunk mode.

Now, the second question here would be this: how can you ensure that no network will saturate the whole link and affect the performance of the others. The solution for that is to use the vSphere traffic shipping. You can simply dedicate the bandwidth to each "port group" per your requirement. Example, for SC you normally don't need more than 1Gbps. For vMotion and FT you would definitely require more bandwidth. To keep things simple, I illustrated in the diagram how the segmentation and bandwidth allocation can be distributed across the two links in an Active/Standby approach.

You will notice here also that we are utilizing the two on board Ethernet ports to have an additional iSCSI SAN (for the Lab environment for example) along with the FC SAN for your production workloads.

CONFIGURATION (4):

In the previous configuration we saw how we leveraged the VLANs to do our network segmentation and how that was quite easy and flexible. But what if the customer has a policy not to use VLANs to consolidate the networks (for a security reason as an example)? Easy, we would still be able to comply with that. Basically we will need to swap here the 2 x 10GbE CFFh card with a 4 x 10GbE card and of course add additional two 10GbE expansion modules to Bay 8 and Bay 10.

Now, what did we achieve by doing that? Two things:

1 – We are compliant with the customer requirement to have a physical segmentation between the Management/FT/vMotion networks and the production networks.
2 – We are using the vSS for our management network while leveraging the vDS for our Virtual Machine networks.

You have also here another two options that were not included in the diagram. You can make use of the two onboard ports to have an additional iSCSI SAN as we did in the previous configuration, or, you can use them as a standby ports for your Management/VM networks in case of a CFFh card failure. Do you see now what I meant above by the "mix and match game"?

Coming Soon – Part 2:

I'll talk about the new HX5 and how you can have a lot more memory or extended IOs to support special workloads or strict design requirements. I will talk about FCoE and CNAs. I will also talk about the new & promising Virtual Fabric from IBM, and how you can basically slice your pNics into almost any protocol or speed you want.

Stay tuned!

VMware vSphere on IBM BladeCenter H – (Part 1 of 2) posted by Hany Michael @ HYPERVIZOR.COM

Motivation

I was having a discussion with one of the large enterprises here in Qatar lately, and I was quite surprised to know from them that they are hesitated to migrate their VI3.5 environment to vSphere because of the associated downtime. What surprised me was not the fact that they can't afford a downtime, I've spent 6 years of my career working in the Telecom sector and I know for a fact that 1 second of downtime could mean a disaster, or even translate to a loss of thousand of $$. What surprised me was that they didn't know that it is possible to do this migration without any downtime!

In this blog post, I will not only show you (and them) how I was able to perform my upgrade without even this single second of downtime, but I will also show how we were able to migrate our storage from one array to another without any service interruption whatsoever in our equally critical environment. To make things even more exciting, what I'm about to show you here is completely achievable using vSphere's built-in features like VMware Converter, EVC, vMotion and Storage vMotion. There was no third-party tools used in this entire migration.

A brief environment overview

There is nothing better than diagramming this for easier follow-up. In the diagram below I'm illustrating a small portion of the environment showing the main components of the old ESX 3.5 hosts as well as the ESX 4.0 hosts. In our case, we decided not to go with in-place upgrade, and preferred to have a fresh install for the ESX hosts in the new vSphere environment.

You might have noticed that I included a video inside the diagram, and probably wondering why on earth would someone do something like that? The answer is simple: I'm showing-off! No seriously, I know many people (from VMware and specific storage vendors) who use my diagrams in their internal meetings with customers (really I'm not showing-off), and I thought it would be nice to have such small clip in the diagram that shows both the vMotion & SvMotion easy point-and-click approach.

Note: This is just an illustration not an S/vMotion architecture diagram! Wait for my A3 if you are interested to see the technology behind this…magic!

The Process

Step 1: We are running here vCenter on a physical server, and we want to utilize the same hardware for the new upgrade. The easiest way to achieve that is to P2V the existing vCenter 2.5 to another standalone ESX host in our environment. After the VM is migrated successfully and all the clean-up is done, the switch over from the physical to virtual can happen in a matter of seconds by disconnecting the physical server from the network, and connecting the VM (which has the same IP address of course) to the same subnet.

Step 2: Now that we have the vCenter 2.5 migrated, the next step is to perform a clean install on the freed physical server. Starting with the OS deployment, all the way to the vCenter 4.0 installation, initial configuration and licensing.

Step 3: The third step is to connect the new vCenter 4.0 to the old vCenter 2.5 licensing server. This part is important because the ESX 3.5 hosts do not leverage the new and improved licensing model that was introduced in the 4.0 release. This step is quite easy: you go to the "Administration" menu on your vSphere client, select the "vCenter Server Settings", and then enter your old vCenter 2.5 hostname into the field as shown in the example below.

Step 4: Now we are ready to create a new cluster for the existing ESX 3.5 hosts on the left side of the diagram. The thing to note here is to create the cluster with the EVC mode enabled as shown below because we will be migrating the VMs between two deferent hardware/CPU generations:

Step 5: We create here a second cluster (EVC enabled as well) and add the new ESX 4.0 hosts to it as shown in the right side of the diagram.

Step6: Now, the trick here is to have one ESX 4.0 host in this cluster connected to both arrays in the environment – the EVA and the V-Max. We achieve that by connecting one HBA to the HP SAN fabric, and the second HBA to the EMC SAN fabric. Once this is done, and all the associated zoning and masking is configured, we can scan the HBAs and have all the datastores/LUNs available on this server that we will call it "Gateway".

Step7: The fun begins. Since the gateway server is having the same shared storage with the ESX 3.5 hosts, all what you need to do here is to drag and drop your VMs from the old cluster to the new one. The vMotion will kick-in and do it's magic to live migrate the VMs to the new gateway server. That's right! We are live migrating virtual machines from ESX 3.5 to ESX 4.0 on the fly.

Step 8: Now to my favorite part in the whole migration process. Here we get to experience one of the most amazing features in vSphere – the Storage vMotion. It has been actually re-written with significant performance improvements that made it one of the most powerful tools for any VMware administrator in my opinion, and the best part is that it's done now with a few mouse clicks through the GUI (checkout the diagram video, or this detailed post). As I mentioned above, we were migrating our workloads from the HP EVA to the EMC V-Max, and we felt quite confident (after intensively testing this in the lab for a week) that the SvMotion would be the best choice for our storage migration. The other reason for using SvMotion was the ability to thin-provision VMs on the fly. I'm not talking here about everything of course, but rather the development VMs that are hardly ever touched. We had so many VMs for our development department with quite huge space requirements, while in fact they are neither actively used all the time, nor they consume the disk space allocated to them. The thin-provisioning for these VMs saved us literally TBs of storage on the new expensive V-Max SAN.

Things to note:

• After you complete this migration you are not quite done yet. You should typically have your VM tools updated, and also the VM hardware upgraded from v4 to v7. While you will still run fine without these upgrades, it's always recommended to be up-to-date in that regard, and to also leverage many of the new vSphere featuers like for example memory hot-add (my personal favorite!). The trick here is that you will need a VM reboot to perform that. In our case, for the less critical VMs we scheduled a planned reboots on weekly basis for the upgrades, and for the high-critical VMs, we just wait for the first possible OS reboot and we perform our upgrades along with it.
• Any storage vendor will tell you to do the thin-provisioning on the array directly, and I kinda agree with them on that, but this is not an option to everyone. Not all arrays come with this feature, or even if they do, not everyone can afford the licensing part. In our case, I simply couldn't rely on the SAN admins for monitoring and maintaining these thin-provisioned LUNs on the array side, and from the other hand, there were some technical limitations associated with that in terms of SRDF replication or FAST v1 (but that is something specific to EMC, and relevant only to the time of writing this post).

Conclusion:

I will finish this post from where I started. The VMware vSphere is a very powerful and a true enterprise class virtualization platform. You've seen here how I was able to migrate the entire VI3.5 environment without one single second of downtime, and also how it was an extremely easy process to migrate our complete storage from one array vendor to another without any interruption in the servers/services whatsoever. There is nothing extraordinary in this scenario (except maybe the embedded video in the diagram), and you've seen how easy the steps are, and how everything we've done here is built in vSphere itself. Just know your requirement, plan your migration ahead, and you will be just fine!

vSphere In Motion: A Real-World Live Migration Scenario posted by Hany Michael @ HYPERVIZOR.COM

I updated the diagram (v1.2) to fix a small typo and adjust also a couple of shapes. Thanks to Joshua Liebster & Bert Bouwhuis for driving my attention to this.

I know everybody skips to the diagram so I'll save you the introduction, just make sure to quickly go through the notes that follow it:

• This is not an introduction to the VMware HA, and it's not a very advanced diagram for it either. I assume here that you have a general idea on the topic before looking into it to appreciate this incredible technology. If you are a VMware professional you may also find this useful to keep your information sharp and present about the topic at any given time. You really don't have to re-read the documentation every time you'd like to remember a small detail about the subject.
• I'm introducing in this diagram the "Layers" feature in Visio for the first time. The diagram may look somewhat confusing at the first glance, so I thought that it might be a good idea to use these layers for you to hide/show the topics that you are going through in the diagram. I can see some other use cases for the Layers in future diagrams, so I hope you will like it.
• This is an A3 diagram, sorry I know most of you just love the traditional A4 from the feedback I get, but seriously, it's just TMI to fit in A4.
• Everything you see in this diagram, and specifically for the admission control, is *not* fictitious. This is a real cluster I built specifically before designing this diagram. I wanted everything to be 100% accurate and more importantly: realistic. If you zoom into the middle of the vCenter shape, you will be able to see the actual screenshot of the vCenter interface showing the HA cluster I used, and its runtime information window as well.
• It's worth mentioning that this is not all the "advanced options" that you can use for VMware HA. I just selected the ones I thought that might be more frequently used. You can always get back to the official VMware documentation for the complete list.
• The Admission Control was probably the hardest part not just to visualize it, but also to understand it in the first place! That being said, I do not expect anyone with no prior reading on this specific topic to just get it from the first glance when he/she looks into the diagram. Duncan Epping has an excellent article that I think everyone already knows about it, but it's worth mentioning that it's the best place you will ever find for VMware HA in general. The diagram should help you though to understand it faster and easier. You can see all the numbers/calculations in front of you in one shot, and how all these numbers are related to each other.
• This HA lab was built in nearly 5 minuets and is 100% virtual. Long live Lab Manager 4.0 ! (more details here)

That's all folks! I hope you will find it useful!

Diagram: VMware High-Availability (UPDATE: v1.2) posted by Hany Michael @ HYPERVIZOR.COM

A Boring Introduction:

It's been a crazy week! A lot of stuff is happening right now at my work, personal life, and my career. For example, I'm building our much-awaited "Private Cloud" at work, using both the ultimate vSphere Cloud OS and the rock-solid IBM hardware that was finally delivered this week. But wait, this is not 'the' exciting thing that was happing this week for me, it's most definitely the news that I've received last Thursday about winning the first round of the vSphere blogging contest. I will not thank John Troyer & Mike Adams for their great idea and their incredible efforts for organizing this contest, and I will not thank Deepak Narain, the man behind this blog existence who kept pushing me to lunch it a year back (more on this soon), I will, instead, thank everyone for their kind words and encouragement (including the names I've just mentioned), I was literally thrilled by the emails, blog posts and "tweets" that were thrown at me since the news was out. THANK YOU!

Now, enough of this boring talk about me, myself and I, and let's get started with this new round of the blogging contest. A heads up first: I was not supposed to participate this week since I've been so busy as you see, but I had a 24 hours after canceling some plans that I had at the last minute. That said, what you see here is not quite final, I believe I need to work more on the diagram especially the IO plane layout in the hidden vSS, and probably add a couple of more configurations to the video to show some cool stuff like the consistent network stats of a mobile VM jumping from an ESX to another. I'll be updating all these stuff hopefully during next week.

The Configuration Video:

The Architecture Diagram:

Another vSphere diagram! I told you, you are going to see these blueprints more than any time before. Quick notes:

• This is an A3 scale diagram in case you want to print it.
• The diagram reflects the exact configuration on the video. I've done this intentionally to make it easier and faster for any one new to the vDS to understand the concept and the various configuration aspects.
• As I mentioned above, due to the very short period of time that I had, I will most probably modify small parts in the diagram to achieve better results. You can come back and check the version number of the diagram to download the latest updates.

MASTER IT!

I love this part at the end of any book/chapter published by SYBEX. It gets down and dirty with all the theoretic parts covered, and guide you through a practical path to try what you've learned. This is what I want to do here as well. The vDS is quite confusing as a concept and configuration for the first time, and I personally didn't get it except when I started getting my hands on it and playing around with the configurations. The challenge here is that you probably won't have the required lab to do this, especially that you need large number of NICs to test all the configurations. If you are one of my regular blog readers, you've probably guessed what I'm getting to. It's the "vSphere in a box"!

Around three month back, I published a series of posts talking about building a vSphere configurations using ESX inside itself. Instead of rewriting the whole story again, here is the links for your consideration. One last thing to note here: the entire lab you've seen in the video was built using Lab Manager 4.0 as you will read in the following posts.

• vSphere in A Box: A "Virtual Private Cloud" Blueprint
• vSphere In A Box: Part (2): Putting the pieces all together
• vSphere In A Box: Part (3): The Lab Manager 4.0 Automation

Special Thanks:

I'd like to thank Duncan Epping for reviewing part of the contents here. I was having some doubts about few points and due to the time constrain, I didn't have the time to research more on them. I asked for Duncan's help and he was very kind to do so.

Additional Recourses:

These are the best resources that I've found so far for the vDS:

• WitePapers: VMware vNetwork Distributed Switch: Migration and Configuration
• VMworld 2009 Sessions: TA2525, TA2105
• Blog Posts: Eric Sloof, Barry Combs, Luc Dekens, ICT-Freak

vSphere 4.0 vNetwork Distributed Switch (vDS) – Video Demonstration + Architecture Diagram posted by Hany Michael @ HYPERVIZOR.COM

This is a response to the new vSphere Blogging contest that was announced in the middle of this month. I truly think that it's a cool idea, and I believe that regardless of winning or losing, the excitement and fun a blogger would have during his/her participation is something awesome by itself.

The rules say that my post need to be compact and straight to the point, so I won't be able to cover all the aspects about something huge like FT. If by any chance I failed to write such a short post, then here are some tips to avoid wasting your time:
1 – If you are one of those people who wear a tie at work, you should jump straight to the "Use Cases" section.
2 – If you are one of those people who are using the words: 10GbE, VMkernel and %RDY, then you probably don't have time for this, but take my advice and have a look on the next two sections.

Fault Tolerance Architecture Diagram:

From the newly launched vSphere Blog on VMTN I quote this part: "[..] they do say a picture is worth a thousand words [..]", well, I believe I've said that also once before on a previous post, and in fact this is the whole concept my blog is built on. So here is a blueprint for the FT with my own tweaks to save you (and me) a thousand words describing how FT works and how it's architectured.. (BTW, this is the first of vSphere blueprints to come):

Fault Tolerance Video Demonstration:

Thank god the contest rules mentioned the possibility of republishing an old content; otherwise I would have been rerecording and video editing this from the scratch, which is a kind of nightmare. I published this video back in April this year when the vSphere was just announced by VMware (the bits were not even available for download at that time), this means it's one of the very first videos ever published about this cool new feature.

Before you hit the play button, let me tell you why this video is deferent from many of the other ones that I've seen later on:
1 – In my scenario, I have three ESX hosts in a cluster rather than two as you may see in most of the FT demonstrations. What is so special about that? Well, it clearly shows the true concept of the "continues availability", where in case of a complete ESX host failure, the FT will not just failover to another host, but will also automatically assign a third host in the cluster to protect itself in case of another host failure until the SysAdmin attend to the incident.

2 – I'm using in this video a continues file copy to the protected VM throughout the host failure process. This is to show you a "real-life" scenario where your VM is busy doing something critical (backup for example). You really don't play movies in your mission critical VMs (I think Microsoft is the one who invented this idea in their Hyper-V live-migration demonstration, kind of weird!)

My Real-life FT use cases:

I'm taking off now my "VMware Evangelist" hat, and putting on the "VMware Customer" hat. What you'll read here is my real-life use cases for the FT, no marketing talk, no political debates. This "is" the real deal:

1 – Blackberry Enterprise Server & RoveIT Mobile Admin:
BES is one of our most business critical applications because it's being used by our higher management in their day-to-day communications. Initially we were depending on HA since we didn't think that our luck would be that bad to have an ESX host failure while one of the executives sending an email.

This continued to be the case until we deployed the RoveIT Mobile Admin & vCenter Mobile Access (with BES/MDS in the backend). We basically wanted to have a 24/7 access for our SysAdmins to our entire IT environment (including VMware) while they are on the go, using their Blackberry smart-phones (given by the corp for this specific purpose). This was mainly to improve our response time for emergency situations, and of course this service makes no sense unless it can tolerate the most severe situations of hardware failures. Enabling FT on both the BES and the Mobile Admin VMs allow us, from one hand, to ensure that our executives will never complain that they can't use their Blackberry whenever they need, and that "IT Suck". From the other hand, we, the IT suckers..er..i mean SysAdmins & consultants, can have a piece of mind that we will always be able to get to our backend systems wherever there is a problem that requires an immediate attention.

2 – ManageEngine Application Manager:
We heavily depend on the ManageEngine Application Manager in our environment, where we get real-time emails and SMS notifications for any issues happening either in the OS layer (e.g. disk usage, service status ..etc) or the applications (e.g. Exchange high local queue, MS SQL DB issues ..etc). In order to maintain this level of real-time notifications, we had to put this application in a very high availability. Although the application comes with optional cluster capabilities, the VMware HA really was doing this trick without paying extra money. In both cases (the cluster option or the ESX HA) if an ESX host fails, we will have to wait approximately 10min for the application to be powered and operational on another host in the cluster. This is not realistic for an application that is supposed to tell us that the ESX has failed at the first place. With FT we are able to have the application up & running all the time with no interruption whatsoever, and consequently send us the notifications of any Host/OS/Application issues no matter what happens across the underlying infrastructure.

3 – Custom Application – Online payment gateway:
We have an online customer payment service consisting of a custom written application integrated with the IBM Websphere MQ and a backend Oracle DB. Everything is in high availability as you would expect, except for the custom application! I must add also that it is poorly written that it needs human intervention every time the VM needs to be rebooted in order to bring it up again. That being said, HA is not even an option in case of host failures. Unfortunately the application developer does not know how to address these issues in his application, and we are stuck with that fact since he's working with the same backend payment gateway provider. We came up with two solutions for that:
a) The Long run: gradually migrate the online service to a new system with a new backend payment gateway. We are around 30% now on this new service.
b) The short run: put the custom application on FT enabled VM where we don't have to suffer from any unplanned downtime associated with the VM and/or the host.

The conclusion:

FT is a "must have" not a "nice to have" feature in any environment. I don't really understand the big debate around it from the so-called "experts" who have been flooding us on twitter or the blogosphere about reasons why it's "not enterprise ready yet". Most of these debates are coming really from people who have not seen enough of these enterprise environments they are talking about and the challenges we have every day with scenarios like the ones I've listed above. Surely enough, FT has a quite long list of limitations that you can find on any of these blog posts (or on the VMware website itself), but you should also know that VMware is working on most of these limitations in future releases. The number one limitation that you will always hear about is the (1 vCPU) restriction for the FT enabled VM, well, let me tell you two things about that to finish up my article:

1 – The vast majority of the applications running in any datacenter do not need, or even make use of SMP. My three use cases above are examples for that.
2 – VMware has published recently this blog post showing how a 1vCPU VM based on the revolutionary Intel Nehalem processor, can perform better than 2vCPUs using older generations.

P.S. This is probably one of my largest blog posts. I'm disqualified from the contest.

vSphere 4.0 Fault Tolerance (Architecture Diagram, Video and Use Cases) posted by Hany Michael @ HYPERVIZOR.COM

I'm a big believer in the saying "A picture is worth a thousand words". If you don't believe in that, then this blog will never be the right place for you. I think there is a fair amount of my blog readers who had actually visited me in my office, and they've seen how I have all sorts of diagrams covering the walls, starting from the infrastructure and solutions architectures, all the way to detailed blueprints for the deferent technologies that I implement in my environment. Beside the extreme fun I have designing these diagrams, just looking at them on daily basis help me identify the areas of improvement and future developments quite easily. Why am I telling you this small story? Well, you are going to see many of this stuff coming on my blog more than any time before folks!

Introducing the "VMware vSphere In The Enterprise" diagram v1.0

Disclaimer: This is a very, very high level "visualization" of the "virtualization" architecture using VMware vSphere. Having said that, this should never be taken for granted or looked at as the perfect design for your vSphere environment. There is no such thing as a "perfect design" at the first place. There is always a customer requirement, and best practices that we follow to achieve the "perfect solution" for the customer. I can't stress enough on this point as I know there are many VMware-newbie visitors on my blog who might be caught in this trap.

A word of appreciation: I'd like to thank Duncan Epping for his great work of choosing the (Top 5 Planet V12n blog posts), which even for me, as a good follower of that RSS feed, I always miss quite a few great posts in there. In week 29, Duncan selected this post that had an incredible list of network ports in the VMware environment, which I have used some of them in my diagram above. It's not the complete list of course since it's out of my diagram scope, although I do intend to do a complete "block diagram" in the future to visualize the entire list.

Printing Considerations:
In case you haven't noticed, this is an A2 scale diagram. I've initially tried to fit it into A3 while designing it but I couldn't. The amount of information and layouts were just too much to fit in the A3 scale. The diagram still prints well on A3, but you'll have a hard time reading some parts like the port numbers. That said, I highly recommend that you print it on an A2 plotter, which will give you the real look and feel of the diagram. In my case, although we have in our GIS department many plotters for printing even larger scales like A1 and A0, I just went to the nearest Xerox center and printed it there just to make sure how it will look like in commercial printing centers, and the printout was phenomenal.

Diagram: VMware vSphere 4.0 in The Enterprise posted by Hany Michael @ HYPERVIZOR.COM

This is most definitely one of the fastest evaluations I've ever done for any VMware product so far…This is what happened yesterday in order (mapped to my local time zone):

07:00am: The product was announced officially by VMware, and the bits were available for download.
08:00am: I was in my office checking the VMW's website to download Lab Manager 4.0, and then I thought why not, let's do the AppSpeed as well.
08:30am: I fired up the VMworld Europe 2009 AP12 session to refresh my memory about the AppSpeed.
09:00am: I was blown away by the presentation, so I decided to forget about the LM4.0 for a bit and give the AppSpeed a spin.
09:15am: the AppSpeed was up & running on my physical lab.
10:00am: I started deploying it on one of my production ESX clusters for tier-2 and dev apps.
11:00am: when the AppSpeed started to do its thing and show some statistics I was utterly stunned!
08:00pm: I created this video at home on my virtual ESX labs for publishing the next day.

If you want to get to the meat and potatoes of the AppSpeed installation and configuration, you can just view this video to get you started. If you want to see how it might look like in real production, and how it personally saved my day, jump to the next section.

Note: As you saw in the video, my lab doesn't have yet any real applications workload, so there is actually no traffic to be metered.

The AppSpeed saved my day!
I recently designed two SharePoint farms running on VI3 as per VMware & Microsoft recommendations and guidelines, each farm consisted of 2 WFE VMs with NLB, 1 Indexing VM server, with 2 Active/Passive physical SQL nodes for the backend database. The SP farms worked perfectly well most of the time, however, sometimes we used to get some strange delay in page loads, and it was happening randomly across deferent pages and not in any particular time. I kept looking and looking for any reason that would be causing these delays from the virtual environment, but nothing. Yesterday when I deployed the AppSpeed in my production, I was able to see that one of my VMs running the Microsoft SCOM was producing a hell of SQL transactions to the SQL cluster. Since the SCOM is not that critical in our environment, where we heavily depend on other monitoring software like the ManageEngine Application Monitor & OpManager, I stopped the SCOM and I instantly saw a noticeable drop down in the Mem & CPU utilization on the SQL cluster. From that moment till the time of writing these words, the SharePoint farms are performing perfectly well. Our DBAs were able to identify that our physical SQL cluster needs an urgent memory upgrade, and in fact considering virtualizing it completely on our new vSphere installation.

Are you a VMware expert?
The AppSpeed is not meant to replace any of your existing monitoring tools that you are happy with. It’s another great visibility tool for doing that, not to mention the SLA part and the integration with the vCloud technologies. I just wanted to tell you that you still need this even if you are a VMware expert who knows how to use the traditional tools for performance analysis. Let me give you a very practical example:
In the SharePoint scenario that I’ve described above, I could still identify the problem as follows:

• I can use the ManageEngine Application Monitor to identify that there is a high memory utilization on the SQL nodes.
• I can use the NetFlow protocol on VMware ESX 3.5 (as demonstrated here) to identify that the SCOM VM is having a high traffic on the vSwitch out to the SQL cluster.
• I can use esxtop to record, play or analyze in high details that the ESX servers and the SharePoint VMs are either ideal or performing perfectly normal all the time.

Now, ask yourself this: why do I need to do all that if there is a tool that can just give me the complete picture? How much time and effort do I need in order to go through this entire cycle?

To conclude all the above, in the AppSpeed presentation there was a part saying this: "AppSpeed reduces finger pointing and allows IT to focus on solving root cause of issues". This is not marketing words, it's a fact I literally experienced yesterday in my production environment.

I really, really wanted to share some screenshots, but since it's running in my production I couldn't do that here. Even with graphics editing, I've never been a fan of doing something like that. I do promise though that I will put some application workloads in my lab and return back with more videos or screenshots.

I extracted some screenshots from the VMworld session I mentioned at the beginning of my post, just to give you a real feel of what you see. It's still not comparable to what you will experience in your own environment. I do encourage you to do that, whether you are on vSphere or still on VI3 like me.

Video: VMware vCenter AppSpeed 1.0 first look! posted by Hany Michael @ HYPERVIZOR.COM

At the time of submitting this post, the press release from VMware should be out. It's July 13^th, 12 midnight NY time, and VMware is announcing a bunch of vCenter products including the much awaited Lab Manager 4.0.

Unfortunately I was not in the beta program of LM4.0 to have an early hands-on experience, but I've been briefed about it, and I must say that I'm quite happy with the new features of the product. I'm exceptionally interested in this product now more than any time before, for three main reasons:
1) I need to implement LM in a wide scale for our development department. Currently our developers are working independently everyone in his/her own island, using things like VMware server or workstation! No effective collaboration or automation whatsoever.

2) I need to implement LM for our infrastructure team to simulate our entire production environment. Again, I need something that can scale out easily (typically on blades), yet with flexible and fenced network configurations.

3) Lastly, I need to build my own labs for my "virtual private cloud" project, which will typically require building multiple vESX clusters, not to mention the importance of having the templates and libraries for deploying these clusters whenever I need to test something new that would require significant changes (e.g. Nexus1000v for networking, or Reflex vTrust for security)

As soon as I wake up from sleep (yes, technically I'm sleeping now, this is a scheduled post) I will download the bits and start playing with it. I will come back with more details later, but between now and then, here are some highlights on the new features of LM4.0 along with two screenshots from the product team:

- Lab Manager 4.0 now fully supports VMware vSphere 4
- Support for the VMware ESX(i) Form Factor (was not supported in previous versions).
- LM is integrated now with VMware vCenter Stage Manager.
- Host Spanning Private Networks: Host Spanning Private Networks, a new technology in Lab Manager 4, creates isolated private networks without the need for setting up VLANs. This new feature requires the Distributed Switch capability of VMware vSphere Enterprise Plus edition.
- Multiple Workspaces: VMware vCenter Lab Manager 4 introduces the concept of multiple workspaces within organizations.
- Archive to Library: Lab Manager 4 provides the ability to keep a particular configuration together with its change history for record within the library.
- Configuration History: Lab Manager 4 mow provides a new configuration history tab for all configurations. The history of a configuration displays the list of the events related to this configuration.

For the complete and detailed list of new features, you can check out this WP from VMware: http://www.vmware.com/files/pdf/VMW_09Q2_WP_vCenter_LabManager4_10_R1.pdf

Screenshot (1): VMware vCenter Lab Manager 4 streamlines application releases from development to production. The self-service interface provides on-demand access to virtualized application environments while IT remains in administrative control.

Screenshot (2): Advanced networking capabilities in Lab Manager 4 allow application teams to create realistic, production-like test environments for complex system and network configurations.

Release: VMware vCenter Lab Manager 4.0 posted by Hany Michael @ HYPERVIZOR.COM

The response to my previous post has been unreal! The amount of tweets, ping backs, hits, linking, emails was quite amazing. I have to admit, I didn't expect to see that much of interest in the subject, but thanks to everyone who participated in promoting this idea on twitter and the blogosphere.

In the second part of this series we will spice things up a bit and explore through the following video many aspects of the idea we've talked about. Following to that an important screenshots and some considerations you should be aware of before and after implementing this in your lab.

What you'll see in this video:
1- Deploy a thin-provisioned vESX(i) VM from a template.
2- Check the required configuration parameter on the vESX to run nested VMs.
3- Customizing the new vESX server (assign password, set a static IP, put the DNS config ..etc)
4- Add the vESX to an existing HA Cluster.
5- Test the vMotion within the same cluster and across deferent clusters in the datacenter.
6- Add the required configuration parameter on the nested VM for enabling the FT.
7- Enable the FT and test the failover across deferent vESX servers.

A Quick note on the hardware used:
Technically speaking, only one server can be used in this whole setup. What is deferent in my setup (as you saw in the diagram) is that I use an external iSCSI array (the CLARiiON AX4) for hosting the vESX VMs. I just needed the flexibility to have them on an external storage to share them later on with other servers, but it is not a requirement. You can simply use the internal storage of the pESX server to host your vESX and you will still have everything you see in the video. As far as the shared storage for the vESX servers is concerned, you can use the Celerra VSA. In my case I use the Celerra only for the SRM labs to do the replication trick. Other than that, I use the OpenFiler as my shared storage for the nested VMs.

Two things to be set on the pESX hosts:
1 – Increase the number of ports on your pESX vSwitch to accommodate the increased number of connections required by your vESX VMs.

2 – Enable the "Promiscuous Mode" on the vSpwitch

The configuration parameters on the Virtual Machines are as follows:
1 – The virtual ESX (vESX) host: monitor_control.restrict_backdoor = TRUE

2 – The nested VM: replay.allowBTOnly = TRUE

The iSCSI vSwitch on the pESX host bound to vmnic1 (phisical NIC 2) and connected to the EMC CLARiiON AX4 iSCSI array

The vMotion internal vSwitch on the pESX host

The Fault Tolerance internal vSwitch on the pESX host

The thin-provisioned ESX(i) size on disk (475MB) + the memory swap:

The thin-provisioned ESX(i) size on disk(3.5GB) + the memory swap:

Other considerations and GOTCHAs:
1 – When enabling the FT, make sure you have your VMs powered off, even if they have eager zeroed disks, I used to get some errors when the VMs were powered on while enabling FT.

2 – Sometimes the network card order and numbering could be confusing. For example, in the vESX VM, you will have the NICs order starting from 1 to 10, but in the actual vESX network configuration tab, you will find the NICs starting from 0 and counting towards 9. This could be confusing when mapping your vnics to the pESX host vSwitches, like the VMotion internal switch, the FT internal switch and so forth. Just make sure you count the nics order accurately.

3 – Deploying vESX from templates may be cool and fast, but it could a bit challenging sometime in troubleshooting the network related issues. The reason behind that is the fact that all the network cards will have the same MAC address, or worst, the Port Groups like (VMotion) could have the same MAC address even if you completely remove the vnics and created brand new ones. The work around for that is to create the vESX template, and then remove all the NICs form it. When you deploy a new vESX, you can just add the new nics as you like, and by that you'll have a new MAC addresses. Beside that, you may need to add a new VMkernel network for the VMotion, and then remove the old one. Of course you may be thinking that deploying a brand new vESX would be easier, you are right, but with the scripting everything could be automated. I will try to write a PS script to automate this network changes/settings and post it here later on.

vSphere In A Box: Part(2): Putting the pieces all together posted by Hany Michael @ HYPERVIZOR.COM

If you are a busy techie like myself, and do not have the time to scratch your head, you can have a quick glance on the following diagram and wait for the next blog post. But if you read on, I promise that this article will inspire you in a way or another.

First off, I take no credit for the basic technical information of running an ESX inside itself. This information has been floating around on the V12n blogosphere and VMTN forums for quite some time now, but no one, as far as I know, has gathered them in one place to put together the missing pieces, and most importantly, illustrated why you should consider doing this at the first place. My main objective from this article(s) is to open your eyes for some cool ideas that you can try in your labs instead of using the traditional physical ESX hosts, or even ESX in VMware WorkStation.

A quick background:
The idea of running virtual ESX servers on top of a physical ESX host has been consuming a lot of my time and research lately. Unfortunately I was doing all these tests and trials on ESX 3.5, and later on I came to know that a special build needed to be used in order to do that. The time passed, and the vSphere came out with all its mind-blowing features and capabilities, but a few of us paid a close attention to the fact that ESX 4.0 came packed and ready to be virtualized inside itself. Eric Gray from VMware broke the news with his great article on VCritical.com, and a week later another cool guy named Itzik Reich, on Chad Sakac's blog, described how FT can be enabled on a virtual ESX (vESX). Great Stuff!

Think out of the box!
Every single time I ask someone about virtualizing ESX inside itself, I always get the same exact question: "Why would you want to do something crazy like that?!" I even remember during my VMware VI3 class I asked the instructor about it, and his answer was: "it's so unsupported", who's talking about support now, and why should I bother if I will never, ever, run something like that in production?! That being said, let's think out of the box for a minute.

Ask yourself the following:

• How long it takes to deploy a classic ESX on a physical host using the traditional way?
• Do I have enough server recourses for building a VMware lab?
• Are my servers up-to-date with the latest CPU technologies in order to test things like FT?
• Do I have enough network resources (switch ports, cables ..etc) to allocate 10 network connections for each ESX servers? Is there any blade server that can have 10 Ethernet ports at the first place?
• Do I have the required storage for building, let's say, 10 classic ESX servers to boot them from the SAN?
• Am I willing to do an aggressive power off for my physical ESX servers just to test the HA mechanism, and its advanced options, to simulate a real life server failure?

If you found the above a bit challenging, you are in the right place. If you don't, you are still in the right place. I told you, think out of the box.

I consider myself lucky enough to have all the resources available for me in my current work environment. When I ask for hardware resources or software licenses I simply get them. Right now I have more than 10 dedicated blades in my lab running deferent ESX flavors and versions. In fact, in our new hardware refresh I will have a brand new set of high specs blades and storage dedicated specifically for the VMware R&D.

What is the deal then?

I need to build a complete private cloud! I need to have everything starting from ESX clusters to dvSwitches, Cisco NX1Vs, virtual appliances, VMsafe based solutions, and last but not least, a working SRM installation between two virtual DCs. I need to have the complete feel of this so-called "private cloud" before I even start an actual PoC, which will be way complicated & a bit challenging in the physical world. I need to go to the management with diagrams and videos and tell them why we need to be 100% virtualized, and why we should start planning for that. Show them where we are headed, and how our IT environment and datacenters will look like one year from now.

Put your ideas back in "the box"
Enough dreaming, the reality is actually one step away. Believe it or not, all what you need is a single server with a reasonable memory and that's all. But I won't go into the technical details today, let's see what ideas we may think of now:

1. Thin-provision your ESX hosts! Yes, all my fat classic ESX hosts are now thin-provisioned, this means I can deploy a 40GB HDD for the ESX, but actually consume less than 3.5GB only!
2. 1 Minute fast deployment! I can now deploy new ESX servers in a matter of seconds from pre-built templates (with special considerations we'll talk about later).
3. No network limitations! A 10 NICs on an ESX server is not something you can test every day.
4. SRM in a Box! Now with the coming SRM 2.0 and the ESX 4.0 memory restrictions, it's going to be extremely hard to maintain the bare minimum of two ESX hosts, nested VM and two VSAs replication on a laptop with 4GB.
5. I'm online 24/7, and still mobile! Yes, I'm no longer running my virtual ESX servers on a laptop, I have them online and under my mouse clicks whenever required. VPN into your Lab or even use your BlackBerry/vCMA on the GO! No more shutdowns, memory restrictions or space issues.
6. Migrate your labs or even share them! One of the very painful situations I went through lately was shifting my physical ESX lab from one of our datacenters to another. If I had my vESX hosts at that time all what I needed to do is to take them on an external USB disk, or even copy them over the WAN and I'm done.
   
7. VMware (YES, THE COMPANY): I still see a great potential of implementing the idea that I proposed sometime back (the on-demand labs). You see here that with this simple setup we can test almost everything in vSphere like HA, DRS, VMotion and even FT & SRM! I really hope that VMware is already working on something like that or at least someone will take the initiative to put this idea in action someday.
   

What's next?
So let's say we've built this "virtual private cloud", will that be it? The answer is No. It is actually the beginning in my opinion. There are so many other cool ideas that I can't get off my head. For example, I can build another/deferent private cloud and start exploring this whole "cloud computing" thing. I can test and develop new approaches for VMotion'ing workloads across two deferent sites. Remember, you have a huge flexibility and control over your "clouds", and most importantly, you are not afraid to screw up anything since you can rebuild your elements quite easily, and almost instantly (snapshots/backups/templates and so forth)

In the next blog post I will get more technical and list the details of my current setup which I illustrated in the diagram above. In the addition to that, I will explore the planned developments for building the complete "VPC". May be also we can start thinking of automating and managing individual labs using the new and long awaited Lab Manager 4.0, expected sometime this month.

Teaser: A vCenter screenshot showing the 10 NICs and their configuration on one of the vESX servers.

Note: If you are as much excited about this idea as I am, and can't wait for my next post (maybe I will close the blog and change my career, who knows?) you can visit these links to get you up to speed with the setup:

• Eric Gray's post that unleashed the beast of the vESX.
• Itzik Reich's reply in Chad's blog, containing the FT configuration parameter on the nested VM.
• Maish Saidel-Keesing has two excellent posts Part 1 & 2 for a similar lab setup.

 
Mike DiPetrillo's response (in case you missed it in the replies below)

Actually, VMware has been doing a lot of this. Why do you think all of the stuff was put into the builds to run nested ESX in the first place? VMware SEs have been running nested VMs for a long time on their laptops using Workstation or VMware Fusion.

VMware also has an internal environment called vSEL that runs nearly all of the products in a nested environment. vSEL (the virtual SE Lab) let’s our tech resources in the field deploy and learn our applications as well as do demos and training with customers and partners. This “cloud” services over 1,200 tech people today inside of VMware. This is being expanded to let our development teams use it for development.

VMware is now working with several service providers around the world to let people request and run ESX environments virtually in the cloud – either for doing trials of the software or for doing actual deployments.

Lastly, there’s a new cloud service being built that will run nested VMware products to service all of the hands on labs for VMworld. Make sure to make it out to VMworld to get some hands on experience with VMware’s implementation of this environment.

Definitely a good picture and thanks for getting the word out. Just wanted to let you know that VMware has been doing this for years and has several different clouds built with this already.

vSphere in a Box: A “Virtual Private Cloud” Blueprint posted by Hany Michael @ HYPERVIZOR.COM