In the second part of this series we will spice things up a bit and explore through the following video many aspects of the idea we’ve talked about. Following to that an important screenshots and some considerations you should be aware of before and after implementing this in your lab.

What you’ll see in this video:
1- Deploy a thin-provisioned vESX(i) VM from a template.
2- Check the required configuration parameter on the vESX to run nested VMs.
3- Customizing the new vESX server (assign password, set a static IP, put the DNS config ..etc)
4- Add the vESX to an existing HA Cluster.
5- Test the vMotion within the same cluster and across deferent clusters in the datacenter.
6- Add the required configuration parameter on the nested VM for enabling the FT.
7- Enable the FT and test the failover across deferent vESX servers.

A Quick note on the hardware used:
Technically speaking, only one server can be used in this whole setup. What is deferent in my setup (as you saw in the diagram) is that I use an external iSCSI array (the CLARiiON AX4) for hosting the vESX VMs. I just needed the flexibility to have them on an external storage to share them later on with other servers, but it is not a requirement. You can simply use the internal storage of the pESX server to host your vESX and you will still have everything you see in the video. As far as the shared storage for the vESX servers is concerned, you can use the Celerra VSA. In my case I use the Celerra only for the SRM labs to do the replication trick. Other than that, I use the OpenFiler as my shared storage for the nested VMs.

Two things to be set on the pESX hosts:
1 – Increase the number of ports on your pESX vSwitch to accommodate the increased number of connections required by your vESX VMs.

2 – Enable the “Promiscuous Mode” on the vSpwitch

The configuration parameters on the Virtual Machines are as follows:
1 – The virtual ESX (vESX) host: monitor_control.restrict_backdoor = TRUE

2 – The nested VM: replay.allowBTOnly = TRUE

The iSCSI vSwitch on the pESX host bound to vmnic1 (phisical NIC 2) and connected to the EMC CLARiiON AX4 iSCSI array

The vMotion internal vSwitch on the pESX host

The Fault Tolerance internal vSwitch on the pESX host

The thin-provisioned ESX(i) size on disk (475MB) + the memory swap:

The thin-provisioned ESX(i) size on disk(3.5GB) + the memory swap:

Other considerations and GOTCHAs:
1 – When enabling the FT, make sure you have your VMs powered off, even if they have eager zeroed disks, I used to get some errors when the VMs were powered on while enabling FT.

2 – Sometimes the network card order and numbering could be confusing. For example, in the vESX VM, you will have the NICs order starting from 1 to 10, but in the actual vESX network configuration tab, you will find the NICs starting from 0 and counting towards 9. This could be confusing when mapping your vnics to the pESX host vSwitches, like the VMotion internal switch, the FT internal switch and so forth. Just make sure you count the nics order accurately.

3 – Deploying vESX from templates may be cool and fast, but it could a bit challenging sometime in troubleshooting the network related issues. The reason behind that is the fact that all the network cards will have the same MAC address, or worst, the Port Groups like (VMotion) could have the same MAC address even if you completely remove the vnics and created brand new ones. The work around for that is to create the vESX template, and then remove all the NICs form it. When you deploy a new vESX, you can just add the new nics as you like, and by that you’ll have a new MAC addresses. Beside that, you may need to add a new VMkernel network for the VMotion, and then remove the old one. Of course you may be thinking that deploying a brand new vESX would be easier, you are right, but with the scripting everything could be automated. I will try to write a PS script to automate this network changes/settings and post it here later on.

First off, I take no credit for the basic technical information of running an ESX inside itself. This information has been floating around on the V12n blogosphere and VMTN forums for quite some time now, but no one, as far as I know, has gathered them in one place to put together the missing pieces, and most importantly, illustrated why you should consider doing this at the first place. My main objective from this article(s) is to open your eyes for some cool ideas that you can try in your labs instead of using the traditional physical ESX hosts, or even ESX in VMware WorkStation.

A quick background:
The idea of running virtual ESX servers on top of a physical ESX host has been consuming a lot of my time and research lately. Unfortunately I was doing all these tests and trials on ESX 3.5, and later on I came to know that a special build needed to be used in order to do that. The time passed, and the vSphere came out with all its mind-blowing features and capabilities, but a few of us paid a close attention to the fact that ESX 4.0 came packed and ready to be virtualized inside itself. Eric Gray from VMware broke the news with his great article on VCritical.com, and a week later another cool guy named Itzik Reich, on Chad Sakac’s blog, described how FT can be enabled on a virtual ESX (vESX). Great Stuff!

Think out of the box!
Every single time I ask someone about virtualizing ESX inside itself, I always get the same exact question: “Why would you want to do something crazy like that?!” I even remember during my VMware VI3 class I asked the instructor about it, and his answer was: “it’s so unsupported”, who’s talking about support now, and why should I bother if I will never, ever, run something like that in production?! That being said, let’s think out of the box for a minute.

Ask yourself the following:

• How long it takes to deploy a classic ESX on a physical host using the traditional way?
• Do I have enough server recourses for building a VMware lab?
• Are my servers up-to-date with the latest CPU technologies in order to test things like FT?
• Do I have enough network resources (switch ports, cables ..etc) to allocate 10 network connections for each ESX servers? Is there any blade server that can have 10 Ethernet ports at the first place?
• Do I have the required storage for building, let’s say, 10 classic ESX servers to boot them from the SAN?
• Am I willing to do an aggressive power off for my physical ESX servers just to test the HA mechanism, and its advanced options, to simulate a real life server failure?

If you found the above a bit challenging, you are in the right place. If you don’t, you are still in the right place. I told you, think out of the box.

I consider myself lucky enough to have all the resources available for me in my current work environment. When I ask for hardware resources or software licenses I simply get them. Right now I have more than 10 dedicated blades in my lab running deferent ESX flavors and versions. In fact, in our new hardware refresh I will have a brand new set of high specs blades and storage dedicated specifically for the VMware R&D.

What is the deal then?

I need to build a complete private cloud! I need to have everything starting from ESX clusters to dvSwitches, Cisco NX1Vs, virtual appliances, VMsafe based solutions, and last but not least, a working SRM installation between two virtual DCs. I need to have the complete feel of this so-called “private cloud” before I even start an actual PoC, which will be way complicated & a bit challenging in the physical world. I need to go to the management with diagrams and videos and tell them why we need to be 100% virtualized, and why we should start planning for that. Show them where we are headed, and how our IT environment and datacenters will look like one year from now.

Put your ideas back in “the box”
Enough dreaming, the reality is actually one step away. Believe it or not, all what you need is a single server with a reasonable memory and that’s all. But I won’t go into the technical details today, let’s see what ideas we may think of now:

1. Thin-provision your ESX hosts! Yes, all my fat classic ESX hosts are now thin-provisioned, this means I can deploy a 40GB HDD for the ESX, but actually consume less than 3.5GB only!
2. 1 Minute fast deployment! I can now deploy new ESX servers in a matter of seconds from pre-built templates (with special considerations we’ll talk about later).
3. No network limitations! A 10 NICs on an ESX server is not something you can test every day.
4. SRM in a Box! Now with the coming SRM 2.0 and the ESX 4.0 memory restrictions, it’s going to be extremely hard to maintain the bare minimum of two ESX hosts, nested VM and two VSAs replication on a laptop with 4GB.
5. I’m online 24/7, and still mobile! Yes, I’m no longer running my virtual ESX servers on a laptop, I have them online and under my mouse clicks whenever required. VPN into your Lab or even use your BlackBerry/vCMA on the GO! No more shutdowns, memory restrictions or space issues.
6. Migrate your labs or even share them! One of the very painful situations I went through lately was shifting my physical ESX lab from one of our datacenters to another. If I had my vESX hosts at that time all what I needed to do is to take them on an external USB disk, or even copy them over the WAN and I’m done.
   
7. VMware (YES, THE COMPANY): I still see a great potential of implementing the idea that I proposed sometime back (the on-demand labs). You see here that with this simple setup we can test almost everything in vSphere like HA, DRS, VMotion and even FT & SRM! I really hope that VMware is already working on something like that or at least someone will take the initiative to put this idea in action someday.
   

What’s next?
So let’s say we’ve built this “virtual private cloud”, will that be it? The answer is No. It is actually the beginning in my opinion. There are so many other cool ideas that I can’t get off my head. For example, I can build another/deferent private cloud and start exploring this whole “cloud computing” thing. I can test and develop new approaches for VMotion’ing workloads across two deferent sites. Remember, you have a huge flexibility and control over your “clouds”, and most importantly, you are not afraid to screw up anything since you can rebuild your elements quite easily, and almost instantly (snapshots/backups/templates and so forth)

In the next blog post I will get more technical and list the details of my current setup which I illustrated in the diagram above. In the addition to that, I will explore the planned developments for building the complete “VPC”. May be also we can start thinking of automating and managing individual labs using the new and long awaited Lab Manager 4.0, expected sometime this month.

Teaser: A vCenter screenshot showing the 10 NICs and their configuration on one of the vESX servers.

Note: If you are as much excited about this idea as I am, and can’t wait for my next post (maybe I will close the blog and change my career, who knows?) you can visit these links to get you up to speed with the setup:

• Eric Gray’s post that unleashed the beast of the vESX.
• Itzik Reich’s reply in Chad’s blog, containing the FT configuration parameter on the nested VM.
• Maish Saidel-Keesing has two excellent posts Part 1 & 2 for a similar lab setup.

 
Mike DiPetrillo‘s response (in case you missed it in the replies below)

Actually, VMware has been doing a lot of this. Why do you think all of the stuff was put into the builds to run nested ESX in the first place? VMware SEs have been running nested VMs for a long time on their laptops using Workstation or VMware Fusion.

VMware also has an internal environment called vSEL that runs nearly all of the products in a nested environment. vSEL (the virtual SE Lab) let’s our tech resources in the field deploy and learn our applications as well as do demos and training with customers and partners. This “cloud” services over 1,200 tech people today inside of VMware. This is being expanded to let our development teams use it for development.

VMware is now working with several service providers around the world to let people request and run ESX environments virtually in the cloud – either for doing trials of the software or for doing actual deployments.

Lastly, there’s a new cloud service being built that will run nested VMware products to service all of the hands on labs for VMworld. Make sure to make it out to VMworld to get some hands on experience with VMware’s implementation of this environment.

Definitely a good picture and thanks for getting the word out. Just wanted to let you know that VMware has been doing this for years and has several different clouds built with this already.