If you are a busy techie like myself, and do not have the time to scratch your head, you can have a quick glance on the following diagram and wait for the next blog post. But if you read on, I promise that this article will inspire you in a way or another.

First off, I take no credit for the basic technical information of running an ESX inside itself. This information has been floating around on the V12n blogosphere and VMTN forums for quite some time now, but no one, as far as I know, has gathered them in one place to put together the missing pieces, and most importantly, illustrated why you should consider doing this at the first place. My main objective from this article(s) is to open your eyes for some cool ideas that you can try in your labs instead of using the traditional physical ESX hosts, or even ESX in VMware WorkStation.

A quick background:
The idea of running virtual ESX servers on top of a physical ESX host has been consuming a lot of my time and research lately. Unfortunately I was doing all these tests and trials on ESX 3.5, and later on I came to know that a special build needed to be used in order to do that. The time passed, and the vSphere came out with all its mind-blowing features and capabilities, but a few of us paid a close attention to the fact that ESX 4.0 came packed and ready to be virtualized inside itself. Eric Gray from VMware broke the news with his great article on VCritical.com, and a week later another cool guy named Itzik Reich, on Chad Sakac’s blog, described how FT can be enabled on a virtual ESX (vESX). Great Stuff!

Think out of the box!
Every single time I ask someone about virtualizing ESX inside itself, I always get the same exact question: “Why would you want to do something crazy like that?!” I even remember during my VMware VI3 class I asked the instructor about it, and his answer was: “it’s so unsupported”, who’s talking about support now, and why should I bother if I will never, ever, run something like that in production?! That being said, let’s think out of the box for a minute.

Ask yourself the following:

• How long it takes to deploy a classic ESX on a physical host using the traditional way?
• Do I have enough server recourses for building a VMware lab?
• Are my servers up-to-date with the latest CPU technologies in order to test things like FT?
• Do I have enough network resources (switch ports, cables ..etc) to allocate 10 network connections for each ESX servers? Is there any blade server that can have 10 Ethernet ports at the first place?
• Do I have the required storage for building, let’s say, 10 classic ESX servers to boot them from the SAN?
• Am I willing to do an aggressive power off for my physical ESX servers just to test the HA mechanism, and its advanced options, to simulate a real life server failure?

If you found the above a bit challenging, you are in the right place. If you don’t, you are still in the right place. I told you, think out of the box.

I consider myself lucky enough to have all the resources available for me in my current work environment. When I ask for hardware resources or software licenses I simply get them. Right now I have more than 10 dedicated blades in my lab running deferent ESX flavors and versions. In fact, in our new hardware refresh I will have a brand new set of high specs blades and storage dedicated specifically for the VMware R&D.

What is the deal then?

I need to build a complete private cloud! I need to have everything starting from ESX clusters to dvSwitches, Cisco NX1Vs, virtual appliances, VMsafe based solutions, and last but not least, a working SRM installation between two virtual DCs. I need to have the complete feel of this so-called “private cloud” before I even start an actual PoC, which will be way complicated & a bit challenging in the physical world. I need to go to the management with diagrams and videos and tell them why we need to be 100% virtualized, and why we should start planning for that. Show them where we are headed, and how our IT environment and datacenters will look like one year from now.

Put your ideas back in “the box”
Enough dreaming, the reality is actually one step away. Believe it or not, all what you need is a single server with a reasonable memory and that’s all. But I won’t go into the technical details today, let’s see what ideas we may think of now:

1. Thin-provision your ESX hosts! Yes, all my fat classic ESX hosts are now thin-provisioned, this means I can deploy a 40GB HDD for the ESX, but actually consume less than 3.5GB only!
2. 1 Minute fast deployment! I can now deploy new ESX servers in a matter of seconds from pre-built templates (with special considerations we’ll talk about later).
3. No network limitations! A 10 NICs on an ESX server is not something you can test every day.
4. SRM in a Box! Now with the coming SRM 2.0 and the ESX 4.0 memory restrictions, it’s going to be extremely hard to maintain the bare minimum of two ESX hosts, nested VM and two VSAs replication on a laptop with 4GB.
5. I’m online 24/7, and still mobile! Yes, I’m no longer running my virtual ESX servers on a laptop, I have them online and under my mouse clicks whenever required. VPN into your Lab or even use your BlackBerry/vCMA on the GO! No more shutdowns, memory restrictions or space issues.
6. Migrate your labs or even share them! One of the very painful situations I went through lately was shifting my physical ESX lab from one of our datacenters to another. If I had my vESX hosts at that time all what I needed to do is to take them on an external USB disk, or even copy them over the WAN and I’m done.
   
7. VMware (YES, THE COMPANY): I still see a great potential of implementing the idea that I proposed sometime back (the on-demand labs). You see here that with this simple setup we can test almost everything in vSphere like HA, DRS, VMotion and even FT & SRM! I really hope that VMware is already working on something like that or at least someone will take the initiative to put this idea in action someday.
   

What’s next?
So let’s say we’ve built this “virtual private cloud”, will that be it? The answer is No. It is actually the beginning in my opinion. There are so many other cool ideas that I can’t get off my head. For example, I can build another/deferent private cloud and start exploring this whole “cloud computing” thing. I can test and develop new approaches for VMotion’ing workloads across two deferent sites. Remember, you have a huge flexibility and control over your “clouds”, and most importantly, you are not afraid to screw up anything since you can rebuild your elements quite easily, and almost instantly (snapshots/backups/templates and so forth)

In the next blog post I will get more technical and list the details of my current setup which I illustrated in the diagram above. In the addition to that, I will explore the planned developments for building the complete “VPC”. May be also we can start thinking of automating and managing individual labs using the new and long awaited Lab Manager 4.0, expected sometime this month.

Teaser: A vCenter screenshot showing the 10 NICs and their configuration on one of the vESX servers.

Note: If you are as much excited about this idea as I am, and can’t wait for my next post (maybe I will close the blog and change my career, who knows?) you can visit these links to get you up to speed with the setup:

• Eric Gray’s post that unleashed the beast of the vESX.
• Itzik Reich’s reply in Chad’s blog, containing the FT configuration parameter on the nested VM.
• Maish Saidel-Keesing has two excellent posts Part 1 & 2 for a similar lab setup.

 
Mike DiPetrillo‘s response (in case you missed it in the replies below)

Actually, VMware has been doing a lot of this. Why do you think all of the stuff was put into the builds to run nested ESX in the first place? VMware SEs have been running nested VMs for a long time on their laptops using Workstation or VMware Fusion.

VMware also has an internal environment called vSEL that runs nearly all of the products in a nested environment. vSEL (the virtual SE Lab) let’s our tech resources in the field deploy and learn our applications as well as do demos and training with customers and partners. This “cloud” services over 1,200 tech people today inside of VMware. This is being expanded to let our development teams use it for development.

VMware is now working with several service providers around the world to let people request and run ESX environments virtually in the cloud – either for doing trials of the software or for doing actual deployments.

Lastly, there’s a new cloud service being built that will run nested VMware products to service all of the hands on labs for VMworld. Make sure to make it out to VMworld to get some hands on experience with VMware’s implementation of this environment.

Definitely a good picture and thanks for getting the word out. Just wanted to let you know that VMware has been doing this for years and has several different clouds built with this already.