Posts Tagged ‘in a box’

We've got quite a bit of stuff to talk about in this post/diagram/video, so I'll keep the introduction as simple as this: It's the third part, it's the final part, and it's my favorite part of the (vSphere In A Box) series.

The Diagram:
This is an overall illustration of the workflow and automation process of the environment using the same single server, and an iSCSI array, that we've used in the previous posts. The diagram may look a bit complicated at the first glance, but I tried to keep things as simple as possible. I used the actual IP addresses, configurations, and workplaces names that I use in my environment and as you will see in the next video, to keep things clear and easier to follow.

The Video:
This is by no mean a Lab Manager tutorial, it's just a very "quick tour" on the components and automation of the my vSphere labs. I intend to do a series of tutorials in the future on Lab Manager 4.0 itself, but for the sake of this subject I created this quick walkthrough video instead of taking more than 20 screenshots that won't make much sense individually.

My Use Cases:
If you've been reading my previous few posts, you've probably known what are my objectives from this, and since I hate to repeat myself, I will copy & paste one of the use cases I mentioned in an earlier post, followed by two new cases I'm working on now:

1. [...] I need to build my own labs for my "virtual private cloud" project, which will typically require building multiple vESX clusters, not to mention the importance of having the templates and libraries for deploying these clusters whenever I need to test something new that would require significant changes (e.g. Nexus1000v for networking, or Reflex vTrust for security) [...]
2. I'm trying these days also to evaluate the VMware Site Recovery Manager running in two deferent and independent setups to simulate a real life scenario instead of using the traditional all-in-one-subnet setup. Since part of these applications are still in Beta, and consequently under NDA, I won't be able to elaborate more on this use case until all the software are in the GA stage.
3. I discovered another great use case while I was working on the Lab Manager, which is training the SysAdmins in my current corp on the new vSphere platform. We are also growing rapidly with our virtual environment (soon to be: private cloud) and we'll definitely need more SysAdmins on board. That being said, I now have an instrumental tool for provisioning independent labs for the SysAdmins to login and have their own hands-on experience with the new platform.

Other Considerations:
As you all know, the vCenter Lab Manager 4.0 was never meant to run virtual ESX hosts inside itself, therefore, there are some tweaks and considerations that you need to know about while you are implementing this in your own environment. But the beauty of VMware is that all its components are highly customizable and work in a true harmony regardless of what you want to achieve at the end.

1) As you know, the vESX hosts need to be connected to a "Promiscuous enabled" vSwitch. When you create new Network Templates in LM, and later on deploy them within your configurations, the LM by default create the new vSwitch with this feature disabled, so you will need to go an manually enable that. I'm trying to figure out a way to automate this or set it to be enabled by default, I will update the post should I manage to do that.

2) When you import your vESX templates you must uncheck the customization part as shown in the screenshot since the VM is not running a supported OS by VMware (like Windows or Linux), however, I managed to install the VMware tools inside the classic ESX 4.0 as shown in the next point.

3) Although I haven’t tested this thoroughly yet, I believe you can customize the classic ESX hosts with COS in Lab Manager by installing the VMware tools as follows:

Mount the VMware tools ISO

unzip/untar the package

Run the perl script and follow the installation instructions:

Conclusion:
By this part, I believe that we’ve completed the entire series of the (vSphere In A  Box) and shown how and why you need to be doing this setup. I hope that everyone can clearly see now the huge deference between running the ESX hosts inside VMware Workstation compared to running it inside ESX itself, as well as the great benefits of doing that in a larger scale with the automation to achieve many of the use cases mentioned across this series of posts.

The response to my previous post has been unreal! The amount of tweets, ping backs, hits, linking, emails was quite amazing. I have to admit, I didn't expect to see that much of interest in the subject, but thanks to everyone who participated in promoting this idea on twitter and the blogosphere.

In the second part of this series we will spice things up a bit and explore through the following video many aspects of the idea we've talked about. Following to that an important screenshots and some considerations you should be aware of before and after implementing this in your lab.

What you'll see in this video:
1- Deploy a thin-provisioned vESX(i) VM from a template.
2- Check the required configuration parameter on the vESX to run nested VMs.
3- Customizing the new vESX server (assign password, set a static IP, put the DNS config ..etc)
4- Add the vESX to an existing HA Cluster.
5- Test the vMotion within the same cluster and across deferent clusters in the datacenter.
6- Add the required configuration parameter on the nested VM for enabling the FT.
7- Enable the FT and test the failover across deferent vESX servers.

A Quick note on the hardware used:
Technically speaking, only one server can be used in this whole setup. What is deferent in my setup (as you saw in the diagram) is that I use an external iSCSI array (the CLARiiON AX4) for hosting the vESX VMs. I just needed the flexibility to have them on an external storage to share them later on with other servers, but it is not a requirement. You can simply use the internal storage of the pESX server to host your vESX and you will still have everything you see in the video. As far as the shared storage for the vESX servers is concerned, you can use the Celerra VSA. In my case I use the Celerra only for the SRM labs to do the replication trick. Other than that, I use the OpenFiler as my shared storage for the nested VMs.

Two things to be set on the pESX hosts:
1 – Increase the number of ports on your pESX vSwitch to accommodate the increased number of connections required by your vESX VMs.

2 – Enable the "Promiscuous Mode" on the vSpwitch

The configuration parameters on the Virtual Machines are as follows:
1 – The virtual ESX (vESX) host: monitor_control.restrict_backdoor = TRUE

2 – The nested VM: replay.allowBTOnly = TRUE

The iSCSI vSwitch on the pESX host bound to vmnic1 (phisical NIC 2) and connected to the EMC CLARiiON AX4 iSCSI array

The vMotion internal vSwitch on the pESX host

The Fault Tolerance internal vSwitch on the pESX host

The thin-provisioned ESX(i) size on disk (475MB) + the memory swap:

The thin-provisioned ESX(i) size on disk(3.5GB) + the memory swap:

Other considerations and GOTCHAs:
1 – When enabling the FT, make sure you have your VMs powered off, even if they have eager zeroed disks, I used to get some errors when the VMs were powered on while enabling FT.

2 – Sometimes the network card order and numbering could be confusing. For example, in the vESX VM, you will have the NICs order starting from 1 to 10, but in the actual vESX network configuration tab, you will find the NICs starting from 0 and counting towards 9. This could be confusing when mapping your vnics to the pESX host vSwitches, like the VMotion internal switch, the FT internal switch and so forth. Just make sure you count the nics order accurately.

3 – Deploying vESX from templates may be cool and fast, but it could a bit challenging sometime in troubleshooting the network related issues. The reason behind that is the fact that all the network cards will have the same MAC address, or worst, the Port Groups like (VMotion) could have the same MAC address even if you completely remove the vnics and created brand new ones. The work around for that is to create the vESX template, and then remove all the NICs form it. When you deploy a new vESX, you can just add the new nics as you like, and by that you'll have a new MAC addresses. Beside that, you may need to add a new VMkernel network for the VMotion, and then remove the old one. Of course you may be thinking that deploying a brand new vESX would be easier, you are right, but with the scripting everything could be automated. I will try to write a PS script to automate this network changes/settings and post it here later on.

If you are a busy techie like myself, and do not have the time to scratch your head, you can have a quick glance on the following diagram and wait for the next blog post. But if you read on, I promise that this article will inspire you in a way or another.

First off, I take no credit for the basic technical information of running an ESX inside itself. This information has been floating around on the V12n blogosphere and VMTN forums for quite some time now, but no one, as far as I know, has gathered them in one place to put together the missing pieces, and most importantly, illustrated why you should consider doing this at the first place. My main objective from this article(s) is to open your eyes for some cool ideas that you can try in your labs instead of using the traditional physical ESX hosts, or even ESX in VMware WorkStation.

A quick background:
The idea of running virtual ESX servers on top of a physical ESX host has been consuming a lot of my time and research lately. Unfortunately I was doing all these tests and trials on ESX 3.5, and later on I came to know that a special build needed to be used in order to do that. The time passed, and the vSphere came out with all its mind-blowing features and capabilities, but a few of us paid a close attention to the fact that ESX 4.0 came packed and ready to be virtualized inside itself. Eric Gray from VMware broke the news with his great article on VCritical.com, and a week later another cool guy named Itzik Reich, on Chad Sakac's blog, described how FT can be enabled on a virtual ESX (vESX). Great Stuff!

Think out of the box!
Every single time I ask someone about virtualizing ESX inside itself, I always get the same exact question: "Why would you want to do something crazy like that?!" I even remember during my VMware VI3 class I asked the instructor about it, and his answer was: "it's so unsupported", who's talking about support now, and why should I bother if I will never, ever, run something like that in production?! That being said, let's think out of the box for a minute.

Ask yourself the following:

• How long it takes to deploy a classic ESX on a physical host using the traditional way?
• Do I have enough server recourses for building a VMware lab?
• Are my servers up-to-date with the latest CPU technologies in order to test things like FT?
• Do I have enough network resources (switch ports, cables ..etc) to allocate 10 network connections for each ESX servers? Is there any blade server that can have 10 Ethernet ports at the first place?
• Do I have the required storage for building, let's say, 10 classic ESX servers to boot them from the SAN?
• Am I willing to do an aggressive power off for my physical ESX servers just to test the HA mechanism, and its advanced options, to simulate a real life server failure?

If you found the above a bit challenging, you are in the right place. If you don't, you are still in the right place. I told you, think out of the box.

I consider myself lucky enough to have all the resources available for me in my current work environment. When I ask for hardware resources or software licenses I simply get them. Right now I have more than 10 dedicated blades in my lab running deferent ESX flavors and versions. In fact, in our new hardware refresh I will have a brand new set of high specs blades and storage dedicated specifically for the VMware R&D.

What is the deal then?

I need to build a complete private cloud! I need to have everything starting from ESX clusters to dvSwitches, Cisco NX1Vs, virtual appliances, VMsafe based solutions, and last but not least, a working SRM installation between two virtual DCs. I need to have the complete feel of this so-called "private cloud" before I even start an actual PoC, which will be way complicated & a bit challenging in the physical world. I need to go to the management with diagrams and videos and tell them why we need to be 100% virtualized, and why we should start planning for that. Show them where we are headed, and how our IT environment and datacenters will look like one year from now.

Put your ideas back in "the box"
Enough dreaming, the reality is actually one step away. Believe it or not, all what you need is a single server with a reasonable memory and that's all. But I won't go into the technical details today, let's see what ideas we may think of now:

1. Thin-provision your ESX hosts! Yes, all my fat classic ESX hosts are now thin-provisioned, this means I can deploy a 40GB HDD for the ESX, but actually consume less than 3.5GB only!
2. 1 Minute fast deployment! I can now deploy new ESX servers in a matter of seconds from pre-built templates (with special considerations we'll talk about later).
3. No network limitations! A 10 NICs on an ESX server is not something you can test every day.
4. SRM in a Box! Now with the coming SRM 2.0 and the ESX 4.0 memory restrictions, it's going to be extremely hard to maintain the bare minimum of two ESX hosts, nested VM and two VSAs replication on a laptop with 4GB.
5. I'm online 24/7, and still mobile! Yes, I'm no longer running my virtual ESX servers on a laptop, I have them online and under my mouse clicks whenever required. VPN into your Lab or even use your BlackBerry/vCMA on the GO! No more shutdowns, memory restrictions or space issues.
6. Migrate your labs or even share them! One of the very painful situations I went through lately was shifting my physical ESX lab from one of our datacenters to another. If I had my vESX hosts at that time all what I needed to do is to take them on an external USB disk, or even copy them over the WAN and I'm done.
   
7. VMware (YES, THE COMPANY): I still see a great potential of implementing the idea that I proposed sometime back (the on-demand labs). You see here that with this simple setup we can test almost everything in vSphere like HA, DRS, VMotion and even FT & SRM! I really hope that VMware is already working on something like that or at least someone will take the initiative to put this idea in action someday.
   

What's next?
So let's say we've built this "virtual private cloud", will that be it? The answer is No. It is actually the beginning in my opinion. There are so many other cool ideas that I can't get off my head. For example, I can build another/deferent private cloud and start exploring this whole "cloud computing" thing. I can test and develop new approaches for VMotion'ing workloads across two deferent sites. Remember, you have a huge flexibility and control over your "clouds", and most importantly, you are not afraid to screw up anything since you can rebuild your elements quite easily, and almost instantly (snapshots/backups/templates and so forth)

In the next blog post I will get more technical and list the details of my current setup which I illustrated in the diagram above. In the addition to that, I will explore the planned developments for building the complete "VPC". May be also we can start thinking of automating and managing individual labs using the new and long awaited Lab Manager 4.0, expected sometime this month.

Teaser: A vCenter screenshot showing the 10 NICs and their configuration on one of the vESX servers.

Note: If you are as much excited about this idea as I am, and can't wait for my next post (maybe I will close the blog and change my career, who knows?) you can visit these links to get you up to speed with the setup:

• Eric Gray's post that unleashed the beast of the vESX.
• Itzik Reich's reply in Chad's blog, containing the FT configuration parameter on the nested VM.
• Maish Saidel-Keesing has two excellent posts Part 1 & 2 for a similar lab setup.

 
Mike DiPetrillo's response (in case you missed it in the replies below)

Actually, VMware has been doing a lot of this. Why do you think all of the stuff was put into the builds to run nested ESX in the first place? VMware SEs have been running nested VMs for a long time on their laptops using Workstation or VMware Fusion.

VMware also has an internal environment called vSEL that runs nearly all of the products in a nested environment. vSEL (the virtual SE Lab) let’s our tech resources in the field deploy and learn our applications as well as do demos and training with customers and partners. This “cloud” services over 1,200 tech people today inside of VMware. This is being expanded to let our development teams use it for development.

VMware is now working with several service providers around the world to let people request and run ESX environments virtually in the cloud – either for doing trials of the software or for doing actual deployments.

Lastly, there’s a new cloud service being built that will run nested VMware products to service all of the hands on labs for VMworld. Make sure to make it out to VMworld to get some hands on experience with VMware’s implementation of this environment.

Definitely a good picture and thanks for getting the word out. Just wanted to let you know that VMware has been doing this for years and has several different clouds built with this already.