• This is the first diagram of more ESX blueprints to come. Previously I was focused more onto the vSphere architectures as a whole, but there will be more granular diagrams on ESX itself as a hypervisor very soon.
• The colors play a major role in this diagram. I thought I should stick with the original colors used in the vCenter Server Recourse monitoring box (on the top right of the diagram) instead of reinventing the wheel. As an example, if you want to scan through the swapping memory reclamation technique, all what you need is to focus your sight on the red colors. Whether it’s in the VMs, hypervisor, esxtop or vCenter, the red color will always relate to the swapping activity, and so forth.
• In the MMU virtualization I did not put much detail (as I was hoping to) because of the diagram limited space, but I’m planning to make more detailed diagrams about this topic in the future, along with other technologies like NUMA and EPT/RVI for example.
• In the memory reclamation techniques, you might have noticed that the “idle memory tax” is not there. I intend to illustrate that in a deferent topic related to shares and resource management.
• The Memory compression and swapping to SSD are future features coming soon in the vSphere generation. This information was mentioned in the VMworld 2009 session TA2627 so it’s public.
• The most interesting part for me while creating this diagram was the ESXTOP. I have a wild idea of creating a crazy huge diagram (poster-like) and add the deferent screens and options for ESXTOP along with real-world numbers and descriptions to show the beauty of this incredible tool. I might also use Duncan’s esxtop section as a reference for the thresholds, I urge you to go there and share your experience!
• For some reason, the PDF converter kept converting the ESXTOP text box into an image and then down-sample it, which ended up not looking as sharp and clean as in the original Visio diagram. I’m trying to figure out why the Acrobat is giving me this grief, so I will probably update the PDF later on (check the version numbers).
• I’ve received a number of emails asking for some descriptions accompanying my diagrams (like the admission control calculations in the HA diagram), and I believe you are right about that, especially for the newbies entering the fascinating world of VMware. I’ll revisit these diagrams in the future with detailed descriptions and write-ups, so stay tuned.

I hope you’ll like this diagram and see the real beauty of the ESX memory management (well a glimpse of it at least). If you have any corrections or suggestions please drop a comment or email and let me know your thoughts.

Recourses:
- Book: Operating System Concepts (Part Three – Memory Management)
- Book: Modern Operating Systems 3rd Edition (Chapter 3 – Memory Management)
- Documentation: vSphere Resource Management Guide
- WP: Understanding Memory Resource Management in VMwareо ESX™ Server
- WP: Memory Resource Management in VMware ESX Server – by Carl A. Waldspurger
- VMworld2009 Session: TA2963 “esxtop for advanced users”
- VMworld2009 Session: TA2627 “Understanding “Host” and “Guest” Memory Usage and Related Memory Management Concepts”
- Blog: Arnim van Lieshout (Part-1 , Part-2, Part-3)

In the second part of this series we will spice things up a bit and explore through the following video many aspects of the idea we’ve talked about. Following to that an important screenshots and some considerations you should be aware of before and after implementing this in your lab.

What you’ll see in this video:
1- Deploy a thin-provisioned vESX(i) VM from a template.
2- Check the required configuration parameter on the vESX to run nested VMs.
3- Customizing the new vESX server (assign password, set a static IP, put the DNS config ..etc)
4- Add the vESX to an existing HA Cluster.
5- Test the vMotion within the same cluster and across deferent clusters in the datacenter.
6- Add the required configuration parameter on the nested VM for enabling the FT.
7- Enable the FT and test the failover across deferent vESX servers.

A Quick note on the hardware used:
Technically speaking, only one server can be used in this whole setup. What is deferent in my setup (as you saw in the diagram) is that I use an external iSCSI array (the CLARiiON AX4) for hosting the vESX VMs. I just needed the flexibility to have them on an external storage to share them later on with other servers, but it is not a requirement. You can simply use the internal storage of the pESX server to host your vESX and you will still have everything you see in the video. As far as the shared storage for the vESX servers is concerned, you can use the Celerra VSA. In my case I use the Celerra only for the SRM labs to do the replication trick. Other than that, I use the OpenFiler as my shared storage for the nested VMs.

Two things to be set on the pESX hosts:
1 – Increase the number of ports on your pESX vSwitch to accommodate the increased number of connections required by your vESX VMs.

2 – Enable the “Promiscuous Mode” on the vSpwitch

The configuration parameters on the Virtual Machines are as follows:
1 – The virtual ESX (vESX) host: monitor_control.restrict_backdoor = TRUE

2 – The nested VM: replay.allowBTOnly = TRUE

The iSCSI vSwitch on the pESX host bound to vmnic1 (phisical NIC 2) and connected to the EMC CLARiiON AX4 iSCSI array

The vMotion internal vSwitch on the pESX host

The Fault Tolerance internal vSwitch on the pESX host

The thin-provisioned ESX(i) size on disk (475MB) + the memory swap:

The thin-provisioned ESX(i) size on disk(3.5GB) + the memory swap:

Other considerations and GOTCHAs:
1 – When enabling the FT, make sure you have your VMs powered off, even if they have eager zeroed disks, I used to get some errors when the VMs were powered on while enabling FT.

2 – Sometimes the network card order and numbering could be confusing. For example, in the vESX VM, you will have the NICs order starting from 1 to 10, but in the actual vESX network configuration tab, you will find the NICs starting from 0 and counting towards 9. This could be confusing when mapping your vnics to the pESX host vSwitches, like the VMotion internal switch, the FT internal switch and so forth. Just make sure you count the nics order accurately.

3 – Deploying vESX from templates may be cool and fast, but it could a bit challenging sometime in troubleshooting the network related issues. The reason behind that is the fact that all the network cards will have the same MAC address, or worst, the Port Groups like (VMotion) could have the same MAC address even if you completely remove the vnics and created brand new ones. The work around for that is to create the vESX template, and then remove all the NICs form it. When you deploy a new vESX, you can just add the new nics as you like, and by that you’ll have a new MAC addresses. Beside that, you may need to add a new VMkernel network for the VMotion, and then remove the old one. Of course you may be thinking that deploying a brand new vESX would be easier, you are right, but with the scripting everything could be automated. I will try to write a PS script to automate this network changes/settings and post it here later on.

First off, I take no credit for the basic technical information of running an ESX inside itself. This information has been floating around on the V12n blogosphere and VMTN forums for quite some time now, but no one, as far as I know, has gathered them in one place to put together the missing pieces, and most importantly, illustrated why you should consider doing this at the first place. My main objective from this article(s) is to open your eyes for some cool ideas that you can try in your labs instead of using the traditional physical ESX hosts, or even ESX in VMware WorkStation.

A quick background:
The idea of running virtual ESX servers on top of a physical ESX host has been consuming a lot of my time and research lately. Unfortunately I was doing all these tests and trials on ESX 3.5, and later on I came to know that a special build needed to be used in order to do that. The time passed, and the vSphere came out with all its mind-blowing features and capabilities, but a few of us paid a close attention to the fact that ESX 4.0 came packed and ready to be virtualized inside itself. Eric Gray from VMware broke the news with his great article on VCritical.com, and a week later another cool guy named Itzik Reich, on Chad Sakac’s blog, described how FT can be enabled on a virtual ESX (vESX). Great Stuff!

Think out of the box!
Every single time I ask someone about virtualizing ESX inside itself, I always get the same exact question: “Why would you want to do something crazy like that?!” I even remember during my VMware VI3 class I asked the instructor about it, and his answer was: “it’s so unsupported”, who’s talking about support now, and why should I bother if I will never, ever, run something like that in production?! That being said, let’s think out of the box for a minute.

Ask yourself the following:

• How long it takes to deploy a classic ESX on a physical host using the traditional way?
• Do I have enough server recourses for building a VMware lab?
• Are my servers up-to-date with the latest CPU technologies in order to test things like FT?
• Do I have enough network resources (switch ports, cables ..etc) to allocate 10 network connections for each ESX servers? Is there any blade server that can have 10 Ethernet ports at the first place?
• Do I have the required storage for building, let’s say, 10 classic ESX servers to boot them from the SAN?
• Am I willing to do an aggressive power off for my physical ESX servers just to test the HA mechanism, and its advanced options, to simulate a real life server failure?

If you found the above a bit challenging, you are in the right place. If you don’t, you are still in the right place. I told you, think out of the box.

I consider myself lucky enough to have all the resources available for me in my current work environment. When I ask for hardware resources or software licenses I simply get them. Right now I have more than 10 dedicated blades in my lab running deferent ESX flavors and versions. In fact, in our new hardware refresh I will have a brand new set of high specs blades and storage dedicated specifically for the VMware R&D.

What is the deal then?

I need to build a complete private cloud! I need to have everything starting from ESX clusters to dvSwitches, Cisco NX1Vs, virtual appliances, VMsafe based solutions, and last but not least, a working SRM installation between two virtual DCs. I need to have the complete feel of this so-called “private cloud” before I even start an actual PoC, which will be way complicated & a bit challenging in the physical world. I need to go to the management with diagrams and videos and tell them why we need to be 100% virtualized, and why we should start planning for that. Show them where we are headed, and how our IT environment and datacenters will look like one year from now.

Put your ideas back in “the box”
Enough dreaming, the reality is actually one step away. Believe it or not, all what you need is a single server with a reasonable memory and that’s all. But I won’t go into the technical details today, let’s see what ideas we may think of now:

1. Thin-provision your ESX hosts! Yes, all my fat classic ESX hosts are now thin-provisioned, this means I can deploy a 40GB HDD for the ESX, but actually consume less than 3.5GB only!
2. 1 Minute fast deployment! I can now deploy new ESX servers in a matter of seconds from pre-built templates (with special considerations we’ll talk about later).
3. No network limitations! A 10 NICs on an ESX server is not something you can test every day.
4. SRM in a Box! Now with the coming SRM 2.0 and the ESX 4.0 memory restrictions, it’s going to be extremely hard to maintain the bare minimum of two ESX hosts, nested VM and two VSAs replication on a laptop with 4GB.
5. I’m online 24/7, and still mobile! Yes, I’m no longer running my virtual ESX servers on a laptop, I have them online and under my mouse clicks whenever required. VPN into your Lab or even use your BlackBerry/vCMA on the GO! No more shutdowns, memory restrictions or space issues.
6. Migrate your labs or even share them! One of the very painful situations I went through lately was shifting my physical ESX lab from one of our datacenters to another. If I had my vESX hosts at that time all what I needed to do is to take them on an external USB disk, or even copy them over the WAN and I’m done.
   
7. VMware (YES, THE COMPANY): I still see a great potential of implementing the idea that I proposed sometime back (the on-demand labs). You see here that with this simple setup we can test almost everything in vSphere like HA, DRS, VMotion and even FT & SRM! I really hope that VMware is already working on something like that or at least someone will take the initiative to put this idea in action someday.
   

What’s next?
So let’s say we’ve built this “virtual private cloud”, will that be it? The answer is No. It is actually the beginning in my opinion. There are so many other cool ideas that I can’t get off my head. For example, I can build another/deferent private cloud and start exploring this whole “cloud computing” thing. I can test and develop new approaches for VMotion’ing workloads across two deferent sites. Remember, you have a huge flexibility and control over your “clouds”, and most importantly, you are not afraid to screw up anything since you can rebuild your elements quite easily, and almost instantly (snapshots/backups/templates and so forth)

In the next blog post I will get more technical and list the details of my current setup which I illustrated in the diagram above. In the addition to that, I will explore the planned developments for building the complete “VPC”. May be also we can start thinking of automating and managing individual labs using the new and long awaited Lab Manager 4.0, expected sometime this month.

Teaser: A vCenter screenshot showing the 10 NICs and their configuration on one of the vESX servers.

Note: If you are as much excited about this idea as I am, and can’t wait for my next post (maybe I will close the blog and change my career, who knows?) you can visit these links to get you up to speed with the setup:

• Eric Gray’s post that unleashed the beast of the vESX.
• Itzik Reich’s reply in Chad’s blog, containing the FT configuration parameter on the nested VM.
• Maish Saidel-Keesing has two excellent posts Part 1 & 2 for a similar lab setup.

 
Mike DiPetrillo‘s response (in case you missed it in the replies below)

Actually, VMware has been doing a lot of this. Why do you think all of the stuff was put into the builds to run nested ESX in the first place? VMware SEs have been running nested VMs for a long time on their laptops using Workstation or VMware Fusion.

VMware also has an internal environment called vSEL that runs nearly all of the products in a nested environment. vSEL (the virtual SE Lab) let’s our tech resources in the field deploy and learn our applications as well as do demos and training with customers and partners. This “cloud” services over 1,200 tech people today inside of VMware. This is being expanded to let our development teams use it for development.

VMware is now working with several service providers around the world to let people request and run ESX environments virtually in the cloud – either for doing trials of the software or for doing actual deployments.

Lastly, there’s a new cloud service being built that will run nested VMware products to service all of the hands on labs for VMworld. Make sure to make it out to VMworld to get some hands on experience with VMware’s implementation of this environment.

Definitely a good picture and thanks for getting the word out. Just wanted to let you know that VMware has been doing this for years and has several different clouds built with this already.

But today I will be talking about the (esxtop modes) which I believe not many of us know much about. With a quick search also over the Planet V12N blogosphere I couldn’t find any posts talking about this topic, so there you have it.

We all know about the traditional way of running the esxtop tool through the esx host service console (or resxtop through RCLI). This is in fact called “interactive mode” where you get to see the statistics or information live in front of you and interact with it using deferent sets of keys (m = memory, d = disk and so forth). But this is actually only one of three modes that esxtop can run into. Here are the other two:

• Replay Mode
  

In this mode, you can record and play the esxtop statistics for a specific period of time, and use also the interval of your choice. But before going into details, you may be wondering why would you want to do that? Well, it could be for support purposes, for example if VMware wants to have a snapshot of your performance statistics they may ask you to run this and send them the output. For me, this came in handy when one of my VMs was having a very strange behavior during an early morning hour. This was happening on daily basis and since I was not in favor of staying awake to see what’s the deal with that, I set a small cron job to run/record the esxtop stats for this specific period of time. The next day I played the stats and I was very grateful for the amount of information I got for troubleshooting the problem.

Now, let’s see how we can do this:

First, you will need to be “root” in order to issue the record command as follows:

vm-support -S -i 5 -d 120

We can see here that the interval is 5 seconds, and the duration for recording the statistics is 120 seconds.

The esxtop will start recording the stats and then compress the output into a .tgz file.

We can issue a command to see the .tgz file as shown below. Make sure you are in the right partition when issuing this command and that you have a reasonable disk space if you will run this command for long time.

Next, we need to uncompress the file as follows:
tar -xzf esx*.tgz

Finally, you need to issue the replay command: (thanks to @vRobM for bringing my attention to this missing command)
esxtop -R vm-support*

And there you have it, the statistics will run as if you are sitting at the point of time the stats were recorded in:

One thing to note here though, the time that will show on the top line, will always reflect the current time you are running in, not the time of the recording, so don’t be confused by that.

• Batch mode
  

In the batch mode, you can dump your esxtop stats into a .csv file for later use. You can utilize either Microsoft Excel, or Microsoft perfmon tool to view these data at any time.

Here is the command to achieve this:
esxtop -b -a > output_file.csv

This command will dump “all” the fields from the esxtop to the file. But what if you want specific columns only? Well, that’s the real beauty of the batch mode. You can always choose what information you are interested in, here is how:

• Run the esxtop command in interactive mode.
• In each of the panels, select the columns you want.
• Save this configuration to a file (by default ~/.esxtop4rc) using the W interactive command.

Now you can use the batch command and it will dump only the comlums that you have selected.

What’s next?
Use Excel or perfmon to analyze your data, I personally prefer to use my Vista’s perfmon for this unless you have a specific requirement for using Excel.

• Go to Start -> Run -> type perfmon.
• Go to Performance Monitor section.
• Click of the “View Log Data” in the tool bar at the right as shown in the screen shot below.

   

After that, you need to use the file you’ve dumped the esxtop data to:

You can choose here which columns/counters you want to graph and analyze

And here you see your selected columns graphed in perfmon

And you are done!

From the other hand I’m really impressed with the free-appliance community edition approach, which reflects a good image of the people running this whole thing!

The HyTrust’s commercial appliance comes in two flavors, a physical rack-mounted appliance ($7,500), or a virtual edition ($3,000) that can run directly inside your VI3 (and soon to support vSphere). I urge you to visit their website to have a closer look and more information. If you are too technical like myself, you may want to start with David Storm’s video review to have an over whole idea.

Speaking about videos, I’ve downloaded the community appliance to test it in my lab, and will most probably come back with some videos showing the great features and capabilities this new security appliance has to offer.

If you are a VMware guru, the first few episodes in this series are not meant for you. You would probably be interested when we reach the level of configuring things like the Cisco Nexsus1000v, create host-profiles, or do some FT magic. Between now and then, it’s all for the newbies coming to our wonderful virtual world, and have no clue where to start and what platform they should adopt.

I’ve been willing to do a similar series in VI3 for quite some time, and start from the basics all the way to advanced configurations, but now that the vS4 is out, well, almost, I said why not, it’s a learning journey for me before anyone else.

You have to be patient with me however, the video recording is fairly easy, but the editing and production is quite hard and time consuming to present in this format. If you have any feedback, corrections or anything that can help improve these videos, please don’t hesitate to drop a comment or an email.

What are we waiting for? Let’s do this:

I highly recommend to you watching these quick demos from intel before reading the eWEEK’s article to have a clear idea about what is written over there:

• Nehalem architecture overview:
  http://www.intel.com/technology/architecture-silicon/next-gen/demo/demo.htm
• Intel QuickPath architecture:
  http://www.intel.com/technology/quickpath/demo/demo.htmiid=tech_nehalemdemo+infolnk_quickpath 

IBM will be officially announcing their next generation of intel-based blades (HS22) tomorrow, March the 31st , which will be equipped with this mind-blowing performance microprocessor, I was fortunate enough to see and touch this blades myself last week which comes with a few other surprises as well.

Video Links: BlipTv | Vimeo | YouTube

 VMware technical resource: http://www.vmware.com/resources/techresources/1014