1

Interactive, Replay & Batch ESXTOP Modes! 16 June 2009 at 10:00 pm by Hany Michael

I’ve been rediscovering the great esxtop tool lately, and I have to say that I’m really impressed with the tremendous amount of information you can get out of this little thing!

But today I will be talking about the (esxtop modes) which I believe not many of us know much about. With a quick search also over the Planet V12N blogosphere I couldn’t find any posts talking about this topic, so there you have it.

We all know about the traditional way of running the esxtop tool through the esx host service console (or resxtop through RCLI). This is in fact called “interactive mode” where you get to see the statistics or information live in front of you and interact with it using deferent sets of keys (m = memory, d = disk and so forth). But this is actually only one of three modes that esxtop can run into. Here are the other two:

  • Replay Mode

In this mode, you can record and play the esxtop statistics for a specific period of time, and use also the interval of your choice. But before going into details, you may be wondering why would you want to do that? Well, it could be for support purposes, for example if VMware wants to have a snapshot of your performance statistics they may ask you to run this and send them the output. For me, this came in handy when one of my VMs was having a very strange behavior during an early morning hour. This was happening on daily basis and since I was not in favor of staying awake to see what’s the deal with that, I set a small cron job to run/record the esxtop stats for this specific period of time. The next day I played the stats and I was very grateful for the amount of information I got for troubleshooting the problem.

Now, let’s see how we can do this:

First, you will need to be “root” in order to issue the record command as follows:

vm-support -S -i 5 -d 120

We can see here that the interval is 5 seconds, and the duration for recording the statistics is 120 seconds.

The esxtop will start recording the stats and then compress the output into a .tgz file.

We can issue a command to see the .tgz file as shown below. Make sure you are in the right partition when issuing this command and that you have a reasonable disk space if you will run this command for long time.

Next, we need to uncompress the file as follows:
tar -xzf esx*.tgz

And there you have it, the statistics will run as if you are sitting at the point of time the stats were recorded in:

One thing to note here though, the time that will show on the top line, will always reflect the current time you are running in, not the time of the recording, so don’t be confused by that.

 

  • Batch mode

In the batch mode, you can dump your esxtop stats into a .csv file for later use. You can utilize either Microsoft Excel, or Microsoft perfmon tool to view these data at any time.

Here is the command to achieve this:
esxtop -b -a > output_file.csv

This command will dump “all” the fields from the esxtop to the file. But what if you want specific columns only? Well, that’s the real beauty of the batch mode. You can always choose what information you are interested in, here is how:

  • Run the esxtop command in interactive mode.
  • In each of the panels, select the columns you want.
  • Save this configuration to a file (by default ~/.esxtop4rc) using the W interactive command.

Now you can use the batch command and it will dump only the comlums that you have selected.

What’s next?
Use Excel or perfmon to analyze your data, I personally prefer to use my Vista’s perfmon for this unless you have a specific requirement for using Excel.

  • Go to Start -> Run -> type perfmon.
  • Go to Performance Monitor section.
  • Click of the “View Log Data” in the tool bar at the right as shown in the screen shot below.

     

 

After that, you need to use the file you’ve dumped the esxtop data to:

You can choose here which columns/counters you want to graph and analyze

And here you see your selected columns graphed in perfmon

And you are done!

+ HyTrust Appliance Community Edition can assess the configuration of your entire ESX environment! By Hany Michael 30 May 2009 at 5:06 pm and have 1 Comment

Today I received an email from the HyTrust Appliance (HTA) team giving me a great tip about the Community Edition that I blogged about earlier this month. This information will be actually announced officially next Monday, so you get to know about it earlier.

As most of you know, the HTA Community Edition protects up to 3 ESX hosts. However, the configuration assessment capability was left intentionally unrestricted. This means you have the ability to assess the configuration of your entire ESX environment for as many hosts as you want!

Here’s how to configure Community Edition for assessment of your ESX hosts:

  1. Log-in to the HTA UI with your web browser, or navigate to the “HyTrust” tab in the Virtual Infrastructure Client (if you have the vCenter Plugin installed.)
  2. Select “Hosts” from the “Hosts” menu.
  3. Press the “Add” button to add a host.
  4. Enter the credentials and details of the ESX host you want to add. Make sure that the “Protected” option is unselected. If you forget, don’t worry! The CE license of the HTA will refuse to add more than 3 protected hosts.
  5. Once you’re done adding hosts to the HTA, you can select the check boxes next to the hosts you want to audit, and press “Assess”.
  6. You’re done!

Thanks to the HTA team for allowing me to blog about this great trick before the official announcement.

+ SCREW YOU Brian Madden…you are not the hero in this story. By Hany Michael 29 May 2009 at 3:11 am and have 5 Comments

Apologies to my dear readers for the language, I had to use Brian Madden’s “own words” when he said that to his own audience in his VDI vs. TS VMworld2008 session. Read on:

Today I got a marketing email telling me about the “new 33 announced tech sessions” of the BriForum2009, and a promotion code to go and register there. I ignored it since I’m not interested, and kept doing whatever I was doing at that time.

Few minutes later my twitter was flooded with all this “VMware banning competing vendors bla bla bla” thing which was originally posted on this link. Again I ignored it because I know the guy is nuts and he always like to appear as if he’s the hero who’s saving the virtualization industry all the time. I said I’ll just wait for the real VMware story.

And sure enough, a small VMware response appeared here, and then some tweets came in with Brian’s new update on his blog. I thought it might be fun to see how he’s going to respond to that, and of course there were lots of “I’m the hero” and “I rock” between the lines, which was fine by me until I reached this last paragraph:

“Now I only have one small complaint about their “official” response: They say that text is industry standard for conferences. I think they mean for vendor conferences, because we have no such text at BriForum. :)”

Well, screw you mister! This is a cheap way to promote your upcoming event which your fellows at techtarget have been marketing about since the beginning of the day, and apparently needed some more attention to the ads you have at the end of your post. Everybody knows that you are a Citrix guy and that’s what you are doing for living, so don’t you dare pretend to be “vendor neutral” now.

It’s not YOU who will come and tell the community that VMware is avoiding the competition in VMworld, and it’s not US who put the pressure on VMware to do otherwise. There are so many examples in the last VMworld events showing that this is not the case, and that they are fair. Go check who won for example the “Best of Show” in VMworld2008, it’s (Reflex) for a product that VMware were working so hard at that time to come up with an equivalent to, which is known now as “vShield Zones“.

Now, on the extremely opposite side, do you remember the Microsoft childish and unprofessional acts at that time with their “Casino chips“? Do you really think VMware will just invite them back and tell them (oh please be my guest, but bring some more this time). They have every right to prevent this kind of things to happen again.

As a matter of fact, I think that VMware is doing a huge mistake inviting both Microsoft and Citrix to VMworld. If I wanted to attend VMworld or even purchase the sessions & Labs for later access, I only do that because I’m interested in VMware solutions and products. Microsoft has its own TechEd, and Citrix has the iForum and Synergy, I would rather go to the TechEd (or again buy the sessions CDs) to have in-depth information from real engineers about Hyper-V rather than watching a bunch of a so-called “myth-busters” coming on the stage talking awkwardly about their product and how it’s better than VMware in its own event.

Seriously, what is that? Is this some kind of a joke or what, are we being looked at as if we are stupid and should accept someone coming and telling us this crap!

To all who is reading these lines, I encourage you to go and register for the BriForum below, maybe he’ll make some money out of that and save us this cheap ways of marketing in the future.

+ Keep your Service Console clean! By Hany Michael 24 May 2009 at 12:11 pm and have No Comments

I just ran into a problem now while I was patching one of my ESX 3.5 servers. I got this message from vCenter:

Insufficient disk space on datastore ’service console’

I SSH’ed into my ESX host and ran the df –ah command which came back with the following:

Apparently I had some big files sitting somewhere on the root partition, and I remembered at once that I was doing some throughput testing from my ESX to the FC SAN and iSCSI targets, so I knew where to go:

And here they are, two iso files consuming 3GB of my precious service console datastore:

Alternatively, you can run the find command with the following syntax to return the file size of your choice, sort them in human readable format and then write them in a text file (here it’s larger than 300MB):

find / -type f -size +300000k -exec ls -lh {} \; 2> /dev/null | awk ‘{ print $NF “: ” $5 }’ | sort -nrk 2,2 > HyperViZor-FileSize.txt

ALWAYS Keep your Service Console FS clean!

+ vSphere 4.0 GA release! By Hany Michael 21 May 2009 at 10:13 am and have No Comments

Al Pacino - (Scent of a Woman): “It’s a wonderful day for singing a song…it’s a wonderful day for mooooooving a long…HOOOWAAA”

Downloads:
Download Page: https://www.vmware.com/download/
Free 60 days Trial: https://www.vmware.com/tryvmware/index.php?p=vsphere&lp=1

Documentations:
http://www.vmware.com/support/pubs/vs_pubs.html

Third-Party Downloads:
Cisco Nexsus 1000V Trial: http://www.cisco.com/go/1000vdownload

Video Tutorials from VMware:
1 - ESX Installation and Configuration
2 - ESXi Installation and Configuration
3 - VMware vCenter Server
4 - VMware vSphere Client
5 - Networking configuration
6 - Storage configuration (iSCSI)
7 - Create and manage virtual machines
8 - VMware Host Profiles
9 - VMware Storage VMotion
10 - VMware vCenter Server Linked Mode
11 - VMware vNetwork Distributed Switch (vDS)

To be updated with the latest and greatest links…Stay Tuned!

+ Finally! Running ESX as a VM inside a physical ESX host! By Hany Michael 19 May 2009 at 9:25 pm and have No Comments

When you find a great blog post that you know from the first glance that it’s going to be big, you really find yourself dragged to refer to it even if you won’t add much.

Eric Gray from VMware, has finally done it! He showed us on his great blog, VCritical.com, how we can run ESX 4.0 as a virtual machine inside a physical ESX host. That’s right, not a VMware Workstation, but a real physical ESX server.

Any regular reader of my blog or a follower on Twitter knows that I’ve been trying to do that since I had my hands on ESX 4.0 starting from the early beta to the RC, but I had no luck doing that. As per Eric, it seems that the new GA build allows you to do this with a little bit configuration tweaks. Eric also mentioned that internally VMware has been running ESX as a VM inside ESX servers with a special build to achieve that.

Which leads me to the question I actually get from people when they know what I want to do: Why would someone need to do something crazy like this?

There are many reasons actually, but here is my personal favorite:
- I want to be able to create a fairly big and complicated ESX labs, and in the same time, I have a fairly huge CPU and memory resources on my existing physical ESX clusters.
- Running ESX on VMware workstation is great, but you are always limited to the amount of memory you’ve got on your PC/Laptop. 4GB or even 8GB won’t really allow you to do much.
- In the Workstation scenario you can’t keep your VMs running all the time, you will always have to shut them down to free up your resources.
- Having your ESX servers as virtual machines gives you extreme flexibility, starting from creating as much NICs/Networks as you want, all the way to creating your own libraries for deferent configurations. Imagine now doing that with VMware Lab Manager and the flexibility you can get for building/sharing these libraries with your peers.

Of course everything has its limitation, and one of the things that I can think of right away, is the fact that you won’t run some features like FT within a virtualized ESX server. I haven’t tried it obviously, but form the restricted CPU requirements for running FT, I don’t think it would work in this scenario.

However, I am as much excited about the GA release as many out there now, where I’ll finally be able to virtualize ESX 4.0. It’s worth mentioning also that, as far as I know, doing the same for other hypervisors is not possible, like Hyper-V or XenServer. So, thanks to Eric for bringing this to our attention, and to VMware for their great flexibility and open mind to allow people to do something great like this.

+ I’ll rob a bank next week to buy these new & must-have virtualization books By Hany Michael 16 May 2009 at 12:36 pm and have 1 Comment

Sorry for the title, although it’s kind of realistic to keep yourself up2date with the latest in the ”V” Technology.

So, if you’re still conservative about Twitter and think that it’s the next evil force on the humanity, or haven’t got the time yet to subscribe to the 123,456 online virtualization blogs (and counting), you may have missed one or two of the announcements lately about these great books. I’m putting them all in one shot, with additional links for getting more information about them:

VMware vSphere and Virtual Infrastructure Security: Securing the Virtual Environment

This book is talking about one of the most important and least talked about topics in the virtualization industry: “Security!”. The great thing about the book also is that it is going to cover the VI3 as well as the vSphere, so you won’t be missing anything from both platforms.

Author: Edward L. Haletky
Amazon: Click here

 

Managing VMware Infrastructure with Windows PowerShell

PowerShell is becoming the universal scripting language now, and gaining more adoption from all kind of vendors and applications. This book is covering the VI3 management with PS.

Author: Hal Rottenberg
Amazon: Click Here
E-Book: Click Here

 

Mastering VMware vSphere 4

The much-awaited vSphere 4 platform from VMware has a detailed book from Scott Lowe available now for pre-order. Scott has a great blog online here , and also Chad Sakac is writing the Storage chapter in this book, which is another reason for me to must-have this book.

Author: Scott Lowe
Amazon: Click Here

 

Information Storage and Management: Storing, Managing, and Protecting Digital Information

Storage is a vital and indispensible part in any virtual environment. This new book from EMC explores deferent storage areas, starting from basic SAN technologies all the way to advanced topics like replication and storage virtualization.

Author: EMC
Amazon: Click Here

 

Virtualization for Security: Including Sandboxing, Disaster Recovery, High Availability, Forensic Analysis, and Honeypotting

 Another very interesting book about security and virtualization, covering all sorts of subjects like: Sandboxing, Honeypotting, Maleware analysis, Application testing, and last but not least Forensic analysis. It’s a must have book for anyone interested in security and how the virtualization can be leveraged in that.

Authors:John Hoopes
Amazon: Click Here

+ VMware’s on-demand and online Labs/PoCs (You wish!) By Hany Michael 11 May 2009 at 8:23 pm and have 2 Comments

[Inspired by a Stevie Chambers' tweet , he really has quite unique and interesting daily tweets, I wouldn't be surprised if he wins the next Golden Globe on twitter this year, if there is anything like that]

So, imagine VMware, as the virtualization leader in the market, implemented this cool idea! An on-demand virtual environment for us, the customers/prospects, to create our own labs, demos, evaluations or even proof-of-concepts! It’s not that challenging really form a technical point of view! Microsoft is already doing that for most of its applications (Hyper-V not included, YET!). As an IT pro, I can tell you that Microsoft TechNet virtual labs have helped me and consequently my employer BIG TIME for testing or evaluating specific MS products/features. It is really the simplicity that makes it incredibly useful. We are talking about a few mouse clicks here and you have a complete environment instantly to test what you need.

But I’m not here to talk about us, the customers, only. I look at it from a VMware perspective as well. This could be a huge step for VMware, mainly to win more customers who are hesitant to adopt the VI3/vS4 environments, or specific applications (like SRM, LM, View ..etc). Let’s take an example here: whenever I talk with anyone who hasn’t worked with VMware or knows much about it, they always tell me that they can’t use the product because they don’t know much about Linux! And then I start spending fairly good time explaining that Linux has nothing to do with us for a day-to-day administration, even for scripting now everything is almost possible through PowerShell. I can’t really explain to a customer who has no experience whatsoever in Linux, about things like the VMkernel and the striped RHEL COS on top of it, or even the ESX(i) and the BusyBox! They really need to have the hands-on experience with the product to be convinced that nearly 99% of their administration time will be spent on a GUI.

Now, let’s try to make more sense out of this idea by simulating an argument (people love to argue for some reason), even if this whole thing is successfully implemented & running by Microsoft as stated above I wouldn’t be surprised if people hated it:

This might not be technically possible.
If I can run on my laptop the entire VI3 stack, including the SRM itself as a multi-site DR solution, I believe VMware can deliver and automate this environments online and on-demand.

VMware doesn’t have the facilities for implementing this.
As far as I know, VMware is building (or already built) a huge datacenter in Washington, and dedicating most of it for R&D, I believe they won’t mind to dedicate just a small part of it for an idea like this.

Not everything can be done remotely, you will still need physical access.
If you took an official online VMware training, you will know that the entire labs are done efficiently on remote facility, and we are talking here about detailed technical training.

VI3/vS4 is not that easy to install/configure by people on their own.
I believe a 10 years old kid can install ESX(i) in less than 15 min. There is also an endless amount of tutorials & videos available online for that matter, even for vS4 that is not yet publicly available you can see people already started creating these videos (ESX4 here & vCenter4 here)

VMware would kill its partners who are already doing this kind of SaaS.
We are talking here about a limited duration labs deployed and wiped-out for testing and evaluations. May be for proof-of-concepts that will require more time and effort, so they could be presented as a part of their PSO services, but online.

Why doesn’t an individual do this on his/her laptop, or an enterprises do this on their own labs.
A 4GB laptop is not that cheap or available for everyone, despite the slowness when running everything in the same time. For enterprises I don’t think a SysAdmin would have the time to prepare and build a lab just to test a software he/she’s not even convinced yet to adopt, while through this idea it takes only a few mouse clicks to test whatever required on the fly without all other complications.

VMware does PoCs and jumpstarts through their PSO.
Indeed, but we are talking here about the initial steps leading to this phase. Not all the managers out there are open-minded and listening to their enthusiastic techies to start PoCs, they may need something to see infront of their own eyes first, and done by their own team. Besides, VMware’s PSO is not available everywhere and this is a fact I’m living in right now.

These are the arguments that I can think of at the moment, if you have any other reasons why this would be a bad idea, drop a comment, I will delete it anyways (just kidding). If you agree, say so or promote the idea on your blog. If you are a VMware employee, present it to your boss and take the credit for yourself, but I’ll hunt you down (I’m NOT kidding!)

+ HyTrust Appliance Community Edition is finally here! By Hany Michael 06 May 2009 at 2:21 pm and have 1 Comment

I’ve been following the HyTrust security appliance for quite some time now, and been in contact with their great development and marketing team as well for a while, waiting with much anticipation & curiosity to evaluate this promising solution. The HyTrust from one hand is expected to address one of the major concerns that I’m having right now with my VI3 environment, which is auditing and compliance , and when I say auditing here I include the “VI3 DC admin” himself!

From the other hand I’m really impressed with the free-appliance community edition approach, which reflects a good image of the people running this whole thing!

The HyTrust’s commercial appliance comes in two flavors, a physical rack-mounted appliance ($7,500), or a virtual edition ($3,000) that can run directly inside your VI3 (and soon to support vSphere). I urge you to visit their website to have a closer look and more information. If you are too technical like myself, you may want to start with David Storm’s video review to have an over whole idea.

Speaking about videos, I’ve downloaded the community appliance to test it in my lab, and will most probably come back with some videos showing the great features and capabilities this new security appliance has to offer.

+ Video: vSphere4 Unleashed: 06 - Storage vMotion By Hany Michael 06 May 2009 at 2:16 pm and have 1 Comment

A new, thin and compact video! Here we’ll demo the new GUI based storage vMotion feature built-in vSphere. Yes, GUI based and built in, no more command lines and hassle. It’s fast, it’s easy and it’s very reliable, especially when doing the thin/thick conversions while you’re migrating the VMs.

Happy viewing!