Posts Tagged ‘esxtop’

• This is the first diagram of more ESX blueprints to come. Previously I was focused more onto the vSphere architectures as a whole, but there will be more granular diagrams on ESX itself as a hypervisor very soon.
• The colors play a major role in this diagram. I thought I should stick with the original colors used in the vCenter Server Recourse monitoring box (on the top right of the diagram) instead of reinventing the wheel. As an example, if you want to scan through the swapping memory reclamation technique, all what you need is to focus your sight on the red colors. Whether it’s in the VMs, hypervisor, esxtop or vCenter, the red color will always relate to the swapping activity, and so forth.
• In the MMU virtualization I did not put much detail (as I was hoping to) because of the diagram limited space, but I'm planning to make more detailed diagrams about this topic in the future, along with other technologies like NUMA and EPT/RVI for example.
• In the memory reclamation techniques, you might have noticed that the "idle memory tax" is not there. I intend to illustrate that in a deferent topic related to shares and resource management.
• The Memory compression and swapping to SSD are future features coming soon in the vSphere generation. This information was mentioned in the VMworld 2009 session TA2627 so it's public.
• The most interesting part for me while creating this diagram was the ESXTOP. I have a wild idea of creating a crazy huge diagram (poster-like) and add the deferent screens and options for ESXTOP along with real-world numbers and descriptions to show the beauty of this incredible tool. I might also use Duncan's esxtop section as a reference for the thresholds, I urge you to go there and share your experience!
• For some reason, the PDF converter kept converting the ESXTOP text box into an image and then down-sample it, which ended up not looking as sharp and clean as in the original Visio diagram. I'm trying to figure out why the Acrobat is giving me this grief, so I will probably update the PDF later on (check the version numbers).
• I've received a number of emails asking for some descriptions accompanying my diagrams (like the admission control calculations in the HA diagram), and I believe you are right about that, especially for the newbies entering the fascinating world of VMware. I'll revisit these diagrams in the future with detailed descriptions and write-ups, so stay tuned.

I hope you'll like this diagram and see the real beauty of the ESX memory management (well a glimpse of it at least). If you have any corrections or suggestions please drop a comment or email and let me know your thoughts.

Recourses:
- Book: Operating System Concepts (Part Three – Memory Management)
- Book: Modern Operating Systems 3rd Edition (Chapter 3 – Memory Management)
- Documentation: vSphere Resource Management Guide
- WP: Understanding Memory Resource Management in VMwareо ESX™ Server
- WP: Memory Resource Management in VMware ESX Server – by Carl A. Waldspurger
- VMworld2009 Session: TA2963 “esxtop for advanced users”
- VMworld2009 Session: TA2627 “Understanding "Host" and "Guest" Memory Usage and Related Memory Management Concepts”
- Blog: Arnim van Lieshout (Part-1 , Part-2, Part-3)

I've been rediscovering the great esxtop tool lately, and I have to say that I'm really impressed with the tremendous amount of information you can get out of this little thing!

But today I will be talking about the (esxtop modes) which I believe not many of us know much about. With a quick search also over the Planet V12N blogosphere I couldn't find any posts talking about this topic, so there you have it.

We all know about the traditional way of running the esxtop tool through the esx host service console (or resxtop through RCLI). This is in fact called "interactive mode" where you get to see the statistics or information live in front of you and interact with it using deferent sets of keys (m = memory, d = disk and so forth). But this is actually only one of three modes that esxtop can run into. Here are the other two:

• Replay Mode
  

In this mode, you can record and play the esxtop statistics for a specific period of time, and use also the interval of your choice. But before going into details, you may be wondering why would you want to do that? Well, it could be for support purposes, for example if VMware wants to have a snapshot of your performance statistics they may ask you to run this and send them the output. For me, this came in handy when one of my VMs was having a very strange behavior during an early morning hour. This was happening on daily basis and since I was not in favor of staying awake to see what's the deal with that, I set a small cron job to run/record the esxtop stats for this specific period of time. The next day I played the stats and I was very grateful for the amount of information I got for troubleshooting the problem.

Now, let's see how we can do this:

First, you will need to be "root" in order to issue the record command as follows:

vm-support -S -i 5 -d 120

We can see here that the interval is 5 seconds, and the duration for recording the statistics is 120 seconds.

The esxtop will start recording the stats and then compress the output into a .tgz file.

We can issue a command to see the .tgz file as shown below. Make sure you are in the right partition when issuing this command and that you have a reasonable disk space if you will run this command for long time.

Next, we need to uncompress the file as follows:
tar -xzf esx*.tgz

Finally, you need to issue the replay command: (thanks to @vRobM for bringing my attention to this missing command)
esxtop -R vm-support*

And there you have it, the statistics will run as if you are sitting at the point of time the stats were recorded in:

One thing to note here though, the time that will show on the top line, will always reflect the current time you are running in, not the time of the recording, so don't be confused by that.

• Batch mode
  

In the batch mode, you can dump your esxtop stats into a .csv file for later use. You can utilize either Microsoft Excel, or Microsoft perfmon tool to view these data at any time.

Here is the command to achieve this:
esxtop -b -a > output_file.csv

This command will dump "all" the fields from the esxtop to the file. But what if you want specific columns only? Well, that's the real beauty of the batch mode. You can always choose what information you are interested in, here is how:

• Run the esxtop command in interactive mode.
• In each of the panels, select the columns you want.
• Save this configuration to a file (by default ~/.esxtop4rc) using the W interactive command.

Now you can use the batch command and it will dump only the comlums that you have selected.

What's next?
Use Excel or perfmon to analyze your data, I personally prefer to use my Vista's perfmon for this unless you have a specific requirement for using Excel.

• Go to Start -> Run -> type perfmon.
• Go to Performance Monitor section.
• Click of the "View Log Data" in the tool bar at the right as shown in the screen shot below.

   

After that, you need to use the file you've dumped the esxtop data to:

You can choose here which columns/counters you want to graph and analyze

And here you see your selected columns graphed in perfmon

And you are done!