Posts Tagged ‘VMware’

We’ve got quite a bit of stuff to talk about in this post/diagram/video, so I’ll keep the introduction as simple as this: It’s the third part, it’s the final part, and it’s my favorite part of the (vSphere In A Box) series.

The Diagram:
This is an overall illustration of the workflow and automation process of the environment using the same single server, and an iSCSI array, that we’ve used in the previous posts. The diagram may look a bit complicated at the first glance, but I tried to keep things as simple as possible. I used the actual IP addresses, configurations, and workplaces names that I use in my environment and as you will see in the next video, to keep things clear and easier to follow.

The Video:
This is by no mean a Lab Manager tutorial, it’s just a very “quick tour” on the components and automation of the my vSphere labs. I intend to do a series of tutorials in the future on Lab Manager 4.0 itself, but for the sake of this subject I created this quick walkthrough video instead of taking more than 20 screenshots that won’t make much sense individually.

My Use Cases:
If you’ve been reading my previous few posts, you’ve probably known what are my objectives from this, and since I hate to repeat myself, I will copy & paste one of the use cases I mentioned in an earlier post, followed by two new cases I’m working on now:

1. [...] I need to build my own labs for my “virtual private cloud” project, which will typically require building multiple vESX clusters, not to mention the importance of having the templates and libraries for deploying these clusters whenever I need to test something new that would require significant changes (e.g. Nexus1000v for networking, or Reflex vTrust for security) [...]
2. I’m trying these days also to evaluate the VMware Site Recovery Manager running in two deferent and independent setups to simulate a real life scenario instead of using the traditional all-in-one-subnet setup. Since part of these applications are still in Beta, and consequently under NDA, I won’t be able to elaborate more on this use case until all the software are in the GA stage.
3. I discovered another great use case while I was working on the Lab Manager, which is training the SysAdmins in my current corp on the new vSphere platform. We are also growing rapidly with our virtual environment (soon to be: private cloud) and we’ll definitely need more SysAdmins on board. That being said, I now have an instrumental tool for provisioning independent labs for the SysAdmins to login and have their own hands-on experience with the new platform.

Other Considerations:
As you all know, the vCenter Lab Manager 4.0 was never meant to run virtual ESX hosts inside itself, therefore, there are some tweaks and considerations that you need to know about while you are implementing this in your own environment. But the beauty of VMware is that all its components are highly customizable and work in a true harmony regardless of what you want to achieve at the end.

1) As you know, the vESX hosts need to be connected to a “Promiscuous enabled” vSwitch. When you create new Network Templates in LM, and later on deploy them within your configurations, the LM by default create the new vSwitch with this feature disabled, so you will need to go an manually enable that. I’m trying to figure out a way to automate this or set it to be enabled by default, I will update the post should I manage to do that.

2) When you import your vESX templates you must uncheck the customization part as shown in the screenshot since the VM is not running a supported OS by VMware (like Windows or Linux), however, I managed to install the VMware tools inside the classic ESX 4.0 as shown in the next point.

3) Although I haven’t tested this thoroughly yet, I believe you can customize the classic ESX hosts with COS in Lab Manager by installing the VMware tools as follows:

Mount the VMware tools ISO

unzip/untar the package

Run the perl script and follow the installation instructions:

Conclusion:
By this part, I believe that we’ve completed the entire series of the (vSphere In A  Box) and shown how and why you need to be doing this setup. I hope that everyone can clearly see now the huge deference between running the ESX hosts inside VMware Workstation compared to running it inside ESX itself, as well as the great benefits of doing that in a larger scale with the automation to achieve many of the use cases mentioned across this series of posts.

This is most definitely one of the fastest evaluations I’ve ever done for any VMware product so far…This is what happened yesterday in order (mapped to my local time zone):

07:00am: The product was announced officially by VMware, and the bits were available for download.
08:00am: I was in my office checking the VMW’s website to download Lab Manager 4.0, and then I thought why not, let’s do the AppSpeed as well.
08:30am: I fired up the VMworld Europe 2009 AP12 session to refresh my memory about the AppSpeed.
09:00am: I was blown away by the presentation, so I decided to forget about the LM4.0 for a bit and give the AppSpeed a spin.
09:15am: the AppSpeed was up & running on my physical lab.
10:00am: I started deploying it on one of my production ESX clusters for tier-2 and dev apps.
11:00am: when the AppSpeed started to do its thing and show some statistics I was utterly stunned!
08:00pm: I created this video at home on my virtual ESX labs for publishing the next day.

If you want to get to the meat and potatoes of the AppSpeed installation and configuration, you can just view this video to get you started. If you want to see how it might look like in real production, and how it personally saved my day, jump to the next section.

Note: As you saw in the video, my lab doesn’t have yet any real applications workload, so there is actually no traffic to be metered.

The AppSpeed saved my day!
I recently designed two SharePoint farms running on VI3 as per VMware & Microsoft recommendations and guidelines, each farm consisted of 2 WFE VMs with NLB, 1 Indexing VM server, with 2 Active/Passive physical SQL nodes for the backend database. The SP farms worked perfectly well most of the time, however, sometimes we used to get some strange delay in page loads, and it was happening randomly across deferent pages and not in any particular time. I kept looking and looking for any reason that would be causing these delays from the virtual environment, but nothing. Yesterday when I deployed the AppSpeed in my production, I was able to see that one of my VMs running the Microsoft SCOM was producing a hell of SQL transactions to the SQL cluster. Since the SCOM is not that critical in our environment, where we heavily depend on other monitoring software like the ManageEngine Application Monitor & OpManager, I stopped the SCOM and I instantly saw a noticeable drop down in the Mem & CPU utilization on the SQL cluster. From that moment till the time of writing these words, the SharePoint farms are performing perfectly well. Our DBAs were able to identify that our physical SQL cluster needs an urgent memory upgrade, and in fact considering virtualizing it completely on our new vSphere installation.

Are you a VMware expert?
The AppSpeed is not meant to replace any of your existing monitoring tools that you are happy with. It’s another great visibility tool for doing that, not to mention the SLA part and the integration with the vCloud technologies. I just wanted to tell you that you still need this even if you are a VMware expert who knows how to use the traditional tools for performance analysis. Let me give you a very practical example:
In the SharePoint scenario that I’ve described above, I could still identify the problem as follows:

• I can use the ManageEngine Application Monitor to identify that there is a high memory utilization on the SQL nodes.
• I can use the NetFlow protocol on VMware ESX 3.5 (as demonstrated here) to identify that the SCOM VM is having a high traffic on the vSwitch out to the SQL cluster.
• I can use esxtop to record, play or analyze in high details that the ESX servers and the SharePoint VMs are either ideal or performing perfectly normal all the time.

Now, ask yourself this: why do I need to do all that if there is a tool that can just give me the complete picture? How much time and effort do I need in order to go through this entire cycle?

To conclude all the above, in the AppSpeed presentation there was a part saying this: “AppSpeed reduces finger pointing and allows IT to focus on solving root cause of issues”. This is not marketing words, it’s a fact I literally experienced yesterday in my production environment.

I really, really wanted to share some screenshots, but since it’s running in my production I couldn’t do that here. Even with graphics editing, I’ve never been a fan of doing something like that. I do promise though that I will put some application workloads in my lab and return back with more videos or screenshots.

I extracted some screenshots from the VMworld session I mentioned at the beginning of my post, just to give you a real feel of what you see. It’s still not comparable to what you will experience in your own environment. I do encourage you to do that, whether you are on vSphere or still on VI3 like me.

At the time of submitting this post, the press release from VMware should be out. It’s July 13^th, 12 midnight NY time, and VMware is announcing a bunch of vCenter products including the much awaited Lab Manager 4.0.

Unfortunately I was not in the beta program of LM4.0 to have an early hands-on experience, but I’ve been briefed about it, and I must say that I’m quite happy with the new features of the product. I’m exceptionally interested in this product now more than any time before, for three main reasons:
1) I need to implement LM in a wide scale for our development department. Currently our developers are working independently everyone in his/her own island, using things like VMware server or workstation! No effective collaboration or automation whatsoever.

2) I need to implement LM for our infrastructure team to simulate our entire production environment. Again, I need something that can scale out easily (typically on blades), yet with flexible and fenced network configurations.

3) Lastly, I need to build my own labs for my “virtual private cloud” project, which will typically require building multiple vESX clusters, not to mention the importance of having the templates and libraries for deploying these clusters whenever I need to test something new that would require significant changes (e.g. Nexus1000v for networking, or Reflex vTrust for security)

As soon as I wake up from sleep (yes, technically I’m sleeping now, this is a scheduled post) I will download the bits and start playing with it. I will come back with more details later, but between now and then, here are some highlights on the new features of LM4.0 along with two screenshots from the product team:

- Lab Manager 4.0 now fully supports VMware vSphere 4
- Support for the VMware ESX(i) Form Factor (was not supported in previous versions).
- LM is integrated now with VMware vCenter Stage Manager.
- Host Spanning Private Networks: Host Spanning Private Networks, a new technology in Lab Manager 4, creates isolated private networks without the need for setting up VLANs. This new feature requires the Distributed Switch capability of VMware vSphere Enterprise Plus edition.
- Multiple Workspaces: VMware vCenter Lab Manager 4 introduces the concept of multiple workspaces within organizations.
- Archive to Library: Lab Manager 4 provides the ability to keep a particular configuration together with its change history for record within the library.
- Configuration History: Lab Manager 4 mow provides a new configuration history tab for all configurations. The history of a configuration displays the list of the events related to this configuration.

For the complete and detailed list of new features, you can check out this WP from VMware: http://www.vmware.com/files/pdf/VMW_09Q2_WP_vCenter_LabManager4_10_R1.pdf

Screenshot (1): VMware vCenter Lab Manager 4 streamlines application releases from development to production. The self-service interface provides on-demand access to virtualized application environments while IT remains in administrative control.

Screenshot (2): Advanced networking capabilities in Lab Manager 4 allow application teams to create realistic, production-like test environments for complex system and network configurations.

The response to my previous post has been unreal! The amount of tweets, ping backs, hits, linking, emails was quite amazing. I have to admit, I didn’t expect to see that much of interest in the subject, but thanks to everyone who participated in promoting this idea on twitter and the blogosphere.

In the second part of this series we will spice things up a bit and explore through the following video many aspects of the idea we’ve talked about. Following to that an important screenshots and some considerations you should be aware of before and after implementing this in your lab.

What you’ll see in this video:
1- Deploy a thin-provisioned vESX(i) VM from a template.
2- Check the required configuration parameter on the vESX to run nested VMs.
3- Customizing the new vESX server (assign password, set a static IP, put the DNS config ..etc)
4- Add the vESX to an existing HA Cluster.
5- Test the vMotion within the same cluster and across deferent clusters in the datacenter.
6- Add the required configuration parameter on the nested VM for enabling the FT.
7- Enable the FT and test the failover across deferent vESX servers.

A Quick note on the hardware used:
Technically speaking, only one server can be used in this whole setup. What is deferent in my setup (as you saw in the diagram) is that I use an external iSCSI array (the CLARiiON AX4) for hosting the vESX VMs. I just needed the flexibility to have them on an external storage to share them later on with other servers, but it is not a requirement. You can simply use the internal storage of the pESX server to host your vESX and you will still have everything you see in the video. As far as the shared storage for the vESX servers is concerned, you can use the Celerra VSA. In my case I use the Celerra only for the SRM labs to do the replication trick. Other than that, I use the OpenFiler as my shared storage for the nested VMs.

Two things to be set on the pESX hosts:
1 – Increase the number of ports on your pESX vSwitch to accommodate the increased number of connections required by your vESX VMs.

2 – Enable the “Promiscuous Mode” on the vSpwitch

The configuration parameters on the Virtual Machines are as follows:
1 – The virtual ESX (vESX) host: monitor_control.restrict_backdoor = TRUE

2 – The nested VM: replay.allowBTOnly = TRUE

The iSCSI vSwitch on the pESX host bound to vmnic1 (phisical NIC 2) and connected to the EMC CLARiiON AX4 iSCSI array

The vMotion internal vSwitch on the pESX host

The Fault Tolerance internal vSwitch on the pESX host

The thin-provisioned ESX(i) size on disk (475MB) + the memory swap:

The thin-provisioned ESX(i) size on disk(3.5GB) + the memory swap:

Other considerations and GOTCHAs:
1 – When enabling the FT, make sure you have your VMs powered off, even if they have eager zeroed disks, I used to get some errors when the VMs were powered on while enabling FT.

2 – Sometimes the network card order and numbering could be confusing. For example, in the vESX VM, you will have the NICs order starting from 1 to 10, but in the actual vESX network configuration tab, you will find the NICs starting from 0 and counting towards 9. This could be confusing when mapping your vnics to the pESX host vSwitches, like the VMotion internal switch, the FT internal switch and so forth. Just make sure you count the nics order accurately.

3 – Deploying vESX from templates may be cool and fast, but it could a bit challenging sometime in troubleshooting the network related issues. The reason behind that is the fact that all the network cards will have the same MAC address, or worst, the Port Groups like (VMotion) could have the same MAC address even if you completely remove the vnics and created brand new ones. The work around for that is to create the vESX template, and then remove all the NICs form it. When you deploy a new vESX, you can just add the new nics as you like, and by that you’ll have a new MAC addresses. Beside that, you may need to add a new VMkernel network for the VMotion, and then remove the old one. Of course you may be thinking that deploying a brand new vESX would be easier, you are right, but with the scripting everything could be automated. I will try to write a PS script to automate this network changes/settings and post it here later on.

I’ve been following the HyTrust security appliance for quite some time now, and been in contact with their great development and marketing team as well for a while, waiting with much anticipation & curiosity to evaluate this promising solution. The HyTrust from one hand is expected to address one of the major concerns that I’m having right now with my VI3 environment, which is auditing and compliance , and when I say auditing here I include the “VI3 DC admin” himself!

From the other hand I’m really impressed with the free-appliance community edition approach, which reflects a good image of the people running this whole thing!

The HyTrust’s commercial appliance comes in two flavors, a physical rack-mounted appliance ($7,500), or a virtual edition ($3,000) that can run directly inside your VI3 (and soon to support vSphere). I urge you to visit their website to have a closer look and more information. If you are too technical like myself, you may want to start with David Storm’s video review to have an over whole idea.

Speaking about videos, I’ve downloaded the community appliance to test it in my lab, and will most probably come back with some videos showing the great features and capabilities this new security appliance has to offer.