Archive for the ‘Diagrams’ Category

Yes, finally! It's been like what? Five months?! Well, the delay in publishing this part was mainly because of the delay in certifying the new IBM HX5 blades on vSphere. It's a quite long process that you can read about it here, but the good news is that the hardware is finally on VMware's HCL, and that I can comfortably blog about the subject now without causing any confusions to the readers.

Before we dig deep into the new designs, I'd like to mention some minor changes in the diagram.

Updated Diagram

I've included the old configurations along with the new ones in one updated PDF. The main difference now is that I'm using normal pages for showing each configuration. In the old version I used the layers to show and hide the configurations as you select them. I thought now that using separate pages for different diagrams would ease the process of browsing through the configurations, and to tell you the truth, to reduce also the high complexity of designing the diagram. It's a crazy process to keep track of all these layers in Visio especially when we are talking about more than 7,000 shapes floating on the same design area!

Now let's get down to business.

Configuration (5) – HX5:

This is the Big Blue's latest two-node blade technology. I emphasized on the "two-node" here because it's the only configuration certified to run with vSphere as of the time of writing these lines. Please note that you can use up to 4 nodes with the HX5 but this won't be supported by VMware. When we talk about two nodes here we mean the following:

• Having the base blade (CPU + Mem + HDD) (+Plus+) the MAX5 expansion try to scale up more memory for the blade.
• Having the base blade (again CPU + Mem + HDD) (+Plus+) another similar expansion board to scale all the blade components, that's 4 x CPUs + 2 x Memory modules + 2 x IO expansion cards.

As you will see in the diagram, I chose the second option to talk about.

Now, what do we have here? it's simply the redundancy at its best! We can place our networks here freely with full redundancy as you see in the layout of the vNICs. For example, if we have a failure in the CFFh expansion card on any of the two nodes, we will still be able to flow the traffic without any issues on the other CFFh card. Same thing holds true for the on-board ports, if for any reason one of these posts fail, the traffic will flow on the other node's board.

Apart from that, I'm introducing here the DMZ networks for the first time. Most of the enterprises prefer to separate the DMZ networks/servers on different chassis for security reasons. While this is a valid decision, we can have with this blade configuration a workaround for organizations that are less paranoid about the DMZ security, yet with good isolation. Let's see how this is done in details:

• For the networks, we have two dedicated blade switches that will be uplinking *only* to the corp DMZ switches (in this case Bay 9 & 10). This means we will have no traffic following from either the internal networks or the VMkernel networks. Same thing for the blade ports, you will always have the NICs 4, 5, 10 and 11 dedicated for the DMZ networks and running in full performance and redundancy.
• For the SAN, we can also ensure that we have a dedicated HBAs as well as an isolation. The uplinks to the SAN switches will be segmented across the two bays 3 and 4, and connected directly/physically to the appropriate SAN fabrics.

Configuration (6) – Virtual Fabric:

Before we start with this configuration, I would like to state that I am not quite sure whether these Emulex Virtual Fabric Adapters (VFA) are supported by VMware or not. While I can't see them clearly on the HCL with the name VFA, I can see some Emulex documents saying that they are. Of course the reference here should be always the VMware HCL itlself, not anything else, but I will double check on that and update this post later. With that said, please refer to this configuration carefully and make sure to confirm this point before engaging with any vSphere design around it.

Now let's dig deep into this cool technology. IBM simply has this Virtual Fabric concept of slicing your CFFh expansion card into 8 different ports. This doesn't only mean that you have the flexibility to adjust the speed, but also the protocol. For example, you can choose to use either Ethernet, Fibrechannel, FCoE or even iSCSI with hardware initiators.

In our case here I used only Ethernet as the protocol for these ports, and then sliced them into 8 different vNICs with various link speeds. Perhaps a screenshot from the diagram would make things more clear.

As you see, we set the bandwidth for the SC to 1GB since we normally don't require high BW for management, while we set 3GB and 5GB link speed for the Fault Tolerance and VM Networks respectively. By default these ports are set to 2.5GB ( 4 x 2.5GB = 10GbE into two ports), but you have the full flexibility to change that as you see.

Configuration (7) – CNA:

A very simple design to wrap up this series with. It's the traditional CNA (oh yeah, it's a common and traditional technology now!). As you see in the diagram, we have here a CFFh expansion card, and it has got four ports:

• Ethernet ports: that's 2 x 10GbE Ethernet ports for the networking traffic. We will treat them here normally as we treat any 10GbE port. We will slice them via the vNetwork traffic shaping in vSphere to achieve the bandwidth that we want.
• FibreChannel ports: that's 2 x HBA ports for SAN traffic. Instead of going into the traditional Bay 3 & 4 as we've see across the whole series and configurations, this time the traffic is multiplexed and pushed to the Nexus 4000 blade switches.

Did I just say Nexus 4000?! yep, that's a specially developed Nexus switches by Cisco to be used only/currently with the IBM BladeCenter H/HT. But here is the catch, you will still need to have the Nexus 5000 switches to segregate the FCoE traffic coming from the Nexus 4000 and then forward the network and FC traffic to the existing LAN and SAN respectively. Of course we should have redundancy here at all layers. In the BCH we have two Nexus 4000 sitting in bays 7 and 9, while we have two Nexus 5000 switches in the back end.

Now what?

Well, as much as I worked really hard in this series to come up with different kind of configurations and design scenarios, as much as I enjoyed it! Now I need to move on to another vendor, but without all these mad options. I initially was planning to jump straight to the HP realm, however, i found myself involved in two different Cisco UCS vSphere designs lately, so it would make much sense to me if I blogged about this platform now. Don't take my word for it though, I might surprise with a Dell or Fujitsu series, who knows?!

Important: In case you haven't done that already, please take a moment to read the first post of this series.

Due to the insane number of expansion modules/options available in the IBM BladeCenter H, I had to split this post into two parts. In fact, I was initially planning to have around 12 different designs for vSphere on BladeCenter H (yes twelve) but I then I started to shrink and skip some designs to fit as many scenarios as possible in a reasonable two-part article. With that said, the following is by no mean a list of all the possible design scenarios you can achieve with this hardware platform. If you started the "mix and match" game, you may literally end-up with uncountable possibilities!

The Diagram

Here is some important notes before using the diagram:

• You will see different configurations in this post and the relevant architecture of each configuration in the diagram. This is done through the PDF layers, which basically means than you should *not* activate more that more layer in the same time.
• By default, "Configuration 1" is the first active layer when you open the PDF file. You can show/hide the other layers by simply clicking on them. Again, you should only show one layer/configuration at a time.
• You will always see two boxes on the right side of the diagram, the upper one will show you the current vSphere configuration, and the lower one will show you the relevant hardware configuration. You should typically start looking at those two boxes before scanning through the diagram to understand the "ingredients" of the design.
• At the time of writing this post, you will see four configurations only in this diagram, however, when I publish the second part, there will be additional configurations that I will add to the existing ones. In other words, the diagram will be updated later on to have those additional configurations so keep that also in mind.

The common design and configurations

You will find in most of the configurations a common design, unless I explicitly state otherwise. I will list them here in details:

The Clusters:

You will see two type of clusters:

• Management Cluster: it is typically a two node cluster running the management and infrastructure services. For example, if you want to virtualize the vCenter Server, the VM should be running on this cluster rather than the actual production clusters. Same thing holds true for other vCenter products like: AppSpeed, CapacityIQ, SRM and so forth. There are two reasons for doing that: the first, we don't want to run into the problem where vCenter Server is not accessible (there are some examples published in the community but my favorites are Jason Boche's Catch22s!). The second reason, we don't want to either affect our workloads' performance with our management virtual appliances or vice versa.
• Production Clusters: You can see here two production clusters (Cluster A and Cluster B). The take away from that is the following:
  • You don't have to stick with that number of hosts per cluster, it depends on what you want to achieve, and also on some configuration maximums that may or may not limit you.
  • The nodes have to be spanned across the two chassis as numbered and illustrated in (Config 1). There are two reasons for that: Firstly, you don't want your whole cluster to fail in an unluckily event when a whole chassis fails. Secondly, you have to keep in mind that VMware HA selects the first 5 hosts in the cluster and promote them as a "Primary" nodes, if they fail, your HA cluster fails.

The Blades:

You will see two consistent blades throughout the first four configurations, the HS22 and the HS22V. Both blade servers share the same IO expansion capabilities, however, there are a some differences between them. For example, the HS22V has no hot swappable HDD but it is superior in the memory capacity (144GB compared to 96GB in the HS22). In part-two of this article, I'll talk in details about the new HX5 and what it can bring to the table in terms of scalability.

The Expansion cards:

Every HSxx blade comes with two onboard 1Gbps Ethernet ports for basic networking. They will always show in vSphere as vmnic0, and vmnic1. These ports are in turn mapped to Bay 1 and Bay 2 in the chassis. Of course no one recommends implementing vSphere using 2 x 1GbE ports in an enterprise environment (although it will technically work), so we will use here what we call: expansion cards. There are two slots for expansion cards in any HS22/V blade, the first one is called CIOv (for vertical expansion modules) and CFFh (for the horizontal fast IO modules). The CIOv is usually used with the FC HBAs (although we will see later how we will utilize it for iSCSI connectivity), and they are mapped to Bay 3 and Bay 4 in the chassis. The CFFh on the other hand is mapped to four fast expansion modules (7, 8, 9 and 10). I say fast because this is the only card that can leverage the 10GbE connectivity (or Infinibad but it's not relevant to our series). Depending on the configuration, you will see how we will use different cards to support our designs, however, the onboard 2 x 1GbE port will be always common, and always there.

Now that we've talked about the common stuff, let's start talking about the unique configurations. Oh yes, we were just warming up!

CONFIGURATION (1):

We have in this configuration 6 x 1GbE pNICs per blade to support our MGMT, VMkernel and Virtual Machine networks. We teamed three pNICs here in a vNetwork Standard Switch (vSS) to serve the SC, vMotion and FT. The other three pNICs are teamed in a vNetwork Distributed Switch (vDS) to serve the VM networks. Let's dig litter deeper on how this is done.

As mentioned earlier, we have three type of IO ports on the blades: the onboard ports, the CIOv, and the CFFh. In order to achieve the maximum availability, we teamed one onboard port with a couple of ports from the CFFh card. In this case, if we had a failure in any IO port (on board or expansion card) we will be able to tolerate that failure.

The second consideration here is to distribute the load and bandwidth for our networks. For example, the SC network will be active on vmnic0 and standby on vmnic1. The vMotion will be active on vmnic1 and standby on vmnic0 and so forth.

You may have noticed also that we grouped the SC + VMkernel network on a vSS, while we grouped the VM networks on a vDS. The reason behind that is to ensure that you would still be able to control your SC network even if your vCenter fails. For the VM networks, you would still leverage the great enhancements and features of the vDS. This is *not* a best practice from VMware, and as far as I know there is no documentation recommending that. It is up to you whether you would go with that setup or simply have everything on a single vDS.

CONFIGURATION (2):

This is nearly identical configuration except for the IP SAN. In Config1 we were running on a FibreChannel SAN, while in this configuration we have an iSCSI. The thing to note here is that you will need to install your Ethernet expansion modules in Bay 3 & 4. We will swap also the CIOv card from being a FC HBAs to a traditional 2 x 1GbE card. Of course you will use in this case the vSphere iSCSI initiator for doing your storage networking. This is fine in nearly most cases, except the one where you will actually need to boot your ESX server from SAN.

Please also note there that you can use NFS with the same layout. Your 2 x 1GbE blade ports + the 2 x expansion modules (bay 3 & 4) will all serve your NFS requirement in a high availability design.

CONFIGURATION (3):

What you will see in this configuration is something a bit different. We are using here a 2 x 10GbE ports through the CFFh expansion card to serve "all" our networks. This card is mapped to two 10GbE expansion modules sitting in Bay 7 and Bay 9.

The trick here is this: how can you have a proper network segmentation if you are using two pNICs only? The answer, of course, is VLANS. As you see in the diagram, we have two production networks and one lab network. All these networks are tagged with a VLAN ID to flow the traffic through the vmnics to pNICS all the way to your enterprise/core switches. The ports on your core switches need to be of course in trunk mode.

Now, the second question here would be this: how can you ensure that no network will saturate the whole link and affect the performance of the others. The solution for that is to use the vSphere traffic shipping. You can simply dedicate the bandwidth to each "port group" per your requirement. Example, for SC you normally don't need more than 1Gbps. For vMotion and FT you would definitely require more bandwidth. To keep things simple, I illustrated in the diagram how the segmentation and bandwidth allocation can be distributed across the two links in an Active/Standby approach.

You will notice here also that we are utilizing the two on board Ethernet ports to have an additional iSCSI SAN (for the Lab environment for example) along with the FC SAN for your production workloads.

CONFIGURATION (4):

In the previous configuration we saw how we leveraged the VLANs to do our network segmentation and how that was quite easy and flexible. But what if the customer has a policy not to use VLANs to consolidate the networks (for a security reason as an example)? Easy, we would still be able to comply with that. Basically we will need to swap here the 2 x 10GbE CFFh card with a 4 x 10GbE card and of course add additional two 10GbE expansion modules to Bay 8 and Bay 10.

Now, what did we achieve by doing that? Two things:

1 – We are compliant with the customer requirement to have a physical segmentation between the Management/FT/vMotion networks and the production networks.
2 – We are using the vSS for our management network while leveraging the vDS for our Virtual Machine networks.

You have also here another two options that were not included in the diagram. You can make use of the two onboard ports to have an additional iSCSI SAN as we did in the previous configuration, or, you can use them as a standby ports for your Management/VM networks in case of a CFFh card failure. Do you see now what I meant above by the "mix and match game"?

Coming Soon – Part 2:

I'll talk about the new HX5 and how you can have a lot more memory or extended IOs to support special workloads or strict design requirements. I will talk about FCoE and CNAs. I will also talk about the new & promising Virtual Fabric from IBM, and how you can basically slice your pNics into almost any protocol or speed you want.

Stay tuned!

• This is the first diagram of more ESX blueprints to come. Previously I was focused more onto the vSphere architectures as a whole, but there will be more granular diagrams on ESX itself as a hypervisor very soon.
• The colors play a major role in this diagram. I thought I should stick with the original colors used in the vCenter Server Recourse monitoring box (on the top right of the diagram) instead of reinventing the wheel. As an example, if you want to scan through the swapping memory reclamation technique, all what you need is to focus your sight on the red colors. Whether it’s in the VMs, hypervisor, esxtop or vCenter, the red color will always relate to the swapping activity, and so forth.
• In the MMU virtualization I did not put much detail (as I was hoping to) because of the diagram limited space, but I'm planning to make more detailed diagrams about this topic in the future, along with other technologies like NUMA and EPT/RVI for example.
• In the memory reclamation techniques, you might have noticed that the "idle memory tax" is not there. I intend to illustrate that in a deferent topic related to shares and resource management.
• The Memory compression and swapping to SSD are future features coming soon in the vSphere generation. This information was mentioned in the VMworld 2009 session TA2627 so it's public.
• The most interesting part for me while creating this diagram was the ESXTOP. I have a wild idea of creating a crazy huge diagram (poster-like) and add the deferent screens and options for ESXTOP along with real-world numbers and descriptions to show the beauty of this incredible tool. I might also use Duncan's esxtop section as a reference for the thresholds, I urge you to go there and share your experience!
• For some reason, the PDF converter kept converting the ESXTOP text box into an image and then down-sample it, which ended up not looking as sharp and clean as in the original Visio diagram. I'm trying to figure out why the Acrobat is giving me this grief, so I will probably update the PDF later on (check the version numbers).
• I've received a number of emails asking for some descriptions accompanying my diagrams (like the admission control calculations in the HA diagram), and I believe you are right about that, especially for the newbies entering the fascinating world of VMware. I'll revisit these diagrams in the future with detailed descriptions and write-ups, so stay tuned.

I hope you'll like this diagram and see the real beauty of the ESX memory management (well a glimpse of it at least). If you have any corrections or suggestions please drop a comment or email and let me know your thoughts.

Recourses:
- Book: Operating System Concepts (Part Three – Memory Management)
- Book: Modern Operating Systems 3rd Edition (Chapter 3 – Memory Management)
- Documentation: vSphere Resource Management Guide
- WP: Understanding Memory Resource Management in VMwareо ESX™ Server
- WP: Memory Resource Management in VMware ESX Server – by Carl A. Waldspurger
- VMworld2009 Session: TA2963 “esxtop for advanced users”
- VMworld2009 Session: TA2627 “Understanding "Host" and "Guest" Memory Usage and Related Memory Management Concepts”
- Blog: Arnim van Lieshout (Part-1 , Part-2, Part-3)

I updated the diagram (v1.2) to fix a small typo and adjust also a couple of shapes. Thanks to Joshua Liebster & Bert Bouwhuis for driving my attention to this.

I know everybody skips to the diagram so I'll save you the introduction, just make sure to quickly go through the notes that follow it:

• This is not an introduction to the VMware HA, and it's not a very advanced diagram for it either. I assume here that you have a general idea on the topic before looking into it to appreciate this incredible technology. If you are a VMware professional you may also find this useful to keep your information sharp and present about the topic at any given time. You really don't have to re-read the documentation every time you'd like to remember a small detail about the subject.
• I'm introducing in this diagram the "Layers" feature in Visio for the first time. The diagram may look somewhat confusing at the first glance, so I thought that it might be a good idea to use these layers for you to hide/show the topics that you are going through in the diagram. I can see some other use cases for the Layers in future diagrams, so I hope you will like it.
• This is an A3 diagram, sorry I know most of you just love the traditional A4 from the feedback I get, but seriously, it's just TMI to fit in A4.
• Everything you see in this diagram, and specifically for the admission control, is *not* fictitious. This is a real cluster I built specifically before designing this diagram. I wanted everything to be 100% accurate and more importantly: realistic. If you zoom into the middle of the vCenter shape, you will be able to see the actual screenshot of the vCenter interface showing the HA cluster I used, and its runtime information window as well.
• It's worth mentioning that this is not all the "advanced options" that you can use for VMware HA. I just selected the ones I thought that might be more frequently used. You can always get back to the official VMware documentation for the complete list.
• The Admission Control was probably the hardest part not just to visualize it, but also to understand it in the first place! That being said, I do not expect anyone with no prior reading on this specific topic to just get it from the first glance when he/she looks into the diagram. Duncan Epping has an excellent article that I think everyone already knows about it, but it's worth mentioning that it's the best place you will ever find for VMware HA in general. The diagram should help you though to understand it faster and easier. You can see all the numbers/calculations in front of you in one shot, and how all these numbers are related to each other.
• This HA lab was built in nearly 5 minuets and is 100% virtual. Long live Lab Manager 4.0 ! (more details here)

That's all folks! I hope you will find it useful!

I'm a big believer in the saying "A picture is worth a thousand words". If you don't believe in that, then this blog will never be the right place for you. I think there is a fair amount of my blog readers who had actually visited me in my office, and they've seen how I have all sorts of diagrams covering the walls, starting from the infrastructure and solutions architectures, all the way to detailed blueprints for the deferent technologies that I implement in my environment. Beside the extreme fun I have designing these diagrams, just looking at them on daily basis help me identify the areas of improvement and future developments quite easily. Why am I telling you this small story? Well, you are going to see many of this stuff coming on my blog more than any time before folks!

Introducing the "VMware vSphere In The Enterprise" diagram v1.0

Disclaimer: This is a very, very high level "visualization" of the "virtualization" architecture using VMware vSphere. Having said that, this should never be taken for granted or looked at as the perfect design for your vSphere environment. There is no such thing as a "perfect design" at the first place. There is always a customer requirement, and best practices that we follow to achieve the "perfect solution" for the customer. I can't stress enough on this point as I know there are many VMware-newbie visitors on my blog who might be caught in this trap.

A word of appreciation: I'd like to thank Duncan Epping for his great work of choosing the (Top 5 Planet V12n blog posts), which even for me, as a good follower of that RSS feed, I always miss quite a few great posts in there. In week 29, Duncan selected this post that had an incredible list of network ports in the VMware environment, which I have used some of them in my diagram above. It's not the complete list of course since it's out of my diagram scope, although I do intend to do a complete "block diagram" in the future to visualize the entire list.

Printing Considerations:
In case you haven't noticed, this is an A2 scale diagram. I've initially tried to fit it into A3 while designing it but I couldn't. The amount of information and layouts were just too much to fit in the A3 scale. The diagram still prints well on A3, but you'll have a hard time reading some parts like the port numbers. That said, I highly recommend that you print it on an A2 plotter, which will give you the real look and feel of the diagram. In my case, although we have in our GIS department many plotters for printing even larger scales like A1 and A0, I just went to the nearest Xerox center and printed it there just to make sure how it will look like in commercial printing centers, and the printout was phenomenal.