This is a legend! Thanks for sharing this. I am trying to build a similar setup, ESX4.1 with HS 22 baldes in 2 different chassis. There's one vDS switch for production traffic and I have used standard switches for SC and vMotion. The commuication from 1 chassis to another chassis is really slow. Internal to chassis is lighting fast although. Where I should be looking at ? Can I have some guidance here ?

Do you know that you are simulating a VCDX defense panel for me now LOL – I’m the one who should be thanking you for your time and discussion, as I told you earlier: I love that on my blog

Here we go again :

>Following that logic you would need two HBAs on two different PCI slots but you don't. Do you trust more Emulex/QLogic hardware vendors than NICs

Actually you do need two HBA cards to ensure the maximum availability! This is common on any rack server (I hardly ever find anyone using one card with dual port). In our case, the blades, you may have noticed that we have one CIOv expansion card, which in turn mapped directly to Bay3 & Bay 4. Now, how would you have two cards in that case? The answer is to use Storage & IO Expansion Unit (SIO) which of course is specific here to IBM. In that case you will have two HBA cards. In the new HX5 (as you will see on my part-two) you also can have two HBA CIOv cards on dual unit blade. I personally do not trust any hardware vendor, and in fact I’ve seen a mission critical system in a large Telco goes down because of a “processor failure” not just an HBA card! It all boils down to what the customer is doing and what need to be achieved. I don’t see why a large bank or a telco would want to have this single point of failure in their system (whether it’s a NIC or HBA).

>At the end all your primaries could be located on one chassis!

You have a valid point here, and the solution for that is to have a 6 nodes per cluster. In that case you will always have three nodes on one chassis and two on the second. Check again what I’ve mentioned in my article regarding the number of hosts per cluster and how that can be determined by many factors.

>Point#7, I don't follow you with the BW thing, if a gigabit card can handle the traffic in an A/P mode, surely it can survive in an A/A mode BW wise!?

If you are having two A/A Nics for SC, FT, and vMotion, while your FT’ed VMs are generating lot’s of logging traffic, and in the same time you are doing lot’s vMotion across the wire, and copying files on your SC links, what would you expect in this situation? The FT network will be saturated or at least have an increased latency. This is why we have the 3 teamed NICs designed in A/P to serve the SC, vMotion and FT in such a way none of them would affect the other.

As per the SC on A/P NIC, this is my preference as mentioned earlier (just like the vCenter VM on a separate cluster) I don’t think there is a written best practice from VMware on this.

Best,
Hany

> you need that to survive a card failure (e.g. the LOM & the CFFh)
Following that logic you would need two HBAs on two different PCI slots but you don't. Do you trust more Emulex/QLogic hardware vendors than NICs

>The nodes of the HA cluster are distributed and spanned across the two chassis..
Yes I noticed that but HA primaries are promoted and demoted dynamically, put a primary in MM or just reboot it, and another primary is promoted, same goes for the 'active' primary. At the end all your primaries could be located on one chassis!

Point#3, 4, and 5, indeed it depends on customers requirements, policies, hardware restrictions, etc…

>Why are you assuming here that the end-users are remote? Why not local?
That wasn't specified in the diagram if I'm not wrong. I use to work for customers where datacenters were physically separated from end users. End users that were spanned World wide, thus WAN is an important aspect of a design in this kind of situation.

Point#7, I don't follow you with the BW thing, if a gigabit card can handle the traffic in an A/P mode, surely it can survive in an A/A mode BW wise!?

Regarding the SC, best in A/P mode, is it a VMware best practice thing?

Thx again for your time on this I just love to share ideas, and discussing on VMware designs
Didier

Many thanks for your detailed reply! I love it when my visitors do that

Although your reply is focused more on your specific preferences, my configurations are showing different set of options, and it all depends on customer requirement are decisions.

Here is my detailed reply anyways:
1 – Why don’t you team two different NICs? In Config 1 & Config2 you need that to survive a card failure (e.g. the LOM & the CFFh)
2 – That’s what we have here. The nodes of the HA cluster are distributed and spanned across the two chassis (the numbering and order are illustrated in Config1)
3 – Again, that’s your preference while some customers policies will strictly require physical segmentation.
4 – Same as point 3.
5 – It depends. In the Config3 we already have a FC SAN for production workload, while the iSCSI is a secondary SAN for supporting the Lab workloads.
6 – Why are you assuming here that the end-users are remote? Why not local?
7 – I don’t agree on that for certain networks like SC. I would always prefer to have the SC on Active/Standby in order to know exactly where it’s connecting in a case you are troubleshooting for example. Besides, just having an Active/Active links for everything can affect and saturate the BW for the networks. Like FT for example, it should be typically running without anything that can saturate its link, and/or increase the latency.

Thanks again for the reply

Here are my thoughts:
1- Do not bond (teaming) different NIC models/vendors like in conf#1 and #2.
2- As long as you have maximum 5 HA primaries do not installed more than 4 blades per chassis within a cluster.
3- I'm in favour of less cables, thus config#3 is my favourite in this case.
4- Use VLANs as much as you can, unless it is mandatory to have traffics physically separated.
5- Config#3 – I would have gone for 2*10GbE for iSCSI SAN.
6- Config#3 – You shape traffic to 2Gb for Prod1. But what's the size of the pipe at the perimeter of your DC? This is many times overlooked, server clients are often remote and the WAN pipe is a T3 (48Mbps), sometimes an OC-3 (155Mbps). I would say shape Prod1 bandwidth according WAN pipe.
7- I would have set active/active NICs in all 4 configs. That will give a bit of work for the network team

My network design would be 2*10GbE active/active doing everything with VLAN tagging and traffic shaping.
Too bad that the onboard NICs can't do 10GbE. We would have even less cables (and switches).

Looks like you're heading VCDX, aren't you

Cheers,
Dider