This blog had so much promise and then the professional life took a 90 degree turn meaning Virtualisation took a back seat.

But not anymore – 2012 is a new Year and with a new year comes some new vResolutions!

1) It’s time to get a new job – one that focuses on Virtualisation exclusively if possible. Its going to be challenging to catch up – but that’s what weekends and spare time is for!

2) Goal 1 allows Goal 2 to succeed – blog more about VMware and the Virtualisation space. There is room for one more blogger who could make positive contribution.

3) Get VCP5 and VCAP certified. Nothing beats learning a product set than studying and playing with the tools.

Let’s go!

However, our VMware infrastructure has been rock solid, I haven’t had to spend much time handholding the infrastructure. It just works and that includes undertaking major infrastructure changes like physically relocating half the SAN and infrastructure during office hours.

We are still on vSphere 4.0 update 1. Can’t wait to upgrade to vSphere 4.1 and see what that is all about.

Hopefully, when I get more time I will be able to post more articles.

We have been having problems over the last few weeks across a specific Nortel stack. Suffice to say the issue has been a pain to troubleshoot since we still can’t isolate it. There seems to be corruption in the backplane and we think it has to do with the MLT links that are configured on the stack.  

The problem that we are experiencing manifests itself as follows:  For example,  port 1/1 may have a problem communicating with a physical server on 4/40. However, port 1/4 has no issues. Their physical switching path means that all traffic has to go via the stack uplink/downlink cables. A similar issue could happen with other ports on other switches in the stack. But it is not consistent! Additionally you may have no issues accessing a server, but certain portions of that server are nigh on unresponsive. Example: Connecting to our SharePoint Site – everything on the site works, but a specific portion of the site just takes forever to return and SharePoint runs off 1 server!

Troubleshooting this issue has forced us to reboot our switch stack to see if the problem could be cleared.

However, a new problem reared it’s ugly head and is the reason for this post.

All ESX (HP DL 360 G5 /380 G5) servers in this site have 6 NICS:- 2 onboard and a Quad Port Intel Gigabit NIC card.

They are configured as follows:

——————————————————————————

pNic0 (On Board) – primary service console

pNic1 (On Board) – Guest VM traffic (3 VLANS)

pNic2 (Quad Port) – Guest VM traffic (3 VLANS)

pNic3 (Quad Port) – Guest VM traffic (3 VLANS)

pNic4 (Quad Port) – Fault Tolerant port

pNic5 (Quad Port) – vMotion Port / Secondary Service Console

——————————————————————————

The Guest VM traffic ports are bonded into a MultiLink Trunk (MLT) which is Nortel’s equivalent of Etherchannel.

 The vSwitch config for this MLT is as follows:

• Use IP Hash Load Balancing
• Notify Switches = Yes

 This configuration had no issues until the recent problem caused us to reboot the switch stack (which is of course a very rare occurrence)

 When the stack comes up, all NIC ports on the Quad Port NIC only negotiate at 100Mbps. Every ESX server exhibited the same problem. And this is consistent since we  had to restart our switch stack a few times over the course of the last few weeks!!

 The Quad Port NICs negotiating at 100Mbps of course causes a few problems:

• vMotion is impossible
• FT doesn’t work
• The MLT seems to be working but it is not! Every member of an MLT link must be at 1000Mbps however we have the on-board NIC negotiating at 1Gbps but the other two members negotiating at 100Mbps. This creates a very unique, unintended and nasty side-effect – all traffic on the MLT is sent out across the entire switch stack causing the switch to act like a hub! Running a WireShark trace on my desktop, I can see all VM guest traffic between each other on my PC (when all I should be seeing is broadcast, multicast and direct IP traffic). I found this out by mistake only as I was trying to troubleshoot some other issue!

The only way to fix this is to take each NIC and force it to auto detect its link speed using the vSphere client which then negotiates at 1Gbps. This then resolves that issue and the VM traffic disappears off my WireShark trace.

On the ESX 3.5 server, all traffic across the MLT and the separate vSwitches (Service Console and vMotion ports) was running through primary port of the MLT. Even if you disabled the vMotion interface on the Switch, we could still ping the vMotion interface since the it was somehow being routed via the MLT. The only way to resolve this was to reboot the ESX server for normal behaviour to resume.

 I don’t know what causes the Add-On Nics to only negotiate at 100Mbps but some good lessons have been learnt as a result of our troubles

 Lessons Learnt:

1)      Configure/Force all switch ports connecting to ESX servers to statically hardcoded to 1000Mbps Full Duplex

2)      Check networking and other config on each ESX server reboot or if another major system event occurs

3)      Pay attention to the events log on ESX servers. We did not notice that vMotions were failing since the vMotion ports were running at 100Mbps.

I would like to thank Michael McNamara for his help in trying to help me figure out our issues with our Nortel Switch stack. The post is here:

http://forums.networkinfrastructure.info/nortel-ethernet-switching/ers-5510e-switch-stack-is-now-running-like-a-hub!

If you guys have any clues as to what would cause some ports not to communicate with another port in a stack, I would appreciate it as we are still experiencing the issue and none of our checks can point to either a faulty switch or a stack cable.

The first deals with  AV deployment considerations for Virtual Desktops.

http://www.vmware.com/files/pdf/VMware-View-AntiVirusDeployment-WP-en.pdf

The second whitepaper deals with View and ThinApp Integration. This is a biggie and requires lots of planning so that you can ensure that you  implement ThinApp properly. I will write more on this soon.

http://vmware.com/files/pdf/VMware-IG-ViewThinApp-EN.pdf

Welcome to my new blog. As this is the first post of what I hope to be hundreds, I thought I would start by setting out the purpose of this blog to ensure that I stay true to my word in the future, as I am quite a forgetful bloke!

So what is the purpose of this blog ?

This blog will be concerned with all aspects of virtualization and my interactions with this technology. As I have been using VMware and most of my experience is based on their technology, the content of this blog, will at this point in time, be skewed towards their product set.

What will I hope to add to the VMware blog universe?

I hope to share my knowledge and experiences in my daily interactions in implementing and managing a virtual infrastructure at my place of work. I sense a shortage of knowledge and experiences with VMview and since we are starting to looking at this technology, I hope my experiences can help others while also allowing me to learn from you.

VRules:

This would be a good time for me to set down a few rules for me to follow:

1. Innovate, don’t duplicate. There are many of us sharing our knowledge. Most of my knowledge has been gained (and I’m still learning daily) from some very good bloggers. So if I am aware of a good post, there is no point me duplicating their effort. I might as well link to the original post, unless I feel I have something unique to add.
2. Consolidate, when necessary. In my travails on this road of virtualization, I’ve found that it is sometimes difficult to find a single source of knowledge on a certain specific topic or action. This is a common occurrence for me on the Experts-Exchange forum where I try to help others with their virtualization problems This blog was mainly created as a result of the many questions I’ve seen being asked time and again. I hope to have create very in-depth “vRepository” on this blog which I hope will help anybody depending on where they are on their road to virtualization.
3. Learn
4. Try not to add a “v” to everything, e.g. “vEverything”  

vKudos:

In this, my manifesto post, I want to take the opportunity to thank some very special people in the VMware universe who have helped me greatly by sharing their knowledge selflessly with all of us over the last few years.

 In alphabetical order:-

• Alan Renouf (http://www.virtual-al.net)
• Chad Sakac (http://virtualgeek.typepad.com)
• Duncan Epping (http://www.yellow-bricks.com)
• Eric Sloof (http://www.ntpro.nl/blog/)
• Forbes Guthrie (http://www.vreference.com)
• Jason Boche (http://boche.net/blog)
• Mike Laverick (http://www.rtfm-ed.co.uk)
• Scott Lowe (http://blog.scottlowe.org/)
• Simon Long (http://www.simonlong.co.uk/blog)
• William Lam (http://www.engr.ucsb.edu/~duonglt/vmware)

Companies I admire

• VMware (www.vmware.com)
• Microsoft (www.microsoft.com) but not for Hyper-V!
• Cisco (www.cisco.com)
• EMC (www.emc.com)
• DataDomain (www.datadomain.com) Now an EMC subsidiary
• Nandos (www.nandos.co.za)

Here’s to living up to my own hype!

Signed

MkH