Networking issues with Hyper-V

Posted: August 28, 2012 in Failover Clustering, Hyper-V
Tags: , , , ,

Just recently I was running into some very strange issues with my Hyper-V Cluster. we have numerous NICs to allow for redundancy and better throughput for our clustered environment. We have a total of 11 NICs on our server to cater for Guest LAN, iSCSI, guest iSCSI (as we present SAN storage directly to the Guests), HeartBeat and Live Migration. So, I was getting a little concerned when I was performing maintenance and/or “Live Migrations” and my VMs were losing network connectivity both on LAN connections and iSCSI connections which was causing major issues with my VMs needing external storage as this was mostly SQL or file shares. So time to start troubleshooting.

First stop would obviously be the NICs on the hosts and I spent some time checking all the NICs and ensuring they are cabled correctly. Not the world’s easiest job considering the amount of network cables we have in our cabinets. We have numerous Hyper-V nodes, all with 11 NICs excluding Management and iDRAC. So we are talking hundreds of cables. However, I trudged through it and put a check mark next to it – no issues there.

So, several hours and days later and cursing and swearing all the network cables, velcro and cable ties in my cabinets, it was time to move on. So, it was NOT a physical network issue. Time to give a little further. My next stop was the Hypervisor layer.

I start diving through the Hypervisor and I found “gold” here, (I actually mean the cause of my issue). Enter Hyper-V Networking. Please allow me to clarify this. The networking was not the issue but what someone/something had done, the truth here will never be known. I found that I had duplicate Virtual Network created.

So, for each physical NIC i had in the server and was using for the Virtualization, I had 2 networks. 1 connected to the “External Network” and here is the kicker, 1 connected to the “PRIVATE” network. So with a little troubleshooting and deal of understanding Virtual Switches, I was able to return my cluster to it’s former 100% redundant glory. After removing ALL the private networks and there were many on all my Nodes, I was able to Live Migrate and patch/maintain to my heart’s content.

Gotta love Virtualization.

Hopefully this will help someone else as well.

The lesson here is to keep a good eye on your Virtual Environment and always check the basics.

Follow me.

facebook-small322252222 twitter-small322252222

MCC11_Logo_Horizontal_2-color_thumb_

Advertisements
Comments

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s