Spanning-tree root switch determination

STP, RSTP and MSTP are well known solutions for both loop detection and redundancy. Spanning-tree will determine the root switch and from that point it will calculate the link cost to the device. After the determination of the shortest path it will disable the more costly paths.

But this won’t always result in the best network topology. So how is this root switch appointed? Root switches are chosen because of two things their priority settings and after that who has the lowest MAC-address. This is why if you just let it determine the paths it won’t always be the best topology.

HP ProCurve (and maybe also other) switches are configured with the default priority 8. This way when you enable STP on them they just look at the lowest MAC-address. To optimize the topology you might want to set the priority of your main switch lower as the default priority (8). This way your switch is chosen as root switch and all paths are calculated from there.

The way to change the priority on HP ProCurve switches from CLI goes as follows:

[hostname]#config
[hostname](config)#spanning-tree priority [n]
[hostname](config)#write mem

[n] can be anything from 0 to 15. 0 being the highest priority and 15 being the lowest.

Beacon probing resulting in excessive broadcasts

Not so long ago I posted a link to a VMWare blog in which beacon probing was demystified. This article stated that you only should use beacon probing when there is no link state tracking on the physical switches and you could consider beacon probing as a nice software solution for replacing it.

Well we’ve got our ESX environment set up by a local supplier and they advised us to use beacon probing instead of link state tracking. But for some reason beginning from that moment I got major events from my ProCurve switches stating excessive broadcasts. This wasn’t often, but especially during peak hours I was getting this notification.

When I started sniffing the network packets on the uplink of the switches I noticed what kind of packets it were. It was an almost continuous flow of RARP packets coming from the ESX servers. RARP packets are meant as MAC address table updates for switches. This way when a node is suddenly available on a different MAC address the switch already knows the new path. This is also what happens when a virtual switch detects a link is not functional. It will switch the uplink and notify the switches. When beacon probing isn’t working as expected ESX constantly thinks the uplink isn’t functional so it is constantly switching the uplink and as a result constantly sending out RARP packets.

While you can define both the notify switches parameter and the network failure detection, it isn’t good to just put notify switches to ‘No’. This way the failover is still constantly changing uplinks and this can result in errors (in my case timeouts with TFTP). The real problem was the failure detection. Beacon probing for some reason just didn’t work in our environment. When I changed the failure detection to ‘Link Status only’ all the RARP packets disappeared and my excessive broadcasts were gone.

In my case just link status only is sufficient, but I can imagine there are cases where you would want to use beacon probing. If you enable beacon probing and this results in excessive broadcasts (or just more broadcasts) I do advise to look if you could find those RARP packets. This can indicate that beacon probing is just not working correctly in your environment.

I want to thank Scott Lowe for giving me a push in the right direction.

Connecting USB HD in ESX for back-up

As the new year begun some servers were due to be moved to another place more permanent burial place. Our ‘expensive’ SAN storage space is too small to keep the old servers on it so it was time to move them to a USB harddisk. But I had no idea how I could attach it to the ESX. Well for those of you who want to know for the future here is how I did it.

Well first of all I hooked the USB drive onto the ESX server (HP Proliant DL380g5). When I logged onto the console the first thing I did was execute cat /var/log/messages. This gave me following lines:

kernel: Initializing USB Mass Storage driver...
kernel: usb.c: registered new driver usb-storage
kernel: scsi3 : SCSI emulation for USB Mass Storage devices
kernel: Vendor: WD Model: 10EAVS External Rev: 1.05
kernel: Type: Direct-Access ANSI SCSI revision: 02


And a couple of lines further:

kernel: VMWARE: Unique Device attached as scsi disk sdc at scsi3, channel 0, id 0, lun 0
kernel: Attached scsi disk sdc at scsi3, channel 0, id 0, lun 0
kernel: SCSI device sdc: 1953525168 512-byte hdwr sectors (1000205 MB)
kernel: sdc: sdc1
kernel: USB Mass Storage support registered.


From this information I concluded that the USB drive was recognised and was emulated as SCSI, hence the sdc. My harddrive already contained a NTFS partition, but ESX doesn’t have the NTFS drivers. Thats why it will mount the partition using FAT32. This would pose a problem because FAT32 doesn’t support files bigger as 4Gb and my vmdk’s are much bigger. So I decided to format the disk with EXT2 so I could move the files.

I first deleted all the existing partitions with fdisk and after that I created a new primary partition with partition type Linux. After formatting it with the command mke2fs /dev/sdc1 I had a working Linux partition. If you by any chance would want to have an EXT3 disk you could use mke2fs -j /dev/sdc1.

So now I got a working disk and a working partition. First I needed a mount point before I could actually mount the USB drive. So I made a directory in /mnt called ‘usb’ with mkdir /mnt/usb and I executed the following command:

mount /dev/sdc1 /mnt/usb

Now I could access the disk at /mnt/usb and copy all the old VM’s from /vmfs/volumes/[datastorename]. If you want to copy entire directory’s make sure you use the '-R' parameter with cp.

Switch meshing ProCurve switches unless…

… you are using those switches for routing. Great so the only option for improving my networking environment for load balancing my ESX environment was crushed by these two HP documents: Switch Meshing and LAN Aggregation Through Switch Meshing.

Apparently there is a whole list of requirements for switch meshing. This list is most clearly defined in Switch Meshing (starting from page to 5 and ending at page 7). Just a short beginning of the list:

• A meshed switch can have some ports in the meshed domain and other ports outside the meshed domain. That is, ports within the meshed domain must be configured for meshing, while ports outside the meshed domain must not be configured for meshing.
• Meshed links must be point-to-point switch links.
• On any switch, all meshed ports belong to the same mesh domain.
• A switch can have up to 24 meshed ports.
• A mesh domain can include up to 12 switches.
• Up to five inter-switch, meshed hops are allowed in the path connecting two nodes through a switch mesh domain. A path of six or more meshed hops between two nodes is unusable. However, in most mesh topologies, there would normally be a shorter path available, and paths of five hops or fewer through the same mesh will continue to operate.
• ….

I have four HP ProCurve 5300 series switches and they all do routing. Switch meshing was my only option left after exploring NIC teaming articles from Scott Lowe and Lukas Kubin. Those articles describe perfectly how you could do NIC teaming and VLAN trunking on one ProCurve switch, but not how to do it with two of them. After some searching the only option to accomplish what I wanted was a HP patented technique called switch meshing.

So this is not an article explaining how to configure switch meshing for use with ESX, but to warn people for all the requirements. If your environment complies with all the requirements made for switch meshing the two documents named above will do sufficiently as far as it goes how to configure and what switch meshing exactly is.

If I ever end up in an environment where I am able to configure switch meshing I will certainly post how I did it. During my search for what switch meshing was I found out about another technology called XRRP. With that technology at least I can make a really good failover for my gateways, but later more on that.

Hypervisor war in virtual land

Not so long ago (14th of December) Mohammed Fawzi posted a blogpost called Hyper-V vs VMWare. In this post he wanted to put together a comparison of Microsoft’s Hyper-V and VMWare’s ESX (Enterprise). Although I don’t think he accomplished what he wanted to accomplish with that post (comparing the two hypervisors), he managed to accomplish something. And that something is what is not in the article itself, but in the comments beneath it. It started a really firm, mostly technical and sometimes even emotional, response of the community that he was comparing apples with grapes and doing it all wrong.

The replies didn’t even stop at Mohammed Fawzi’s blogpost, but even resulted in new articles stating the faults he made when he compared the two. The writers (Scott Lowe and Jason Boche) of those articles are writers whose blogposts I read with great care and are (in my opinion) really interesting.

This rumble in hypervisor land remembered me at the console wars where fanboys were flaming that their console is the best and the rest of the gaming consoles were rubbish. Although the comments and articles really don’t represent the behaviour of those fanboys flaming about their favourite console, I recognised some of the emotions that appeared and how heavily the community reacted on it.

I myself work a lot with HP equipment and use VMWare ESX as my hypervisor and I think it was the best solution in my situation. Just as with gaming consoles I believe the best hypervisor is the one that fits best in your situation. And above all I like the idea of more than one choice in hypervisor land. It will keep everyone awake and new functions will be developed to have that one unique selling point.

What the community doesn’t need is a break because there isn’t a trustworthy datasource. What the community does need is a factual datasheet comparing the two so that when we have to make that choice or when we have to write that business case we all have the facts. I like to know what Microsoft is doing and how it solves problems with their solution as well as how VMWare does those things. I hope I can learn from that and one day I can use that to my advantage and solve a problem or make the correct decision using all my knowledge.

The only question that remains is: Who is going to make that factual datasheet that is trustworthy?

Making inactive storage active in VirtualCenter

Today a power outage hit our datacenter for several hours. This was too long for the UPS so everything was shut down since we don’t have any backup generator or something like that. However when power came back on the ESX hosts booted faster as our storage controllers. I logged into VirtualCenter and I saw that none of my virtual machines were powered on automatically. They were greyed out with “(inaccessible)” behind the names. I googled, but couldn’t find anything related to my problem.

When I looked at the ESX hosts they weren’t showing my SAN datastores. So I went to my commandview EVA server and looked if the LUN’s were there. In fact first they weren’t there, but after a while they finally appeared. I turned back to my VirtualCenter and looked if the datastores also would appear automatically again, but that apparently was too much to ask. Not knowing what to do I started looking around what I could do. I looked at the storage adapter configuration of one of the ESX hosts and the HBA’s were nicely listed, but still no LUN’s. Finally just before I wanted to reboot the entire host I discovered I could rescan the HBA with right clicking on the HBA and then click “Rescan”. After the rescan was finished my datastores returned and I was back in business.

For the people who like to do everything at the commandline you can use the command:
esxcfg-rescan vmhba[0...n]