Fundamental Xen-AoE Properties

Aside from utilizing Xen and AoE in general there are a few guiding principles used in the design of a Xen-AoE cluster which makes the concept work so well:

• Diskless CPU nodes - This ensures that your data/virtual machine is not trapped inside any one physical CPU unit. If any particular CPU dies you can instantly restart the virtual machine on a different CPU. It also means there are no disks to fail and bring down the machine, no disks to add more heat to the chassis, and no disks to block airflow to the CPU's. The CPU nodes themselves are PXE booted. So you only have to change one disk image on the network to change how all of the systems boot. Then the virtual machines can be network booted or handed block devices from the Domain0.
• Gigabit ethernet - Ethernet has gotten fast enough and cheap enough that it has become feasible to essentially replace the traditional IDE cable in our servers with a piece of ethernet. This is great as it provides us the ability to combine lots of disk using a simple, fast, cheap network technology and make this disk available to many CPU's on a network.
• SATA disks - Although you could use SCSI the price/performance difference these days suggests that it makes more sense to simply buy more SATA disks and get more spindles than to spend a bunch of money on SCSI disks and controllers. "A recent study":http://www.cs.cmu.edu/~bianca/fast07.pdf has shown that there isn't even as much difference, if any at all, in reliability between SATA and SCSI or FC disks as we may have thought.
• RAID - This is an important part of ensuring that the cluster survives disk failures. With a Xen-AoE cluster you can use RAID 5 inside a disk node exporting block devices via AoE to prevent a single disk failure from causing data loss and then you can even RAID 1 two disk nodes together to ensure that even if a whole disk node failed (or the switch port it is connected to) you do not have any interruption in service. Plus this gets you lots of spindles for random accesses which is exactly what you need for performance as it is nearly always disk head seek time which slows things down.
• Zero single points of failure - If the CPU/RAM/motherboard/power supply dies, you restart the domain on a different CPU. If a disk dies RAID 5 covers it up. If a disk node dies RAID 1 gets it. If a switch dies, only the equipment on that switch goes down but the rest of the equipment stays up. Make sure you have each RAID 0 disk node pair connected to different switches. Make sure you divide up your CPU nodes too.
• Commodity hardware - The mobos, cases, cpus, switches, everything here is commodity. You can get spare parts in a matter of hours and swap them out yourselves with little more skill than the guy down at the corner mom and pop PC hardware store has. This keeps price down and maintainability up.