Disk Server Hardware

Feel free to shop around but I found some good deals on Newegg. The prices here have likely changed by the time you read this. The url may have changed also. Google the model number for current vendors and prices.

We want a lot of disk and we want it to be reliable. And we want two of these so we can mirror across them. Here's what I used:

• SUPERMICRO X7DVL-E-O Dual 771 Intel 5000V ATX Server Motherboard

  http://www.newegg.com/Product/Product.aspx?Item=N82E16813182097&Tpk=X7DVL-E

  $306.99

  The led panel ribbon cable and 8 pin EPS12V power cable that comes with the PSU won't reach the connector on the mobo. You can use the 4 pin EPS12V and just stick it in to one side of the 8 pin connected on the mobo. Also the ribbon cable is unneccessary. The chassis came with an "octopus" which is a bunch of individual colored wires to hook up all of the LED's. It is long enough to reach.

  The X7DVL-E-O works well for me but also consider the X7DB8 without onboard scsi. It has two additional memory slots, the PCI hole is 256MB instead of 1GB, it is the board designed to go into the we are using chassis and fits perfectly, and it has got extra fan inputs so you can monitor those from the OS.

  Also seriously consider buying the IPMI board for whatever mobo you get. It can really help with remote administration.

• Intel Xeon 5110 Woodcrest 1.6GHz 4M shared L2 Cache Socket 771 Active or 1U Processor - Retail

  http://www.newegg.com/Product/Product.aspx?Item=N82E16819117081

  $196.99

  This CPU works with the X7DVL-E-O motherboard. I only bought one. We generally won't need a whole lot of CPU power to run the disk server.

• Kingston 2GB (2 x 1GB) 240-Pin DDR2 FB-DIMM ECC Fully Buffered DDR2 667 (PC2 5300) Dual Channel Kit Server Memory - Retail

  http://www.newegg.com/Product/Product.aspx?Item=N82E16820134330

  $106.99

  You may want to load these boxes out with more RAM. 2G here seemed sufficient for my purposes, at least initially. It all ends up being used as a disk cache. More RAM here won't help you for long sequential reads/writes but can help tremendously in reducing disk head seeks for database applications. Between the RAM in the CPU server and RAM in the disk server you may even be able to cache your whole database.

• SUPERMICRO CSE-932T-R760B Black 3U Server Case 760W Triple-Redundant - Retail

  http://www.newegg.com/Product/Product.asp?Item=N82E16811152069

  $739.99

  This case is awesome and really makes the disk server possible. You can fit 15 disks into these hot-swap bays. You want lots of spindles for lots of disk performance. Don't you dare fail to load this thing up with all 15 disks if you want performance. Buy smaller size disks if you have to. It's all about spindles. Note that this thing has triple-redundant power supplies which is really great. Not only does this thing need the power but you don't want a single power supply failure to bring down your disk. You also want to make sure this whole setup is on a UPS. You are going to be caching a lot of data in the RAM in this thing and you don't want it going down hard. The datacenter where I have my machines located has two circuits to each cabinet and I split the load across them. It also has a huge battery backup and twin diesel generators.

• Seagate Barracuda 7200.11 ST3750330AS 750GB 7200 RPM 32MB Cache SATA 3.0Gb/s Hard Drive - OEM

  http://www.newegg.com/Product/Product.aspx?Item=N82E16822148298

  $169.99 * 15 = 2549.85

  This drive and the Adaptec 31605 both do SATA II which is 3.0Gb but the Seagate drives come jumpered to only do 1.5Gb SATA-I for backwards compatibility reasons. If you want 3.0Gb (and why not since the hardware supports it) you need to remove a jumper from the drive.

• LSI Logic SAS3081E-R - storage controller (RAID) - SATA-300 / SAS - PCI Exp

  http://www.cdw.com/shop/products/default.aspx?EDC=1312454

  $249.99 * 2 = 499.98

  Nobody had the LSI Logic. Nor the Areca controller that comes highly recommended. I ended up going with the Adaptec 31605 which seems to work well. Make sure you get the appropriate cables. Our Adaptec came with cabling to connect 8 HD's and I needed more which added a little to the price.

DISK SERVER TOTAL 4400.79 each * 2 for redundancy and performance = 8801.58

CPU Server Hardware

• SUPERMICRO MBD-H8DME-2-O Dual 1207 NVIDIA nForce Professional 3600 Extended ATX Server Motherboard - Retail

  http://www.newegg.com/Product/Product.aspx?Item=N82E16813182108

  $359.99

  I went Intel in the disk server but I like AMD for the cpu server. Opteron is an awesome chip and I really like the onboard memory controller and the fact that we can have a separate memory bank for each CPU. This lets us load these things up with a ton of RAM. You really want to put as much RAM in each one as you can as that allows us to get the maximum number of virtual machines per U of rack space. It is unlikely that you are going to max out the CPU's unless you are running some serious number crunching apps.

• Dynatron A87G CPU cooler

  http://www.performance-pcs.com/catalog/index.php?main_page=product_info&manufacturers_id=154&products_id=22633

  $24.95 * 2 = $49.90

  It's rather hard to find a 1u cpu cooler with the 3.5" mounting hole spacing. This is the only one I could come up with and so far it seems like a quality piece of equipment. They come with thermal compound so it isn't necessary to buy your own.

• Kingston 4GB(2 x 2GB) 240-Pin DDR2 SDRAM ECC Registered DDR2 667 (PC2 5300) Dual Channel Kit Server Memory - Retail

  http://www.newegg.com/Product/Product.aspx?Item=N82E16820134329 (32G total)

  144.99 * 8 = 1159.92

  This is a critical component of the whole system. Do not cheap out on memory! Definitely get ECC and registered. And get as much of it as you can. The motherboard could actually handle 64G of memory but you have to step down to a slower speed and that seems like an awful lot of eggs in one basket if you actually made 64 1G virtual machines. Next best is 32G so that's what I went with. Each of these packages contains 2 DIMMS of 2G each. That is 4G per package. 8 packages makes for 32G in 16 individual DIMMS which maxes out the number of DIMM slots in our motherboard with 8 per CPU.

• AMD Opteron 2218 Santa Rosa 2.6GHz 2 x 1MB L2 Cache Socket F 95W Dual-Core Processor - OEM

  http://www.newegg.com/Product/Product.aspx?Item=N82E16819105034

  379.99 * 2 = 759.98

• hec RA164A00 1.0 mm Thickness 1U Server Case 1 External 5.25" Drive Bays - Retail

  http://www.newegg.com/Product/Product.aspx?Item=N82E16811121005

  $124.99

  Since this is only going to contain a motherboard and PSU and no drives get the smallest case you can. I haven't yet been able to find a decent 1u case that fits the mobo and PSU that doesn't have drive bays. So I got this case and just leave the front part of it where the drives go empty. It is better for cooling if we had a smaller case and didn't have to suck air through the drive bay area where it meets unnecessary resistance. If you have a better case please let me know. At the moment newegg has a combo deal with one of these cases and a 400W PSU. DO NOT get this deal. The equipment we have spec'd out requires at least 430W of power plus you want some safety margin. Instead get a good 500W EPS12V 1u power supply like the one I list below.

• 500 Watt 1U EPS12V PFC power supply - Sparkle FSP500-80BU

  http://www.rackmountpro.com/product.aspx?prodid=203&catid=211

  $117.00

CPU SERVER TOTAL 2535.95 each * 2 for redundancy and performance = 5071.90

So if you really want to go all out and have a zero single point of failure system we're looking at nearly $15k in hardware. That sounds like a lot but if you have a bunch of systems to virtualize and considering that this setup gets you 64 machines worth (if we assume 1G per machine which also leaves you with 175G of disk per machine) the actual cost per machine comes out to $216 per virtual machine. The cost of the disk system is actually a very large percentage of the system and 175G per virtual machine is a lot. If you add more cpu servers and allocate 50G per virtual machine you can amortize the cost of that expensive disk system over a lot more virtual machines and get your cost per virtual machine down to closer to $100. In my latest deployment each of these virtual machines is replacing a $2500 Dell server. $2.500 times 64 minus the $20,000 for the hardware investment means my client is saving $140000 over deploying Dell boxes! Not to mention everything fitting in 8u instead of 64u.

Networking hardware

If you don't already have a couple of quality gigabit ethernet switches you will need to get some. I would not recommend anything other than Cisco or HP. Netgear, Linksys, and pretty much anything you can buy at Fry's is RIGHT OUT. At the very least any switch used in this setup needs to be gigabit, support jumbo frames, have hardware flow control, be capable of VLAN, 802.3ad channel bonding, and who knows what else. The Cisco Catalyst and HP Procurve series do these things. I really like the Cisco 3560G.