Friday, October 8, 2010

On building SANs from scratch

One of my exciting projects has been building a new SAN from scratch. I've gone through a lot of design ideas, but in the end settled on Nexenta Enterprise. In fact, I built two appliances: a single unit for our DR site and a dual-controller HA Cluster with redundant interconnects for the primary site. Here's what I got for the main unit:

2 Supermicro SC823TQ-R500LPB enclosures for the controllers. They have 7 low profile slots in the back, and 6 3.5" disk bays in the front, and fit in 2U.
2 Supermicro X8DTH-iF motherboards. Dual socket Nehalem, 12 ram slots and 7 PCIe 8x slots are perfect for what I need, and I opted not to get onboard SAS controllers. The "i" in the name stands for a built-in IPMI controller, so I don't have to drive all the way to the datacenter if something comes up.
2 Xeon E5620 CPUs. Same clock speed as E5530, but cheaper and with more cache. At this time I don't think I need dual CPUs in the controllers.
2 2U passive heatsinks. Don't forget these - the parts are specific to the enclosures, so make sure to get the correct part number.
2 LSI 9200-8e SAS controllers. 48Gbps of combined theoretical throughput in two external mini-SAS ports should be plenty.
2 QLogic 2460 controllers in order to export the storage to my servers via Fibre Channel. Interesting observation: unlike with hard drives, vendor-specific FC cards from Dell or IBM are much cheaper than "retail" ones. While this may be borderline unsupported, the chips are the same and it generally makes no difference.
4 3x4Gb DDR3 ECC 1333Mhz RAM sticks. 24Gb per host should be plenty for now.
4 cheap 160Gb 3.5" 7.2k rpm drives from Western Digital for mirrored boot drives.

Here comes the interesting part: the storage enclosure and the drives. In order to get more spindles in less space, I opted for a 2U unit with 24 2.5" drive bays. The quest of choosing the enclosure has been kind of a bummer: the cheaper Supermicro SC216 that I originally wanted to get does not support SATA to SAS interposers (drive mounts are too short and there are no screw holes for the interposer-offset drives), and I needed those for the SATA SSD's I was planning to use for read and write cache in dual-head configuration. SAS SSD's weren't an option because they would cost more than everything else combined.

After some googling, I have found the recently released LSI-620J unit, which is double the price of the Supermicro one, but supports 6Gbps SAS on the backplane, and doesn't need to be assembled manually. Also got a second SAS expander for it, as well as rails - neither of those are included by default.

Aside from that, all I really needed for the base setup were 4 3ft SFF-8088 miniSAS cables.

For the drives, I ordered 17 600Gb Savvio 10k.4's - they also have native 6Gbps SAS support. I originally planned to get similar Toshiba's because of higher performance, but got a really good deal on the Savvio's, so I couldn't resist.

Originally, the plan was to get four 160Gb Intel X25M SSDs for read cache, and two 32Gb X25E's for write cache, but I have decided to wait a little: these SSDs are sold out in a lot of places, and there are rumors that Intel is about to release bigger and faster SSD drives, hopefully at a lower price.

And now, eye candy:







No comments:

Post a Comment