How to build a 16TB box for less than $5000

[WinstonTJ]

** Disclaimer - the RAID calculators will tell you that this is 18TB but the actual usable space of 8x 3TB drives in RAID6 is only 16TB.


$3600 for 9x 3TB drives with tax & shipping

$250 Dell Precision 690 workstation
Just buy anything complete with at least one CPU and all the caddys

$200 for CPU and heatsink upgrades for P690
Plan on buying two Xeon x5160 CPUs and at least one FULL heatsink, not the cheap kind with the holes in the middle (look at pictures to follow)

$150 for 16GB of DDR2 RAM
Get 2Rx8 and make sure it's fully buffered not just ECC

$200 for a Intel Pro 1000 PT 4-port NIC
You can go cheap-seats and get two of the MT cards (PCIX64) and bond 8 gigabit ports but the PT will bond better and sustained speeds will be better than the MT NICs.

$150 for a LSI MegaRAID 8708EM2 8-PORT SAS RAID CARD W/ BBU
There are others out there but for the money I've used this quite a bit. Make sure it comes with the battery backup and test the audible alarm as well as the battery charge.

$150 for a 128gb SSD (system cache)
Anything works. Cheaper the better because you will blow through these with read & writes. Plan on replacing it in 20-30 months.

$60 for a 64gb SSD for system OS
You can also run the OS off a USB flash drive if you want. Does not need to be a SSD


$100 for other junk like SATA cables, thermal compound, the $30 fancy dell stand for these things so it won't tip over and for a few of the 5.25" to 3.5" drive bay converters that are snap-in from dell.

$0 for the OS (FreeNAS)

Pictures and how-to to follow.

** NOTE: 9x drives required so that there is a local spare. If you are considering building something like this I hope you are smart enough to go with RAID6 over RAID5 and that you are smart enough to have a spare drive on hand. Rebuilding a single 3TB drive is enough to put the others at risk just with the sustained reads.

I'll post a bunch of pictures and I'm happy to answer questions.

 

[WinstonTJ]

I started to take and post pictures but my Chief Technology Officer had other plans so I guess that means give her a bit of attention and then head out for some non-computer related fun this evening.

^^ that's the box I got on eBay. Was about $180 but it needs parts and stuff that I either have (because I build a bunch of these) or will order from Dell. If you look at the box it's missing the bottom feet to hold the Dell stand. This is a pretty beat up box and I'm not happy. In hind sight I would have rather paid more for something in better condition. I'm going to use this for the build and initial testing/burn-in because I have a deliverable but then I'm going to need to swap everything over to a better chassis. This one seemed too good to be true and it ended up being beat to crap.

Have these around also. The blue caddys snap into the chassis and then the metal trays slide into and snap into the chassis. They are actually worth it and I end up buying them a lot and taking tin snips to them and using them in other boxes like HP.

I'll take and post more pictures but my CTO is being annoying and it seems that I have an appointment at the pub.

Couple more pics before I head out. I bought this thing on eBay so it's pretty sight unseen. I will need to burn it in and do load testing for a solid 3-4 days to make sure it's OK before I put it into production or deliver it to a client.

^^ Initial opening of the box. Nothing special but good news is it comes with pretty much everything. It's missing a bracket (which I will show later) that just keeps wires & cables in line. Not the end of the world. It has all the blue caddys.

Looking into the wire harness coming off the PSU I see that the P4 (dell specific) power cable is missing. Will need to order one of those up on eBay or from dell (or might have one). That P4 connector will have 3-4 SATA power connectors. Most modern HDD's can take a MOLEX to SATA power adapter converter (see below) but they omit the low power voltage which some spinny HDD's require. I only use the MOLEX to SATA converters on SSD's because they draw lower power and it's never been an issue.

^^^ It has a video card. No idea if it works nor do I care what it's specs are - as long as it works enough for the initial install. The FreeNAS OS (BSD OS) has it's own web GUI that's awesome so after the initial install you can pull the video card and run it 'headless' to save power. I would only recommend running headless if you know how to use a serial cable or telnet or SSH incase something breaks - but you can pull the card and run it. Dell's most recent BIOS allows you to run the machine totally without a video card.

Has one CPU heatsink. Maybe when I pull the CPU I'll get a decent one but chances are the seller has something that's junk in there. This is the CPU that you want for these - one that's solid in the middle. If you look where the white sticker is there normally is a diamond or oval or rectangle cut-out or stamped out of the heatsink metal that goes down to the CPU. You want the solid heatsink because they will keep the CPU cooler during sustained copying or writing to the database. It isn't that big of a deal but since the heatsinks are before the memory the cooler the heatsinks (spread more heat so more of lower temperature vs. less of a higher temp) the cooler the temps will be as they pass over the memory sticks.

Pull the memory fan and see the seller gave me junk. Two sticks of something cheap and slow. I'll be throwing that memory into the junk pile and buying faster DDR2 with full heatsink sleeves on the sticks of RAM.

Another interesting thing to note about this box is that the memory fan has double sided foam stuck to it. You can see the little tabs next to the holes on the photo with the CPU heatsink. No idea why that's the way it is but between that and the missing legs I wonder if this was in an industrial application. This thing was used and abused so it will NOT be going to a client. I'll build and burn-in the HDD's and RAID cards then switch them over to another machine that I buy. At least this will give me a head start for my deliverable and if worst comes to worst I'll use this chassis as my own personal media/music/data server but based on the initial impression I'd never give this to a client.

 

[bluematrix]

or you could just get this this? why bother?

 

[WinstonTJ]

Quote from bluematrix:or you could just get this this? why bother?

Well...

What you posted is a cancelled one-off from Dell and does not come with hard drives.

I'm not trying to sell 16TB arrays. I am simply documenting the purchase and build and performance process so that others can do exactly the same (if they want) and not pay crazy money for things on eBay that don't come with hard drives.

 

[FCXoptions]

What in the world do you need 16TB for? That is a lot of videos or information

 

[dyson]

Re: How to build a 26TB box for less than $3000

I built a similar server back in August/September 2011 (before the Thailand flooding/hard drive shortage). I visited Newegg to compile a list based on the same specifications and current pricing. If you plan the build (and buy parts over a couple of months), you can also take advantage of coupons and rebates.

A few substitutions are needed because of deactivated items. Example: I am using an Intel Core i3-2100T (35W) CPU which is no longer available. Energy consumpsion was important since I am running the server 24/7 and did not want a spike in energy bills. Current energy usage is between 110W and 140W.

I am also using Hitachi 5K3000 (0F12117) 2TB drives. At the time they were going for $49 ($69 with $20 rebate). They are no longer available as Hitachi is now HGST (Western Digital).

I am using Linux (Ubuntu) as my OS with ZFS as the file system. I prefer the software RAID (zfs) vs hardware for compatibility reasons.

Here is the parts list with Newegg item numbers. Total cost is $2,968.77.

16 x SAMSUNG EcoGreen F4 HD204UI 2TB 32MB Cache SATA 3.0Gb/s 3.5" Internal Hard Drive -Bare Drive
$119.99
FREE SHIPPING
http://www.newegg.com/Product/Product.aspx?Item=N82E16822152245
Notes: I have the drives split as 8 x 2 (LSI Controller), 6 x 2 (Motherboard Chipset SATA ports), and 2 x 2 (Motherboard Marvell SATA ports). The 8 x 2 is used for my media server, the 6 x 2 is for work and business, and the 2 x 2 is a temporary workspace.

1 x Fractal Design Define XL Black Pearl w/ USB 3.0 ATX Full Tower Silent PC Computer Case
$143.99
$14.99 (Shipping)
http://www.newegg.com/Product/Product.aspx?Item=N82E16811352017

1 x CORSAIR Professional Series HX650 (CMPSU-650HX) 650W ATX12V v2.2 / EPS12V 2.91 SLI Ready 80 PLUS BRONZE Certified Modular Active PFC Power Supply
$89.99
FREE SHIPPING
http://www.newegg.com/Product/Product.aspx?Item=N82E16817139012

1 x GIGABYTE GA-Z68XP-UD3 LGA 1155 Intel Z68 HDMI SATA 6Gb/s USB 3.0 ATX Intel Motherboard
$119.99
$7.87 (Shipping)
http://www.newegg.com/Product/Product.aspx?Item=N82E16813128512

1 x Intel Core i3-2120T Sandy Bridge 2.6GHz LGA 1155 35W Dual-Core Desktop Processor Intel HD Graphics 2000 BX80623I32120T
$134.99
FREE SHIPPING
http://www.newegg.com/Product/Product.aspx?Item=N82E16819115094
Notes: I wanted a low power setup. Processor is only 35W.

1 x COOLER MASTER Hyper 212 Plus RR-B10-212P-G1 "Heatpipe Direct Contact" Long Life Sleeve 120mm CPU Cooler Compatible Intel Core i5 & Intel Core i7
$19.99
FREE SHIPPING
http://www.newegg.com/Product/Product.aspx?Item=N82E16835103065

1 x G.SKILL Ripjaws Series 8GB (2 x 4GB) 240-Pin DDR3 SDRAM DDR3 1600 (PC3 12800) Desktop Memory Model F3-12800CL9D-8GBRL
$45.99
FREE SHIPPING
http://www.newegg.com/Product/Product.aspx?Item=N82E16820231314
Notes: I find 8 GB to be perfect. Memory is cheap if you want 16 GB.

1 x ASUS 8400GS-1GD3-SL GeForce 8400 GS 1GB 64-bit DDR3 PCI Express 2.0 x16 HDCP Ready Video Card
$32.99
$2.99 (Shipping)
http://www.newegg.com/Product/Product.aspx?Item=N82E16814121484
Notes: Anything cheap will work since graphics are not needed; the monitor is off 99.9% of the time.

1 x LSI Internal SATA/SAS 9211-8i 6Gb/s PCI-Express 2.0 RAID Controller Card, Single
$252.99
$6.98 (Shipping)
http://www.newegg.com/Product/Product.aspx?Item=N82E16816118112
Notes: If you wait for a Newegg sale, you can get this for around $219.

2 x 3WARE Cable Multi-lane Internal Cable (SFF-8087)
$18.60
Free Shipping
http://www.amazon.com/3WARE-Cable-Multi-lane-Internal-SFF-8087/dp/B000FBYS2U

1 x Patriot Xporter XT Boost 16GB Flash Drive (USB 2.0 Portable) Model PEF16GUSB
$20.99
Free Shipping
http://www.newegg.com/Product/Product.aspx?Item=N82E16820220253
Notes: Linux OS is installed to this USB flash drive.

1 x APC Back-UPS ES BE550G 550 VA 330 Watts (4) NEMA 5-15R (Battery Backup) (4) NEMA 5-15R (Surge Protection) Outlets UPS
$66.99
FREE SHIPPING
http://www.newegg.com/Product/Product.aspx?Item=N82E16842101343
Notes: If power drops, OS will power off machine safely. Otherwise about 10 to 15 minutes backup power.

Other cables and accessories purchased at Newegg/Amazon/Monoprice for approximately $50.

With some price searching and substitutions, you can get the price below $2500. With fewer drives, you can get the price below $2000.

 

[Landis82]

I LOVE these kinds of threads on ET.

This is the initial reason why I came to Elitetrader in the first place years ago... The HARDWARE Section has always been filled with tons of VALUE over the years and it still keeps me coming back!

Very cool stuff.

 

[377OHMS]

Quote from Landis82:

I LOVE these kinds of threads on ET.

This is the initial reason why I came to Elitetrader in the first place years ago... The HARDWARE Section has always been filled with tons of VALUE over the years and it still keeps me coming back!

Very cool stuff.

Agree, great thread. The hardware forum has been valuable to me over the years too. From smartphones to desktops there has been some great information offered and the OP has done a nice job here.

 

[WinstonTJ]

Quote from FCXoptions:What in the world do you need 16TB for? That is a lot of videos or information

Normally my answer would be PORN...

But this particular box was requested as a desktop/tower so that it can be physically located on the trading floor of a very small hedge fund so that the guys can be directly connected to the machine. The setup is a bit quirky but think of it as a massive NAS that 5 guys are connected to via cross-connects.

They have one data server (in a rack and a climate control room) that is only 8TB that has their tick data on it. This thing is just for the programmers to push (junk, garbage, porn, files... I have no idea) over to where they can have essentially a workspace or workbench.

This machine needs to be fast enough that it acts the same as a mapped network drive on a local machine or LAN - but 16TB of mapped network drive.

 

[WinstonTJ]

Quote from dyson:Re: How to build a 26TB box for less than $3000

I built a similar server back in August/September 2011 (before the Thailand flooding/hard drive shortage). I visited Newegg to compile a list based on the same specifications and current pricing. If you plan the build (and buy parts over a couple of months), you can also take advantage of coupons and rebates.

A few substitutions are needed because of deactivated items. Example: I am using an Intel Core i3-2100T (35W) CPU which is no longer available. Energy consumpsion was important since I am running the server 24/7 and did not want a spike in energy bills. Current energy usage is between 110W and 140W.

I am also using Hitachi 5K3000 (0F12117) 2TB drives. At the time they were going for $49 ($69 with $20 rebate). They are no longer available as Hitachi is now HGST (Western Digital).

I am using Linux (Ubuntu) as my OS with ZFS as the file system. I prefer the software RAID (zfs) vs hardware for compatibility reasons.

Here is the parts list with Newegg item numbers. Total cost is $2,968.77.

What are you using this for? I've got to say that's a great price for that much space however your example and mine are for two totally different purposes, use different types of hardware and (I would venture to guess) have two different intentions.

The biggest differences between your example and mine is the drives and raid cards. There is a major difference between 5400rpm drives and 7200rpm drives and an equally large difference between a RAID card/controller with battery back up and onboard RAM (so that write caching can be enabled) and a non-BBU card that is just being used for extra SATA ports.

Normally when I build a database such as my example the intent is to have many different processes as well as many different users accessing the array at the same time. This means it will be bombarded with constant high loads of reads and writes therefore the overall I/O of the disks and the RAID cards is very important to me. Using a 5400rpm green drive with 32mb of cache is not an option. Most of these things are being used for modeling or optimization or backtesting and with all of that going on at once the I/O to the array is paramount.

Overall, for $5,000 you get server-grade components (motherboard, NICs, CPUs, RAM (fully buffered, ECC), RAID card, etc.) with hardware RAID which is going to outperform most other retail grade options and be more reliable for 24/7 sustained use.

One more question, how do you get to 26TB with 16x drives? Is that just the usable space you get from 16 2TB drives?


On another note, I finished building the thing and it's burning in so I have more pictures and the rest of the build & configuration as well as benchmarks to post shortly.