Building a home-made out-of-shelf SuperComputer.

M

mcgene4xpro

Hi,

I am dreaming to build a really powerful supercomputer cluster that could do backtesting, optimization and modeling very quickly and efficiently. However, the main constraint for me is that i am not a computer science specialist.

http://www.calvin.edu/~adams/research/microwulf/design/
http://www.clustercompute.com/

I am planning to build my supercomputer from out-of-shelf component which i could get it very cheap and cluster them together to work as one unit.

I need to have very detailed resources of how to do this.
Suggestions from experienced people are highly welcomed.

Thanks

McGene
 
You can build what the links describe pretty easiy. The trouble is to create software to usitlise all of this power. From teh view point of most operating system thsi cluster will look exactly like 4 independednt computers connected with gigabit ethernet. From there it's up to you to amke sure you can split any computations.

Much more interesting a concept is CUDA. A high-end grpahics card contains thousands of computational units. Here comes a home-made super-scalar computer. The difficulty in utilising this power is graphics cards use single-precision floating point arithmetics and most modern parallel algorithms require double precision to be useful.

Historically, the reason for this is till the last 10 years or so RAM was the most scarce resource. Consequently, there was lots of research done on "sparse" matrices and algorithms that modify data in-place without copying. Now that memory is cheap but improvements in computational power seem to have hit the cap we could see more research in algorithms involving low-precision arithmetics.
 
Quote from mcgene4xpro:

Hi,

I am dreaming to build a really powerful supercomputer cluster that could do backtesting, optimization and modeling very quickly and efficiently. However, the main constraint for me is that i am not a computer science specialist.

http://www.calvin.edu/~adams/research/microwulf/design/
http://www.clustercompute.com/

I am planning to build my supercomputer from out-of-shelf component which i could get it very cheap and cluster them together to work as one unit.

I need to have very detailed resources of how to do this.
Suggestions from experienced people are highly welcomed.

Thanks

McGene
More...

I'm not sure why you feel like you need a super computer to do back testing. I build my own computers by buying parts at http://www.newegg.com/. I try to stick to Intel mother board and processor and I often call the help number at Intel for comparability and suggestions. I would think a top of the line I7 win 64bit Win7, a 10K hard drive or a SATA III MLC Internal Solid State Drive (SSD), good power supply, DVD-r with as much ram as you can afford would work really well in a standard ATX nid-tower. If your requirements are higher, you can build a dual processor server with a similar chip in the XEON family. This will add a lot of money the the end machine. Should be nasty fast.

In the end, try to find someone that's done this before, you will get a clean win 7 install with no other crap running. The machine will be VERY fast and easy to service, because you know what's inside.

Good luck....
 
yeah..i'm also curious what kink of back testing is that and how you going to split the load equally..hardware imo should be least of your worries. it's all come down to the actual fill ratio at exchange..
like i said before-i can pick the winners all day long,but if i can't get my shares-it's pointless to do any backtesting. last few years i did pretty pretty good on stocks, using couple old dual core AMD's and basic nvidia cards(NVS 285)

here-dual 6 core..12 total..i hope that this is enough for ya

http://www.newegg.com/Product/Product.aspx?Item=N82E16813151227
 
Quote from LeeD:

You can build what the links describe pretty easiy. The trouble is to create software to usitlise all of this power. From teh view point of most operating system thsi cluster will look exactly like 4 independednt computers connected with gigabit ethernet. From there it's up to you to amke sure you can split any computations.

Much more interesting a concept is CUDA. A high-end grpahics card contains thousands of computational units. Here comes a home-made super-scalar computer. The difficulty in utilising this power is graphics cards use single-precision floating point arithmetics and most modern parallel algorithms require double precision to be useful.

Historically, the reason for this is till the last 10 years or so RAM was the most scarce resource. Consequently, there was lots of research done on "sparse" matrices and algorithms that modify data in-place without copying. Now that memory is cheap but improvements in computational power seem to have hit the cap we could see more research in algorithms involving low-precision arithmetics.
More...
Excellent post, thank you for the information.
 
Quote from mcgene4xpro:

I am planning to build my supercomputer from out-of-shelf component which i could get it very cheap and cluster them together to work as one unit.
More...

And then what?
 
Quote from LeeD:

The difficulty in utilising this power is graphics cards use single-precision floating point arithmetics and most modern parallel algorithms require double precision to be useful.
More...

Actually, (also in the link you provided) NVidia cards that support CUDA 1.3 or above have double-precision.
 
Quote from LeeD:

You can build what the links describe pretty easiy. The trouble is to create software to usitlise all of this power. From teh view point of most operating system thsi cluster will look exactly like 4 independednt computers connected with gigabit ethernet. From there it's up to you to amke sure you can split any computations.

Much more interesting a concept is CUDA. A high-end grpahics card contains thousands of computational units. Here comes a home-made super-scalar computer. The difficulty in utilising this power is graphics cards use single-precision floating point arithmetics and most modern parallel algorithms require double precision to be useful.

Historically, the reason for this is till the last 10 years or so RAM was the most scarce resource. Consequently, there was lots of research done on "sparse" matrices and algorithms that modify data in-place without copying. Now that memory is cheap but improvements in computational power seem to have hit the cap we could see more research in algorithms involving low-precision arithmetics.
More...

Great info. I have heard that there is a product called " jacket" can run matlab on CUDA.
And this is one of the ways i can go. Please check this website and share your opinion.
http://www.accelereyes.com/

Thanks LeeD

McGene
 
You must log in or register to reply here.
Back
Top