Automated Trading for a Living

[garachen]

Quote from NetTecture:

No problem at all.

See, we are:
* Storing them on a RAID system that is refreshing every month.
* On two locations
* In a checksum secured file system

If tings get bad we get a read error and regenerate them.

I think I just have super bad luck with these things. Everything that's not supposed to happen on a RAID system seemed to happen to me. So one particularly bad weekend we went on a RAID crusade and got rid of it everywhere.

 

[NetTecture]

Quote from garachen:

I think I just have super bad luck with these things. Everything that's not supposed to happen on a RAID system seemed to happen to me. So one particularly bad weekend we went on a RAID crusade and got rid of it everywhere.

Check Windows Server 2012 or Linux with ZFS - that plus backtups should handle all.

Server 2012 I would use ReFS on a 3 copy setup in a storage pool

 

[garachen]

Quote from NetTecture:

Check Windows Server 2012 or Linux with ZFS - that plus backtups should handle all.

Server 2012 I would use ReFS on a 3 copy setup in a storage pool

I do Linux with ZFS. Tried to switch to ReFS when it came out. I have about 70TB of data so after a few failed attempts I gave up. Someday we should probably prune out what we store.

 

[NetTecture]

Quote from garachen:

I do Linux with ZFS. Tried to switch to ReFS when it came out. I have about 70TB of data so after a few failed attempts I gave up. Someday we should probably prune out what we store.

Stick witz ZFS for the moment. MS finally gets it - but it is a process and with ReFS and storage pools they are not even starting to be serious. Another 2 generations

 

[ofthomas]

interesting that you guys are having issues with storage... but without knowing what vendors you are using, etc... hard to express an opinion...

any trading activity/business generates tons and tons of data... but I dont think that should be a huge issue, my RDO teams manage about 5PB+ and the storage is tiered depending on the source, compliance req, etc...

anyhow, I dont want to clutter this thread with a storage discussion...

in any event, our grids are built with HP SL6x000, we go for density, 256 Cores or 512Threads per 4U block... each rack has a NetApp filer... the scheduler are usually DL580s... recently we deployed a full rack of DL560Gen8 Grid with K20's as a test case for a group...

in other places, the grids were built using DL380 servers, given you dont want too much compute on a single node or rack and if that goes out you are out a certain % of capacity...

anyhow, they come in really handy when doing MCS and do RT analysis/pricing and risk management... I realize you are using it more for back testing, which is also used for, but more of a secondary effect...

 

[vincegata]

Quote from NetTecture:


We split the backtest into tasks - a task is always
* One week (Sunday to Saturday - we are always flat on weekends)
* Max. x combinations (128 at the moment).
.......................

Ok, thank you.

 

[dholliday]

Quote from QuantWizard:

Question is: There are code libraries available for C/C++ for Tesla; shouldn't be too tricky to convert...?

With Intel's 50 and 60 core processors coming out this year, Tesla (I assume running on a GPU) will no longer even be a consideration for many.
-David

 

[vicirek]

Quote from QuantWizard:

Question is: There are code libraries available for C/C++ for Tesla; shouldn't be too tricky to convert...?

There are both commercial and free like Thrust, Accelereyes etc. Also C++ AMP in VS2012 will let you use GPU through DirectX drivers which are hardware independent.

 

[vicirek]

Quote from garachen:

I think I just have super bad luck with these things. Everything that's not supposed to happen on a RAID system seemed to happen to me. So one particularly bad weekend we went on a RAID crusade and got rid of it everywhere.

Agree. RAID is not reliable and high maintenance. Same or better results can be achieved without RAID.

 

[ofthomas]

Quote from vicirek:

Agree. RAID is not reliable and high maintenance. Same or better results can be achieved without RAID.

LMAO.... hmmm... please tell me how to store 70TB of data without raid... last I checked, the biggest drive on the market is 4TB... I can of course use a Violin with 20TB, but there is cost associated with that (unless you are an actual bank that can afford it, and we dont even deploy it everywhere)...

it amazes me the stuff people spew out at times... please understand the tech before saying non-sense... RAID is necessary, what most get hurt by is the implementation of said technology because they usually pick the wrong choice and assume they are protected when they might not be...