Memory requirements

[abattia]

I want to run backtests for an intraday strategy :
• single instrument - SPY, i.e. SPDR S&P 500 ETF
• single timeframe – 15 second bars

By my estimate, each year of SPY tick data equates to about 5 GB of data (well it does if it’s in .txt format).

My backtesting software (NinjaTrader) loads all data into memory before commencing a backtest. My testing machine can’t handle this amount of SPY tick data in memory all at once, with the result that my backtest results for this strategy are unreliable (yes, even more unreliable than backtest results always are anyway!).

I’m trying to understand what’s required to upgrade to a machine that could handle 5 GB (or 10 GB, or 15 GB, etc …) of SPY tick data in memory all at once.

Am I looking at something like a top of the range DELL PC?
Or a chunky and more expensive server??
Or even a supercomputer??? :eek:

 

[Big AAPL]

Ninja has crippled my machine with backtests of similar granularity. Try breaking it up into weekly, or better yet, daily timeframes and see if you can get better performance.

The only problem you would now face is plotting the results for the year manually.

 

[abattia]

Thanks.

Quote from Big AAPL:
... Ninja has crippled my machine with backtests of similar granularity...

I don't think it's an NT problem in the case of SPY tick data: I believe SPY's daily volume makes it among the most actively traded instruments on any global exchange, and in the recent past I've also had trouble with SPY tick data backtests on another more heavy-duty "simulator" for the same reasons.

In the case of SPY, I think it's a matter of having/not having enough memory ... I just don't know how much would be needed, and whether attaining that level is practical or not ...

 

[Scataphagos]

Quote from abattia:

I want to run backtests for an intraday strategy :
• single instrument - SPY, i.e. SPDR S&P 500 ETF
• single timeframe – 15 second bars

By my estimate, each year of SPY tick data equates to about 5 GB of data (well it does if it’s in .txt format).

My backtesting software (NinjaTrader) loads all data into memory before commencing a backtest. My testing machine can’t handle this amount of SPY tick data in memory all at once, with the result that my backtest results for this strategy are unreliable (yes, even more unreliable than backtest results always are anyway!).

I’m trying to understand what’s required to upgrade to a machine that could handle 5 GB (or 10 GB, or 15 GB, etc …) of SPY tick data in memory all at once.

Am I looking at something like a top of the range DELL PC?
Or a chunky and more expensive server??
Or even a supercomputer??? :eek:

Most x58 and i7 mobos can run 24GB or more of RAM. You don't need to buy all of that at once... just get 6 GB to start on a W7-64bit machine... then add RAM as you think you need. RAM is dirt cheap now days. There is a Dell T3500 with Xeon W3550 CPU on their outlet site for $809. If you don't buy that, use its price as a guideline for power and price on any machine you buy.

 

[abattia]

Quote from Scataphagos:

Most x58 and i7 mobos can run 24GB or more of RAM. You don't need to buy all of that at once... just get 6 GB to start on a W7-64bit machine... then add RAM as you think you need. RAM is dirt cheap now days. There is a Dell T3500 with Xeon W3550 CPU on their outlet site for $809. If you don't buy that, use its price as a guideline for power and price on any machine you buy.

Thanks!

 

[mgookin]

I'd suggest you load half of what you think your system can handle. If it handles it ok, double that and find the limit. And of course if it still does not work, cut that in half until you find that limit. Once you know how much data you can crunch with what you presently have, you can determine how much data you will be able to handle with x qty ram.

I just threw 16gb DDR3 into a box for $89 free shipping at Newegg.

For the peace of mind you can't beat it.

http://www.newegg.com/Product/Produ...6050 600006073&IsNodeId=1&name=16GB (4 x 4GB)

 

[Scataphagos]

Quote from mgookin:

I'd suggest you load half of what you think your system can handle. If it handles it ok, double that and find the limit. And of course if it still does not work, cut that in half until you find that limit. Once you know how much data you can crunch with what you presently have, you can determine how much data you will be able to handle with x qty ram.

I just threw 16gb DDR3 into a box for $89 free shipping at Newegg.

For the peace of mind you can't beat it.

http://www.newegg.com/Product/Produ...6050 600006073&IsNodeId=1&name=16GB (4 x 4GB)

For that kind of price, one could justify "starting with 16GB RAM". That will handle almost anything in the trading world.

I can recall spending more than that for 1GB RAM on a P4 rig... (Actually, I'm old enough to recall spending more than that on 1MB of RAM... yes... 16, 64K RAM chips... on a rig with a 40 MB hard drive that cost $400 (HDD cost alone)... and it even had lots of bad sectors... can remember remarking, "Damn, this thing is FAST"... LOL!)

 

[mgookin]

Quote from Scataphagos:

For that kind of price, one could justify "starting with 16GB RAM". That will handle almost anything in the trading world.

I can recall spending more than that for 1GB RAM on a P4 rig... (Actually, I'm old enough to recall spending more than that on 1MB of RAM... yes... 16, 64K RAM chips... on a rig with a 40 MB hard drive that cost $400 (HDD cost alone)... and it even had lots of bad sectors... can remember remarking, "Damn, this thing is FAST"... LOL!)

I paid $2,000 for the four sticks I pulled out.

It broke my heart to give them to a younger person who wanted more ram. They were 4x1GB DDR3 on the first day DDR3 came out.

$500 each and I had the smokinest system one could build.

 

[abattia]

Just reporting back on where I’ve got to:

I had a third-party compress the SPY tick data (any continuous sequence of ticks at the same price was converted to a single tick with same total volume as all the combined ticks), so that approx 5GB of data (1 year of SPY tick data) was reduced to approx 500MB (in the case of SPY, this sort of compression doesn’t affect the generation of 15-second bars, which is what I need).

When loaded into NT7, the backtest ran without a problem.

So some combination of tick compression and using NT7 (rather than NT6.5) resolved the issue without needing to purchase a more powerful machine.