How to Validate Your Back Testing Software

SimpleMeLike

SimpleMeLike

Hello,

We trading strategies developers depend on back testing software to return us accurate as possible back test results.

How do you validate your back testing results when testing a strategy?

I plan to do the follow:

1. Run strategy in sim mode for 3 months.
2. Back test those same 3 months.
3. Compare results of 1 and 2.
4. Make a business decision if my back testing software is accuracy.

Thanks for the help.
 
SimpleMeLike said:
Hello,

We trading strategies developers depend on back testing software to return us accurate as possible back test results.

How do you validate your back testing results when testing a strategy?

I plan to do the follow:

1. Run strategy in sim mode for 3 months.
2. Back test those same 3 months.
3. Compare results of 1 and 2.
4. Make a business decision if my back testing software is accuracy.

Thanks for the help.
More...

And why would you wait three months to validate backtest results?

Put your backtest results into a spreadsheet, then run back through your data, or charts, and log the entries and exits results by hand to compare. Or even better, use a data source other than that being accessed by your backtesting, to log by hand and compare to the backtest.

Doing so a number of years ago revealed that what I was seeing in backtesting, and what I had then been pursuing in real time, was an illusion based on data that wasn't accurate.

The lesson I learned the hard way was not to completely trust data, or trust anything. Verify, verify.
 
SimpleMeLike said:
Hello,

We trading strategies developers depend on back testing software to return us accurate as possible back test results.

How do you validate your back testing results when testing a strategy?

I plan to do the follow:

1. Run strategy in sim mode for 3 months.
2. Back test those same 3 months.
3. Compare results of 1 and 2.
4. Make a business decision if my back testing software is accuracy.

Thanks for the help.
More...
I don't think you would need to wait that long to validate the backtesting software itself. Let's take a really simple trading system that just uses a moving average. Let's say this system gets Long above the moving average and Short below it. You would simply match the time stamp/date that the backtesting software shows you for initiating a new position and match that with what you see manually.

If manual validation aligns with the backtesting software's output then you can make the assumption that everything is working properly. There is no getting around this manual validation testing. You can test it by hand or you can write your own software just to test. If you can't write your own software and are dependent on a vendor then your only option is manually checking things with pen and paper.
 
I think there will be 20-30 trades enough to "Make a business decision". But in case your strategy triggers 2-3 trades per month, the term will be much longer than 3 months. ... :sneaky:
 
lindq said:
And why would you wait three months to validate backtest results?

Put your backtest results into a spreadsheet, then run back through your data, or charts, and log the entries and exits results by hand to compare. Or even better, use a data source other than that being accessed by your backtesting, to log by hand and compare to the backtest.

Doing so a number of years ago revealed that what I was seeing in backtesting, and what I had then been pursuing in real time, was an illusion based on data that wasn't accurate.

The lesson I learned the hard way was not to completely trust data, or trust anything. Verify, verify.
More...

Thank you lindq for response.

You are correct. I can easily manually back test about 100-200 trades and compare/validate those same trades from the back testing simulation. No sense in waiting months. Or just let it run in market replay.

what do you mean by data that wasn't accurate?

Thanks
 
You should do it with forward tested data if you are trading futures. You might run into continuous/nearest contract data disagreements that might slightly make the results different even for intraday. So look at the output and manually test it that very same day before there is any possible "shift" in data to validate your signals.
 
SimpleMeLike said:
Thank you lindq for response.

You are correct. I can easily manually back test about 100-200 trades and compare/validate those same trades from the back testing simulation. No sense in waiting months. Or just let it run in market replay.

what do you mean by data that wasn't accurate?

Thanks
More...


The data was presenting bar lows that represented price points that were below what was actually printed in the markets. I only discovered it one day when I watched the price action in real time, then saw that the backtest would have had me in a trade that was not possible to have achieved. I then found that it was a consistent problem in the timeframe I was trading.

A way to avoid this, in any event, is to be cautious of pegging possible trades to what a bar low showed looking back at historical data. Random data spikes up or down can really throw off backtesting. If it is just one or two symbols, that's pretty easy to track. But if you're trying to align many symbols in short timeframes, best to do a workaround like only peg trades to bar opens/closes. It is more reliable.

Nothing hard is ever easy.
 
By running multiple tests, checking entries and exits, forward testing.
 
What I also like doing is using several different data feeds - there are subtle differences between providers and you want to make sure your results are close no matter what data you use.

But in the end I've come to the conclusion the only real test is with real money... forget paper accounts, just use a tiny size and trade live.
 
You must log in or register to reply here.
Back
Top