How to Validate Your Back Testing Software

Your plan is the correct approach. And, it depends on your data time frame. Those who test on daily data may be getting spurious results. I've never seen correct daily data for stocks, though it may exist.

Obviously you can do all kinds of testing with historical data, which you should do, but many people would be shocked at the real-time differences.

The reason is simple, the daily data is "corrected", and corrected as far as a trader is concerned is incorrect. Why? Because when financial institutions trade among themselves in large blocks they don't have to report their trades until after the market closes. These large block trades often trade far away from the daily range creating long tails on the daily bar. So even though the trade did happen during the day, you, or anyone else for that matter, could not participate. The closing (and opening) price is another possible problem. Is the closing price on the daily bar the price at 4:00, 4:15 (whenever trading on that symbol ends), or the Market on Close price. Can you get that price?

You may be able to see the results of this "corrected" data on your trading platform if it uses real-time tick data during the day. When you open your chart the next day, the previous days bar visibly corrects itself. I see this when I have a daily bar database and a real-time tick data feed. The previous day corrects itself to use the now "corrected" daily bar instead of the correct tick data.

My only point is that if you have not already done testing like this, you may learn a lot of useful information about your data via your testing plan. Note also that manually checking your results against "corrected" data does not catch these issues.

Also keep in mind that if you are trading stocks you can buy/sell 1 share for testing. Yes it costs you commissions but you can learn some more useful information like;

Can you short the stock? In a timely manner?

What is the slippage like? You may think that you have reasonable liquidity, but many would be surprised at how long a stock can go without trading in the middle of the day. It may very well gap past the price you thought you would get or, if you put in a market order, get a horrendous fill.

-David

SimpleMeLike said:
Hello,

We trading strategies developers depend on back testing software to return us accurate as possible back test results.

How do you validate your back testing results when testing a strategy?

I plan to do the follow:

1. Run strategy in sim mode for 3 months.
2. Back test those same 3 months.
3. Compare results of 1 and 2.
4. Make a business decision if my back testing software is accuracy.

Thanks for the help.
More...
 
MoneyMatthew said:
There is no getting around this manual validation testing.
More...

I agree completely. I tested my final system by getting thru 1000 intraday trades MANUALLY. It was a big job, but for me it had the advantage that I saw every signal and every stop. I saw the evolution from start till finish for each trade. I had the feeling that I really mastered all I was doing.
I have a friend who is programmer. He always started by programming and testing, but the real progress came only when I told him to go manually thru his trades. Then it was much clearer what and where the problems were and which adjustments he had to make.

About having correct data. I was once told that a stable system would see almost no difference in performance if the data was slightly changed or ticks were missing (depends of course of the timeframe you are trading in). Stability comes from finding the best setup, not from optimizing parameters for a certain period as then you will have to optimize again on a regular basis. I can change the parameters of my system (within reasonable limits) without deteriorating my returns or drawdowns.
 
Mtrader said:
I agree completely. I tested my final system by getting thru 1000 intraday trades MANUALLY. It was a big job, but for me it had the advantage that I saw every signal and every stop.
More...

Thank you for the advice Mtrader,

Few questions for you please:

1. Did you manually back test 1000 intraday after programmed and automated back testing results showed profitability? Or do you prefer manually back testing a strategy/idea to closely understand the strategy/idea before spending time programming?

2. Did the results from the 1000 manual intraday trades profit/loss match the 1000 trades from the back testing software?

Thanks
 
SimpleMeLike said:
Thank you for the advice Mtrader,

Few questions for you please:

1. Did you manually back test 1000 intraday after programmed and automated back testing results showed profitability? Or do you prefer manually back testing a strategy/idea to closely understand the strategy/idea before spending time programming?

2. Did the results from the 1000 manual intraday trades profit/loss match the 1000 trades from the back testing software?

Thanks
More...


I first watch charts with only prices.

I try to understand why prices reverse on certain places. I try to find the logic behind it.

I use purely math to find the “magical formula”” that will tell me where to go long and where to go short. The entries and exits should ideally be such that the market never reverses against my position while in a trade.

I mean:I put a stop at 3 points (ES) and this stop should never be touched (if possible). So no entries at tops or bottoms, but a bit later to minimize the risk of fake tops and bottoms. I always need confirmation of the trend. That will in the long run make more money then trying to catch tops and bottoms. I checked it mathematiccally for my system.

I check the classical stats on 1000 trades, and to see how many times the stop is hit and what the returns would be. Avoiding hitting stops for 100% is impossible.

I optimize then by switching to a modular system, because no system can be optimal for all possible market conditions. Modular systems can be customized for certain market conditions and perform much better in that specific situation, no need to change each time the parameters. All this will result in a number of modules.

I built then a monitoring system that will tell all the time which module is optimal for the given market situation. The monitoring system will also switch to another module when market conditions change. So it is a dynamical system as market are dynamical too. This avoids that certain parameters are not working well anymore. So never any need to optimize anything.

I test the total system again and can still make changes in each module without interfering the other modules and their result. So I can test each module separately and after a change in 1 module I don’t have to retest the entire thing, just the module I changed.

The last step is automation, or in my case partial automation.

After that I check the programmed version against the manual version to see if they match.
 
Mtrader said:
......So no entries at tops or bottoms, but a bit later to minimize the risk of fake tops and bottoms. I always need confirmation of the trend. That will in the long run make more money then trying to catch tops and bottoms. I checked it mathmatically for my system......
More...
Very good :D
In actual fact all of it is very good.
 
MoneyMatthew said:
If manual validation aligns with the backtesting software's output then you can make the assumption that everything is working properly. There is no getting around this manual validation testing. You can test it by hand or you can write your own software just to test. If you can't write your own software and are dependent on a vendor then your only option is manually checking things with pen and paper.
More...

Thanks MoneyMatthew,

Instead of manual back testing to validate the backtesting software, what about using market replay (in fast mode) as well?
 
You haven't mentioned what backtesting software you are using nor what you are using to build your ATS. I built a few robots on meta trader, which for my strategies works great. This is the process I go through in order to put real money.

1. Backtest using at least 2 years worth of historical data.
- Here I analyze the trades made, win ratio, profit ratio, times where strat was most/least successful, sharpe ratio, whether it made money going long/short, avg time on winning trades vs longs, etc.

I then spend time tweaking the algo until I improve these metrics and overal result.

2. Foward test a few months
Once I am satisfied with the results, I let the ats run on sim for a few months tweaking the algo if necessary.

3. Go live risking the least amount possible.

You might want to visit meta trader's website as it will give you plenty of resources to backtest and develop robots. You can even copy people's trades and analyze them.
 
Last edited:
bashatrader said:
Forward test is waste of time because system has higher probability to work in near term and fail afterwards.
More...

Thanks bashatrader,

Can you explain why "system has higher probability to work in near term and fail afterwards."

Thanks
 
You must log in or register to reply here.
Back
Top