How to Validate Your Back Testing Software

[RobertPeterson]

Interesting link to my development lab where you can see Live my new algo strategy for volatility.
https://forums.collective2.com/t/direct-link-to-watch-my-strategies-development-lab/10508

or direct link to the lab
www.screenleap.com/tradingmachine

I will be happy to answer any question.

 

[mushinseeker]

This is how I approach my backtests and its validation. Let's say I want to backtest the performance of an ATM fly purchase on ETf's that crossed its 50-day moving average. I would create the filters for the 50Ma, create the option payoff that simulates an ATM fly purchase, run the formula on years of data, pick out 30 trades randomly, eyeball 20 of them making sure that there is nothing funky wit the results (ex... entry price of SPY is $200, 3 weeks later, SPY settles at $203 - implies that my ATM fly made $$) . I then take the top 5 losses and top 5 gains, plug the ATM fly into thinkorswim on demand feature and see tick-by-tick how the position made or lost money. There is drift between the TOS on demand vs. the backtest but not enough to invalidate the backtest.

 

[bashatrader]

Thanks bashatrader,

Can you explain why "system has higher probability to work in near term and fail afterwards."

Thanks

Sure, I'll try. Decay and higher probability of market changing as time passes. Then there is the problem of data-snooping bias. If you forward test a few strategies you may think one matches your backtests by luck. There is an expert in this area that recently wrote an article in Medium. In the first Figure 1 replace out-of-sample test by forward test and you get the same effect of data-snooping bias.

 

[water7]

I optimize then by switching to a modular system, because no system can be optimal for all possible market conditions. Modular systems can be customized for certain market conditions and perform much better in that specific situation, no need to change each time the parameters. All this will result in a number of modules.

I built then a monitoring system that will tell all the time which module is optimal for the given market situation. The monitoring system will also switch to another module when market conditions change. So it is a dynamical system as market are dynamical too. This avoids that certain parameters are not working well anymore. So never any need to optimize anything.

I test the total system again and can still make changes in each module without interfering the other modules and their result. So I can test each module separately and after a change in 1 module I don’t have to retest the entire thing, just the module I changed.

very well designed


i want to know your opinion..
is it wise and possible to create self-adjusted modules within a specific market condition?

is it wise and possible to have a dynamic monitoring system that capable of recognizing market characters rather than using static measurements?

with minimum statistical bias and minimum operator involvement..