Hey guys just wondering how many out of sample trades would be necessary to validate a system as a positive performer in out of sample tests? per period? I have a few systems that are designed to trade about once a day and the result is that they trade about 100 times a year based on 250 trading days out of sample 3 years, in sample 3 years.
How many trials necessary to validate a system?
- Thread starter EliteTraderNYC
- Start date
IMHO ...
... out-of-sample testing on historical data is useless: if the system performs well, you keep it, if it doesn't, you go back to the drawing board, as a result you are not in any better position than if you were doing only in-sample backtesting.
I would say if your system's performance is consistent across all of those 6 years you're working with, you have some chance of being on something usable.
... out-of-sample testing on historical data is useless: if the system performs well, you keep it, if it doesn't, you go back to the drawing board, as a result you are not in any better position than if you were doing only in-sample backtesting.
I would say if your system's performance is consistent across all of those 6 years you're working with, you have some chance of being on something usable.
If I understand you correctly you have designed some systems based on historical data and you want to know how many trades are required to cross-validate their performance.
Note that if market conditions develop not encountered before then the oos testing is useless.
Even worse, if you go back to the drawing table you are subjecting you whole process to data-mining bias. If you have done this already, then you might have found the system just by luck because if you keep developing systems, testing them oos and going back to the drawing table at some point you will find some random system that passes the oos cross-validation step. This is a great read.
Note that if market conditions develop not encountered before then the oos testing is useless.
Even worse, if you go back to the drawing table you are subjecting you whole process to data-mining bias. If you have done this already, then you might have found the system just by luck because if you keep developing systems, testing them oos and going back to the drawing table at some point you will find some random system that passes the oos cross-validation step. This is a great read.
Sergio I agree with you up to the point of saying that the referenced article is "a great read". It s tough not be harsh on the blogger holding himself out as an expert on systems and then contending that the number of trades in your test does change the probability of random results. Lets say a lack of understanding of markets and/or system development?
He uses the coin flip system. If you flip a coin 10 times the probabilities of 10 straight heads is north of 1000 to 1. If you get a steady trend of heads and the occasional tail in this system, then it only shows that you have an extreme outlier.
It certainly does NOT show that the number of iterations is not relevant to the randomness of the system. If you are testing systems in the markets your number of iterations is absolutely correlated to your probability of a non random result.
There are other factors as well of course, but even those other factors are mitigated by a high number of iterations, because it hard to get that high number of transactions if you have over curve fit the rules, hence reducing the universe of qualified trades.
=================================
That's helpful, 6 or 7 years;
6 or 7 bear /bull markets/trends probably will also.
Wisdom is profitable to direct.
I have an auto system that trades about 30 Round trips in the Fx market 24/7 .Currently, in the final step before live trading which means moving from bar-by-bar backtesting to firing real orders to the IB simulated TWS. I find that my pnl jumps form 0 to about +$300-$500 using on average about USD 12000 in margin. After about 10 hours, it would give up 70% of the pnl where I reoptimize a couple of parameters such as lookback period for ADX and DMI and widen or tighten my buy/sell zones using bollinger bands. I would restart my ATS , it would do the similar thing again ie good pnl off the gates then 8-12 hours later give back 50-70% of it. Other than reoptimizing every 5 hours (which is not a big deal to do since I just merely have to change the ATS parameters ) are there any other suggestions into making it truly a stand alone system? Thanks.
How long was the backtest period if your having to reoptimize so often.
What was your Walk forward Analysis result ?
I've been running for close to two month and haven't had to reoptimize any of my system params yet
I trade 5 RT futures daily.
Thanks for you reply adrian. I am not familiar with walk forward yet although my system-Amibroker has a button for it. I have a lookback of 2 months worth of data and use 5m at entry time frame with 30m as my longest time frame for long term trend analysis.
Murray Ruggiero
Sponsor
One of the most important tools is to look at the optimization surface and performance over a valid range of parameters. One simple test I do is to use the average net profit or average trade and the standard deviation of net profit /trades to judge, You want a system to be profitable at -1X standard deviation over a valid parameter range, ideally over -2X. So if the net profit over the optimization range is $200,000 with a standard deviation of $65,000 that good. if it's $200,000 with a standard deviation of $250,000 that bad.