Back test your decision to use back testing

[Redneck]

Back testing is simply one of the tools at our disposal - it has its uses .., time and place

also has its limitations - like all other tools


ETA - one could view sim trading as back testing - nothing wrong with that

RN

 

[danielc1]

IMHO, backtesting helps to understand systems that might be profitable. Specially if you are automating a trading plan. Looking at those systems in bulk can save you a lot of time, when you are a programmer you can speed up your learning process. My background is in software development and I had zero experience in finance when I started trading. Backtesting routines gave me an understanding of what might be the path to go, but more importantly what not to do and what are the critical routes, specially about risk and funds management.

I get your point that our data is not accurate and it misses the whole picture of the market, but for purely price action systems it is useful. If you are after the order book in order to understand which players moved the market, that is obviously missing in any data set as it can't be captured.

I see backtesting as a way to understand what I shouldn't do instead of a way to find a winner system. I don't see how someone would test ideas if it is not through backtesting. Maybe spending countless hours in from of the tape would give you that experience, but running automated routines can definitely speed thing up.

In the book "Quantitative trading" by Ernest Chan he speaks about the flaws that our data has, and he points something very similar to what you've said, so you are definitively on point on that one. The book goes on and gives really good reasons to apply a backtest to your ideas, maybe is a read that might change your mind if you are interested.

You have put some necessary nuances in this discussion, thank you. Nothing is black or white and for every one way someone will show the exact opposite way and also make a succes out of it. I will read the book, thank you for the suggestion. With that said, I started my trading career with a computer and software too... backtesting. It was only after stop searching for the edge through software tests, I saw the light, 5 years later... It was not the system, it was me. When that was clear, I could use something simple as moving averages and make money of it. I felt that those 5 years where a rud of trying everything under the sun, to see if something would stick.

 

[savoir]

Person A: I have a strategy.
Person B: Test the strategy with 100 to 150 trades.

Person A: I'm too lazy for that, I'll just use a back-testing machine.
Person B: I see

Person A: This sucks, back-testing gave negative results
Person B: Is it possible if you actually trade the strategy you might get better results?

Person A: Dude, that's so antiquated.
Person B: I see

Person A: I have a strategy.
Person B: Test the strategy with 100 to 150 trades.

Person A: Why do you keep telling me that?
Person B: It might actually help you learn how to trade.

Person A: Freak that, I trust software more than my own ability.
Person B: I see

Person A: This sucks, back-testing gave negative results
Person B: Have you ever seen the movie Groundhog Day?

If you think backtesting or anything else people do is not useful, shouldn’t you stay quiet and let them continue? Futile traders contribute to the profits of others.

I’m not saying backtesting is futile. I have not found the need or desire to backtest anything, but about 10 years ago I read Larry Connors’ How Markets Really Work and liked how it dispelled some common myths about the stock market through backtesting.

Trading index futures is a zero-sum game. I don’t think there are any secrets. Well if there are, I guess I’m one of the people not in on the secret.

I think it’s a good idea to just observe people and how they go about doing things, including those who have strong feelings about others doing it wrong. One member here repeatedly laments the high failure rate of those who attempt to trade stock index futures as if that’s a negative. Where does he think the profits of the winners come from? Without losers, there is no game.

 

[SunTrader]

If you think backtesting or anything else people do is not useful, shouldn’t you stay quiet and let them continue? Futile traders contribute to the profits of others.

I’m not saying backtesting is futile. I have not found the need or desire to backtest anything, but about 10 years ago I read Larry Connors’ How Markets Really Work and liked how it dispelled some common myths about the stock market through backtesting...

Got it also when it came out. AFAIK 2nd hand copies are all that can be bought nowadays. Its a keeper for me. Still refer to it every so often, even if it is no longer based on recent data - it holds up. Not much really changes.

 

[SimpleMeLike]

You have put some necessary nuances in this discussion, thank you. Nothing is black or white and for every one way someone will show the exact opposite way and also make a success out of it. I will read the book, thank you for the suggestion. With that said, I started my trading career with a computer and software too... backtesting. It was only after stop searching for the edge through software tests, I saw the light, 5 years later... It was not the system, it was me. When that was clear, I could use something simple as moving averages and make money of it. I felt that those 5 years where a rud of trying everything under the sun, to see if something would stick.

Hello danielc1,

Great persistence you displayed.

 

[Drawdown Addict]

You have put some necessary nuances in this discussion, thank you. Nothing is black or white and for every one way someone will show the exact opposite way and also make a succes out of it. I will read the book, thank you for the suggestion. With that said, I started my trading career with a computer and software too... backtesting. It was only after stop searching for the edge through software tests, I saw the light, 5 years later... It was not the system, it was me. When that was clear, I could use something simple as moving averages and make money of it. I felt that those 5 years where a rud of trying everything under the sun, to see if something would stick.

Another point that is discussed in the book. Chan went through examples to prove that simplest algorithms can be as good if not better than complex ones.

There's a tendency to think that financial institutions have extremely complicated models to beat the market. They do have them but not to gain any edge, those models are there to excuse the firm in front of their clients regarding the massive fees they are charging to handle funds.

He went to test multiple systems with extremely complicated machine learning models and found that a simple buy and hold on the SP500 yielded better results.

He is a fan of mean reversion strategies using moving averages, so yes, there's nothing wrong with keeping it simple.

 

[SimpleMeLike]

Another point that is discussed in the book. Chan went through examples to prove that simplest algorithms can be as good if not better than complex ones.

There's a tendency to think that financial institutions have extremely complicated models to beat the market. They do have them but not to gain any edge, those models are there to excuse the firm in front of their clients regarding the massive fees they are charging to handle funds.

He went to test multiple systems with extremely complicated machine learning models and found that a simple buy and hold on the SP500 yielded better results.

He is a fan of mean reversion strategies using moving averages, so yes, there's nothing wrong with keeping it simple.

Hello Drawdown Addict,

Dollar cost average the sp500 index is the best the decision I ever made in my life.

Day trading is just quick way to get rich

 

[mikeriley]

Hello mikeriley,

Person A is a very wise and logical and good business person. And doing the right thing IMO.


Question:

Why would Person B want to manual take 150 trades, when a computer can back test 20 years of those same trades for nearly XXX to XX,XXX trades?

Thank you,

If you remember, ask me on Tuesday, Sept 5th before 9am which way
the NQ futures market is going. Then you'll have your answer what
your mind can do, that a computer can't.

 

[virtusa]

There is no getting around a manual validation testing.
I tested my final system by getting thru 1000 intraday trades MANUALLY. It was a big job, but for me it had the advantage that I saw every signal and every stop. I saw the evolution from start till finish for each trade. I had the feeling that I really mastered all I was doing.
I have a friend who is programmer. He always started by programming and testing, but the real progress came only when I told him to go manually thru his trades. Then it was much clearer what and where the problems were and which adjustments he had to make.

I check the classical stats on 1000 trades, and check how many times the stop is hit and what the returns would be. Avoiding hitting stops for 100% is impossible.

I optimize then by switching to a modular system, because no system can be optimal for all possible market conditions. Modular systems can be customized for certain market conditions and perform much better in that specific situation, no need to change each time the parameters. All this will result in a number of modules.

I built also a monitoring system that will tell all the time which module is optimal for the given market situation. The monitoring system will also switch to another module when market conditions change. So it is a dynamical system as markets are dynamical too. This avoids that certain parameters are not working well anymore. So never any need to optimize anything.

I test the total system again and can still make changes in each module without interfering the other modules and their result. So I can test each module separately and after a change in 1 module I don’t have to retest the entire thing, just the module I changed.

The last step is automation, or in my case partial automation.

After that I check the programmed version against the manual version to see if they match.

About having correct data.
I was once told that a stable system would see almost no difference in performance if the data was slightly changed or ticks were missing (depends of course of the timeframe you are trading in). Stability comes from finding the best setup, not from optimizing parameters for a certain period as then you will have to optimize again on a regular basis. I can change the parameters of my system (within reasonable limits) without deteriorating my returns or drawdowns.