How far do you manually back test?

Z

Zr1Trader

Discretionary traders,

How far do you manually back test a method by hand before starting to consider it may be a good enough one to push into the sim, then live if that pans out?

Automated guys usually say 10 years of data or over 1000 trades. Then test it on out of sample data ect,


I go with the 1000 trade rule myself but dang it can take a very long time and be very tedious! Just testing a new idea this weekend and thought I'd ask what other people use as a rule of thumb.


The nice thing about manual strats is that most the good ones can't be automated but the testing phase can take forever!
 
An answer to your question is: Through several bull and bear market phases, low and high volatility, and low and high liquidity.

I second the second poster's sentiment, however. If I could do this all over again, I would try to automate some of my rules sooner. That would have saved me quite a bit of failures and allowed me to identify what works quicker.

As soon as I found out about GA optimization, I realized how futile it was for me to be doing what I was doing before.

Also, I don't know of any systems that "can't" be automated. Only systems that are complex enough to make automization extremely difficult (although still probably worth it).

I am working on automating a fairly complex strategy based entirely on PA trading. I have a friend helping me with this. If you told me several years ago, that I would have been able to automate this, I probably would have said that would be impractical. Yet, here I am. Mostly done. It trades much better than I do.
 
What is the time frame of your system in terms of the length (holding time) of the average trade. Is it seconds, minutes, hours, or days. This is a significant factor in determining the back-testing time period required.
 
Quote from Zr1Trader:

Discretionary traders,

How far do you manually back test a method by hand before starting to consider it may be a good enough one to push into the sim, then live if that pans out?

Automated guys usually say 10 years of data or over 1000 trades. Then test it on out of sample data ect,


I go with the 1000 trade rule myself but dang it can take a very long time and be very tedious! Just testing a new idea this weekend and thought I'd ask what other people use as a rule of thumb.


The nice thing about manual strats is that most the good ones can't be automated but the testing phase can take forever!
More...

Why bother manually backtest? No-one will ever trade real-time like it was assumed in a manual backtest. Lots of reasons for that, for example:
- all types of breaks (lunch, bathroom, coffee)
- all types of interrupts (tel., door bell, animals)
- all types of distractions / loss of focus (you name them)
- inability to execute trades following 100% of the rules (unless these are really simple)
- lack of discipline, break a rule here or there for whatever "good" reason

Not to mention, part-way through that backtest, you realize there are better rules than the ones you started with, but you don't restart that backtet from trade 1.

Getting to the point you can backtest automatically thousand of trades on a new strategy might take as long as the initial manual backtesting (although, with some practice, it will get better), but you gain the ability to trade real-time exactly as backtested.
 
As an example, in the last couple of days, I have been evaluating different ways of making my system's trading decisions, using the same set of patterns, just working through conflicts differently.

I have been evaluating 7 new ways of making those decisions, for the basic system (~ 5500 trades in 6.5 years) and using all trade signals (~ 11000 trades in same period).

And the results are quite interesting.
 
Quote from Zr1Trader:

That is for automated backtesting correct?


I'm talking about testing strats that can't possibly be automated.
More...

After so many thousands of backtests, I just know if something will work and work for my personality. So I will go back 100 sample size manually, and if it does good there, then comes the process of coding it for 10 years of tick data if available, but to put my money at risk based on manually doing it, stopped doing that 20 years ago when I use to lose on consistent basis.

And anything can be automated, depends on your skills at programming and months/years of doing it, by time you finish, if you finish, you have discovered 90 other patterns you want to test. I have found it better to do exits first, and same exits for all methods used, plus any more based on new ideas. This way I have my exits all ready for new testing of anything I want to backtest.
 
I have been backtesting one trading strategy from past two months and i have scaled up $13 to $58 during this time and i feel that this is really good. Put in my real money $13 for backtesting the method. Which has given me good result.
 
Funny how EVERYONE used to trade without computers and now your a dumby if you don't use one for automation :).

I like being the minority in the market place.

I can automate too and I do have a couple mediocre strats that are completely auto but I have found manual trading to fit me better and to be more profitable. Maybe if I had a few million to blow on the best programmers in the world I could automate it but I'll leave that to smarter people than I like LTCM, Bats, Knight, Goldman ect. In fact when their shit is breaking that is likely when I will benefit the most. ie Flash crash.

There are so many different variables involved, some intangible to a computer that it would literally take an army of programmers to try to do it.


I like handles Idea of keeping exits the same. I have found that some types of exits work with almost anything including random entry.


No matter what , I think it is up to the "trader" (or programmer) to determine when to shut down or reevaluate . I think no matter what everyone agrees that when a DD more than 2 times your max tested DD occurs, it is time to make a discretionary decision. No matter how far you back tested doesn't matter anymore at that point.

Thing that is scary is that some strats on automation look absolutely great for a certain period then go to hell the next. With an idea that one is manually back testing they think in the back of their head, what if this period I am testing is just "luck"?. I suppose that is a dilemma everyone faces until they've tested live with confidence for a long time.


Here are some stats from my manual back testing across three separate instruments for the month of August (tf, cl , gc). If it worked on three separate instruments , going long as well as short , 2 min time frame then I think it's a good start but still a long way to go and a lot of wasted time if at some point it "breaks". It's a variation of something I use now so I'm fairly certain it should work but how do you really know for sure it's not just a lucky period ! I don't think anyone knows for sure. There are only probabilities it will work.
=============
CL
-10
-16
-14
54
50
37
33
-14
15
37
-11
29
13
21
10
22
16
272 total ticks for August
=========================
TF
-15
-10
40
28
34
25
-6
12
13
-5
-6
-5
-6
19
118 total ticks for August
====================
GC
53
20
61
-12
-9
77
20
-9
-15
39
4
34
-8
-9
-16
-13
-16
-11
-16
60
234 Total ticks for Aug
 
You must log in or register to reply here.
Back
Top