Manual Backtesting

T

TriPack

I'd like to start a thread on manual backtesting. To start things off let me ask a few questions to get some discussion going.

Do you find manual backtesting useful, and if so what are some of the ways this has been helpful?

What is the best way you have found to do manual backtesting?

What process do you go through to make manual backtesting as objective as possible?

Are there any pitfalls of manual backtesting that others should be aware of?

How do manual backtesting and computerized backtesting compare with regards to insights learned about the market's character, new patterns found, robustness of the systems tested, and overall productivity for testing a data series?

Thanks for your comments and responses on this topic.
 
I will just start out by saying that manual backtesting has been a very rewarding exercise for me, but I should have started doing this a long time ago. I have always been the type of person who tries to automate everything to avoid doing repetitive tasks that can just as easily be performed by computers. I've owned a backtesting platform ever since I got into trading and so I guess it was a natural for me to test my ideas by writing code in TradeStation or some other testing environment.

The problem is that the results you get are not always equivalent to the idea you have in your head for what the code should do. So from that standpoint alone, when writing code it is important to manually verify that the system is executing the trades where they *should* be executed. Or in other words it is wise to manually verify that the code is executing according to plan. So even when the testing is automated, the verification should be manual anyway.
 
With fully manual backtesting you eliminate the code layer. Your programming ability does not come into play. All that exists is the base ruleset for entries and exits compared with what is actually going on on the price data. Presuming that you have an analytical mind capable of understanding the logic of your system, and executing it on each and every bar in the same way, then you have the ability to manually backtest your system ideas. This has forced me to simplify my systems which has also been a good side effect of manual backtesting.

While manual backtesting takes longer because by its very nature it isn't automated, it seems to me that this inherent lack of efficiency is more than made up by the vast multitude of trading ideas and insights that are gained by manually going through a series of days and weeks chart by chart, bar by bar and applying a logical set of rules to each bar. In my opinion, the next frontier in system development is going to be in getting back to doing more testing by hand. The Empirical Revolution is HERE!!!

I have gained far more confidence in and intimate knowledge of my methods and systems through manual backtesting than through computerized testing. Empirical patterns start to become visible when manually backtesting that may not be seen any other way - patterns that can become the basis for trading systems. I have also gained a lot from keeping a manual system journal of the "ideal" system results for each trade. I go through the chart at the end of the day and record in Excel all the trades that the system signalled for the day. It then is easy to compare my trading performance of the system against the system itself.

So my view used to be to use computerized system backtesting to whittle out what works, and then to manually verify the results. Now I think using manual backtesting first to come up with the patterns and system logic and test results over a smaller set of data is the best approach. And then after the system is refined and ready for longer term tests, using the computerized testing to test the results over longer time periods and to ultimately automate the system (if that is the goal).

Quote from TriPack:

I'd like to start a thread on manual backtesting. To start things off let me ask a few questions to get some discussion going.

Do you find manual backtesting useful, and if so what are some of the ways this has been helpful?

What is the best way you have found to do manual backtesting?

What process do you go through to make manual backtesting as objective as possible?

Are there any pitfalls of manual backtesting that others should be aware of?

How do manual backtesting and computerized backtesting compare with regards to insights learned about the market's character, new patterns found, robustness of the systems tested, and overall productivity for testing a data series?

Thanks for your comments and responses on this topic.
More...
 
On the controversial Woodies website there is a utility to be found by Kiwi that will advance a historical TradeStation 2000i chart (any timeframe) 1 bar at a time. Am unable to check it but there are some other charting packages that are supported as well. If used properly this avoids "fooling" yourself by looking forward.

In addition "real time" manual testing is possible by using a simulator eg AutoTrader, BracketTrader (the two are brothers), Ninjatrader, TSim+ etc. for 1minute, 5 minute, tick charts.

When using simulators one has to realise that in real trading there will be execution delays, slippage and fills at different levels than in the simulators. This is due to the size of the bids and asks.
It pays to add at least 1 tick of slippage on each side and if you are using limit orders add another tick. (Futures trading)

6 Months ago I did not really believe in doing simulations / backtesting but now I most strongly suggest that any newby contemplating trading experiments with simulations. As usual we discover the good things always too late and as mentioned by Tripack I should have started earlier with it.

Marinus
:cool:
 
If it's a stochastic model the decision process should be quite simple so I don't see the interest of doing it manually.

When it is something discretionary that involves some informal knowledge that need to be formalised down to an algorithm first so that doing it manually is necessary at least in the first phase because if you want to program the stuff well you will encounter a problem. The most difficult part is to formalise market context : it's a set of variables that condition the probability of realisation of a signal or level. That's what is in black in my dialect journal prototype : http://www.elitetrader.com/vb/showthread.php?s=&threadid=30651 or explicitely mentioned with the keyword "context" when it is a local context. The reason it is difficult is that a brain is processing differently from a sequential machine and when you do things you don't even realise the many implicit information you are processing notably when it has already become a habit. Also a human process many redundant informations that's the way he compensate his weakness compared to a computer. Btw an interesting book from Nobel Prize Edelman I have already mentioned http://www.amazon.com/gp/reader/046...dir/002-9698837-8448866?v=look-inside&s=books I read in fact to help me decompose my own brain processing for automating the whole decision trading process :p
 
What should be done manually is to really understand your indicators so as not take it as magic. Then calculating by hand a few times may be a way to oblige you to understand the underlying stuff :D.

Quote from harrytrader:

If it's a stochastic model the decision process should be quite simple so I don't see the interest of doing it manually.

More...
 
Quote from TriPack:

With fully manual backtesting you eliminate the code layer.
More...

you also eliminate the ability to test across long time periods. i routinely test dow strategies going all the way back to 1930, that would not be possible if i wanted to "eliminate the code layer".
 
Quote from harrytrader:

What should be done manually is to really understand your indicators so as not take it as magic. Then calculating by hand a few times may be a way to oblige you to understand the underlying stuff :D.
More...

Where did you get that graphic?
 
Quote from TriPack:

I'd like to start a thread on manual backtesting. To start things off let me ask a few questions to get some discussion going.

Do you find manual backtesting useful, and if so what are some of the ways this has been helpful?

What is the best way you have found to do manual backtesting?

What process do you go through to make manual backtesting as objective as possible?

Are there any pitfalls of manual backtesting that others should be aware of?

How do manual backtesting and computerized backtesting compare with regards to insights learned about the market's character, new patterns found, robustness of the systems tested, and overall productivity for testing a data series?

Thanks for your comments and responses on this topic.
More...




Personally, I get almost all my trading ideas from manual backtesting. After I see a "new" (to me) pattern or setup form while trading live, I then go back and look for the same pattern or setup over and over historically.

Then I try to come up with some kind of parameters for trading it -and then finally, backtest it manually (with substantial room for slippage and indicator lag) to see if it is a viable edge. I will literally go back and use "tick" marks to mark the entries/exits and tally up the p/l. Very time consuming.

Regards,

Paul


PS With embarassment but in all honesty I should point out that I'm a Tradestation user that can't program it yet so this is why I've done it manually (I'm going to EasyLanguage classes this month to remedy that)
 
in my short experience as a trader, i have found manual backtesting to be the largest waste of my time ever. wish i would have switched over to automated testing long, long, time ago....

the human mind is terrible at perceiving odds.
 
You must log in or register to reply here.
Back
Top