Manual Backtesting

[hypostomus]

...thanks for the very eloquent description of your thought/development process, which I suspect reflects your elegant thinking in general.

My only comment is that down and dirty rapid testing of an extremely simple rule which fails to test out may lead to an unsuspected rule which succeeds. This can happen because the hypothesized rule is "logical", leading to its opposite, an unsuspected market illogic.

It can also lead to a system through a coding error made in extreme haste (I have one such now that I am very proud of, in which I mistakenly coded a rule which is totally illogical).

Finally, a fast optimization over the full range of all the rule parameters may lead to an unsuspected system totally different from what you had in mind, such as discovering a retrace buy when your looking for the best sell stop. (The very best thing I have came about that way, and is laughably stupid, and market illogical).

In summary, not having your discipline, I get my best ideas from grubbing around erratically in the testing process looking for anomalies and making mistakes. As we say in the South, "even a blind hog occasionally finds an acorn."

 

[DblArrow]

Quote from TriPack:

Do you find manual backtesting useful, and if so what are some of the ways this has been helpful?

What is the best way you have found to do manual backtesting?

What process do you go through to make manual backtesting as objective as possible?

Are there any pitfalls of manual backtesting that others should be aware of?

How do manual backtesting and computerized backtesting compare with regards to insights learned about the market's character, new patterns found, robustness of the systems tested, and overall productivity for testing a data series?

Thanks for your comments and responses on this topic.

Unable to automate backtesting I cannot compare the two.

I have found manual backtesting to give me the confidence in seeing the setup and the results of the setup.

Setup the chart with what is desired and scroll back a few months and then scroll forward one bar at a time (day, 5 min, 89 tick, whaterver) analyzing each bar for the proper setup before scrolling to the next, writing down any action to be taken before scrolling to the next bar.

One pitfall perhaps could be overoptimization(?). But I also believe that if it worked yesterday and the day before it should have worked 23 yrs ago, and should also work into the future.

For me going back over the charts in my time frame - 5 min - I get a better feel for the market itself. Not just some results that say 59% of the time this worked. Well what did it look like when it worked and what did it look like when it did not work.

Make 'em pretty, Chris

 

[harrytrader]

This is in theory; in real it's often like that: you do first then you can make an idea/thesis . I have never been able to create any great idea/thesis before experimenting first by using a pure exploratory process.

Quote from TriPack:

Just for the record, none of my praise of manual backtesting should be taken to mean that I've abandoned automated or computerized backtesting. I think I've just gained a slightly better perspective on how the whole process fits together. Everything has its place and proper order. Prophet and a few others talked about that in their posts.

As I see it, the basic system development process flows something like:

Information + Thought -->

Idea/thesis (brain) -->

comparison of thesis with empirical data (manual backtesting chart/data) -->

revision of thesis (intuition - repeat until thesis becomes rule based or is discarded) -->

testing (backtesting - automated is more efficient at this point with larger data sets) -->

refinement (iterative cycle of testing and refinement based on spot checking of results)

After this point you get to the phase where parameters are set, the logic is complete and the out of sample testing, and forward testing can be done. Then it is just a matter of what form will work best for trading the system (manual / automated / hybrid / other).

 

[Sam Mcgee]

I find in most cases a manual backtest is lot faster. I can scroll through a chart and check a method for about 30 trades much quicker than I could program something. 30 trades is usually enough to tell you if a method is worth further investigation.

I've attached a simple spreadsheet that I use for manual backtesting.

 

[Gringinho]

Quote from prophet:

IB's data does seem superiour in many ways.

Yes, except that sometimes trade ticks at both bid and ask seems to be aggregated/summed. It's really nice with a minimum of delay though. I guess it can give an edge on automated systems.

Matlab’s strengths are ease of coding, rapid prototyping, and a huge set of vector/matrix manipulation and visualization methods. Once you have an infrastructure in place, manual experimentation in Matlab is a pleasure compared to experimenting in C. Performance-wise, vectorized, interpreted Matlab code can be just as fast as (loop-based) compiled C code. Anything scalar or non-vectorizeable can be done as a compiled MEX function. So you have the best of both worlds available… low level and high level.

That was something I was hoping to achieve as well; with ease of visualization as one of the strong incentives for me - becuase that's normally a pain, and not something really core to what one want to spend time on.

Sorry I don’t know much about third party sockets packages. I write everything using publicly available code as examples, such as the IB TWS TestSocketClient program, and various other TCP socket codes. You can also avoid sockets a great deal by receiving data via the TWS API in TwsSocketClient.dll.

Yes, that's true. Although then you are basically adhering to their standard.
BOOST is an attempt to standardize a lot of routinely stuff done in C++, like htey have standardized things in the standard Java libraries. Having to construct various types of exceptions and morphing of data can become very tedious, and error-prone too, when everyone do their own implementations. Therefore e.g. Sockets are being related to other standard classes like IO Streams, which greatly enhances and simplifies handling of Sockets in this case. It also has a well-thought out exceptions model, which is rather important.

The specific library which looked like the best candidate a few months back was something called socketstreams. The charter for Boost sockets are on : http://www.crystalclearsoftware.com/cgi-bin/boost_wiki/wiki.pl?BoostSocket .
Boost are part of the continous C++ standardization process, so they will become more mainstream after a while. Their libraries are frequently offered with newer C++ compilers too.
The Socketstream library I got was from http://groups.yahoo.com/group/boost/ - file area => http://groups.yahoo.com/group/boost/files/socketstream/. You can read more about it on the Boost developers/users mailing list archive on their site.

Here's another interesting article on another useful C++ topic: Threads - http://www.cuj.com/documents/s=8470/cuj0205kempf/ (C/C++ User's Journal - Boost has a subsection there).

The lack of a complete Boost-library is one of the reasons why I prefer doing my prototyping in Java for the time being.

 

[gramps]

dblarrow said it better for me.

It is like posting daily charts by hand in the olden days.

 

[TriPack]

Quote from DblArrow:

I have found manual backtesting to give me the confidence in seeing the setup and the results of the setup.

Make 'em pretty, Chris

Yes, the value of confidence in the system, in seeing the setups, in being able to execute according to the ruleset is no small thing. How do you put a value on that? This is really a valuable side-effect of in-depth study of the market and deep understanding how price moves from bar to bar within the context of the system. Excellent observation.

 

[TriPack]

Quote from Sam Mcgee:

I find in most cases a manual backtest is lot faster. I can scroll through a chart and check a method for about 30 trades much quicker than I could program something. 30 trades is usually enough to tell you if a method is worth further investigation.

I've attached a simple spreadsheet that I use for manual backtesting.

Yes, I've noticed the same thing.

 

[Turok]

DblArrow:
>Unable to automate backtesting I cannot compare the two.

*snip*

>But I also believe that if it worked yesterday and the day
>before it should have worked 23 yrs ago, and should also
>work into the future.

The conclusion in the second quote is naive considering the content of the first.

JB

 

[DblArrow]

Quote from Turok:

DblArrow:
>Unable to automate backtesting I cannot compare the two.

*snip*

>But I also believe that if it worked yesterday and the day
>before it should have worked 23 yrs ago, and should also
>work into the future.

The conclusion in the second quote is naive considering the content of the first.

JB

The second in not a conclusion of the first. The second is simply a statement that I have come to believe.

I use a couple of indicators to trade with - they, as we all know, are based on price action. Regardless of what price does the indicator will respond the same way. i.e. a 20 ema will always be a 20 ema and the market will either trade above, below or at.

Now take a chart 23 yrs back and you will see that the market was either above, below or at the 20.

I have taken a chart from many years in the past, many various points, and manually checked my system. It has given me the confidence that I require in my trading.

What may change over time is how the market is played or reacted to by the various players, but that 20 ema sure doesn't know or care.

Why, might I ask, would you consider that naive, based on the fact that I have not the ability to automate backtesting?

Make 'em pretty, Chris