Walk-Forward Testing and Optimization

[Wide Tailz]

Quote from intradaybill:

If you optimize a system just before a choppy market turns into a trend, then you miss the trend. Trading system performance is path dependent.

This is why part of the process of being successful involves gauging the time frame to capture the most relevant paths which encompass trends and chop. I had my strategies fail if trained on only 1 year of data, but hold up if given 4+. The difference: variance in the signals therein gave the opportunity to adapt to more possible paths.

Same thing seen in nature. Super specialized organisms can only make it in their natural environment. An example of the reverse would be the mongoose, which is considered highly dangerous to non native ecosystemz because of how successful a predator it is. So too shall be your trading system....if you know what environment to design it for.......

 

[savagemp5]

I started writing algorithms and strategies with forward testing only after I had a stable and complete system, and knowing in mind what I wanted to write (hypothesis).

Tested them onto several strats on a wide array of securities, stocks, and futures in global markets.

Next I started to optimize. Its interesting to find some strats work well for some, not for all.

What is optimizing you may ask ? Some parameters:

- Start/stop time.
- Duration
- Contract size
- Suitability
- Stop Losses big/small
- Sensitivity for individual graphs
- Combining triggering of signals

How do I know what and how to optimize ?

- Win/loss ratio / percentage
- Patterns or reasons why it wins or loses
- Probabilities / fat tails
- Accuracy of signal detection and trade management
- Algorithms to intelligently manage the trade


Lastly: Pls don't believe my BS since im also on the 95% side, but if it makes sense for you, good.

Some other info can be found on my website:

http://managemoneyonline.com/2012/01/research-results-of-systematic-algorithmic-intraday-trading/

 

[trade4ever2day]

Quote from savagemp5:

I started writing algorithms and strategies with forward testing only after I had a stable and complete system, and knowing in mind what I wanted to write (hypothesis).

Some other info can be found on my website:

http://managemoneyonline.com/2012/01/research-results-of-systematic-algorithmic-intraday-trading/

How is life these days in Singapore? Is it getting too expensive? How much a two-bedroom apartment costs?

Back to the issue of optimization I think it worths reading Harris's paper:

http://www.priceactionlab.com/Blog/2011/09/curve-fitting-and-optimization/

I too think that if when you are optimizing you are essentially moving entry points around then you may have nothing and you are just trying to find a best fit of a random system.

 

[Rationalize]

Quote from intradaybill:

I agree but since it is very difficult to measure DoF and added complexity of non-stationary systems, optimization must be avoided at all costs. This is my point. Mathematically you are correct (as always) but in practice it hardly works.

Edit: Plus I hope you understand that "optimal amount of optimization" leads to an infinite regression and thus it cannot be resolved to a sound deductive procedure.

Bill - I agree with your comment quite a lot.

Adding my 2c:

Without a solid microstructure or market structure based explanation, there's usually very little "there". Optimisation outside of fair value, flow and inventory models are generally at best a time consuming distraction. To take it one step further, I could propose that all the information about the next price is in the last price(s), and the order book(s). Nothing else adds new information.

 

[ssrrkk]

Quote from Rationalize:

Bill - I agree with your comment quite a lot.

Adding my 2c:

Without a solid microstructure or market structure based explanation, there's usually very little "there". Optimisation outside of fair value, flow and inventory models are generally at best a time consuming distraction. To take it one step further, I could propose that all the information about the next price is in the last price(s), and the order book(s). Nothing else adds new information.

I agree with this -- if there is no real signal, of course, optimization will only mislead (i.e., cause you to trust a system that is guaranteed to lose money). But if there is a real physical phenomenon that you are capturing that is defined by a physical parameter (e.g., the fair value as you say, or the amount of order imbalance, etc.), then I believe it is worth obtaining accurate estimates of that parameter. Also, I don't believe there is such a thing as a parameter-less system. There are always parameters: you may have cleverly made them proportional to (or some function of) other properties of the system, but you still need to calibrate that proportionality (or function).

 

[savagemp5]

Quote from alexandermerwe:

How is life these days in Singapore? Is it getting too expensive? How much a two-bedroom apartment costs?

Back to the issue of optimization I think it worths reading Harris's paper:

http://www.priceactionlab.com/Blog/2011/09/curve-fitting-and-optimization/

I too think that if when you are optimizing you are essentially moving entry points around then you may have nothing and you are just trying to find a best fit of a random system.

Well it's quite expensive. Just recently the "COE" (a piece of blank paper that says you can drive on Singapore roads) costs about SGD$80,000, not including the car.

A 2 bed-room, assuming its a private apartment or a condominium costs somewhere 700k-1.2mil, depending on location. Foreigners would need an additional 13% levy, so you do the maths.

btw USD/SGD @ USD1 : SGD1.266

 

[savagemp5]

I always believe optimization is something like = algorithms

where you write specific details about how do a particular action or calculation, smarter and better inteligent than if there wasn't any in place.

A trial stop is one that beats a dead stop loss. A percentage-base stops loss may be better or worst than a fixed point/price-based stop. Add that with time ticks and other filtering mechanisms, and it would be much better than just a stop loss as an example.

 

[wwatson1]

If you torture the data long enough, it will confess to anything