Back Testing Tip: Give your Algo a Placebo

[AutomatedTrades]

I wanted to pass on a tip for those of us that are testing Automated Models.

When developing Automated Models it is very easy to develop "Tunnel Vision".

We all want our models to be profitable so sometimes we unknowingly "curve fit".

Applied to your model now: Here is an easy solution by taking your queue from Drug Companies and how they decide whether a drug has legs in clinical trials.

Applying this to your model:

Test your model by assuming "Entries" are uniformly distributed between longs and shorts and assume a Poisson Distribution for time of trade. This randomness will shed light on whether your Algorithm works as a result of predictive behavior or (I hate when this happens) you curve fit some parameters or use risk management to make a "non predictive" model look good.

Run your back test with the Placebo. If the randomly distributed Entries perform similar to your actual model (use something simple like the Sharpe Ratio).

Well... "Houston we have a problem"

Save yourself some aggravation later on. Do this before forward test.


Cheers

 

[lindq]

Run your back test with the Placebo.

An interesting point, and one seldom discussed.

I discovered a few years ago that simply running backtests on various days of the week helps to uncover potential problems. And of course simply accessing various levels of the VIX when running backtests on anything associated with equities can be helpful.

 

[AutomatedTrades]

I agree with you. But a good discussion to have, especially for those of us that have more time sensitive Algorithms. Drawdowns are my enemy and to be minimized to the extent possible.
By using this methodology, it narrows the field of place to look to lower draws and hence be in a position to use moderate leverage.

So in my case, I run a medium frequency (FIX) algorithm that uses microstructure to enter.
This type of Algorithm really needs to be tested real time. Back testing is only useful to a point to get an idea of the Algorithms scope of profits.

What you are testing by using a "Placebo" is the whether your entries are "better" than random. Everything else that you assume in your model remains the same.

There are two major components of an Algorithm, Entries and Risk Management. On the risk side, you can choose to get out in a variety of ways.