Monitoring an ATS's performance

dom993

dom993

I have developed an ATS using 18months of data, then did some blind testing of it on the prior 18month which showed similar performance, and I have been using it live for about 5months. It is a relatively low frequency intraday trading system, the total 36month backtesting has about 430 trades, a win% of 65% for an average payout ratio of ~1 (avg win ~= avg loss).

The live results for about 65 trades are about the same for win%, but the payout ratio is down to ~0.75.

To monitor the system's performance level ... so far what I have been doing is to compare the actual live performance (win%, peak equity, max drawdown) with MonteCarlo simultations using the backtesting distribution, looking at how many standard-deviations away are the live results vs. the mean in the MonteCarlo simulation. I am far from a statistician expert, and I understand that this method would work if the process modeled did follow a gaussian distribution, which is not the case for most trading related distributions.

What would you suggest in order to get a better assessment of the current live performance vs the backtested performance?

Thanks in advance
 
one quick thing you can do without burning any more data; just to compare if what you are seeing live is in sync with the data you tested is take all 5month rolling periods in the 36months you tested and see the mean profit per trade.

I am sure all of them wont be 1; some will be less; some more. That would give you about 30rolling periods. From there you could if you want get a probability for you seeing 0.75 avg. But once you see the numbers for each of the rolling periods it should be intutive whether what you are seeing is inline with that particular backtest period or not.

But without even looking at all just based on the number of trades; i would guess that it probably is in line.

-gariki
 
Winston:

yes, live money - I do monitor the difference between live result & "official" result (essentially stop slippage, although occasionally a lucky fill), that difference is insignificant.


gariki:

good suggestion ... I will definitely do that ... thanks
 
What would you suggest in order to get a better assessment of the current live performance vs the backtested performance?
More...

You can try different sampling methods, ie. instead of using totally random sampling, add some clustering variable to the mix.

I have personally used a method where instead of using random points from the curve, I just use the next datapoint for random X% of points.
 
If you are really running this thing live with live money (not running live in a SIM) then I would suggest you go back and backtest the days you have run live.

Are your fills & P&L similar?

The biggest thing guys try to do is over-optimize and spend way too much time crunching the numbers. Even if your backtest is 180* off from your real life fills - if they are consistent that is more important. As long as you know that the backtest is consistent vs. the live fills don't do anything.

For example, if you find that your backtest has a fill rate that's 90% higher - as long as that's consistent then keep it as it is. Just adjust your backtest results to accurately reflect the real numbers.
 
Quote from WinstonTJ:

If you are really running this thing live with live money (not running live in a SIM) then I would suggest you go back and backtest the days you have run live.

Are your fills & P&L similar?

The biggest thing guys try to do is over-optimize and spend way too much time crunching the numbers. Even if your backtest is 180* off from your real life fills - if they are consistent that is more important. As long as you know that the backtest is consistent vs. the live fills don't do anything.

For example, if you find that your backtest has a fill rate that's 90% higher - as long as that's consistent then keep it as it is. Just adjust your backtest results to accurately reflect the real numbers.
More...

Infact, if you are trading live; another thing you should do (which i did and its very instructive) is code in the fills into the system - worst case. Its easy to do; delay your signals by a tick. Ex: say you are testing on /ES and you get a short signal is at 1225.50; test for 1225.75 between entry and exit of signal to filter out only signals that are guaranteed to be filled. This is worst case; since ocassionally you will be filled without the market moving past you.

-gariki
 
Thanks to all for your suggestions & comments.

- My backtesting engine requires price to go through LMT orders to provide a fill. It also assumes 1-tick systematic slippage on STP orders.

- The live performance on last 5 months vs backtested performance on the same 5 months is pretty much identical - there are a few differences here & there (actual slippage, a couple exits fills right at the LMT, and a few discipline errors on my part), but all in all this is normal

- The main drag on the performance in the last 5 months is a higher frequency of max loss (and lower frequency of smaller losses). Of course, I can relate that to the big swings we have had since August.

I have charted a 50-trades MA for average-per-trade & P/F (attached). I am actually quite surprised by it, I didn't suspect such big variations from looking at the cumul. P&L chart (will post this one in next post). The last big drop starts pretty much beginning of August (a couple weeks after I started trading it live ...).


mikkom: this is an interesting idea, but I don't see how I can do that with my MonteCarlo simulator (I use YASAI) - I guess I need to think a little more about it.
 

Attachments

(2nd try with this post)

Thanks to all for your suggestions & comments.

- My backtesting engine requires price to go through LMT orders to provide a fill. It also assumes 1-tick systematic slippage on STP orders.

- The live performance on last 5 months vs backtested performance on the same 5 months is pretty much identical - there are a few differences here & there (actual slippage, a couple exits fills right at the LMT, and a few discipline errors on my part), but all in all this is normal

- The main drag on the performance in the last 5 months is a higher frequency of max loss (and lower frequency of smaller losses). Of course, I can relate that to the big swings we have had since August.

I have charted a 50-trades MA for average-per-trade & P/F (attached). I am actually quite surprised by it, I didn't suspect such big variations from looking at the cumul. P&L chart (will post this one in next post). The last big drop starts pretty much beginning of August (a couple weeks after I started trading it live ...).


mikkom: this is an interesting idea, but I don't see how I can do that with my MonteCarlo simulator (I use YASAI) - I guess I need to think a little more about it.
 

Attachments

You must log in or register to reply here.
Back
Top