Automated trading system outcomes and real trading results

V

vlad

The main goal of this thread is to demonstrate the real obstacles arising during automated trading the proprietary developed strategy. I will provide some history of this system and will shortly describe:
1. Total system trading ideology without any specific disclosing details,
2. Development history,
3. Hardware and software realization,
4. Back testing system results and forward system simulation,
5. Forward testing system results,
6. System real trading results,
7. System verification and correction,
8. Comparing the automated trading system calls with the real trading results.

Then, I will try to represent daily and monthly system statistics with the analysis of divergence between the system calls and the real trading results from 05/01/05 to 12/31/05. This analysis will include the reasons of this discrepancy and the ways of removing or decreasing its value. Let’s consider system degradation and how to detect it.

I believe this thread will give to the beginners some understanding the game they are going to be in. I wait for myself from this thread some advices and discussions of the listed problems from the more experienced participants, if it will be possible.

Some remarks. I do not sell something. No software codes, no Website references, no trading signals, and no review letters.
 

Attachments

Here's an example that drives system traders nuts. The data feed for TradeStation spits out at least three or four bad SPY prints every day (three refreshes today), triggering false signals in some cases. The difference between backtesting and trading in real-time is that when you're backtesting all data have been cleaned up on an end-of-day basis. In these cases, you have to verify the data and validate the signal on an ongoing basis.
 
That's not necessarily true. I record all data in realtime to a postgresql database and reply it as-is for cleaning, I don't go back and scrube the data first.

Quote from PoundTheRock:

Here's an example that drives system traders nuts. The data feed for TradeStation spits out at least three or four bad SPY prints every day (three refreshes today), triggering false signals in some cases. The difference between backtesting and trading in real-time is that when you're backtesting all data have been cleaned up on an end-of-day basis. In these cases, you have to verify the data and validate the signal on an ongoing basis.
More...
 
Jesus tap dancing christ. Let me try that again.

I don't clean the data before replaying and backtesting.

Quote from stephencrowley:

That's not necessarily true. I record all data in realtime to a postgresql database and reply it as-is for cleaning, I don't go back and scrube the data first.
More...
 
Quote from PoundTheRock:

Here's an example that drives system traders nuts. The data feed for TradeStation spits out at least three or four bad SPY prints every day (three refreshes today), triggering false signals in some cases. The difference between backtesting and trading in real-time is that when you're backtesting all data have been cleaned up on an end-of-day basis. In these cases, you have to verify the data and validate the signal on an ongoing basis.
More...

I just ran an identical test on daily and 390 minute data in TS8 and got 76 trades out of the daily and 84 trades out of the 390. This system is worst case in that it depends on data highs and lows wherein lie much of the troubles.

I'm thinking you test a system and then you roll it out with small capital allocated to it to "really" test it. Not much point in going with the "in sample-out of sample" routine if you are not too sure of the data anyhow.
 
Quote from stephencrowley:

How does one post calls on a millisecond timescale in the case of automated trading systems that make tens or hundreds of trades a day?
More...

To do this, you simply need a way to timestamp events to the nearest millisecond. I'm not sure if the off-the-shelf automation tools allow this, but this should not be difficult in custom built automation systems. For me, I record the timestamp of ever order BEFORE it is sent out. Then, I also timestamp the moment the order was confirmed by the brokerage order server, confirmed by the market, as well as when it was executed or cancelled. By knowing your system lag times, you can help troubleshoot additional slippage that may have been caused by additional delays in your order speed, allowing you to troubleshoot the cause of the delay.

For example, the following are some of the results I have gotten from speed tests with various brokers:

*************************
<b>Broker A:</b>
.... <u>INET:</u>
........ OS Confirm: ............... 0.305 seconds
........ Market Confirm: ......... 0.305 seconds
........ OS Cancel Confirm: .... 0.253 seconds
........ Market Cancel Confirm: 0.253 seconds

.... <u>BRUT:</u>
........ OS Confirm: ............... 0.273 seconds
........ Market Confirm: ......... 0.298 seconds
........ OS Cancel Confirm: .... 0.250 seconds
........ Market Cancel Confirm: 0.250 seconds

.... <u>ARCA:</u>
........ OS Confirm: ............... 0.294 seconds
........ Market Confirm: ......... 0.494 seconds
........ OS Cancel Confirm: .... 0.258 seconds
........ Market Cancel Confirm: 1.258 seconds

*************************
<b>Broker B:</b>
.... <u>INET:</u>
........ OS Confirm: ............... 0.047 seconds
........ Market Confirm: ......... 0.236 seconds
........ OS Cancel Confirm: .... 0.016 seconds
........ Market Cancel Confirm: 0.188 seconds

.... <u>BRUT:</u>
........ OS Confirm: ............... 0.047 seconds
........ Market Confirm: ......... 0.214 seconds
........ OS Cancel Confirm: .... 0.016 seconds
........ Market Cancel Confirm: 0.186 seconds

.... <u>ARCA:</u>
........ OS Confirm: ............... 0.047 seconds
........ Market Confirm: ......... 0.205 seconds
........ OS Cancel Confirm: .... 0.015 seconds
........ Market Cancel Confirm: 0.192 seconds

*************************
<b>Broker C:</b>
.... <u>INET:</u>
........ OS Confirm: ............... 0.048 seconds
........ Market Confirm: ......... 0.048 seconds
........ OS Cancel Confirm: .... 0.015 seconds
........ Market Cancel Confirm: 0.015 seconds

.... <u>BRUT:</u>
........ OS Confirm: ............... 0.047 seconds
........ Market Confirm: ......... 0.047 seconds
........ OS Cancel Confirm: .... 0.016 seconds
........ Market Cancel Confirm: 0.016 seconds

.... <u>ARCA:</u>
........ OS Confirm: ............... 0.047 seconds
........ Market Confirm: ......... 0.047 seconds
........ OS Cancel Confirm: .... 0.016 seconds
........ Market Cancel Confirm: 0.016 seconds
*************************

Note: OS = Order Server, Market = the specific ECN

Each number shown is the average of ten orders placed through the specific broker to the specific route. As you can see, the speeds can vary considerably. For some trading styles, an extra 0.25 seconds may be insignificant and not affect results in the slightest. However, for other trading styles, the speed difference can be important.

Best of luck,
-Eric
 
You must log in or register to reply here.
Back
Top