How to tell an ATS will be profitable in the future

I

Iwilldoit

Hi there,

I am considering in investing in an ATS, no I will not disclose the vendor or the price.

I have read some articles on C2 (no that is not where I will be purchasing the ATS), which roughly describes to choose one that has favourable metrics over a long period (N).

1. What are the telltale signs its a scam/fraud? (important question to ask when dealing with vendors)

2. What particular relationships between metrics should I be analyzing? win ratio versus drawdown?

3. How can I audit the Tradestation performance summary to look for manipulation?

It currently has a 86.7% win ratio over a period of 160 trades (N=160)

It is not a blackbox.

Please help me, as I do not want to buy a 'lemon'.

Thank You.



@Visaria @TheNotSoGreatGatsby @rmorse @marketsurfer @Victor Newman @greaterreturn @Autodidact @globalarbtrader @fullautotrading @Handle123
 
Iwilldoit said:
Please help me, as I do not want to buy a 'lemon'.
More...

Nothing is guaranteed in this trading world -- you'll have to basically just jump in, and learn fast on your own o_O
...don't necessarily expect anyone to hold your hand, or provide you a road map to riches.
indian_jones_trader.jpg

...kind of think of yourself...as an Indiana Jones trader...
 
Last edited:
I am very skeptic about any backtest with Tradestation when they involve LMT orders (by default, these are deemed filled on a price touch instead of waiting for price to go through le limit).

Also, if all you have to judge is 160 trades, backtest results are too easily overfit to mean anything. If it is 160 live trades, which match closely the backtest for the same period, then it is worth deeper analysis of the results, in particular if extending the backtest to a much longer period or other markets shows consistency with the 160 results.

In any case, take your time, ask for a trial, and don't bet the farm on it, as past performance are not necessarily indicative of future results.
 
Also, is the system tradable live? How does it recover from a loss of connectivity, a local crash, etc ... backtest is one thing, live trading has a lot more issues
 
dom993 said:
Also, is the system tradable live? How does it recover from a loss of connectivity, a local crash, etc ... backtest is one thing, live trading has a lot more issues
More...

Well it runs as a "strat" that you import into NinjaTrader...

Thank you for posting a reply which actually addresses the issue at hand.
 
dom993 said:
I am very skeptic about any backtest with Tradestation when they involve LMT orders (by default, these are deemed filled on a price touch instead of waiting for price to go through le limit).

Also, if all you have to judge is 160 trades, backtest results are too easily overfit to mean anything. If it is 160 live trades, which match closely the backtest for the same period, then it is worth deeper analysis of the results, in particular if extending the backtest to a much longer period or other markets shows consistency with the 160 results.

In any case, take your time, ask for a trial, and don't bet the farm on it, as past performance are not necessarily indicative of future results.
More...

I can get a two week free trial...........so if I am profitable after two weeks should I seriously consider investing?
 
dom993 said:
I am very skeptic about any backtest with Tradestation when they involve LMT orders (by default, these are deemed filled on a price touch instead of waiting for price to go through le limit).

Also, if all you have to judge is 160 trades, backtest results are too easily overfit to mean anything. If it is 160 live trades, which match closely the backtest for the same period, then it is worth deeper analysis of the results, in particular if extending the backtest to a much longer period or other markets shows consistency with the 160 results.

In any case, take your time, ask for a trial, and don't bet the farm on it, as past performance are not necessarily indicative of future results.
More...

Which specific metrics are most important to analyze? sharpe? Drawdown?

Time period?
 
@dom993

Ok lets roleplay, lets pretend I am selling you this ATS......what questions are you going to ask me?


What metrics will you look at it? and what will be the focus of your audit?
 
Win ratio is very high and that's not a good thing. What was the backtest period? what order types were used? what instrument? Unless it's ES you will have some market impact if you have any size at all.
 
It sounds like you have no experience at all with automated trading systems. On the other hand, I am one of NinjaTrader's partners, specialized in automated trading systems - which puts me in the vendor category, but I can also give you a few pointers.

First, some systems are not tradable - by that, I mean one of the following:
  1. Their realtime behavior is different from their backtest behavior - very often the case for strategies on NinjaTrader which require CalculateOnBarClose=false, but can also happen with "specialty" bar-types, which H/L in backtest doesn't match the realtime H/L.
  2. They hold positions for extended periods of time (days or weeks), but offer no support to stop & restart the strategy without losing the management of the open-trade.
By running that system during a few weeks on Sim101, and backtesting at the end of each week for the week just ended, you'll get an idea whether any of this applies or not.

As for the backtest performance, you should ensure that the backtest is done using at least 1-tick of slippage for MKT & STP orders, and that the fill-type used is at least Default (never Liberal), but I would recommend using a filltype which actually guarantees the requested slippage (that's not the case for Default). Feel free to use the attached "BetterThanDefaultV3FillType", which you need to manually unzip, then copy to NinjaTrader/bin/Custom/Type, and after that recompile any indicator or strategy and it will become available.

Once you have a backtest with a guaranteed 1-tick slippage, look at the average net per-trade, in ticks. The minimum viable is a matter of taste, I trade some systems with a 5-tick avg.net/trade, but it is much better to see 10+ ticks, and probably anything at or under 3-ticks is a waste of money.

Another aspect to look at, is the entry-stop size, in relation to the entry-target size, as well as the % of entry-stop hit, vs % of entry-target hit. No hard & fast rules there (for me), but keep in mind that the % of entry-stop hit might increase over time.

A key aspect, which is very difficult to judge, is the likelihood of the system to continue performing decently in the future. I use a pretty elaborate method for my own systems, but without access to the system design, you are pretty much left with judging based on the number of trades in backtest (for a single setup, which you never know for sure is the case when you purchase a blackbox). Anything under 1000 trades in backtest will experience significant variations in the future, 1000-2000 would be the place to start, 3000+ gives you better chances of long-term stability.
 

Attachments

You must log in or register to reply here.
Back
Top