Frosty's trading bot goes live part 2

[TSGannGalt]

Quote from walterjennings:

not true for all cases. i found one difference between sim and live which effected me was when my system attempted to grab liquidity which faster systems got to first. obviously that had a negative effect on the system and has nothing to do with curve fitting.

IB gives you a sim fill for a market order at the next 'last price' after you transmit (not realistic), with limit orders IB gives you a fill only if someone else trades at your limit price (also not realistic).

If you're talking about high frequency trading (and some scalping), then sure... liquidity, latency and computer will greatly affect the performance.

If minor data issue was a problem, he should be setting up his trading environment differently. More-so, he "really really really" needs to step-up his testing skills and equipment going into every detail from acquiring access to institutional level quotes to execution...

Frosty is using a time bar chart making only 3 trades a day. He's trading style is supposedly trend-following (intraday swing?). The problem doesn't lie in the quote and execution platform. If this were true...

He's hypothetically made around 60 trades... if it's a point slippage for each... he's still deep down... or is he getting +5 tick slippage for each trade????

Considering the system to be working... let's all be serious... he's not losing because of unnecessary slippage....

 

[maxpi]

Quote from BigBubba:

i think (but can't prove it yet) that the data from the IB's simulated account vs the "real" account is slightly less than identical. i base it on 2 identical systems running side by side on each account. i have not looked closely at it however, so.. you prolly will be able to confirm (or deny) this assertion when you move fwd w/ your testing..

it's most likely insignificant..if at all. just something i've noticed.
bb

I actually talked to the developers about differences between the sim acct and the real acct. They said that they tried to make it as close to real as they could. Obviously you are correct in that it will take a little experience with it to know what the differences really are. I suspect that it would not be hard to have the data be the same but getting filled on a limit order is another story possibly. There is supposed to be a way to log into both accounts on the same computer, I might try that tomorrow since currently I am stuck waiting for a software order to come through and have nothing but playtime all day...

 

[nonlinear5]

Quote from TSGannGalt:

Considering the system to be working... let's all be serious... he's not losing because of unnecessary slippage....

I think it is slippage. As I calculated and posted before, if the slippage is only two ticks greater in live trading than what it was assumed to be in Frost's backtest, the entire spectacular backtest equity curve would collapse. From the trading stats that have been posted (less than 0.5 points average profit per trade in backtest), it can be easily seen that the bot would be hypersensitive to fill prices: lose two ticks on entry, another two ticks on exit, and now your average profit per trade is 1 tick. So there.

 

[TraDaToR]

Quote from TSGannGalt:

Does this seem right?

Also, there is no such thing as "true edge". They will always go away, just like the trend in a market will change.

Yes, but there are niches were you really feel you got "something" and you can hope it will last a certain time.

For example, one of my systems I'm trading for 4 months just made one losing ( almost breakeven ) week, making 30-40 trades per week, with extremely stable results. It's the first time I feel like I have somewhat of a "true" edge.

 

[mujoh]

Quote from brownsfan019:

I'm really curious where the bot can actually profit and not a couple hundred bucks either. It doesn't really profit on huge moves to the upside and not so great on choppy days either.... Frost, you have yet to see ONE day where the bot can make you SUBSTANTIAL gains. It's very good at chipping away and then BAM, big loss(es) ensue.

Needs to be the other way around in my opinion.

That's exactly what I was thinking. We have seen all different kinds of markets the last weeks: Range bound markets with low and/or high volatility, directional days with a slow and/or fast pace etc. In none condition the bot was able to generate big profits but big losses occur on regular basis.
From what I can see here your problem has nothing to do with data issues or something similar. It is your "edge" that seems to be flawed. So instead of burning any more money I would really recommend to stop trading this system with real money.

 

[GTS]

I could be way off but I thought that frosty's backtesting engine does the simulated fills so it is not relevent how IB's simulator handles market/limit orders.

I also thought that frosty ran recent sim-collected data through the backtest engine for the last couple of weeks and got nearly the same results as the live/real forward-testing.

If that's the case then all this talk about slippage being the problem or a differences with the datafeed being the big issue are all barking up the wrong tree.

This isn't rocket science. The only person who knows if the backtesting was done in a valid way or if the parameters were over-optimized is frosty. To everyone else in this thread it is a blackbox, data goes in and we get to read about the net daily result (not even the trade by trade breakdown).

All of these hypothesis about what is probably wrong are made without the benefit of the details - there is so much information that frosty hasnt shared.

Lastly, I'm not sure why everyone has their panties in such a bunch. Is losing $3k on ER2 the end of the world? Its only been trading a little over 3 weeks. Would you have a heart attack if you flipped a coin heads five times in a row?

If I were frosty and if I felt that I had developed the system correctly (didn't over-optimize) then I would continue to let it run. The sample size is still far too small to draw the conclusion that the system is fundamentally flawed, at least from an outsider perspective.

 

[walterjennings]

iv been thinking about the problem of over-optimization a bit today. i think people might be putting too much weight on it when it comes to trading.

ill use an analogy to explain. imagine you are writing a bot which visually classifies fruits. say your learning data has mostly apples and oranges (apples and oranges from 2006 to boot!). the system will learn to classify apples and oranges very well, but will be doomed to misclassify all other fruits for the rest of its existence. this system has been over optimized to the unique fruits in the data set right? though it fails its true goal of classifying fruits, it still can classify apples and oranges correctly outside the data set.

in trading this is what we want, you dont need to know the correct choice of every possible new tick. you just need to be very good at picking out the apples and oranges. and you need to make every trade you make based on apples and oranges.

if my data set was large enough, and the grammar my a.i could use was simple enough, and the vapnik-chervonenkis dimension was small enough. id have no issue saying 'regardless of training time, over-optimization, curve-fitting, if the hypothesis was profitable on everyday in the sample set of a year, the max seen drawdown for any day was less than 100$, and I can clearly define the max loss on any given trade, this hypothesis is most likely a good predictor of a subset of profitable trades in the current market.'

sure if you have a hypothesis that says, if time = 5:43:21 and size = 32 then buy. you'll have a uselessly overfit system, but by limiting the vc-dim you should prevent that. if you do 1000s of trades and your hypothesis is fairly simple, there is no way you are defining each individual trade and must be defining a much more useful subset of trade situations, ochams razor at work.

thoughts?

 

[sulli]

Walter, are you saying that you believe if one were to optimize over a very large sample of data, that it would be hard to over optimize?

Just trying to get a handle on your point?

 

[frostengine]

I would like to point out the issue is NOT slippage. As I have stated before everyday I verify my actual results with what the backtesting engine said it SHOULD be for that day. Those results are well within reason. Therefore the slippage is not causing the issue. In fact more times than not I am OVER estimating slippage in backtesting compared to the actual results.

The differences I speak of are occuring between the live and sim mode accounts. They are for the most part pretty steady, but there are several instances where they go VERY different.. as in one may trade 3 times while the other traded 4 times..... also, the sim account is not always doing better than the real account. Neither account seems to be doing that much better/worse than the other... BUT some individual days are very different than each other. This just supports the fact that there are some very real differences between the data being collected from both sim and real accounts.

I am not making excuses as to why my bot has performed so poorly. Instead, i'm simply looking at all angles and discrepancies that are showing up and trying to find a solution.

 

[rdg]

Quote from walterjennings:

ill use an analogy to explain. imagine you are writing a bot which visually classifies fruits. say your learning data has mostly apples and oranges (apples and oranges from 2006 to boot!). the system will learn to classify apples and oranges very well, but will be doomed to misclassify all other fruits for the rest of its existence.

what if it uses color as the determining factor? if it only knows apples and oranges, then any orange fruit must be an orange and every non-orange fruit must be an apple. and when you put it to work, it thinks all bananas are apples. and all watermelons. and kiwi. and strawberries. and grapes. so now you have a bot that can't really identify any fruit correctly even though you think it can identify apples and oranges perfectly.