Frosty's auto-trading bot goes live with REAL money

Quote from walterjennings:

you would be saying the exact opposite if the fed meeting blew up your account. better safe than sorry. i understand interfering with the bot invalidates the results obtained by backtesting. but I think its counter productive to automatically assume any input you have to give to your bot will have a negative effect, even if the total profit at the end of the day is reduced somewhat. I often exit positions early when I feel its right, if I end up with less profit b/c of it, I just look at it as paying for piece of mind. my 2c on the subject.
More...

I disagree with this philosophy for two reasons.

1) Wild, unpredictable moves while you are in a position may hurt, but they can also go in your favor. If they move against your algorithm's current position, it is completely feasible to add some logic in to bail on the position once the movement goes wild beyond a certain threshold (how you decide that is left to personal preference).

Stopping it as a one-time precaution like frost did, perhaps knowing that that logic is not yet in place is fine, but doing this on a continual basis is just leaving a key part of the system un-automated.

But more importantly...

2) The development of an ATS is, in most cases, with the goal of the system becoming FULLY automated (i.e. unsupervised). Any "babysitting" of the system beyond monitoring and correcting for errors that were technical in nature (i.e. due to disconnections, bad data, software bugs etc.) is, in my opinion, masking behaviour that will tell you more about how you need to augment your trading algorithm more than many other times during the days/weeks of live testing.

Eventually, even those "technical" bugs should not need babysitting if the hope for a system is to be running (nearly) unsupervised at some point. I add the "nearly" because every system is supervised on some timescale, when I say unsupervised, I'm referring to the practical goal of not having to watch the system trade on a daily basis.
 
waxwing is correct about a common pitfall in curve fitting systems. You always want to try and develop your strategy using as little data as possible, but still enough to be significant.....

If you are currently in strategy develoment mode you must be careful about avoiding this. For example lets say everytime you make a change to your parameters or entry/exit critiera... anything.. you make a change... Now if you take that change and run it through all of your historical data you have and then compare those results to the last time you ran this test then you are going to end up curve fitting if you do this everytime.... What you really want to do is take a small subset of this data. Then every change you make you run it on this data and compare the results to develop your strategy....... After many many iterations on this small sample of data and when you finally have what you believe to be your best strategy, then go and test the strategy on another subset of data or your whole historical database.... did your strategy hold up and continue to perform as expected? Or did it fall apart as soon as it hit some out of sample data? If it held up then chances of being curve fitted is much lower... if it fell apart.. well you know what that means....

Another thing to keep in mind is if you getting your strategy ready too often and then end up testing on all of your data you could inadvertly be using the results from your big backtest and end up curve fitting to all of your data... so you really want to avoid testing on all of your data until you really think you got a strategy ready to go...

At least this is my take on the whole backtesting issue....I am open to other ideas bout this issue.. I just know that a long time ago I ended up with some curve fitted strategies by testing on all of my data too often......
 
Incase anyone was wondering I just checked it out: since the bot has been live it hast lost a total of -$417 or somewhere there about...

Had 7 up days, and 11 down days.....

Two days to go before it reaches a month since i started on nov 14...... 1 or 2 good days and it could end up being positive for its first month LOL ;)
 
Frost - not to be a downer, but who knows... if the bot as it is now loses money over the long run, you can just do the opposite of what the bot would normally do. You can just fade the bot.

Sounds like a joke, but you never know...
 
There's been a good DVD published by Keith Fitschen called "Developing Tradable Systems". One thing that stands-out from that presentation: successful systems built for intraday trading are likely to be COUNTER trend models vs. trend-following (Frosty mentioned this was his premise initially). This guy has been around for a while (20+ years), so I trust his judgement on that observation. He has a somewhat famous system called Aberration that is a longer term trend-following system. So I expect he would have tried it on an intraday basis and found it not to have been profitable.
Just my 2 cents.
 
Quote from frostengine:

waxwing is correct about a common pitfall in curve fitting systems. You always want to try and develop your strategy using as little data as possible, but still enough to be significant.....

....
More...

I disagree. I know there is no way to show ES can be modeled as an i.i.d system in the short term.

but generally if you look at the chernoff bound, the larger the sample size m => the smaller the probability that the value is an epsilon outside of the expected value. implying there is no upper limit to sample size for any given epsilon. i think i am not off base with that example for building a trading system, i took that course half a year ago and hav'nt really used the material since so i am a bit rusty.

http://www.cs.princeton.edu/courses/archive/spr03/cs511/scribe_notes/0220.pdf

that being said, I know its easy to think using a bigger sample set will lead to curve fitting, but you have to realize that the sample set is supposed to representative of your whole problem space. so if the 4 year sample set trends up, then your ML system might learn to just buy and hold because that is generally the correct decision. you might say 'what happens if the trend breaks and we go into a recession.' well then the sample set becomes invalid to the new problem space. you are basically gambling that the sample set still is representative. that is the best you can hope for.

note the above is based on viewing the ats as a classification agent which attempts to assign a label of 'buy' or 'sell' to each piece of data in the sample set. the data points being a combination of the tick + previous knowledge.

dont get me wrong, over training / over fitting is a huge problem. thats why in conjunction to a large sample set, the hypothesis space, or rather the language used to describe the hypothesis space has to be as small as possible, the smaller the hypothesis space, the more likely it is to be correct. also known as ockham's razor, which has a mathematical proof in this situation.

example. you have 4 years of data. your system finds the best hypothesis to produce maximum profit. doing 400 trades. if you define your hypothesis as 'if tick = tick1. trade. if tick = tick2. trade." and so on. your hypothesis will consist of 400 rules and will perform well over sample set but will not be statistically valuable due to the sheer size of the definition. simple hypothesis + large amounts of test data. will give you the best estimate of the problem space, and if the problem space remains consistant your hypothesis should perform in the future.
 
Walter,

Good point. I was simply basing my view on what I had experienced previously.... perhaps using a large amount of your data would be a good route to go. However, keeping some that you did not use during active development can serve as a way of verifying to yourself if nothing else that "maybe" its not curve fitted.. after all the system never saw that data before... so it could serve as a way to put your mind at ease concerning over fitting..

I do agree that for an intraday system with several years of data it would be really hard to curve fit that system.... IF the trades are a lot... if it makes 40 trades over the 4 years then its probablly curve fitted.. you really have no way of knowing without some out of sample data to look at.. but if its making several thousand... then yea I doubt its curve fitted....

So i guess the bottom line to this is, if your developing a high frequency trading strategy then it probablly would not hurt to use all of your data.. but if its low frequency, at least in my opion some out of sample data could go a long ways in proving to yourself its not curve fitted.
 
I mentioned earlier in the day that I would show a screen shot of my bot's origional ES trading strategy.... so here it is, from first q of 2000 to now..


This is not the exact strategy I use but very close...... I adapted the one I use to better fit the way the ER2 trades.....

One thing your going to notice about this PL chart is it hit a MAJOR period where it just choped around for a while...... but over the long run.... it looks good...


Also wanted to point out that this assumed trading only 1 contract the whole time.. no matter how much equity it has it still assumed only 1 contract... which is unrealistic...... Had it increased contracts every 10k, which is what I would do, then it would have made several million over this time period..... but that isn't revelant.. thefore I show it trading only 1 contract as that is the performance I am interested in... I can always add contracts later and the performance should stay similar per contract unless i add too many...
 

Attachments

Quote from frostengine:

I mentioned earlier in the day that I would show a screen shot of my bot's original ES trading strategy.... so here it is, from first q of 2000 to now..


This is not the exact strategy I use but very close...... I adapted the one I use to better fit the way the ER2 trades.....

One thing your going to notice about this PL chart is it hit a MAJOR period where it just chopped around for a while...... but over the long run.... it looks good...
More...

it seems that the choppy area lasts a bit over 2 years? true that the long term performance looks good, but I'd be worried that after the first few years of good performance the system becomes invalid for those 2 years and is recently picking up validity again for whichever reasons. it might be an idea to consider trying to find a system that performs consistently throughout the 5 years of data. maybe within a certain standard deviation of weekly profit..

try plotting a moving sum with a window of 1 week in the past along side of your equity curve. the flatter / consistant plot would show long term validity. if not flat, then an increasing weekly return leading up to present time would be an even better indication of strong validity.

as an investment it might be fine. if you were using this system as a form of income those two years might leave you a bit hungry.

edit. interesting idea i was thinking about. modify your system to factor in changes in the moving window sum. allowing the system to switch between multiple systems or none depending on the how the moving window is changing. note the smoother the equity curve the better this method would work i believe. opinions?
 
just realized a mistake im my previous post. instead of graphing a moving sum window. graph the a weighted average of trade p/l over the past week. weighted by the time spent in the position.

the result would be the system trading the market and trading its own profitability.
 
You must log in or register to reply here.
Back
Top