Yah. it definately is.. I'm taking a few days off here and there but its "crunch time" for a huge project that is about to go live.. cant really take a lot of time off. If I knew I could quit anytime then it wouldnt matter.. but I really don't want to screw anyone nor have all the pressure of having to live off trading income (yet), also I'd prefer to let everything compound rather than have to take money out for living expenses. 0.25%/day regularly would be outstanding, 1% would be absurd.. it'll probably be somewhere in etween. I keep finding more great research material and books and I keep refining the models rather than throwing it out there.. I the 'project manager' in me to just hit me over the head and get the 1st release out already.
1% a day consistently: possible?
- Thread starter stephencrowley
- Start date
Frankly, I couldn't give a shit less what most of you think. (With the exception of a few) I've found people to correspond with outside of ET that actually do this type of thing for a living and that are infinitely more helpful and knowledgable than most of the jackasses here.
How much extra profits will this get you? By doing this you are creating a feedback system and given yourself at least an additional 2 parameters to optimize - the learning rate and the time lag of the new training data you feed it. If the learning rate is too high you could end up with an unstable system as it tries to adapts to quickly to new data. Additionally, your backtesting would now also need to incorporate model optimization - with 2 additional parameters adding 2 additional degrees of freedom and more risk of curve-fitting.
If on the other hand you think that new data would not change ths system too much during any one trading day, they why even bother with this.
You are absolutely right about speed of adjustment choice. There is a precise way to find the optimal "model re-estimation rate" using maximum-likelihood methods.
Say you have some model for predicting 'something', and you calibrate the model paramters, use statistical tests to make sure they arent overfitted (using regularization methods, cross validation, out-of-sample-testing) whatever.
Those parameters are the 'best' estimate over the period in which you analyzed, so, you apply them to trading during your next period, those parameters will, at best, slowly drift over time.. hopefully they drift slow enough for you to make some money before they are useless.
So, how do you estimate these parameters in the first place? Take a sample of the past N days/months/years? Standard techniques apply equal weight to every point in time, e.g. something that happened 6 months ago will have as much ifluence as parameters on something that happened yesterday. This sucks, to say the least.
You could apply some sort of weighted regression, where the times in the past are weighted by how long ago they happened.. btu the problem here is that something that happened in the middle of the sample, might indeed be more important than something that happened yesterday, but would still have less weighting applied. Better, but still sucks.
Or you could implement stochastic parameter estimation, where you start with some 'initial estimate' and 'initial covariance', normally you set the initial estimates to just 0 and the covariance to some huge number on the diagonals of your parameter cov matrix. Now, every new 'sample' that arives (one day, hour, year, month, nanosecond, etc), you calculate some updated estimate that would fit this sample. and then you reconcile that with the current state estimate. e.g., some of the new observation is an actual model change, and some of it is noise.
Then the choice becomes what you mentioned earlier, track too fast and the model is extremely volatile and useless, track too slow and the model doesnt keep up with the data and becomes useless as well. The tracking speed is determined by the ratio between your observation covariance and your state estimate covariance.
So, using this method you dont estimate moel parameters directly, you only set 'hyper parameters' which affects how the algorithm modifies parameters in real time. These hyper parameters can be optimized via maximum likelihood methds which find the exact point where the tradeoff between bias and variance is minimized.
To sum it up: implementing this actualy reduces the number of parameters i need to optimize, it doesnt add to them.
Also, the stuff I trade does indeed drift during intraday timescales.. so dynamic estimation is critical, also, large overnight changes can affect stuff.. realtime estimation optimally combines all stuff from even point in the past, with everything in the near future (pre-market, opening hour.. etc) so that it tracks better during the day without being biased to whatever the market did during the last week.
Say you have some model for predicting 'something', and you calibrate the model paramters, use statistical tests to make sure they arent overfitted (using regularization methods, cross validation, out-of-sample-testing) whatever.
Those parameters are the 'best' estimate over the period in which you analyzed, so, you apply them to trading during your next period, those parameters will, at best, slowly drift over time.. hopefully they drift slow enough for you to make some money before they are useless.
So, how do you estimate these parameters in the first place? Take a sample of the past N days/months/years? Standard techniques apply equal weight to every point in time, e.g. something that happened 6 months ago will have as much ifluence as parameters on something that happened yesterday. This sucks, to say the least.
You could apply some sort of weighted regression, where the times in the past are weighted by how long ago they happened.. btu the problem here is that something that happened in the middle of the sample, might indeed be more important than something that happened yesterday, but would still have less weighting applied. Better, but still sucks.
Or you could implement stochastic parameter estimation, where you start with some 'initial estimate' and 'initial covariance', normally you set the initial estimates to just 0 and the covariance to some huge number on the diagonals of your parameter cov matrix. Now, every new 'sample' that arives (one day, hour, year, month, nanosecond, etc), you calculate some updated estimate that would fit this sample. and then you reconcile that with the current state estimate. e.g., some of the new observation is an actual model change, and some of it is noise.
Then the choice becomes what you mentioned earlier, track too fast and the model is extremely volatile and useless, track too slow and the model doesnt keep up with the data and becomes useless as well. The tracking speed is determined by the ratio between your observation covariance and your state estimate covariance.
So, using this method you dont estimate moel parameters directly, you only set 'hyper parameters' which affects how the algorithm modifies parameters in real time. These hyper parameters can be optimized via maximum likelihood methds which find the exact point where the tradeoff between bias and variance is minimized.
To sum it up: implementing this actualy reduces the number of parameters i need to optimize, it doesnt add to them.
Also, the stuff I trade does indeed drift during intraday timescales.. so dynamic estimation is critical, also, large overnight changes can affect stuff.. realtime estimation optimally combines all stuff from even point in the past, with everything in the near future (pre-market, opening hour.. etc) so that it tracks better during the day without being biased to whatever the market did during the last week.
I donât believe in your âmodel re-estimation rateâ.
The first thing to do is to build indicators that work well. This means that they should always be profitable (not on a daily basis but at least on a monthly basis). The indicators should function well WITHOUT ever changing the parameters. A good indicator is an indicator that stays stable even if the parameters are changed within a reasonable range. This is necessary to avoid signals based on noise. Re-optimising indicators is adapting them to the past, without any guarantee that they will work in the future.
I have indicators that are profitable since 10 years without ever having to change anything neither in the formulas nor in the parameters. They are self-adapting based on a trend system. But this is not what you do. You just take a lot of quotes and try to find the optimal set up for this past period. If you combine that with a weighted regression it might even get worse, because if you re-optimise at the end of a bull market you will get slaughtered in the coming bear market.
The basic of all the good trading systems is the trend. Based on the trend you built indicators that are fitted for that kind of trend. This implies automatically that the tracking speed will vary according to the strength of the trend. Strong trend will give other parameters than weak trend or no trend. In this way you never have to optimise anything because each kind of trend has its own set of indicators. It is not logical to develop a system that covers all kind of markets, but most traders are too lazy or unaware of this.
Compare it with the set up of a racecar. For all kind of weather types there are different set ups, even per track there can be differences. So the ideal set up doesnât exist and can never be developed for all kind of situations. If you use a set up for a Californian hot summer race and drive it in Sweden in the middle of the winter, you can change the parameters as much as you want, but you will never achieve a satisfying result. Same things is true in trading.
Develop set ups for each type of trend and apply them only in the corresponding type of trend. You will not have to re-optimise your system, because the optimisation is in the link between the trend and the applied set of indicators.
What you mentioned might very well work, but with all due respect I don't think you understand what I'm doing (part of that is by design).
I don't "re-optimize", it is a continual optimization process (in my case, happens about once per second). No matter what sort of "system" you have, that system isnt perfect at every moment in time in which it is inoperation. The "true/most profitible/perfect" parameters (which cant be reached in reality, but that is what you are shooting for) are time-varying.
So essentialy, I have a "bunch of numbers" that define my trading rules, and so do you, even if the ideas in your head are numbers. Now, these numbers that define the strategy arent static (you say yours are, but i find that highly suspect). Then these parameters must change over time. If the parameters are changing over time then you can model HOW these parameters change, and it might very well be that the movement of these parametrs over time are more predictable than the things you are prediciting. In fact, this will be the case.. if your parameters are more volatile than the things you are predicting then that doesnt even make sense..
I don't "re-optimize", it is a continual optimization process (in my case, happens about once per second). No matter what sort of "system" you have, that system isnt perfect at every moment in time in which it is inoperation. The "true/most profitible/perfect" parameters (which cant be reached in reality, but that is what you are shooting for) are time-varying.
So essentialy, I have a "bunch of numbers" that define my trading rules, and so do you, even if the ideas in your head are numbers. Now, these numbers that define the strategy arent static (you say yours are, but i find that highly suspect). Then these parameters must change over time. If the parameters are changing over time then you can model HOW these parameters change, and it might very well be that the movement of these parametrs over time are more predictable than the things you are prediciting. In fact, this will be the case.. if your parameters are more volatile than the things you are predicting then that doesnt even make sense..
I think you will find that successful professional money managers, CTA's, CPO's, HFs etc probably have a lower return on average than successful individual traders. The reason for this is that most successful traders are comfortable with themselves and their edge and thus are willing to accept larger drawdowns than most investors are. When you are playing with OPM you have to account for the clients risk tolerance, and adjust accordingly. Doing this will lower overall returns in most cases.
That said I don't really know what is with the sudden rash of XX% per year threads. Maybe a lot of newbies coming back in, which is great. At the expo I saw a booth that claimed you could make 5% per day trading their system, I was tempted to just hand over all my money to them and offer them a 90/10% in their favor if they could do that for me
If you read the returns posted by all the market wizards they average out to around 60% per year. I can't remember exactly now what it is, but I did average it once and it was in the 60% area if I recall correctly. If you can do more than that you are set.
I do know a number of daytraders and swingtraders who have returns in excess of 10% per month, which I feel is a reasonable goal for a person trading a small account (under 250K). But, keep in mind that this is not scalable, so you will not be compounding it. 300K per year is nice but not anything to write home about either.
The best return I know of is a friend of mine who did ECN arbs. He averaged around 9% per day, but his typical trade was 77 shares. The system only required $8000 in it, and anything above that was not going to add anything to the return at all. So, I guess with very small edges anything is possible, but as the accounts/trades get bigger then its less and less likely you will be doing 10% + per month. Again I know a number of people who do that with under 250K, a couple who do it with a few million and none who do it with more than $10. Maybe I just need to get out more, but that is my experience.
I don't "re-optimize", it is a continual optimization process (in my case, happens about once per second). [/QUOTE]
A continual optimization is re-optimising, only your frequency is much higher.
The "true/most profitible/perfect" parameters (which cant be reached in reality, but that is what you are shooting for) are time-varying.
[/QUOTE]
That is your opinion, but that doesn't have to be the truth. Drawdown and return are objective criteria to judge on. Based on these criteria i am sure they are NOT time-varying. My parameters NEVER changed over time. But i agree that the optimal indicators don't exist. All we can do is try to reach the highest level of perfection.
A continual optimization is re-optimising, only your frequency is much higher.
The "true/most profitible/perfect" parameters (which cant be reached in reality, but that is what you are shooting for) are time-varying.
[/QUOTE]
That is your opinion, but that doesn't have to be the truth. Drawdown and return are objective criteria to judge on. Based on these criteria i am sure they are NOT time-varying. My parameters NEVER changed over time. But i agree that the optimal indicators don't exist. All we can do is try to reach the highest level of perfection.