Optimizing or curvefitting
- Thread starter indahook
- Start date
Quote from OddTrader:
Mike, you knew that I alone was just trying to mislead you. Didn't you?
Many thanks for your sharing.
BTW, how would/do you evaluate the robustness of individual systems? Backtesting alone? What/How is the basic approach?
Excuse for too many questions.
Q
It all depends on how you look at it.
If you look at each series of trades on a specific market as only that, then you are evaluating robustness.
But if you look at each series of trades without taking into consideration where and when they are produced, then you evaluate the system's consistency.
--- Trading Systems and Money Management, by Thomas Stridman
UQ
...your excellent advice came at a time when it was clear to me that a better approach to abandoning a failing system was needed. Prior to that I had given up when the number of down days exceeded the maximum number in backtesting, but this tended to produce losses in excess of the maximum historical drawdown. So I started designing for a tighter stop at the expense of lower net profit. That makes playing a system until it fails a little less costly. IMO you have to do that, because systems tend to turn around dramatically just when you're ready to kiss them off.
Robustness? Continuing to work six months after the original design! Also not much variation in parameters from optimization to optimization. And being able to associate some market logic with the system.
Robustness? Continuing to work six months after the original design! Also not much variation in parameters from optimization to optimization. And being able to associate some market logic with the system.
I thought robustness is how a system would do over a long period of time over a variety of assets.
I would think a system that is only working over 1 asset, is less robust and more optimized.
Could you please explain this to me,
Thanks much,
larryTAKOUT@seldin.net
"there are only three variables in trading (time, price, and volume), so with sufficient algorithm development skill IMO there is no hypothesized pattern in market action which cannot be coded and tested."
These three variables you are quoting are again based on all the factors and players in the market. The market also reacts to news and other external influences (e.g. mortgage rates).
When I was making models to predict 24h energy markets, it was rather straightforward in most cases because of very strong correlation to date-of-year, day-of-week-, hour-of-day (consumption fluctuates in a very predicatble way - e.g. when work hours start, people return home, go to bed), temperature (sometimes also weather forecasts - but they may leverage errors), level of reserves (water-basins in this case, because of heavy influence of hydro-power) and average aggregates of previous prices. Using this and neural networks with a population mutated using genetic algorithms (which is a common technique) we got quite good results, beating most analysts every day.
But as you can see from the description above, we did not use only market-internal data like trades/volume, price and time/date.
I don't belive the majority of traders only use time, price and volume/trades in their decision-making. Therefore, you would at least need differently geared models for various types of markets and trading hours, or you would need some type of seeding your models. I'm not only talking about only neural network models here, but any regression-based or mathematical based models.
It also depends on the goals of the model, where trying to predict prices or price-levels is pure folly in my opinion, because of the complexity of the markets.
For instance, on average, any model just based on the three basic variables would be mostly useless anticipating some major news or macro-economical data.
On the other hand, simplicity is king, trying to avoid periods of "lag" when the model needs to readjust.
Watching the markets, some aggregated variables are quite dominant at times (like the stochastics-types), showing that many follow the patterns of those. Still, it is in my opinion best to calibrate any trading model into looking for small gains (however trailing any continuing "trend"), because of the extremely difficulty in predicting price levels or momentum going forward. There are just simply innumerous traders - each with many strategies - out there.
What are the aggregated variables you use ? Do you guys e.g. use raw acceleration/deceleration in deltas of variables ? Or size of deltas ?
Quote from OddTrader:
Q
It all depends on how you look at it.
If you look at each series of trades on a specific market as only that, then you are evaluating robustness.
But if you look at each series of trades without taking into consideration where and when they are produced, then you evaluate the system's consistency.
--- Trading Systems and Money Management, by Thomas Stridman
UQ
Many thanks for your input.
Q
A robust system is a system that works equally as weel, on average and overtime, on several markets and market conditions.
...
But it's also fair to say that we looked at each system's consistency, or how well the system is likely to perform in several continuous and well-defined time periods, such as quarterly or yearly.
--- Trading Systems and Money Management (Stridsman)
UQ
Would you have any comments on the above from your experience?
...re your Stridsman quote, I would have to disagree in light of my understanding of Katz and McCormick, who found little applicability of any one system across all markets. But I only trade NQ, which is confusing enough for me. My definition of robustness is "the system doesn't make me feel like I was a fool when I developed it". Right now I am a twenty-fold fool.
Please note I'm only a newbie with some opinions.
Basically I think your understanding of the topic above would be seemingly better than mine.
imo, a "good" system would be very much determined by the designer's mindset and knowledge about the characteristics (you name them) of the target markets/issues.
...I have a home-grown definition of robustness. I am not smart enough to understand even the one instrument I trade, much less the entire market. I approach system design statistically, and attempt to understand why a system might work only after it is developed. I find that most "logical" system ideas fail ignominiously.
My definition is:
- the equity curve is monotonically increasing
- the equity curve doesn't change slope much when volatility changes
- the optimal parameters don't change much over time
- the maximum number of losses in a row doesn't change much over time
- flat spots in the equity curve don't last more than about ten trades
- profitability doesn't depend on one big trade out of many losers or scratches
Obviously this definition is highly idiosyncratic to my anxiety-prone psychology, but it works for me.
My definition is:
- the equity curve is monotonically increasing
- the equity curve doesn't change slope much when volatility changes
- the optimal parameters don't change much over time
- the maximum number of losses in a row doesn't change much over time
- flat spots in the equity curve don't last more than about ten trades
- profitability doesn't depend on one big trade out of many losers or scratches
Obviously this definition is highly idiosyncratic to my anxiety-prone psychology, but it works for me.