You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.
You should upgrade or use an alternative browser.
10 months fulltime and still no edge
- Thread starter estim
- Start date
I was in what sounds like a similar position you were after 10 months. I was very discretionary and just winging it. I was making a little money, enough to cover comms which were very high, but not taking home much and swinging way too much relative to what I was taking home.
About 14 months in something just clicked. I started focusing on just a few key setups... things that were very tangible and defined... I began to take more size on fewer trades instead of less size on more trades. I'd wait for a few trades per day to make my day, because I had seen that pattern over and over and over and over. But only having traded tens of thousands of times over 1000+ hours, did my discretionary become mostly mechanical. It just clicked. Since then I've had 9 straight profitable months... all (for me) good months. There is no saying when things will click with you, but if you keep it at and manage the risk, you will find your edge, your niche, and focus on that. Your patience will increase, your ability to take size will increase, and your confidence will grow. From what you write it sounds like you are on the right path, just keep at it.
About 14 months in something just clicked. I started focusing on just a few key setups... things that were very tangible and defined... I began to take more size on fewer trades instead of less size on more trades. I'd wait for a few trades per day to make my day, because I had seen that pattern over and over and over and over. But only having traded tens of thousands of times over 1000+ hours, did my discretionary become mostly mechanical. It just clicked. Since then I've had 9 straight profitable months... all (for me) good months. There is no saying when things will click with you, but if you keep it at and manage the risk, you will find your edge, your niche, and focus on that. Your patience will increase, your ability to take size will increase, and your confidence will grow. From what you write it sounds like you are on the right path, just keep at it.
Quote from estim:
Rabbitone,
Split your data set into eg. 4 equal size. Optimise your model 3 times on the first 3 data sets and save all the parameters for each model. To select the optimal parameters you chose those that fit best to all 3 datasets. Finally you test in on data set 4.
This procedure should be applied as walk forward meaning you will have to introduce a new parameter namely the size of each data set. This parameter is optimised as well and is sometimes called the learning curve, that is, how much data do you need to train your model on before it produces a stable result.
Hope this make sense.
Estim,
interesting insight on this modified version of walk-forward. But maybe you can clarify a few points. You say to divide the data set into 4 parts and test on 3. I'm a bit confused when you say "chose those that fit best to all 3 datasets". Are you saying to use the set of parameters that performed best on the first 3 data sets and then test it on the 4th?
When you say "chose those that fit best to all 3 datasets" can also imply a single optimization test for the 3 datasets. If that's the case then there is really no reason to test the 3 data sets separately. I'm not sure which one...
hope you can clarify so we can expand on the issue.
Thanks
Estim â You have a very interesting model. The curse of dimensionality was not quite the optimization problem I was referring to with security price data. I understand Bayesian statistics can be a factor in hierarchical models with more than a few parameters and data needs to be increased to avoid over fitting. This was not the mathematical model problem that I was referring to.
The problem with testing with a 100 atypical securities is the data is heterogeneous with regard to directional and volatility characteristics. This directly influences the conclusions that the trader can draw from any signal produced from a set of optimization parameters.
Let me back track a few notes. In my banking days in IT here is what the Trust department of the bank did to buy and sell securities using models. This was a top down approach where they used data from Standard and Poors. The method was:
1. Markets are scanned to obtain buy and sell signals in the weekly indexes.
2. Next sector and industries where scanned for buy and sell signals in the weekly indexes using statistical modeling.
3. Then the trading models ran through a combined set of stocks from one or more similar industries that gave buy or sell signals.
4. Individual stocks were then traded through an automated step program (where large buys and sell were broken down into smaller trades).
5. Actively trade stocks were sold quickly when sectors or industry data gave exit signals reversing the original signal.
I adopted this model for trading my own positions and intermediate term trades. A key part of this bank model was that securities tested (optimized) with similar directional and price characteristics as the parent sector or industry gave superior buy and sell signals.
The guy who ran the trust department (a former mutual fund manager) told me just a few facts. Here are some key ones:
1. The primary statistical model is at the sector or industry index level for stocks. The moments about the mean where done at this level as part of their model. This drove the buy and selling of all securities.
2. Trading models were signal driven when the primary statistical model for sectors and industries when it confirmed market direction.
3. Individual stocks that were run through signal models for automated trading had similar characteristics for both direction and volatility.
What this leads to is testing with price data with a common statistical inference. This is more like dynamic programming where there are over lapping sub problems and an optimal substructure. What I asked before is do you consider using all homogeneous price data in your data set before it is split. Correlation is always a concern with clustered price data. I prefer compact and understandable models that incorporate the relationships between index and securities as simple predictors. Higher models like neural nets have never worked for me.
The problem with testing with a 100 atypical securities is the data is heterogeneous with regard to directional and volatility characteristics. This directly influences the conclusions that the trader can draw from any signal produced from a set of optimization parameters.
Let me back track a few notes. In my banking days in IT here is what the Trust department of the bank did to buy and sell securities using models. This was a top down approach where they used data from Standard and Poors. The method was:
1. Markets are scanned to obtain buy and sell signals in the weekly indexes.
2. Next sector and industries where scanned for buy and sell signals in the weekly indexes using statistical modeling.
3. Then the trading models ran through a combined set of stocks from one or more similar industries that gave buy or sell signals.
4. Individual stocks were then traded through an automated step program (where large buys and sell were broken down into smaller trades).
5. Actively trade stocks were sold quickly when sectors or industry data gave exit signals reversing the original signal.
I adopted this model for trading my own positions and intermediate term trades. A key part of this bank model was that securities tested (optimized) with similar directional and price characteristics as the parent sector or industry gave superior buy and sell signals.
The guy who ran the trust department (a former mutual fund manager) told me just a few facts. Here are some key ones:
1. The primary statistical model is at the sector or industry index level for stocks. The moments about the mean where done at this level as part of their model. This drove the buy and selling of all securities.
2. Trading models were signal driven when the primary statistical model for sectors and industries when it confirmed market direction.
3. Individual stocks that were run through signal models for automated trading had similar characteristics for both direction and volatility.
What this leads to is testing with price data with a common statistical inference. This is more like dynamic programming where there are over lapping sub problems and an optimal substructure. What I asked before is do you consider using all homogeneous price data in your data set before it is split. Correlation is always a concern with clustered price data. I prefer compact and understandable models that incorporate the relationships between index and securities as simple predictors. Higher models like neural nets have never worked for me.
NY0BScalper,
When I do discretionary trading I pretty much break even and my style is just as you describe it - many trades small profits. Yesterday I was trading APOL. I found my entry shorted it with a limit order and right after the trade went in my direction. I moved my stop to the break even level. Since I don't have an edge = confidence, my trade was only 100 shares. Because the position was that small I wanted the price to go even further down otherwise my profit was to small.... So after 15 min or so my stop was hit and again I went break even. However with larger size and more confidence I would have scaled out and taken some profit.
When I do discretionary trading I pretty much break even and my style is just as you describe it - many trades small profits. Yesterday I was trading APOL. I found my entry shorted it with a limit order and right after the trade went in my direction. I moved my stop to the break even level. Since I don't have an edge = confidence, my trade was only 100 shares. Because the position was that small I wanted the price to go even further down otherwise my profit was to small.... So after 15 min or so my stop was hit and again I went break even. However with larger size and more confidence I would have scaled out and taken some profit.
Quote from NY0BScalper:
I was in what sounds like a similar position you were after 10 months. I was very discretionary and just winging it. I was making a little money, enough to cover comms which were very high, but not taking home much and swinging way too much relative to what I was taking home.
About 14 months in something just clicked. I started focusing on just a few key setups... things that were very tangible and defined... I began to take more size on fewer trades instead of less size on more trades. I'd wait for a few trades per day to make my day, because I had seen that pattern over and over and over and over. But only having traded tens of thousands of times over 1000+ hours, did my discretionary become mostly mechanical. It just clicked. Since then I've had 9 straight profitable months... all (for me) good months. There is no saying when things will click with you, but if you keep it at and manage the risk, you will find your edge, your niche, and focus on that. Your patience will increase, your ability to take size will increase, and your confidence will grow. From what you write it sounds like you are on the right path, just keep at it.
Rabbitone,Quote from Rabbitone:
A key part of this bank model was that securities tested (optimized) with similar directional and price characteristics as the parent sector or industry gave superior buy and sell signals.
I like your top-down approach and will try to adapt it. From previous working experience I have done many top-down approaches on data modeling which always seems to make the final models easier to understand and use. I need to read your post again a couple of times because you give so much information in them, but here I will give my comment on the testing/classifying of indices/sectors/stocks. First, I have not segmented my data regarding volatility and direction but will probably do it after been reading your posts. This is how I would approach it to keep it simple:
As you write it has to updated dynamic in order to reflect the current market conditions. The timeframe I chose is 60 min because you wrote it as an option for better entry. Objective: to define at each 60 min bar the volatility and direction for all sectors and equities and at the same bar to classify the sectors and equities into segments of volatility and direction.
At each bar calculate the ATR and ADX for all symbols. Let say you have 8000 symbols corresponding to equities and sectors. You then have a 8000x3 matrix where the columns are ATR, ADX,SYMBOL.
There exist a ton of different ways to bin/group âcontinuesâ column data but here we will bin each column ATR and ADX into 3 bins/groups using a fixed number. Donât mind the numbers I just made them up. Note I am using percentages for the ADX in order to segment across varies price levels.
ADX_HIGH if ADX>60
ADX_MEDIUM if ADX<60 & ADX>30
ADX_LOW if ADX<30
ATR_HIGH if ATR>2%
ATR_MEDIUM if ATR <2% & ATR >0.5%
ATR_LOW if ATR <0.5%
This will produce 9 different ATR/ADX segments at each bar update. You then have to look at the correlation of the price action within each segment and decide if you only will optimize on the correlated stocks.
Corelio,
My explanation wasn't that clear so I will give it a go here with different words.
I am not just taken the parameters that fit best on the combined three data set, but the parameters that fit bests on the individual data set 1,2 and 3. So basically you need to sort the parameters after performance for each data set this will give you 3 tables consisting of x number of optimisation runs after that you merge the 3 tables and select the parameters that first appear in all three coulmns.
My explanation wasn't that clear so I will give it a go here with different words.
I am not just taken the parameters that fit best on the combined three data set, but the parameters that fit bests on the individual data set 1,2 and 3. So basically you need to sort the parameters after performance for each data set this will give you 3 tables consisting of x number of optimisation runs after that you merge the 3 tables and select the parameters that first appear in all three coulmns.
Quote from Corelio:
Estim,
interesting insight on this modified version of walk-forward. But maybe you can clarify a few points. You say to divide the data set into 4 parts and test on 3. I'm a bit confused when you say "chose those that fit best to all 3 datasets". Are you saying to use the set of parameters that performed best on the first 3 data sets and then test it on the 4th?
When you say "chose those that fit best to all 3 datasets" can also imply a single optimization test for the 3 datasets. If that's the case then there is really no reason to test the 3 data sets separately. I'm not sure which one...
hope you can clarify so we can expand on the issue.
Thanks
Mr Denmark, you need to understand the importance of looking at bigger time frames to see if there are more noticeable trends. Alot of times, the smaller time frame you use, the more randomness and noise you encounter, thus making it harder for any system to follow and capture a move with good profit potential. Start studying larger time frames, 15min, 1 hr, 4hr, daily, etc to see where, if any, there are better defined trends and concentrate on those time frames. The smaller time frames should only be used for entry and exits. Also you should consider other investment instruments like index futures and currency pairs...some trend better than others. Of course trading the news can get you some serious $$ if you know how to analyze and react to economic report consensuses other breaking market news and prepare accordingly.
Estim â You have an excellent grasp of using ATR an ADX but you have to understand the opposite side of the equation which is the characteristics of price data. This is the side of automated trading that when I grasped it - changed all of my results. The 60 minute time interval is only for entry and exits. Basic testing is done in higher time fames. I will explain
Most traders are unaware that the trading solution they create is dependant on the characteristics of price data that they test with and not the other way around. When they trade it is often with price data that is not compatible with their original trading solution. This happens because the market has changed and price data has new characteristics. The testing optimization they perform historically finds profitable trades that are specific to price direction and volatility events which do not reoccur in future trading (because price data characteristics constantly change).
All trading solutions are built from price that have two direct influences â direction and volatility and one indirect influence volume. I have seen trader after trader build their solution with historical price data and apply it to current price data that is irrelevant to their trading solution. A simple case in point is trading solutions with indicators that follow trend using ADX. For example a solution is historically tested in a bull market and shows great profits. In the congestion that follows the bull market going to the bear market this trading system loses ALL of the profits of the bull market because the trader had no idea the directional and volatility dependencies of their code were specific to trend following.
Try and find a decent book on how to categorized and examine price data. It does not really exist. We are forced to use are statistical measures in our code that are often irrelevant unless precise price data events occur. We are consumed as traders with the thought that any price data is sufficient to make our code work. This is just NOT the case.
Do a small exercise. Pick a stock you want to trade. Build a weekly chart for the last 10 years add ATR or a volatility indicator and a directional indicator like ADX. Then examine the periods of price direction and volatility in combination (there are 4 possible states in combination) that change. These are periods I look at. What you grasp is each separate period has a different trading solution. In one its swings and high volatility, in another it trend an medium volatility and still another no trend and low volatility.
Another purpose of using sectors and industries is that most are cyclical and prior historical cycle data is relevant, for the most part in the future, for sectors and industries. That is the model I adopted from the bank pros. The premise is simple. For example the bear computer hardware industry in early 2000s can be used as a model for computer hardware in a bear market today.
1. Obtain the current markets directional and volatility characteristics.
2. Also define the current sector and industry directional and volatility characteristics.
3. Trading model identifies prior market, sector and industry directional and volatility periods the same as the current period.
4. Test stocks in that prior historic period.
5. Walk forward multiple settings from that historic period with the stocks.
6. Automate trading in the current period using the most relevant settings.
Example: I identified an historic weekly period a few years back like todayâs environment that matched Apples price data. I tested my system in that prior period. Then I walk forward tested those settings to trade in todayâs market. The common tread was volatility conditions matched perfectly (my directional component was only half way there).
If I had tested Apple with the last 2 years of price data 70% of it was in a bull market that is irrelevant to todayâs bear market. In this process I even tested my strategy against the sector index apple was in to see if the results were the same. They were the same.
To summarize the concept of using sectors and industry stocks:
- Stocks in a grouping have a similar homogeneous directional and volatility components at any one point in time. This makes processing them with an automated system easier because the trader can isolate direction and volatility in an industry index and match that with the stocks they trade.
- Sectors and industry groupings can provide historical periods similar to todayâs markets that can be used to determine how to trade stocks in todayâs markets.
Estim here is how you can apply this second method (above) to your current testing. In this test it will be with daily rather than hourly data. This is just one of many sector trading techniques. But I think it may apply to you (probably wrong but worth a try).
Let us do this example in bulk using a multi stock back test. Identify a sector that should thrive in this current bear market. Let us say you chose the Consumer Cyclical sector. You would then examine market data that matched the current market data - say you chose 2001. You can then bulk test the consumer cyclical stocks of that period with your system. Then you can walk forward to today and evaluate the same sector. Stocks that immerge from testing that show the same characteristics as prior periods are your trading candidates.
Estim you sure are good at pressing my buttons and getting me to write.
Most traders are unaware that the trading solution they create is dependant on the characteristics of price data that they test with and not the other way around. When they trade it is often with price data that is not compatible with their original trading solution. This happens because the market has changed and price data has new characteristics. The testing optimization they perform historically finds profitable trades that are specific to price direction and volatility events which do not reoccur in future trading (because price data characteristics constantly change).
All trading solutions are built from price that have two direct influences â direction and volatility and one indirect influence volume. I have seen trader after trader build their solution with historical price data and apply it to current price data that is irrelevant to their trading solution. A simple case in point is trading solutions with indicators that follow trend using ADX. For example a solution is historically tested in a bull market and shows great profits. In the congestion that follows the bull market going to the bear market this trading system loses ALL of the profits of the bull market because the trader had no idea the directional and volatility dependencies of their code were specific to trend following.
Try and find a decent book on how to categorized and examine price data. It does not really exist. We are forced to use are statistical measures in our code that are often irrelevant unless precise price data events occur. We are consumed as traders with the thought that any price data is sufficient to make our code work. This is just NOT the case.
Do a small exercise. Pick a stock you want to trade. Build a weekly chart for the last 10 years add ATR or a volatility indicator and a directional indicator like ADX. Then examine the periods of price direction and volatility in combination (there are 4 possible states in combination) that change. These are periods I look at. What you grasp is each separate period has a different trading solution. In one its swings and high volatility, in another it trend an medium volatility and still another no trend and low volatility.
Another purpose of using sectors and industries is that most are cyclical and prior historical cycle data is relevant, for the most part in the future, for sectors and industries. That is the model I adopted from the bank pros. The premise is simple. For example the bear computer hardware industry in early 2000s can be used as a model for computer hardware in a bear market today.
1. Obtain the current markets directional and volatility characteristics.
2. Also define the current sector and industry directional and volatility characteristics.
3. Trading model identifies prior market, sector and industry directional and volatility periods the same as the current period.
4. Test stocks in that prior historic period.
5. Walk forward multiple settings from that historic period with the stocks.
6. Automate trading in the current period using the most relevant settings.
Example: I identified an historic weekly period a few years back like todayâs environment that matched Apples price data. I tested my system in that prior period. Then I walk forward tested those settings to trade in todayâs market. The common tread was volatility conditions matched perfectly (my directional component was only half way there).
If I had tested Apple with the last 2 years of price data 70% of it was in a bull market that is irrelevant to todayâs bear market. In this process I even tested my strategy against the sector index apple was in to see if the results were the same. They were the same.
To summarize the concept of using sectors and industry stocks:
- Stocks in a grouping have a similar homogeneous directional and volatility components at any one point in time. This makes processing them with an automated system easier because the trader can isolate direction and volatility in an industry index and match that with the stocks they trade.
- Sectors and industry groupings can provide historical periods similar to todayâs markets that can be used to determine how to trade stocks in todayâs markets.
Estim here is how you can apply this second method (above) to your current testing. In this test it will be with daily rather than hourly data. This is just one of many sector trading techniques. But I think it may apply to you (probably wrong but worth a try).
Let us do this example in bulk using a multi stock back test. Identify a sector that should thrive in this current bear market. Let us say you chose the Consumer Cyclical sector. You would then examine market data that matched the current market data - say you chose 2001. You can then bulk test the consumer cyclical stocks of that period with your system. Then you can walk forward to today and evaluate the same sector. Stocks that immerge from testing that show the same characteristics as prior periods are your trading candidates.
Estim you sure are good at pressing my buttons and getting me to write.
A number of traders ask me to summarize what we have been stating about automated trading in this forum. So here it is:
Most discretionary traders fail to understand why automated systems do not work for them. This is especially true for discretionary traders who have had some success in the past. The standard belief is that optimizing a set of rules successfully with any set of price data will produce good automated trading results going forward. This is normally not the case with automated trading systems. Automated trading systems are much more temperamental and dependent on the markets than trades suspect.
Automated trading is all about timing as I learned from watching a banks trust department trade. Here are 5 areas the bank paid attention to in automated trading using a longer term top down approach:
- Start automated trading in higher time intervals â Daily and weekly automated trading strategies can produce great results and trading signals. Automated traders often use buy or sell signals from daily or weekly automation (in tune with markets) as a queue to start trading intraday automated trading systems.
- Analyze your strategies behavior â What conditions does it respond best in? Back test in a variety of conditions, such as directional price data with low to medium volatility and in non directional and high volatility. This often throws traders for a loop when they find out their strategy runs only under specific conditions.
- Define how your strategy performs historically â Find market conditions similar to todayâs market conditions to back test your strategy in. Then walk these settings forward to todayâs markets. If the settings produce reasonable results use them. I
- Start automated trading at longer term market entry points â Starting automation under just any market conditions is unacceptable. The market conditions must match you strategies temperament for direction and volatility. Look to start automation when markets begin to move like your strategy performs. For example use bounces off S&P 500 support or resistance as starting points. Use weekly index charts to do this analysis or for longer term trades use monthly index charts. It gives clearer signals.
- Pick a winning stock sector and industry to trade from - Trying to pick individual stock winners to trade with automation can be tough. Learning sector rotation can increase the odds to finding stocks that perform as expected from better performing sectors as they rotate. When longer term markets shift at support and resistance drill down through sectors and industries to find the stocks that are moving up or down. These are the candidates for automated trading.
This method managed several billion in the 1990s.
But what about other methods than this top down approach? Things are always changing what works for you today?
Most discretionary traders fail to understand why automated systems do not work for them. This is especially true for discretionary traders who have had some success in the past. The standard belief is that optimizing a set of rules successfully with any set of price data will produce good automated trading results going forward. This is normally not the case with automated trading systems. Automated trading systems are much more temperamental and dependent on the markets than trades suspect.
Automated trading is all about timing as I learned from watching a banks trust department trade. Here are 5 areas the bank paid attention to in automated trading using a longer term top down approach:
- Start automated trading in higher time intervals â Daily and weekly automated trading strategies can produce great results and trading signals. Automated traders often use buy or sell signals from daily or weekly automation (in tune with markets) as a queue to start trading intraday automated trading systems.
- Analyze your strategies behavior â What conditions does it respond best in? Back test in a variety of conditions, such as directional price data with low to medium volatility and in non directional and high volatility. This often throws traders for a loop when they find out their strategy runs only under specific conditions.
- Define how your strategy performs historically â Find market conditions similar to todayâs market conditions to back test your strategy in. Then walk these settings forward to todayâs markets. If the settings produce reasonable results use them. I
- Start automated trading at longer term market entry points â Starting automation under just any market conditions is unacceptable. The market conditions must match you strategies temperament for direction and volatility. Look to start automation when markets begin to move like your strategy performs. For example use bounces off S&P 500 support or resistance as starting points. Use weekly index charts to do this analysis or for longer term trades use monthly index charts. It gives clearer signals.
- Pick a winning stock sector and industry to trade from - Trying to pick individual stock winners to trade with automation can be tough. Learning sector rotation can increase the odds to finding stocks that perform as expected from better performing sectors as they rotate. When longer term markets shift at support and resistance drill down through sectors and industries to find the stocks that are moving up or down. These are the candidates for automated trading.
This method managed several billion in the 1990s.
But what about other methods than this top down approach? Things are always changing what works for you today?