Seeking feedback on my rules-based system

nooby_mcnoob said:
Intellectually I understand what you're doing. But my brain screams overfit. I assume you are doing cross-validation but I haven't heard anything about that in this thread yet though my apologies if I missed it.
More...

Can you please describe what you mean by overfitting? I think I understand but want to make sure that I get it right.

When I think of overfitting, I think of an absurdedly narrow "signal" window, with a very small sample size. Something like this:

upload_2019-4-23_13-56-59.png


The signal is a narrow range of the 200-day MA of the USD. A small number of trades, concentrated on a few lucky spikes in VIX. Seems to me that this strategy probably was just a lucky one, since I'm scanning thousands of potential strategies.

I'm actually not doing any cross-validation. Half because I'm lazy. ;-) Half because I think I can visually observe stationarity in the profit curve. Here's a profit curve that looks much better: over 1000 historical trades and reasonable performance in all market conditions.

upload_2019-4-23_14-5-38.png


As a measure of stationarity, I actually compute the variance of the profit about a straight line connecting t=0,p=0 to t=tmax,p=p(tmax) and use it in the strategy screening process. In other words, if the profit curve is "blocky" or has lots of drawdowns, it will be penalized relative to a nice steady-marching profit curve.
 

Attachments

  • upload_2019-4-23_14-3-18.png
    upload_2019-4-23_14-3-18.png
    92.5 KB · Views: 20
  • upload_2019-4-23_14-5-16.png
    upload_2019-4-23_14-5-16.png
    72.3 KB · Views: 20
morganpbrown said:
Thanks for this, I will dig in deeper to your post later today. RSI is one attribute that I've been too lazy to incorporate into my models, but it's very high on the list. For one, I have a hard time getting a firm definition of RSI itself!
More...

firm definition of RSI

https://stockcharts.com/school/doku...hnical_indicators:relative_strength_index_rsi
Calculation
upload_2019-4-23_15-52-54.png


To simplify the calculation explanation, RSI has been broken down into its basic components: RS, Average Gain and Average Loss. This RSI calculation is based on 14 periods, which is the default suggested by Wilder in his book. Losses are expressed as positive values, not negative values.

The very first calculations for average gain and average loss are simple 14-period averages:

  • First Average Gain = Sum of Gains over the past 14 periods / 14.
  • First Average Loss = Sum of Losses over the past 14 periods / 14
The second, and subsequent, calculations are based on the prior averages and the current gain loss:

  • Average Gain = [(previous Average Gain) x 13 + current Gain] / 14.
  • Average Loss = [(previous Average Loss) x 13 + current Loss] / 14.
 
morganpbrown said:
Can you please describe what you mean by overfitting? I think I understand but want to make sure that I get it right.

When I think of overfitting, I think of an absurdedly narrow "signal" window, with a very small sample size. Something like this:

View attachment 201338

The signal is a narrow range of the 200-day MA of the USD. A small number of trades, concentrated on a few lucky spikes in VIX. Seems to me that this strategy probably was just a lucky one, since I'm scanning thousands of potential strategies.

I'm actually not doing any cross-validation. Half because I'm lazy. ;-) Half because I think I can visually observe stationarity in the profit curve. Here's a profit curve that looks much better: over 1000 historical trades and reasonable performance in all market conditions.

View attachment 201341

As a measure of stationarity, I actually compute the variance of the profit about a straight line connecting t=0,p=0 to t=tmax,p=p(tmax) and use it in the strategy screening process. In other words, if the profit curve is "blocky" or has lots of drawdowns, it will be penalized relative to a nice steady-marching profit curve.
More...

More or less, if you have a training data set and a test data set you should be fine. So you mine the last 10-9 years for rules and then run them over the next year. Or constantly update the rules as you go based on past info only. I'm sure I missed your architecture somewhere and you're doing the sensible thing.

An extreme example of over fitting would be that you train a random forest on events up to yesterday and run the forest today to predict the movement. What you've got will work really well in the past but not the future.
 
dreamer2019 said:
Here is the result:
(long / short)entry time -> closure time, entry price -> close price, profit, funds(initial 10000)
v20180102 09:59:00 -> 20180102 10:05:00, 1181.310 -> 1183.360, -16.399, 9983.601
v20180102 11:16:00 -> 20180102 11:19:00, 1184.455 -> 1183.590, 6.920, 9990.521
v20180102 12:08:00 -> 20180102 12:10:00, 1187.670 -> 1187.660, 0.080, 9990.601
^20180102 12:39:00 -> 20180102 12:46:00, 1185.315 -> 1186.990, 13.400, 10004.001
......
v20190405 13:25:00 -> 20190405 13:28:00, 1832.300 -> 1831.980, 11.843, 69357.602
v20190405 13:57:00 -> 20190405 14:00:00, 1834.780 -> 1834.785, -0.185, 69357.414
^20190405 14:16:00 -> 20190405 14:32:00, 1833.170 -> 1833.625, 16.833, 69374.250
v20190405 14:53:00 -> 20190405 14:56:00, 1834.885 -> 1834.895, -0.370, 69373.883
More...

The result is based looking forward bias , which is wrong.
Seems rsi is not that predictive.
 
dreamer2019 said:
The result is based looking forward bias , which is wrong.
Seems rsi is not that predictive.
More...

Is anything predictive? There is only "when ABC happens, there is a X% probability that Y happens at most N days afterwards". Market can stay irrational... etc... Not at all mad.
 
morganpbrown said:
Thanks for pointing this out. Whenever I can, I'm trading the leveraged ETF's (UPRO vs SPY, TVIX vs VIXY, TMF vs IEF, TECL vs QQQ), but I keep hearing the horror stories of these ETF's flash crashing. It makes me very nervous. Could actually be safer to just trade the lower-leverage underlying security on margin!
More...
IF that was a problem+ i dont think its is much, why risk a margin call=panic sellers never win??
AS you partly implied , morgan B,when you have a loss + that's part of it=Leverage is not your friend ; trend = friend. My cash position in DVY did get a flash crash once; but i seldom watch 100% intraday; i use 10 year paper charts + all data much more, than any flash crash which could be a bad quote.
I know Black Rock called GS out[ in public] on that flash crash , as some kind of error.Planned sellers can win; panic sellers never win.Thanks:cool::cool:,:cool::cool::cool::cool::cool::cool:
 
morganpbrown said:
Can you please describe what you mean by overfitting? I think I understand but want to make sure that I get it right.

When I think of overfitting, I think of an absurdedly narrow "signal" window, with a very small sample size. Something like this:
More...

Another example of overfitting: you have a EMA crossover strategy. You data mine the best fast/slow window and use that as your rule. In backtests, this is going to be b-e-a-youtiful. But in real trading, it's likely going to fail after some time if you're lucky, or immediately if you're unlucky.

It is quite possible that you can optimize these parameters on a daily basis and pick the best rule but at that point, I think you're statistically not doing any better than picking random rules.

To convince yourself make sure you are running backtests that avoid forward bias.
 
You must log in or register to reply here.
Back
Top