Would You Trade This System?

[dom993]

More comments / ideas to consider:

- Day of Week (DoW) analysis ... what is the best/worst DoW performer on the entire backtesting period, and is this consistent on a year-per-year basis? Do the analysis for Longs & Short separately as well.

- Time of Day (ToD) analysis ... analyze the system's performance using 15min windows (I do this using the entry-time, but sometime the entry-signal time is more appropriate). In particular, how does the 10:00am-10:15am window perform (news) ?

- Expiration Week analysis (3rd Friday of the month) ... day by day performance analysis for that week

- Employment Report analysis (1st Friday of the month) ... you should analyze performance on the day before & the following Monday as well.

- Trading hours ... if your system takes or manage trades overnight, your backtesting results are highly suspect for those trades (I am talking stocks here, not futures) ... it is not because price printed beyond a limit order before/after RTH that any resting order would have been filled

- My favorite test for "directional" systems ... use a strict 1:1 Risk:Reward ratio (stop size = target size), no trade management at all, and just look at win% ... anything under 55% doesn't have enough of a directional edge IMO ... note that different size for the stop/target will give you different win%

 

[ronblack]

Quote from jayjay121:

Max. Shares/Contracts Held 250
Total Shares/Contracts Held 1508000
Account Size Required $1,744.00
Total Commission $12,064.00


Trading Period 9 Yrs, 11 Mths, 22 Dys, 5 Hrs, 37 Mins
Percent of Time in the Market 3.90%

Nobody noticed the above?

Account size required is $1,1744 for 250 SPY max? Give me a break...

Percent time in the market 3.90% for 1 min timeframe? This is highly fitted.

 

[jayjay121]

Quote from ronblack:

Nobody noticed the above?

Account size required is $1,1744 for 250 SPY max? Give me a break...

Percent time in the market 3.90% for 1 min timeframe? This is highly fitted.

Mate if ive curve fitted then fine but please show me how and where?, i want to get to the bottom of this myself................It is not a one minute system. It uses at least 10 min timeframe for entry not one minute, but it uses 1 min timeframe to consider a new entry or exit as this is as far as i can go in tradestation, but i do need any further than this as im not micro scalping or even scalping imo, kind of middle scalping if there is such a thing. So it is not a 1 minute scalper or anything like that. If it was id agree with you completely.

The good thing with synthetic data is my code does not have to wait say 60 mins to get new value of an indicator, it updates the values each one minute so highly realistic.


As for those stats you commented on, you are clearly not a tradestation user. Account size means only the worst drawdown and they are saying this is what the system requires, of course it requires more, but this stat only suggests the room the test require (worst drawdown). As for percent in the market, how is this un reasonable, i really don't understand what you are implying. Have you seen how long this test was?, almost 10 years and if it only makes 2 to 3 trades per day for 20-30 cents targets, then of course its not in the market for long....unless ive mis understood you.



If you look at my average time in trades, winning and losing trades you should understand more clearly.

 

[jayjay121]

Quote from dom993:

More comments / ideas to consider:

- Day of Week (DoW) analysis ... what is the best/worst DoW performer on the entire backtesting period, and is this consistent on a year-per-year basis? Do the analysis for Longs & Short separately as well.

- Time of Day (ToD) analysis ... analyze the system's performance using 15min windows (I do this using the entry-time, but sometime the entry-signal time is more appropriate). In particular, how does the 10:00am-10:15am window perform (news) ?

- Expiration Week analysis (3rd Friday of the month) ... day by day performance analysis for that week

- Employment Report analysis (1st Friday of the month) ... you should analyze performance on the day before & the following Monday as well.

- Trading hours ... if your system takes or manage trades overnight, your backtesting results are highly suspect for those trades (I am talking stocks here, not futures) ... it is not because price printed beyond a limit order before/after RTH that any resting order would have been filled

- My favorite test for "directional" systems ... use a strict 1:1 Risk:Reward ratio (stop size = target size), no trade management at all, and just look at win% ... anything under 55% doesn't have enough of a directional edge IMO ... note that different size for the stop/target will give you different win%

..i too start at a basic of 1-1 to judge an edge and use the same concept if the win rate is above 55% i know i can start looking into it more.......If if can't meet at least that then its not worth my time as no real edge there.

As for days of the week and others....Ive never wanted to do this as i believe this can bring about curve fitting in itself. I want to trade all days and all periods. If i did what you say yes i could improve but is that not curve fitting?.........

I know for a fact system is better when no sentiment driven days but how will i ever predict this before hand?, and hence why i do not try.

It holds nothing over night.


thanks for your comments.

 

[jayjay121]

Quote from intradaybill:

If this is a 1-min timeframe, prifits from trades should be almost normally distributed. But you start making profit after the 150 trade or so for this year.

This does not make any sense unless you are using several filters with optimized parameters.

Are you using work-forward optimization in your testing?

Your results are highly unusual.

1) Over-fitting far and beyond

or

2) Look ahead in backtest

or

3) Unrealistic settings

thanks for your feedback.......

Something must be wrong then, but please help me get to the bottom of it. I would say though that system can have periods of making no money but at the same time not losing much if you know what i mean, just chopping. Not sure if this makes a difference.

Ive gone through a trade list page of my last trades and all seems fine with slippage, comms calculations. All the candles seem to have passed through price ok, and not entering on very last tick of candle or anything like that.....Every fill has been passed through in same entry/exit candle. So no problems there.

Im doing alot of research right now and it does seem it is possible to curve fit such a large number of trades to which i did not realise honest and truely. So im a bit confused right now because all my beliefs were the opposite, and why ive always tested with long term sample sizes, as i thought i couldn't possible curve fit that long back.

Im just trying to figure out how and if ive done that.

Please let me post a WFO test, would this help determine?, if so please suggest a walk forward in and out of sample.

What about the robustness and that it works on other symbols?, how is this then?
regards,

 

[deaddog]

I may have missed it but has anyone suggested testing the system on a different market?

Have you tried it on the Q's or the DIA.

 

[dom993]

Quote from jayjay121:

As for days of the week and others....Ive never wanted to do this as i believe this can bring about curve fitting in itself. I want to trade all days and all periods. If i did what you say yes i could improve but is that not curve fitting?.........

10am news release, Employment Friday, Options Expiration week, 9:30am open / 4pm close, 1st / last day of the week (or month) ... all of these are patterns which may or may not impact your system performance.

I understand you are cautious of not over-optimizing your system, but deliberately ignoring recurring events / patterns is like fighting with 1 arm tied in your back. I believe you should do the analysis anyway, then make a decision based on the analysis.

Let's say you discover that the 9:30-9:45 window is unprofitable in 80% of the years, and slightly better than BE on the other 20% ... would that tell you something useful?

You have a large enough sample size to make useful statistical analysis ... I would take advantage of it (not to mention, in your case you can cross-check that analysis between various instruments)

 

[dom993]

Quote from jayjay121:

..i too start at a basic of 1-1 to judge an edge and use the same concept if the win rate is above 55% i know i can start looking into it more.......If if can't meet at least that then its not worth my time as no real edge there.

55% win% at 1:1 translates in a P/F of 1.22

Your system backtest on 10 years shows P/F of 1.28, and the average win is much smaller than average loss ... the trade management part of your system is weak IMO

 

[jayjay121]

Quote from dom993:

55% win% at 1:1 translates in a P/F of 1.22

Your system backtest on 10 years shows P/F of 1.28, and the average win is much smaller than average loss ... the trade management part of your system is weak IMO

Hi, when testing i start at 1-1 but im very willing to go below. Im not one of these types that automatically thinks you must have 2-1 or higher. I traded that way for a long time and constantly lost. If you look at most RTM professional traders, they too are around 1-1 or lower.

It depends on so many factors. I would love 3-1 to be my preference but my stats have formed odds and probabilities to suggest im better off capturing a little at a time giving space to work out.

you said the sample size was good.........well ive used this sample size to find the best probabilities of targets, exit levels, volatility etc......its all averages i know and yes you do get the odd black swan event but you can't form a system based on black swan events...........my PF is low because im not looking for perfect, im looking for the average to form a odd to. This is like a baseline strategy that can be improved im sure.

I just hope i have not curve fitted.

 

[jayjay121]

Quote from dom993:

55% win% at 1:1 translates in a P/F of 1.22

Your system backtest on 10 years shows P/F of 1.28, and the average win is much smaller than average loss ... the trade management part of your system is weak IMO

thanks for your feedback...........when i get time i will try what you say.

Do you have any suggestions to improve trade management without using small stops?...........Ive tested 3/4's off, stop to break even etc etc but it did not improve it. One thing i have not tried is trailing stop. Also i was reading up the other day about a average price style which is not averaging down but breaks the usual size (so in this case 2500), into 3 to give 3 entries to try and capture the best average fill. Thats quite vague but it did sound a good method. Also i have fixed shares right now. I have not tested optimal f, fixed fractional, kelly, etc. Not sure if it would improve trade management though.

any suggestions?