backtest variables, compare between two systems

S

Shay

Hi,

what do you think we should check when doing backtest ?

Total PnL
% Profit trades
Largest Loss
Largest Win
Therotical % Profit

what else ?

here are two examples
Both systems has -
total PnL 150
% profit 87%
Largest Loss -150
Largest Win 25
Therotetical % Profit 83%

the SAME results, but i think you agree with me, that we will prefer the system #2

so i wonder, what other calculations should we include in the backtest, so we can compare them and see the "better" one ?
upload_2016-3-6_20-51-8.png


upload_2016-3-6_20-51-14.png


Thanks,
Shay
 
Shay said:
Hi,

what do you think we should check when doing backtest ?

Total PnL
% Profit trades
Largest Loss
Largest Win
Therotical % Profit

what else ?
More...
Also very important are these:
- Final PnL%
- Equity curve and/or PnL% curve
- Max drawdown (MDD%)
- StdDev of the equity and/or of the PnL%

MDD% can be extracted from the equity curve or the PnL% curve, but that should be done by the test performer.

Professional backtesting environments have much more metrics, like Sharpe ratio, Sortino ratio etc...
You can check these metrics: http://www.amibroker.com/guide/h_report.html
 
Last edited:
Hey shay,
I would suggest including average win- when the trade is profitable
and average loss - when the trade loses.
this kind of details may illustrate the feeling you about to experience during your trading time!
further more, if your strategy includes stop losses- make sure you count them and compare them against a strategy without stop losses
 
I often get asked "if you had to narrow it down to one metric, which one would it be?"
Would that even be doable?

I guess for me I would need at least 3 to do anything
MDD%
%Profit Trades
Avg Profit per Trade
 
Shay said:
here are two examples
Both systems has -
total PnL 150
% profit 87%
Largest Loss -150
Largest Win 25
Therotetical % Profit 83%

the SAME results, but i think you agree with me, that we will prefer the system #2
More...
Hmm. I think #1 has a better MDD than #2...

Update;
No, MDD seems to be about the same in both cases; w/o numbers it's not easy to calc the MDD...
Ok, I overlooked the text where it says largest loss is in both cases -150...
As said, I would need the StdDev as well...
 
Last edited:
Gimpyron -
Avg Win Vs. Avg Loss - doesn't say a lot here,
system#1 Avg Win = 25, Avg Loss = -125
system#2 Avg Win = 20.2, Avg Loss = -93.75

sysdevel99 -
Avg Profit per trade is the same in both systems,
because both systems have Avg profit of 5 (let me know if you wanted to calculate it differently)

%Profit Trades are the same for both systems = 87%

botpro -
MDD in system #1 = -175
MDD in system #2 = -150

StdDev of PnL in system #1 = 53.09
StdDev of PnL in system #2 = 41.27

what do you mean by
- Final PnL%
- Equity curve and/or PnL% curve


i thought about some parameters that show large difference -

the slope of the linear regression of PnL
and the R square of it.
both should be high (high slope for profitable trend, and high R2 to know how good is our linear line)
upload_2016-3-6_22-42-11.png


upload_2016-3-6_22-42-18.png
 
Shay said:
botpro -
MDD in system #1 = -175
MDD in system #2 = -150

StdDev of PnL in system #1 = 53.09
StdDev of PnL in system #2 = 41.27

what do you mean by
- Final PnL%
- Equity curve and/or PnL% curve
More...
- I meant that in the general case the equity curve should be made available to the verifying person.
This can be done by providing the trade list, or in case of portfolio trading (ie. more than 1 instrument gets traded), the daily account values, so the verifying person can just load it into excel, mark the column, and click on graph...
Of course providing the ready-made chart is a good idea as well.

- Backtests should prefer presenting most results as percentages.

So, you started in both cases at 25 and ended up at 150.
Do you know how much that makes, ie. how much percent profit it makes?
Sure you can calculate it in this simple case easily (it's 500% profit),
but in the general case that information belongs into the backtest report.

- Sharpe ratio for #1: (150 - 25) / 53.09 = 2.35
Sharpe ratio for #2: (150 - 25) / 41.27 = 3.03
--> so system #2 seems indeed to be better.

If I didn't make a calc error then MDD% in #1 seems to be 140%...
That's gigantic! A value >=100% means normally total loss of the investment (not neccessarily with options if they still have time till expiration)...
MDD% of the investment should be kept under about 15%, IMO....

MDD% in #2 seems to be 335%.
But I think in both MDD% cases I could have made some big calc errors. Anybody interessted should calc it him/herself...

What you can learn is this: you need to apply "diversification", ie. trade many titles (at least 4 I would say, the more the better) at the same time, and make the calcs over your shared account value (ie. all trades done from the same account), then your MDD% of your investment, ie. your risk, will then be much lower...
 
Last edited:
botpro -
it's not an actual system !
sorry if i misled you.

i just entered some values in excel,
so i should get the same PnL, same profit %, ...
and i wanted to here what else should i check,
for future back test
so we will know better how to compare systems.
 
Shay said:
botpro -
it's not an actual system !
sorry if i misled you.

i just entered some values in excel,
so i should get the same PnL, same profit %, ...
and i wanted to here what else should i check,
for future back test
so we will know better how to compare systems.
More...
I know it's not real, just a simulation model, that's no problem...
 
You must log in or register to reply here.
Back
Top