my Backtest results different than TastyTrade

[Shay]

Hi,

i have SPY data that i purchased from iVolatility
i inserted the data to MSSQL database, so i could do backtests easily

i checked some strategies, and i am trying to compare the results to what i see in TastyTrade.
and the results are different, and i wonder why.
the reasons can be -
1. my data is not good
2. TastyTrade did a mistake and publish wrong results
3. something else that i am missing

here is the tastytrade results for :
Selling Straddle
Enter 1st Trading Day of the month, around 45 DTE
Held until Expiration
56 trades
from 01/01/2010 until 30/09/2014

they checked 4 scenarios (exit 25%,50%,75%, Expiration)
but i am looking right now only on Expiration.

you can see their segment here -
https://www.tastytrade.com/tt/shows/market-measures/episodes/straddle-stops-09-30-2014

TastyTrade results :
PnL : 7083$
# of Wins : 42/56
Biggest Loss : -1577$

and my results (the same parameters) :
PnL : 4338$
# of Wins : 31/56
Biggest Loss : -1575$

as you can see,
my biggest loss is equal to TastyTrades
but there is a large difference between the number of the winning trades (and the PnL)
Why ? How ? Where is 11 trades that they say they are profitable ?

i attached my results,
i hope someone can do the same test, and tell me what is different.

Thanks,
Shay

 

[botpro]

I think you have the TT PnL copied wrong from their text. Can you vrfy that first?

Here's the full text from the given link:
"
Many investors believe that using stops are a viable way to prevent large draw downs on their account. However here at tastytrade, we believe in managing our winning trades as opposed to managing our losing trades. Due to this methodology, we will rarely ever use stops and are out to prove why not. Today, Tom Sosnoff and Tony Battista look at a 5 year study based on SPY. The guys place an at the money (ATM) Straddle and look to manage the trade at 25% of max profit. Next, they look at the effect a stop has when placed at 25% of the credit received. This means that if the strangle was sold for $4, it was closed when it was trading for $3 (managing a winner) or if it was stopped at when it was trading for $5 (managing a loser). They also look at these two management levels combined, meaning what was the results when seeing what came first. Tom and Tony confirm what they already knew: stops are not as effective as managing your winning trades. By just managing the straddle, your total profit/loss (P/L) would have been $5,611, with a win rate of 88%. If you were using a stop, your P/L would have only been $1,670 with a win rate of 46%. When looking at both of the management styles together, you still have a lower P/L of $4,221 and a win rate of 59%. After looking at all these results, it proves that managing a winning trade is more effective than managing a losing trade.
"

Update:
ok, there are 9 slides besides the text. Slide #9 summarizes it, and yes then I see the discrepancy you mean:

 

[botpro]

I haven't seen a tradelist of them on their web pages.
If it was shown in the video (which I haven't watched), then simply compare their trades to your trades to find the cause, ie. debug the 2 lists...

The discrepancy could also be caused maybe due to the use of different entry and exit prices.
Just try also these combinations: avg(Bid, Ask) and avg(Bid, Ask, Last)

 

[Shay]

I haven't see a tradelist of them on their web pages.
If it was shown in the video (which I haven't watched), then simply compare their trades to your trades to find the cause, ie. debug the 2 lists...

The discrepancy could also be caused maybe due to the use of different entry and exit prices.
Just try also these combinations: avg(Bid, Ask) and avg(Bid, Ask, Last)

they don't show the tradelist.
only the summary that you can see in the slides.

i checked avg(Bid, Ask), and the results are :
PnL :6064$
# of Wins : 34/56 (still, too much missing winning trades)
Biggest Loss : -1531$

i don't have "last" in my data.


i hope someone can do the same backtest and tell me results

Shay

 

[botpro]

they don't show the tradelist.
only the summary that you can see in the slides.

i checked avg(Bid, Ask), and the results are :
PnL :6064$
# of Wins : 34/56 (still, too much missing winning trades)
Biggest Loss : -1531$

i don't have "last" in my data.


i hope someone can do the same backtest and tell me results

Shay

I think not many people keep/collect/buy options data of 5 years.
I would suggest to contact them over the discrepancy. Maybe they can give you their tradelist for analysis...

You could gain more insight into the used strategies if you feed them with GBM+BSM data (same data for all the strategies used),
and do that say 1000s of times with different GBM data (ie. seed the RNG properly each time, or just seed once at pgm start),
and then average the results and/or count the win and loss cases etc...
You can find source codes for both GBM (Geom. Brownian Motion) and BSM (BlackScholesMerton) here on ET, and also on the web.

The historical volatility over the last 3 months of SPY is about 20%, and about 18% over the last 6 months. So, you can take just 20%.
Or if you need it exact, then just recalculate it in each bar/tick/day for the past period of days
(usually over the past 30 or 50 or 100 days; this is IMO unfortunately not standardised in the industry;
I personally would take 3 months, ie. 63 business days or 91 calendar days)...

 

[Shay]

i know i can take random numbers and check with them.
but i want to compare apples to apples,
i already have data, and i wrote backtest code,
now i want to check if my code is ok,
and the best way to do that, is to take known results and see if i get similar results with my code.
this is why i want to check the same quote with the same parameters on the same dates.

Shay

 

[botpro]

i know i can take random numbers and check with them.
but i want to compare apples to apples,
i already have data, and i wrote backtest code,
now i want to check if my code is ok,
and the best way to do that, is to take known results and see if i get similar results with my code.
this is why i want to check the same quote with the same parameters on the same dates.

Shay

Like I already said, just get their tradelist and simply compare the two lists, then you will have the answer...

 

[stepandfetchit]

Shay:
It is refreshing to observe someone posting some of their detailed work. I did a similar exercise trying to use TastyTrade report to compare my back testing against. I had similar experience as you. I'll go back and see if I can find my work.
One issue I had was "imprecise" information on their test. Such as what time of day were the trades entered, and was this consistent. (I seem to recall my results were skewed similar to yours (lower success rate than stated by TT).
I would be surprised if TT will provide the details of their study as mentioned by "botpro", however I have never tried to obtain it from them. If you do, I would greatly appreciate it if you would share the info either here, or to me privately!

 

[botpro]

The probability that they made an error is lower than in the opposite case... because they have many viewers and so they can't afford making such a dilettante error...
But of course, nobody is perfect...

 

[botpro]

But people, please take such testing results, that use past data, always with a grain of salt... ;-)
...because it can mean what is widely known as "curve-fitting", ie. just try hundreds or more stocks (or trade sets)
and then present just the best one... ;-)
Or just picking specific timeframes where the system gives best results... ;-)

Backtesting is good in early stages of developing a system, but it never should be the basis for real trades.

Live-testing (also called forwardtesting) is always better... For this, professionals use simulated market data to perform countless tests quickly and cost-effectively,
and then present all the single results together with statistics (number of winning runs, number of losing runs, and averages etc.); it's called "GBM data" (Geometric Brownian Motion)...
.