my Backtest results different than TastyTrade

OK, i got answers from TT research team -
they replied -
I didn't run the study back then, so I just ran the same study again. Here is my results:
Total PnL:$5,510
Win: 35
Max Loss: -1,445

My numbers are actually in between yours and our previous study. And the max loss is even smaller.

I personally believe this is due to how you choose the date and duration. Back to 2014, we manually pulled the data from trading platform, so there could be a couple days difference than yours and mine based on personal selection, which might change the results. Also since weekly options become more actively traded recently, and that could change the expiration date too.

Please look at our recent segment conducted after Sep 2015, we use intensive program to test all those strategies, and the results should be more reliable.

Hope this is helpful!
 
I think Tom has mentioned several times that they use EOD data for larger studies. I cannot provide where and when he said it but that's what I recall.
 
Diamond Geezer said:
I have saved dozens of Tastytrade back-test summary results and I have noticed inconsistencies many times. They often make the error of leaving out details of the back-testing assumption in their slides and then sometimes fill in the details on air. One huge failing in their presentation is the absence of equity curves and drawdown details. Max loss is not good enough.
More...
Oh, oh! Without an equity curve any system result is practically worthless.
Just imagine a big drawdown... and the psycho of the trader and his/her reaction or action to that... ;-)
 
TT answered -
We only use EOD data. We only use intraday data when the study is focusing on intraday stuff.
 
Shay:
Thanks for the info from TT! It is very nice to find they used EOD data! That implies we "should" be able to have apples to apples comparisons! Also, interesting that when they "re-ran" the test, they got different (less profitable results). The response suggesting that they may or may not have used "weekly" options as opposed to "monthly" options is a bit bothersome, and seems to imply the person responding does not know (may not imply errant data). It makes no logical sense to me that they would include any weeklies in that test (a mixture of weeklies with monthlies should void that test). However, it is possible they "botched that first run", the one they reported on, as the re-run seems close to what we would expect. That seems to imply they had/have inconsistent criteria in their process which should make us take their info with a grain of salt (but improving with time?)! Seems their re-run of the test had same number of Wins as my test, which kind of implies my test may be OK or close!
While I am unable to "export" from ThinkScript in TOS, If you want to manually observe my trade data, I can re-run, and zoom in on the time intervals to allow observation of all the trades. Let me know if you want me to do this.
I hope you continue to post your work and questions such as this!

It would be nice if they re-ran the entire test with the profit target exits with the "new and improved" back test, to allow us to also compare (our back tests) with their profit target tests, as I would suspect those tests results may have the same flaw as the one you were focusing on.
Thanks
 
Last edited:
Shay said:
Hi,

i have SPY data that i purchased from iVolatility
i inserted the data to MSSQL database, so i could do backtests easily

i checked some strategies, and i am trying to compare the results to what i see in TastyTrade.
and the results are different, and i wonder why.
the reasons can be -
1. my data is not good
2. TastyTrade did a mistake and publish wrong results
3. something else that i am missing

here is the tastytrade results for :
Selling Straddle
Enter 1st Trading Day of the month, around 45 DTE
Held until Expiration
56 trades
from 01/01/2010 until 30/09/2014

they checked 4 scenarios (exit 25%,50%,75%, Expiration)
but i am looking right now only on Expiration.

you can see their segment here -
https://www.tastytrade.com/tt/shows/market-measures/episodes/straddle-stops-09-30-2014

TastyTrade results :
PnL : 7083$
# of Wins : 42/56
Biggest Loss : -1577$

and my results (the same parameters) :
PnL : 4338$
# of Wins : 31/56
Biggest Loss : -1575$

as you can see,
my biggest loss is equal to TastyTrades
but there is a large difference between the number of the winning trades (and the PnL)
Why ? How ? Where is 11 trades that they say they are profitable ?

i attached my results,
i hope someone can do the same test, and tell me what is different.

Thanks,
Shay
More...

I suspect that the tastytrade researchers often determine the answer they desire, and then construct a study to support their predetermined opinion. They often take investment maxims, present them out of context, and then support that context with trading studies, ignoring that investing and trading are at best, cousins; but certainly not the same discipline.

I chuckle when I watch their dough related segments as they present the strategic use of stop-losses for small accounts after years of listening to Batista and Sosnoff arrogantly dismiss the legitimacy of stop-losses.
 
a few updates.
i got the detailed trade list from TT, so i could compare each trade.

there were some problems that i saw -
1. i had a mistake when closing the trade at expiration (i didn't calculate it correctly)

2. they are using Mid price, while i used Ask/Bid.
meaning - every single trade they did is more profitable than mine
the total difference between my PnL and TT's PnL is $1700
when i run my backtest using Mid Price, the difference was only $21.
some of the trades turn to be winners just because of the mid price.
the research team and other people say i should use mid price.
but i prefer to use pessimistic approach, and i think it's more accurate to use bid/ask (not in the expiration where i should calculate if i am ITM or worthless)

3. there were two trades of TT that the expiration was not correct - they exited on a weekly expiration, one week before the monthly expiration.
one of the trades was a winner in TT, while it was a loss in mine

4. i think there was a problem with another trade they did (the strike and the result of this trade seem weird to me)

to summarize :
1. very nice of TT research and support team to answer so quickly and with details. +1/Like to TT

2. we should treat TT results with care,
they have very good ideas !
but we need to check every result ourselves (like all ideas we hear in forums/sites/...)

i will recalculate my results after i will fix my bug from point #1 above, and i will publish it here.

Shay
 
as promised -
my results (after correcting my bug in expiration)
i use Bid/Ask (and not Mid Price)
for :
Selling Straddles
Enter 1st Trading Day of the month, around 45 DTE
Held until Expiration
56 trades
from 01/01/2010 until 30/09/2014

PnL : 5595$
# of Wins : 33/56
Biggest Loss : -1584$
Biggest Win : 976$
Avg BPE : 3511$
if you want me to write more values, just name them

Shay
 
You must log in or register to reply here.
Back
Top