Done. I had fixed it before, but due to some accident, reverted the change. Fixed it even better now. Thanks for the motivation
Nooby McNoob becomes a quant
- Thread starter nooby_mcnoob
- Start date
This is a half-journal entry. Trades are going through pretty smoothly now, and I have narrowed down the area which causes backtest data to be different from live data. Getting some help from the backtrader author on this.
I have confirmed that my "fix" for cancelling the trade after 30 seconds did not work because I used the wrong parameters. I have attempted to fix it again. This time I verified that it works in backtesting.
This will probably be the closest I get to a hands-off week.
The next thing I have to figure out is: how do I get reports out of IB? IB shows buys and sells, but doesn't pair up trades. I would like to be able to programatically extract a report from IB that shows each trade and the profit/loss. I would also like this to show up in my monitoring dashboard (at the moment, only live trades show up and those are susceptible to crashes so I would like the data to come from IB directly.)
I have confirmed that my "fix" for cancelling the trade after 30 seconds did not work because I used the wrong parameters. I have attempted to fix it again. This time I verified that it works in backtesting.
This will probably be the closest I get to a hands-off week.
The next thing I have to figure out is: how do I get reports out of IB? IB shows buys and sells, but doesn't pair up trades. I would like to be able to programatically extract a report from IB that shows each trade and the profit/loss. I would also like this to show up in my monitoring dashboard (at the moment, only live trades show up and those are susceptible to crashes so I would like the data to come from IB directly.)
Do you mean getting reports via the API? Or getting them from online account management and then scraping the screen?
@HobbyTrading I mean whichever would allow me to achieve those goals. I'd like to start running analysis on the trades outside of the backtests. Perhaps pushing it into something like pyfolio?
I have not implemented anything myself. But I've read that you can't get much historical trading data via the API. The maximum seems to be that you can only get data of the last couple of days. The solution, if you need a longer trading history, is to use the online report generator at account management and then parse the XML report into a format that you can further process.
From a first glance, it looks as though you're right. I couldn't even automate it because the 2FA would be needed.
Journal entry
The themes of this week:
1. My architecture around live trading is working well
2. The backtrader platform is not well-suited to live trading
3. Backtests are not matching live trades due to (what I believe) is a technical issue
I have discovered that the reason the live trades do not match the backtest is that there is a fundamental mismatch between how the software handles live trading and backtesting. Unfortunately, I don't think the author is using it for live trading which makes it tough for him to help or discover issues. Also, the code is horrifying and I don't think I could fix it in a meaningful amount of time.
Thoughts/current state:
1. I have backtested on two separate platforms that give similar results. I believe in the strategy.
2. I don't want to use backtrader to live trade anymore.
3. I am happy with the architecture I have built. It gives me the information I want, in a format I need, which allows me to verify if the algorithm is working as intended.
Questions for me to answer:
1. Should I give up?
2. Should I backtest using a third platform (that would make it two commercial, one open source)?
3. Should I find or build another backtest system (pysystemtrade, qtpylib)
My initial answers are:
1. No
2. Yes. Note that I have had a forward test running on a commercial platform this whole time which matches backtest entries, but not paper trading (they don't support IB?) Might as well get MT4 or NinjaTrader or something that integrates with IB and see how it fares.
3. I have briefly looked into pysystemtrade and qtpylib. My inclination is towards qtpylib primarily because he has built in all the things I have already built in to my architecture around backtrader. Notifications, blotter, shared data, event-based, etc.
This is kind of tough to swallow. I'm really sad that the backtests don't match live trades using backtrader. The emotion I'm feeling right now is just disappointment. It's like taking off a bad trade where you didn't have a stop loss. What's the lesson? What did I do wrong? Is it using an open source system? Is it waiting too long to paper trade? I think the thing I did wrong was not listen to others saying to use a commercial system.
Mais c'est la vie, n'est-ce pas?
I write in French when I'm really trying to avoid the emotion.
So my plan for this upcoming week is to fight my urge to use an open source system, or build my own, load up a NinjaTrader trial and see how it fares on the backtest. I will still let my paper trading run in order to validate the architecture, but I will no longer care about the results.
The themes of this week:
1. My architecture around live trading is working well
2. The backtrader platform is not well-suited to live trading
3. Backtests are not matching live trades due to (what I believe) is a technical issue
I have discovered that the reason the live trades do not match the backtest is that there is a fundamental mismatch between how the software handles live trading and backtesting. Unfortunately, I don't think the author is using it for live trading which makes it tough for him to help or discover issues. Also, the code is horrifying and I don't think I could fix it in a meaningful amount of time.
Thoughts/current state:
1. I have backtested on two separate platforms that give similar results. I believe in the strategy.
2. I don't want to use backtrader to live trade anymore.
3. I am happy with the architecture I have built. It gives me the information I want, in a format I need, which allows me to verify if the algorithm is working as intended.
Questions for me to answer:
1. Should I give up?
2. Should I backtest using a third platform (that would make it two commercial, one open source)?
3. Should I find or build another backtest system (pysystemtrade, qtpylib)
My initial answers are:
1. No
2. Yes. Note that I have had a forward test running on a commercial platform this whole time which matches backtest entries, but not paper trading (they don't support IB?) Might as well get MT4 or NinjaTrader or something that integrates with IB and see how it fares.
3. I have briefly looked into pysystemtrade and qtpylib. My inclination is towards qtpylib primarily because he has built in all the things I have already built in to my architecture around backtrader. Notifications, blotter, shared data, event-based, etc.
This is kind of tough to swallow. I'm really sad that the backtests don't match live trades using backtrader. The emotion I'm feeling right now is just disappointment. It's like taking off a bad trade where you didn't have a stop loss. What's the lesson? What did I do wrong? Is it using an open source system? Is it waiting too long to paper trade? I think the thing I did wrong was not listen to others saying to use a commercial system.
Mais c'est la vie, n'est-ce pas?
I write in French when I'm really trying to avoid the emotion.
So my plan for this upcoming week is to fight my urge to use an open source system, or build my own, load up a NinjaTrader trial and see how it fares on the backtest. I will still let my paper trading run in order to validate the architecture, but I will no longer care about the results.
Following you along remains an interesting read. Seeing the progress you make, and the setbacks you experience.
I'm not precisely sure what is causing the mismatch between your backtest setup(s) and the live paper trading. However, I do hope that you are aware that IB's paper trading behaves differently than "the real thing". You should make sure that the discrepancies you observe are not caused by anomalies due to IB's paper trading setup.
Having said that, if the issues are indeed caused by the way how IB's paper trading executes orders, your only respite would be to use a live account. There would be not much else you could do. But you would want to start small, with small position sizes to avoid incurring large losses in cases there are some bugs remaining in your code.
Bonne chance.
I'm not precisely sure what is causing the mismatch between your backtest setup(s) and the live paper trading. However, I do hope that you are aware that IB's paper trading behaves differently than "the real thing". You should make sure that the discrepancies you observe are not caused by anomalies due to IB's paper trading setup.
Having said that, if the issues are indeed caused by the way how IB's paper trading executes orders, your only respite would be to use a live account. There would be not much else you could do. But you would want to start small, with small position sizes to avoid incurring large losses in cases there are some bugs remaining in your code.
Bonne chance.
Sorry to hear of your struggles, but your experience is a common one. I went through the same thing years ago, when I first tried my own backtest platform. Eventually, I went with a retail platform (Tradestation in my case).
But I was scared to death at first. All the order functions, switches, etc. How would I know if backtest results would be replicated live?
So, I started simple. A basic moving average crossover, with daily bars, with simple "buy next bar at market" order types. I then ran it live with real money. Then I compared 4 results:
1. The real money results
2. The real time strategy engine results
3. The backtest results from the strategy engine
4. Calculating trades by hand, just using price data
All 4 of those SHOULD be the same. If they aren't, you have an issue somewhere. I assume you know and have seen that.
Once I was satisfied that all 4 agreed, I then slowly introduces more complicated things (it took me a few years before I was happy with "setstoploss" for example. Even now 10+ years later, I still stick with time based bars - no exotic bars for me).
For me, then, a retail platform did the trick. For you, depending on what you are doing, it might also be good. Just start out simple, get agreement with results, and build from there.
But I was scared to death at first. All the order functions, switches, etc. How would I know if backtest results would be replicated live?
So, I started simple. A basic moving average crossover, with daily bars, with simple "buy next bar at market" order types. I then ran it live with real money. Then I compared 4 results:
1. The real money results
2. The real time strategy engine results
3. The backtest results from the strategy engine
4. Calculating trades by hand, just using price data
All 4 of those SHOULD be the same. If they aren't, you have an issue somewhere. I assume you know and have seen that.
Once I was satisfied that all 4 agreed, I then slowly introduces more complicated things (it took me a few years before I was happy with "setstoploss" for example. Even now 10+ years later, I still stick with time based bars - no exotic bars for me).
For me, then, a retail platform did the trick. For you, depending on what you are doing, it might also be good. Just start out simple, get agreement with results, and build from there.
Thanks! I am generally unemotional to things, and I would say the closest approximation is "sadness". But it's mostly a reaction akin to "ugh, more work before I reach my goal." Heavy shoulders, etc. I greatly appreciate the feedback. I am now in "figure it out mode" so that feeling is gone.
That being said, I should have been more clear as to why there is an issue:
The back tested data is different from the live data in that the live data appears to be off by a period leading to incorrect entries when live trading. I am generally fine with IB's paper trading execution for my strategy and I don't believe it will be an issue in the long term. That is to say there is a relation which appears to be roughly ohlc_live(t+1) ~ ohlc_backtest(t). They are offset by a period. This isn't exactly true, as some values are somewhat different but I haven't yet figured out why.
For example, here is a log from live trading:
DEBUG:root:2018-04-06 16:12:00 ... bar1_ohlc=[1.27749,1.27864,1.27659,1.27783], bar2_ohlc=[1.27782,1.27857,1.2765,1.27678], ...
And here is the same bar from backtesting:
DEBUG:root:2018-04-06 16:12:00 ... bar1_ohlc=[1.27791,1.27857,1.2765,1.27681], bar2_ohlc=[1.27681,1.27822,1.2759,1.2782], ...
These values should match, but they don't. So the question is: which bars match the live values from backtesting?
DEBUG:root:2018-04-06 15:12:00 ... bar1_ohlc=[1.2775,1.27864,1.27659,1.27791], bar2_ohlc=[1.27791,1.27857,1.2765,1.27681] ...
The backtested bar that occurs one hour before the live bar (the bars are hourly). That is: there is an off-by-1 issue somewhere. The backtested data occurs an hour earlier than the corresponding live data. Note that the offset bars don't match _exactly_ especially with regards to open and close values. In live trading, those may be one value, whereas IB may rejigger historical data due to various issues. So I'm not too worried about that. I have also confirmed that the backtesting has last price occurring one period earlier than live trading.
In my eyes, this explains 100% why live entries don't match backtests.
Maybe I should be debugging this a bit more, even if it is a bug in the platform...