Well, no. You yourself wrote in your 2M lines backtest article that for a complete backtest, order objects are created -- these take CPU cycles and memory. So it's not exactly separate is it. The approach you have with Backtrader is relatively straightforward to implement, with pypy and great implementation it might be as fast as vectorization, I'll use BT as a benchmark.
My goal is to have something that's cheap on resources and large data, not something that is extremely versatile and implements all strategies. My (relatively simple) trading logic doesn't take more than a few seconds to apply on half a million lines or so, that's fine for me. I'm quite sure it takes longer with anything that loops.
I disagree that writing a framework is important at all. It is the last task one should try unless very certain that you need automation and have good strategies. Here is a post from reddit with the arguments that I won't replicate
Some final words
I am a professional quantitative portfolio manager, who has been in the industry for a very long time, and works on the bleeding edge of ML and applied mathematics with focus on the capital markets - I manage $100mn+ these days. I created this account to write on r/algotrading so that I can interact with a few people on this sub, but as I have seen, this sub is filled with amateurs and it is just annoying reading the feeds most days. I am going to delete my account and I wanted to leave a few points that I hope with help a few people here,
Anyways. Good luck and goodbye.
- BTC and other crypto-coins are nothing more than another asset. Stop putting it on a pedestal or thinking its anything different.
- ML is super hard when applying to financial markets, and its not something anyone can figure out very easily. Most amateurs can play around with RNNs and have a descent strategy, but don't think its going to give you anything extraordinary. It's just another tool in your toolbox to create a strategy.
- ML can be used to make some amazing automated trading systems, but it won't be possible for 99.999% of people. People have been doing ML for trading for a very very very very very very long time. You are being exposed to it just now because there are lots of tools and lots of resource that wasn't accessible before. Do not think taking tensorflow, sklearn, <insert library name here> and it will magically make you money. It takes a very long time, ie. decades to get anything automated to the level most of your dreamers think.
- Most of you are software engineers here. Stop thinking like one. Writing a new shiny backtesting tool or trading framework is not going to do anything than waste your time. Stop talking about languages, it really doesn't matter. Work on your alpha. Yea, its the thing that you don't know how to build, work on that. Trading frameworks come after.
- Anything that works on the intraday time-frame is considered HFT. Stop thinking that its only low-latency stuff, its basically what timeframe most of you are trying to make money in. People can do this, but, you need to find that thing that most of you avoid - alpha. Most people can't succeed here, so most of you, do yourselves a favor, trade daily+ timeframes, it will save you some frustration.
- If you have capital, make a portfolio of a few nice assets. Start with management accounting principals and work from there to figure out what makes one asset worth more than another.
- Stop asking people where to begin, how their stuff works. MONEY is involved here, no one will help you with anything. No one is going to tell you anything more than what I have said in the few points above. And the people who tell you things, are usually negative such as TA is bullshit or ML won't work or HFT is only latency sensitive stuff - well, most them are idiots who don't know what they are talking about. Let me tell you clear and simple here - TA is not bullshit, it's just mathematical transforms and features that MIGHT contain predictive power, ML can be used very well to make a lot of money, and HFT is anything on the sub-daily timeframe and a lot of strategies are not latency sensitive.
- Lastly, there are VERY smart people in the world, who have spent their entire lives studying, building and creating technological and scientific advances more than most of the people here can fathom. These people work in this industry and make a ton of money. I am happy that you saw some documentary of how a lot of people made money in the 70-80s trading and you want to be like them. Sorry, the world is different, with the availability of information and higher education standards, the bar to be good in this industry is very very very very high. So, you need to be a good scientist or have that mentality today to be good in this industry. Its great you want to be like the best of this industry, so start with being humble.
- xxzam
That's all great but off the shelf frameworks don't do it for some of us. It's sort of like telling a racing driver "don't worry about what car it is, just drive it". We all know the car matters and will impact end results.
Having a data, backtesting and automation tool that you built means you can implement a strategy fast, know the details of the operation, can fix bugs immediately and implement new features within minutes. No retail software allows for this. I've seen bugs pop up in the morning and I've patched it before market open. If I didn't know how or didn't have access to code, I'd be in some forum pleading a developer to do it for me...
