Trading System Building Learning Curve?
- Thread starter thefuturestrader
- Start date
Yeah, I found those books and found them to be invaluable. Another thread on here also convinced me that I really need to learn a programming language, so I have been spending a ton of time with R. It's been fascinating yet frustrating at times. Still, I am beginning to see how important statistical analysis is for financial time series.
R is a great starting point! it's really easy to use, and there's a ton of stuff out there to help you get started. if you're interested in automated trading, it's important to have some programming ability. Even if you use something like ninja trader or the like to deploy orders, having the ability to use a research environment like R is gonna be helpful. Here's a list of steps you might be interested in taking if you wanna start systematically trading.
- Learn to program a bit, seems like that's what you're doing and that's good. You can use all sorts of different approaches and languages. In the beginning, just use whatever makes your life easiest. If you want to get into specifics about what to learn, you can PM me for my advice and opinions. There are too many contradictory opinions about this here and it's a waste of time to debate what the best is. So just use whatever is easiest and if you need help deciding what that is, ask me privately.
- You need data. This can come in a number of forms...you can get some free data from brokers, paid data services, etc. There's a vast range of qualities and price points for data. Tons of options out there, you can PM me for my recs again if you like.
- You'll likely want a research environment. MATLAB, R, Octav, Python...stuff like that. This will allow you to explore data and make inferences that you can use for forming trading strategies. Seems like you're well on your way there and understand the utility of these languages.
- A backtesting environment. There's, as always, a range of options here. Everything from something that comes from your broker, to expensive paid services, to building it yourself. Backtesting is very important, so make sure you understand how it works and some of its shortcomings.
- A clearing firm. Depending on how you want to attack this problem that may be down the line. As always, range of options, yada yada...etc. the key here is that you understand your commission structure,margin reqs, rules, and everything of the like because you can very easily come up with strategies that perform fantastically until you account for these variables.
- THE BIGGEST ONE: you need respect for the process. It's not easy. It won't happen overnight. There tons of reading, coding, bouncing questions off professionals, education, failure, trial and error, and most importantly time that will be spent here. I would strongly discourage getting into this sort of endeavor if you're just interested in easy profits, because that's just not how it is. You're gonna throw away almost everything you ever code up many many times. You're gonna be so terribly wrong about some things that you'll want to hide...I myself have some ET Posts from early in my two or three month career here that make me cringe when I reread them. If you aren't 'about the journey,' you're almost guaranteed to fail (and I'm one of the more optimistic people you'll find here), so make sure you have a childlike fascination with this subject and commensurate work ethic as well as ample free time before you invest any money into this, because it's very consuming of a both human resources and money. When I really think about it, it's almost out of character for me to preach about how much work this is. Only after a talk with another ET member last night did I really reflect on how much effort this takes, because to me it, it's just a hobby. Some kids play Xbox, I work on quantitative trading systems. I couldn't imagine success here really being achievable without this kind of affinity for trading, because I'm not such a patient guy in other areas of life. If you're in for the long haul, I wish you the best of luck. Don't hesitate to PM me, I'm on the same journey as you!
Last edited by a moderator:
This is a great response, thanks. I have access to decent data, since I work at a small futures brokerage. I have actually gotten a little "burned out" on system development so am taking a break. I did come to the realization you discuss that this is a longer term process, and I need to break that process down to manageable bites. Between the learning statistics and learning R, I got somewhat overwhelmed. Further, as good as all the different sources for R and trading systems out there are, I don't really have someone in the next office I can get help with from. Thus, when I get stuck on something, I tend to get really stuck and it may take a while to resolve even a small issue. I can't tell you how difficult it was, and still is, for me to convert a data source into an xts object. I really want to get to the point that I can effectively collaborate with others on projects, and be an effective contributor. I would like to design a spread trading system for trading ag and energy commodities. I believe I am committed for the long-haul, but my overloaded brain needs to take a small break by watching some sports and playing some video games.
Haha well just because I like playing around with these systems doesn't mean that I don't play sports or video games either! I hate to come of as berating you into thinking this is going to be an insurmountable task. If you work at a futures brokerage you already have a huge leg up on me because you can get exact specification on how clearing works, commissions, etc etc. as well as access to data. I presume, if your firm records its own data, theres some kind of IT infrastructure around you that can help you figure out how to database it yourself and call it into R and stuff. That's more than 90% of the folks on ET can do, including myself some of the time. Maybe I initially misjudged your level of involvement in the industry, but that's a good thing. You should have no problems crossing some major bridges that have impeded me in the past like paying 400+ a month at times for data on a college budget.
I wouldn't really say that R is that difficult to learn at all, frankly. I took my intro programming classes in C++, so whenever I look at a new language, it seems like much easier because the level of abstraction is always so much higher. If you ever have any R questions, feel free to ask me; if I can't solve them, I'll pretend they're my own and have some of my TA's look at them haha. As I'm sure you're aware, it's not like guys at Microsoft and Google are pushing out production code in R; its strength is abstraction and ease of use, it's weakness is speed, and flexibility. I realize that I'm...again...sounding like an elitist, but I totally understand that programming is very difficult to learn on your own. I spent a lot of time teaching myself before I got to college, and even now I get embarrassed regularly during interviews because I have huge holes in my knowledge. I've been through the frustrations that you're going through learning, and what I'd tell you is a) it's totally normal b) don't to be afraid to ask c) preserver. Stackoverflow and sites like that might seem daunting at first, but those doods put their pants on one leg at a time too. When I started asking questions and built up a little network of friends online and around school to help me solve programming questions, it really helped me. Just ask. Literally the worst thing they can do is close the thread haha. If you want even more online help, you can use python with/in place of of R. There are plenty of people using python for all sorts of things, so finding help and youtube videos will be even easier. I don't personally use it much, but people love python.
Put bluntly: the level of difficulty of this process is directly related to the level of abstraction/control you want in your trading systems. Do you want to trade higher timeframes and not stress execution or any of that? That's easy. Do some data analysis in R, and code up your findings in something like ninja or amibroker, backtest it, and trade it. That kind of process could take you only a couple months. The flip side of that, is if you want to build something more similar to what a prop trading desk or a similar entity might deploy, you're looking at reinventing several of the 'wheels' that come with a platform like ninjatrader etc. With your currently learning curve, that might take several years, honestly. Just be honest with yourself and your objectives. I'm one of maybe 10 people actively posting on this forum who are hardheaded enough to try to build a lot of this stuff themselves, but there are hundreds of people here who will certainly line up out the door to tell you how profitable they all are using some of the off the shelf options. I have sort of a different paradigm than most of the people here; where they look for software or tools to perform a task, I look at the software and ask myself 'what are the limitations?', 'could I come up with something better?', 'will I understand how it works if I don't build it myself or see the source code?', and (of course) 'how am I supposed to afford this?!' for a lot of the higher end offerings. That's just personal preference...it often holds me back. I'm not saying that it's necessary, in fact I'm really saying just the opposite.
Also, I love that one of your goals is to get to a level of competency where you can collab with others in this field! This site seems to give such an austere and negative vibe, whenever anyone purposes a project, more people come to bash the idea than to help haha (granted a lot of the ideas I've heard on here for projects are...far fetched). That's just the nature of trading I suppose, it attracts a lot of people who believe they're right more often than other people kinda by definition, right? I personally have benefited greatly from folks on here and elsewhere who offer information and have more of a team/collaborative attitude. It will serve you well as you progress. If you want to collaborate on anything, don't hesitate to PM me and I'll happily consider participating or offer advice. I think part of the reason this site is so dead is this pervasive dick measuring attitude in trading circles. I think if we as a site are collectively more welcoming and collaborative we'll probably all benefit. Anyway, let me just also quickly say, don't be afraid to start collaborating with people now. No one's gonna tell you your help isn't wanted no matter what your ability is. People tend to exaggerate their abilities on the internet. Your level of R competency is probably not that different from mine, and mine probably isn't that different than some of my professors, etc. etc. Don't be dissuaded by what you see on the internet, as long as you stay grounded about your abilities and legitimately want to help, no one is gonna turn you away from a one of these trading projects...it's not anyone here is writing Boost or some big open source project with top notch contributors or something.
I wouldn't really say that R is that difficult to learn at all, frankly. I took my intro programming classes in C++, so whenever I look at a new language, it seems like much easier because the level of abstraction is always so much higher. If you ever have any R questions, feel free to ask me; if I can't solve them, I'll pretend they're my own and have some of my TA's look at them haha. As I'm sure you're aware, it's not like guys at Microsoft and Google are pushing out production code in R; its strength is abstraction and ease of use, it's weakness is speed, and flexibility. I realize that I'm...again...sounding like an elitist, but I totally understand that programming is very difficult to learn on your own. I spent a lot of time teaching myself before I got to college, and even now I get embarrassed regularly during interviews because I have huge holes in my knowledge. I've been through the frustrations that you're going through learning, and what I'd tell you is a) it's totally normal b) don't to be afraid to ask c) preserver. Stackoverflow and sites like that might seem daunting at first, but those doods put their pants on one leg at a time too. When I started asking questions and built up a little network of friends online and around school to help me solve programming questions, it really helped me. Just ask. Literally the worst thing they can do is close the thread haha. If you want even more online help, you can use python with/in place of of R. There are plenty of people using python for all sorts of things, so finding help and youtube videos will be even easier. I don't personally use it much, but people love python.
Put bluntly: the level of difficulty of this process is directly related to the level of abstraction/control you want in your trading systems. Do you want to trade higher timeframes and not stress execution or any of that? That's easy. Do some data analysis in R, and code up your findings in something like ninja or amibroker, backtest it, and trade it. That kind of process could take you only a couple months. The flip side of that, is if you want to build something more similar to what a prop trading desk or a similar entity might deploy, you're looking at reinventing several of the 'wheels' that come with a platform like ninjatrader etc. With your currently learning curve, that might take several years, honestly. Just be honest with yourself and your objectives. I'm one of maybe 10 people actively posting on this forum who are hardheaded enough to try to build a lot of this stuff themselves, but there are hundreds of people here who will certainly line up out the door to tell you how profitable they all are using some of the off the shelf options. I have sort of a different paradigm than most of the people here; where they look for software or tools to perform a task, I look at the software and ask myself 'what are the limitations?', 'could I come up with something better?', 'will I understand how it works if I don't build it myself or see the source code?', and (of course) 'how am I supposed to afford this?!' for a lot of the higher end offerings. That's just personal preference...it often holds me back. I'm not saying that it's necessary, in fact I'm really saying just the opposite.
Also, I love that one of your goals is to get to a level of competency where you can collab with others in this field! This site seems to give such an austere and negative vibe, whenever anyone purposes a project, more people come to bash the idea than to help haha (granted a lot of the ideas I've heard on here for projects are...far fetched). That's just the nature of trading I suppose, it attracts a lot of people who believe they're right more often than other people kinda by definition, right? I personally have benefited greatly from folks on here and elsewhere who offer information and have more of a team/collaborative attitude. It will serve you well as you progress. If you want to collaborate on anything, don't hesitate to PM me and I'll happily consider participating or offer advice. I think part of the reason this site is so dead is this pervasive dick measuring attitude in trading circles. I think if we as a site are collectively more welcoming and collaborative we'll probably all benefit. Anyway, let me just also quickly say, don't be afraid to start collaborating with people now. No one's gonna tell you your help isn't wanted no matter what your ability is. People tend to exaggerate their abilities on the internet. Your level of R competency is probably not that different from mine, and mine probably isn't that different than some of my professors, etc. etc. Don't be dissuaded by what you see on the internet, as long as you stay grounded about your abilities and legitimately want to help, no one is gonna turn you away from a one of these trading projects...it's not anyone here is writing Boost or some big open source project with top notch contributors or something.
There are plenty of resources outthere that can teach you to write code, like MOOC's and books. But at the end of the day nothing beats practice, it takes some effort to learn how to think about problems programatically and at the beginning you'll struggle with syntax... but one day, out of the blue, something goes click in your head and all of a sudden you're no longer thinking about the language you're using, instead you're thinking about the problem... and that's when things get really fun. 
Thanks for the great responses again. For a potential system, I am thinking of anywhere from 15 minute to 60 minute bars, as well as using spread trades. For the most part, I think Ninjatrader would be fine for a execution/trading program. I don't want to trade higher time frames, or deal with execution strategies at this point, as I realize this is beyond what I am currently capable of. There are some ideas with ARMA models, as well as Hidden Markov Models that I would like to play around with. R is such a powerful tool for development, as well as learning statistics. I still have a long ways to go in both programming and statistics.
