Great thread!
Runningbear
Runningbear
homegrown trading software
- Thread starter dchang0
- Start date
0) What vital features, capabilities or characteristics of your software drove you to write it yourself rather than use some preexisting trading software?
---
Wanted to capture some specific post trade data. Needed to integrate with some existy legacy analytics software.
---
1) Is it written entirely from scratch, or did you use a framework/API such as Quantstudio or IB's TWS to help you along? If so, which framework?
---
Completely from scratch.
---
2) What programming language did you use? Is it an interpreted language like perl or a compiled language like C?
---
C++ - dont want to get into a holy war, but I still think this is the fastest. VB is catching up and maybe be neck and neck now. Most of what we are during is VERY speed sensative.
---
3) What OS(es) is your platform based on, and is your software multithreaded or multiprocessor-capable?
---
Windows 2000. Multi-threaded and multi-processor capable. Looking at Linux for some applications due to high cost of MS licenses. Havent done speed tests yet though.
---
4) Are you using grid computing or a server farm for the heavy lifting?
---
Nope. More trouble than it was worth. With the high power computers today, we use some dual processors boxes for options modelling and trading apps, but they only hit about 10% of the CPUs at max.
---
5) Can your software platform handle lots of different trading systems/indicators, or does it have to be rewritten/modified to handle new trading systems/indicators?
---
Can handle some simple changes to the systems/indicators. But a large shift in the model requires some rewriting of code. We can generally turn around the code changes within a trading session if needed.
---
6) How fast is your homegrown trading software? (A highly subjective question, I know, so please pick a reference point to give us a comparison.)
---
Not sure what reference point you want. We write directly to the exchange APIs to take out any middle step - (TT, PATS or other APIs). For some exchanges this requires you to write an exchange specific server. Depending on how critical speed is to you and if you have access to exchanges lines, writing directly to the exchanges is the best way to cut down on your round trip time. Can see orders and fills in terms of millesecond round trips.
---
7) How long did it take to *initially* develop the first acceptably-working version of your software?
---
Simple order entry/trade fill application - 3 days
Complex trading application (Spreads, multi-legs, auto-stops, etc)
3 weeks
Exchange Server - Depends on the complexity of the exchange.
Simple Exchange - Island (INET) for example - about 2 weeks.
LIFFE/eCBOT - about 3-4 weeks, but then you have to go through their conformance testing.
---
8) Optional question: what is the estimated cost of developing your homegrown trading software?
---
Costs estimates can vary. Do you have to hire someone? How complex is it?
For simple C++ applications, we generally estimate the cost at around $50-$75 per man hour. When you factor in salary, health insurance, etc.
For the more complex applications (exchange servers) your looking at closer to $100 per man hour.
So from this, you can see if custom building the servers/applications makes sense for you. For us it did. We did cost justifications and some ROI studies before and after. We have been using some ActiveX controls into a trading app and we were getting creamed. We built our own and consistently make money. Same exact trading model but we were now first to the order rather than getting run over.
If your trading isn't as time sensative, it probably doesn't make sense to write your own servers. May be easier to use the hooks from TT, Pats, etc.
---
9) Optional question: did you reverse-engineer anything in order to develop your software?
---
No. Took some ideas from other applications, but I'd say 90-95% our own ideas.
---
10) Optional question: is your software good enough to be released to the public and sold as a competitor to today's trading software platforms?
Sort of. We have some bugs in the applications, but most of the time we just tell the traders not to use that feature because its not working. With a one or two weeks of some more hardcore system testing it would be commericial viable product. But we are not looking to sell.
---
Wanted to capture some specific post trade data. Needed to integrate with some existy legacy analytics software.
---
1) Is it written entirely from scratch, or did you use a framework/API such as Quantstudio or IB's TWS to help you along? If so, which framework?
---
Completely from scratch.
---
2) What programming language did you use? Is it an interpreted language like perl or a compiled language like C?
---
C++ - dont want to get into a holy war, but I still think this is the fastest. VB is catching up and maybe be neck and neck now. Most of what we are during is VERY speed sensative.
---
3) What OS(es) is your platform based on, and is your software multithreaded or multiprocessor-capable?
---
Windows 2000. Multi-threaded and multi-processor capable. Looking at Linux for some applications due to high cost of MS licenses. Havent done speed tests yet though.
---
4) Are you using grid computing or a server farm for the heavy lifting?
---
Nope. More trouble than it was worth. With the high power computers today, we use some dual processors boxes for options modelling and trading apps, but they only hit about 10% of the CPUs at max.
---
5) Can your software platform handle lots of different trading systems/indicators, or does it have to be rewritten/modified to handle new trading systems/indicators?
---
Can handle some simple changes to the systems/indicators. But a large shift in the model requires some rewriting of code. We can generally turn around the code changes within a trading session if needed.
---
6) How fast is your homegrown trading software? (A highly subjective question, I know, so please pick a reference point to give us a comparison.)
---
Not sure what reference point you want. We write directly to the exchange APIs to take out any middle step - (TT, PATS or other APIs). For some exchanges this requires you to write an exchange specific server. Depending on how critical speed is to you and if you have access to exchanges lines, writing directly to the exchanges is the best way to cut down on your round trip time. Can see orders and fills in terms of millesecond round trips.
---
7) How long did it take to *initially* develop the first acceptably-working version of your software?
---
Simple order entry/trade fill application - 3 days
Complex trading application (Spreads, multi-legs, auto-stops, etc)
3 weeks
Exchange Server - Depends on the complexity of the exchange.
Simple Exchange - Island (INET) for example - about 2 weeks.
LIFFE/eCBOT - about 3-4 weeks, but then you have to go through their conformance testing.
---
8) Optional question: what is the estimated cost of developing your homegrown trading software?
---
Costs estimates can vary. Do you have to hire someone? How complex is it?
For simple C++ applications, we generally estimate the cost at around $50-$75 per man hour. When you factor in salary, health insurance, etc.
For the more complex applications (exchange servers) your looking at closer to $100 per man hour.
So from this, you can see if custom building the servers/applications makes sense for you. For us it did. We did cost justifications and some ROI studies before and after. We have been using some ActiveX controls into a trading app and we were getting creamed. We built our own and consistently make money. Same exact trading model but we were now first to the order rather than getting run over.
If your trading isn't as time sensative, it probably doesn't make sense to write your own servers. May be easier to use the hooks from TT, Pats, etc.
---
9) Optional question: did you reverse-engineer anything in order to develop your software?
---
No. Took some ideas from other applications, but I'd say 90-95% our own ideas.
---
10) Optional question: is your software good enough to be released to the public and sold as a competitor to today's trading software platforms?
Sort of. We have some bugs in the applications, but most of the time we just tell the traders not to use that feature because its not working. With a one or two weeks of some more hardcore system testing it would be commericial viable product. But we are not looking to sell.
NinjaTrader_Dierk
ET Sponsor
nitro,
1) The pats adapter is included in the download.
2) As with complete TM 2, the TM 2 Pats adapter is free for personal, non-commercial use.
3) It's based on the Patsystems proprietary API.
4) The TM Pats adapter is Patsystems certified.
5) You still have to certify your TM based app at Patsystems. The reduced Pats fee (not TM!) is $250.
1) The pats adapter is included in the download.
2) As with complete TM 2, the TM 2 Pats adapter is free for personal, non-commercial use.
3) It's based on the Patsystems proprietary API.
4) The TM Pats adapter is Patsystems certified.
5) You still have to certify your TM based app at Patsystems. The reduced Pats fee (not TM!) is $250.
jlm890:
Fantastic answer, John! I realize some of the questions may not be specific enough to get a clear answer, but I left them that way to try to cast as wide a net as possible, since I have no idea how varied the types of homegrown apps there are out there are.
So, for #6, anyone can choose whatever point of reference one wants. You answered it very concretely, using milliseconds to describe the speed, which is a far better answer than I expected. I was expecting most people to use a subjective comparison such as: "it backtests on average four times faster than it takes TradeStation to go through a basket of 500 stocks, going over one year of 1-min bars," or, "it trades automatically about twice as fast as Wealth-Lab and about half the speed of Amibroker across the entire universe of stocks."
And on #8--also a highly subjective question--new respondees are encouraged to describe how much it cost to develop their specific systems in any concrete terms they wish. For instance, if you are using your own in-house (unpaid) programmers, you could say that it cost "45 man-hours" or "2 weeks of two programmers working full time." For those efforts where professional programmers were used and actual project costs are known, please share those with us too, such as: "total of $53,457.81 with all software/API licenses, programming and data costs, not including hardware costs and OS licenses."
Thanks for all the great responses so far! (And thanks for answering all the questions too!)
Fantastic answer, John! I realize some of the questions may not be specific enough to get a clear answer, but I left them that way to try to cast as wide a net as possible, since I have no idea how varied the types of homegrown apps there are out there are.
So, for #6, anyone can choose whatever point of reference one wants. You answered it very concretely, using milliseconds to describe the speed, which is a far better answer than I expected. I was expecting most people to use a subjective comparison such as: "it backtests on average four times faster than it takes TradeStation to go through a basket of 500 stocks, going over one year of 1-min bars," or, "it trades automatically about twice as fast as Wealth-Lab and about half the speed of Amibroker across the entire universe of stocks."
And on #8--also a highly subjective question--new respondees are encouraged to describe how much it cost to develop their specific systems in any concrete terms they wish. For instance, if you are using your own in-house (unpaid) programmers, you could say that it cost "45 man-hours" or "2 weeks of two programmers working full time." For those efforts where professional programmers were used and actual project costs are known, please share those with us too, such as: "total of $53,457.81 with all software/API licenses, programming and data costs, not including hardware costs and OS licenses."
Thanks for all the great responses so far! (And thanks for answering all the questions too!)
Quote from damir00:
perl is the glue that holds the universe together.somebody mentioned python, also a useful too esp. if lots of database stuff. and excel (or one of the many free/cheap clones) is a great way to start. Mathematica isn't even a tool, really, it almost qualifies as a way of life, lol.
If you want speed, do C++. If you want to get stuff done quickly and don't mind that it might not execute the fastest, try perl or ruby. Ruby is more pleasing to the senses than perl (especially syntax). And if you code in an object oriented fashion, ruby is for you.
My program is not completed yet, but I figured I would answer anyways.
0) More control than commercial apps since I control the code. Little tweaks will be easy with my program but may be more difficult with someone elses. Also, no commercial program does *exactly* what I want.
1 & 2) It is written in Java utilizing the IB TWS API. I use Eclipse as my IDE.
3) I use Windows, but I assume that since it is Java, it will work anywhere TWS will work.
4) No, it is not complex enough to warrant that kind of stuff.
5) As of now, I would need to program each indicator I want.
6) Fast enough for me.
7) It is still in development and has been for 9 months. I program it in my spare time but am starting to concentrate more on it.
8) The only costs were my time and whatever I spent on books, which I probably would have bought anyways. The way I see it, even if my program does not work, I will have learned a lot about trading and programming so I feel it is worth it in the long run.
9) No, but I intend to scour the TWS API source code to see if there is anything useful.
10) Not yet, but I hope one day it will be.
0) More control than commercial apps since I control the code. Little tweaks will be easy with my program but may be more difficult with someone elses. Also, no commercial program does *exactly* what I want.
1 & 2) It is written in Java utilizing the IB TWS API. I use Eclipse as my IDE.
3) I use Windows, but I assume that since it is Java, it will work anywhere TWS will work.
4) No, it is not complex enough to warrant that kind of stuff.
5) As of now, I would need to program each indicator I want.
6) Fast enough for me.
7) It is still in development and has been for 9 months. I program it in my spare time but am starting to concentrate more on it.
8) The only costs were my time and whatever I spent on books, which I probably would have bought anyways. The way I see it, even if my program does not work, I will have learned a lot about trading and programming so I feel it is worth it in the long run.
9) No, but I intend to scour the TWS API source code to see if there is anything useful.
10) Not yet, but I hope one day it will be.
I also wrote and am using my own trading software. To summarize, I did this to eliminate dependencies on other companies, programming models etc.
0) Mostly the ability to express anything. I had found metastock and tradestation too simplistic.
1) From scratch, but using eSignal and TWS for quotes and execution
2) In Java where possible. eSignal required a bit of C++. Started out in C++
3) JVM primarily on XP
4) Yes, the software is fully distributed and generally runs on 3 computers.
5) No limits on indicators or systems
6) Fast enough. Can typically run 5 systems on ~150 securiteis in RT on a 1GHz PIII
7) I've been at this since 96; so a LONG long time. Started taking signals in 97 or so.
8) Insanely high. But lots of fun.
9) No.
10) Absolutely; but why would I?
0) Mostly the ability to express anything. I had found metastock and tradestation too simplistic.
1) From scratch, but using eSignal and TWS for quotes and execution
2) In Java where possible. eSignal required a bit of C++. Started out in C++
3) JVM primarily on XP
4) Yes, the software is fully distributed and generally runs on 3 computers.
5) No limits on indicators or systems
6) Fast enough. Can typically run 5 systems on ~150 securiteis in RT on a 1GHz PIII
7) I've been at this since 96; so a LONG long time. Started taking signals in 97 or so.
8) Insanely high. But lots of fun.
9) No.
10) Absolutely; but why would I?
0) What vital features, capabilities or characteristics of your software drove you to write it yourself rather than use some preexisting trading software?
available software can be easily tricked by the market (block trades, bid/ask issues), it's convenient to have software dedicated to 'execution' (as opposed to pure signals like TS, eSignal etc.. ).
1) Is it written entirely from scratch, or did you use a framework/API such as Quantstudio or IB's TWS to help you along? If so, which framework?
yes in C++ and linked to IB API.
IB API is not a framework, it's only a message carrier. ie it makes it possible to send a request to buy, sell, etc.. (as well as getting market data).
2) What programming language did you use? Is it an interpreted language like perl or a compiled language like C?
C++
actually they are 2 parts :
one is in WealthLab for the signals.
and one in C++ for the execution (WL is not used to send the orders at all ).
I Like WL because it's basically like Pascal and you can generate output files very easily as well as coding pretty complex systems and backtest them.
WL is not as good at realtime and actual order handling.
the C++ program can also be used standalone and used to trade futures.
3) What OS(es) is your platform based on, and is your software multithreaded or multiprocessor-capable?
windows. multithreaded but not needed (specially with IB).
4) Are you using grid computing or a server farm for the heavy lifting?
nope.
I see others are doing very complicated stuff.
in my case, simple is good. I love simple. no need for high computation.
5) Can your software platform handle lots of different trading systems/indicators, or does it have to be rewritten/modified to handle new trading systems/indicators?
very flexible.
copy / paste, modify things for the next system. roll out and repeat.
testing a new idea usually does not take more than an hour.
when a system is promising though, work can last weeks, if not months.
6) How fast is your homegrown trading software? (A highly subjective question, I know, so please pick a reference point to give us a comparison.)
again, in my case, speed is no issue.
you have to be careful when your trading is solely dependent on speed imo.
cause, someone will manage to be faster. speed as edge is troublesome imo.
also, you can have the fastest computer in the universe, your internet connection will be the bottleneck both for data and execution.
granted, you can overcome this by moving your trading farm to your broker. and then again, I know some friends who need to push this very far and make sure the link with the broker API is super fast AND the data is super fast and the computer is super fast. any part slower and the entire operation is slower.
7) How long did it take to *initially* develop the first acceptably-working version of your software?
one month I guess. although it's very different from what I have now.
fine tuning takes forever. not mandatory, but important.
8) Optional question: what is the estimated cost of developing your homegrown trading software?
I used to be software architect. I am glad I don't have to contract myself, it would cost an arm. not for the original, first working software. it's the maintenance and constant enhancements which makes it very expensive.
9) Optional question: did you reverse-engineer anything in order to develop your software?
reverse engineered market makers.
10) Optional question: is your software good enough to be released to the public and sold as a competitor to today's trading software platforms?
custom solutions like mine (and probably most) are so specific they can't be sold.
traders are interested, but people with capital are not. they can hire people to build for them, or even simply prefer to allocate to funds and have nothing to do with this plumbing.
don't believe the others though. anyone here with a working system is not rich enough that he can't be bought. only this week 2 on ET were hired.
everyone has a price. and it's fine.
I wanted to add something.
I used to think that trading is a 'problem to solve'. so I came from my engineering background sure that I could build trading machines.
wrong.
I was able to build profitable trading systems only when I learned how to trade without an automated system. manual, discretionnary trading first.
maybe some can come from outside, unable to trade on their own, and research and find an edge and be profitable.
I did it the other way around.
I learned how to trade (after trying system building without success) one trade at a time, one lot at a time, tape reading and all.
It is much easier to build systems when you can trade, the kind of ideas you get are not the same.
again, maybe others approached that only statistically and made, and now I can see how.
and then, when you can build your own trading machines, you start to trade ideas, concepts that are very different from what you would trade manually (although the characteristics of the systems probably suit your psyche).
tntneo
available software can be easily tricked by the market (block trades, bid/ask issues), it's convenient to have software dedicated to 'execution' (as opposed to pure signals like TS, eSignal etc.. ).
1) Is it written entirely from scratch, or did you use a framework/API such as Quantstudio or IB's TWS to help you along? If so, which framework?
yes in C++ and linked to IB API.
IB API is not a framework, it's only a message carrier. ie it makes it possible to send a request to buy, sell, etc.. (as well as getting market data).
2) What programming language did you use? Is it an interpreted language like perl or a compiled language like C?
C++
actually they are 2 parts :
one is in WealthLab for the signals.
and one in C++ for the execution (WL is not used to send the orders at all ).
I Like WL because it's basically like Pascal and you can generate output files very easily as well as coding pretty complex systems and backtest them.
WL is not as good at realtime and actual order handling.
the C++ program can also be used standalone and used to trade futures.
3) What OS(es) is your platform based on, and is your software multithreaded or multiprocessor-capable?
windows. multithreaded but not needed (specially with IB).
4) Are you using grid computing or a server farm for the heavy lifting?
nope.
I see others are doing very complicated stuff.
in my case, simple is good. I love simple. no need for high computation.
5) Can your software platform handle lots of different trading systems/indicators, or does it have to be rewritten/modified to handle new trading systems/indicators?
very flexible.
copy / paste, modify things for the next system. roll out and repeat.
testing a new idea usually does not take more than an hour.
when a system is promising though, work can last weeks, if not months.
6) How fast is your homegrown trading software? (A highly subjective question, I know, so please pick a reference point to give us a comparison.)
again, in my case, speed is no issue.
you have to be careful when your trading is solely dependent on speed imo.
cause, someone will manage to be faster. speed as edge is troublesome imo.
also, you can have the fastest computer in the universe, your internet connection will be the bottleneck both for data and execution.
granted, you can overcome this by moving your trading farm to your broker. and then again, I know some friends who need to push this very far and make sure the link with the broker API is super fast AND the data is super fast and the computer is super fast. any part slower and the entire operation is slower.
7) How long did it take to *initially* develop the first acceptably-working version of your software?
one month I guess. although it's very different from what I have now.
fine tuning takes forever. not mandatory, but important.
8) Optional question: what is the estimated cost of developing your homegrown trading software?
I used to be software architect. I am glad I don't have to contract myself, it would cost an arm. not for the original, first working software. it's the maintenance and constant enhancements which makes it very expensive.
9) Optional question: did you reverse-engineer anything in order to develop your software?
reverse engineered market makers.
10) Optional question: is your software good enough to be released to the public and sold as a competitor to today's trading software platforms?
custom solutions like mine (and probably most) are so specific they can't be sold.
traders are interested, but people with capital are not. they can hire people to build for them, or even simply prefer to allocate to funds and have nothing to do with this plumbing.
don't believe the others though. anyone here with a working system is not rich enough that he can't be bought. only this week 2 on ET were hired.
everyone has a price. and it's fine.
I wanted to add something.
I used to think that trading is a 'problem to solve'. so I came from my engineering background sure that I could build trading machines.
wrong.
I was able to build profitable trading systems only when I learned how to trade without an automated system. manual, discretionnary trading first.
maybe some can come from outside, unable to trade on their own, and research and find an edge and be profitable.
I did it the other way around.
I learned how to trade (after trying system building without success) one trade at a time, one lot at a time, tape reading and all.
It is much easier to build systems when you can trade, the kind of ideas you get are not the same.
again, maybe others approached that only statistically and made, and now I can see how.
and then, when you can build your own trading machines, you start to trade ideas, concepts that are very different from what you would trade manually (although the characteristics of the systems probably suit your psyche).
tntneo
0) I wanted a "swiss-army-knife" of trading platforms, as well as advanced features like neural-networks of many kinds, and other methods for pattern recognition, and then finally a method for making simple rules-based "reasoning". The reason for this was partly curiosity, and then a wish for automated trading and a very specialized and flexible tape-reading environment. However, I probably still use a lot of time on program enhancements, rather than exploring scripted indicators or models. Hey, I got to trade too, and I started this software not-so-long-ago.
http://www.elitetrader.com/vb/showthread.php?s=&threadid=30307
1) It's written from scratch, first version in a scripting shell. Then a "proper version" with compiler. I used a lot of free and open APIs - especially for the AI-stuff. Then I also included hooks to MatLab through another free API. Therein I have access to many pre-made methods for analysis and visualization. Some of the AI stuff is Joone.org, Hidden-Markov-models, Bayesian networks, Prolog-style-rule-set and more. MatLab also allows for even more.
2) Java - which is "interpreted, but with Just-In-Time compiler" for last-minute optimized native-platform execution. The first version (prototype) was coded using BeanShell which is "instant Java" without compiling, just goofing around, and with most Java 1.5 features already available.
3) Most OSes. But I use IB TWS API, Quote.com and a forex supplier for data. Multi-threaded for independent data-feeds, data-scripts and updates of graphical components.
4) No, although several of the APIs support this. I mainly concentrate on short-term, even for pattern recognition. Markets change fast. The software perhaps has too many threads right now, but something like Linda-tuples could be implemented if the need for a server farm came up. Otherwise the current computing grid frameworks are very fresh, perhaps save the IBM commercial grid.
5) Yup, although it's very work-intensive to go through the AI-training cycles. For normal TA-indicators, they're all available through the ~274KB BeanShell scripting extensions API, meaning that new indicators can be scripted/loaded/saved dynamically while running the software. All AI-models are also loaded through the scripting part.
6) Fast enough for me. It runs several threads. Although, I did a lot of tweaking on the GUI-side to get very fast graphics moving in Java. Took a lot of hints from gaming-optimizations, especially J2ME techniques (mobile phone Java). Back-testing systems can still become faster, but the practicality of using enormous heaps of memory also needs to be considered.
Many are biased towards speed of Java, but the fact is that it's quite slow on floating-point maths because of emulation in some cases, mostly the JIT-compiler takes care of that. Remember, huge list handling is fastest in functional languages like Lisp. All languages have perks. Java is the most modern, full-fledged and run-time secure language around. It's hard to do obscure mistakes, unless you're doing unsafe thread-mixing.
C++ would be an option if there were better standardized sockets-handling - which they're getting to with the BOOST.org initiative, but Java is the most complete environment for now.
7) Prototype: ca 1-2 days. Proper verison: ca 1 month. Enhancements: forever. I've been programming since the age of 10 since Basic 2.0 and MC 6502/6510, and also founded 3 computer consultant companies - mainly focused on programming.
8) In programming hours: ca 500 hours @ whatever price.
In acutal spent money: 0 (all expenses @fixed costs)
9) Java is fun, and JAD is rad!
10) Would never want to do this because of support-nightmares. Otherwise, open-source is always possible. I am thinking of another type of software which groups traders, and can "coordinate traders and trades".
PS! I first decided not to reply to the questionare because there has been a serious lack of any programming-oriented threads as of late and mostly just BS, but I was encouraged by the interesting responses.
PPS! How many others have include scripting capabilities for adding indicators etc. ? Are you accessing all internals or using a pre-defined API ( I use an API to not get too messy in my scripts) ?