Creating own trading system

[Nab]

Quote from gmst:

Have you coded up your own trading software? If yes, then I would like to ask you following:-

I am curious how long in your experience does it take to code up a basic trading engine sufficient for many purposes? Please also write the specific capabilities that such a basic trading engine would have, so that it allows for a proper mapping with the time required to code this system up. Thank You for your response.

Yes, I developed from the scratch my own system.
(In Java. But please no discussions about all this no-knowledge BS like GC etc. . With the right JRE it *is* even used for real-time military grade applications like radars, avionics etc. ... ).

I don't want to start discussing elementary technical details, since there are already a lot of threads on this board about that ... But let me say what I mean under basic:

1. Your data frequency is low enough for standard db storage
(I don't want to start this discussion, but for most retail brokers it does not make much sense to go lower than 1s resolution as the execution anyway lags roughly 1/2-x secs)

2. Your system is *not* able to automatically solve eventual problems/discrepancies (for example a disconnect at an unfortunate moment) but rather requires user intervention (therefore just notifies an admin via mail/sms or whatever)

3. No GUI BS, but just a socket based command & control console

Such a basic engine is simple and does not take long ... but number of hours strongly depend on the skill/talent/knowledge of the programmer. A good one can do this in less than a week ...

 

[gmst]

Quote from Nab:

Yes, I developed from the scratch my own system.
(In Java. But please no discussions about all this no-knowledge BS like GC etc. . With the right JRE it *is* even used for real-time military grade applications like radars, avionics etc. ... ).

I don't want to start discussing elementary technical details, since there are already a lot of threads on this board about that ... But let me say what I mean under basic:

1. Your data frequency is low enough for standard db storage
(I don't want to start this discussion, but for most retail brokers it does not make much sense to go lower than 1s resolution as the execution anyway lags roughly 1/2-x secs)

2. Your system is *not* able to automatically solve eventual problems/discrepancies (for example a disconnect at an unfortunate moment) but rather requires user intervention (therefore just notifies an admin via mail/sms or whatever)

3. No GUI BS, but just a socket based command & control console

Such a basic engine is simple and does not take long ... but number of hours strongly depend on the skill/talent/knowledge of the programmer. A good one can do this in less than a week ...

First, thanks for a useful post. This thread has been going astray.
Agree with all the 3 points above. I am surprised you mention a week, I would have thought at least couple of months to develop a basic backtesting and execution without GUI framework. But then my IT skills are fairly limited.

Some specific questions regarding your setup:
What DB do you use? Do you store 1 second resolution data in that DB? For how many markets do you store data for? Do you store market depth? What specific features have you implemented that are beyond what Multicharts or Ninja has?

I am a long-term MC user but my biggest gripe with MC is that I have outgrown their capability. So, I need to start looking into enhancing or writing a program suited to my needs.

 

[Nab]

Quote from gmst:


What DB do you use? Do you store 1 second resolution data in that DB? For how many markets do you store data for? Do you store market depth? What specific features have you implemented that are beyond what Multicharts or Ninja has?

Actually, for my purposes it is sufficient to store long-term only 1 min resolution (~150 contracts) and keep everything else in memory. A simple mysql db is sufficient for that.

As I said before, the trading engine and backtesting are two separate things. Using for the latter something like R or Mathematica is far more time efficient and sufficient for many purposes. Also, coding the strategy once in a high-level framework and then again in a lower-level framework in order to implement in the trading engine is quite useful to uncover stupid mistakes which screwed up the previous backtests ...

For instance, one of the things I couldn't do with off the shell products, as far as I know, are interactions between different strategies like internalization of orders and risk distribution.

 

[gmst]

Quote from Nab:

Actually, for my purposes it is sufficient to store long-term only 1 min resolution (~150 contracts) and keep everything else in memory. A simple mysql db is sufficient for that.

As I said before, the trading engine and backtesting are two separate things. Using for the latter something like R or Mathematica is far more time efficient and sufficient for many purposes. Also, coding the strategy once in a high-level framework and then again in a lower-level framework in order to implement in the trading engine is quite useful to uncover stupid mistakes which screwed up the previous backtests ...

For instance, one of the things I couldn't do with off the shell products, as far as I know, are interactions between different strategies like internalization of orders and risk distribution.

Thanks Nab for sharing your experience. I need to expand my backtesting and execution framework. This is what I am thinking as a start for streamlining my systematic trading framework.

For backtesting, I will continue to use Multicharts and MATLAB. For auto-execution, portfolio construction and global risk configuration, I am going to construct a prototype in excel+vba, with 3 months of data loaded in vba from db. I will probably start with setting up a 1 minute database for 10 futures contracts. I might use sqlserver express edition or Access for that matter - again for a start and link this db to excel.

After loading last 3 months data from the database into vba, I will stream the real time 1 minute data into excel. Next step would be to code all my MC strategies into VBA. Finally, construct the trading strategies portfolio and then execution on IB using this excel+vba interface.

Sounds like old school, but should get the job done for my needs. Your comments are welcome.

 

[hftvol]

Amibroker's symbol database and data storage facility is very clumsy to deal with. No rolling of futures contracts, no detailed exchange settings for margin nor anything related to how to maintain futures and options contracts. So, let me ask you if you need all those interfaces why not directly coding something on your own in one language that you yourself understand from the ground up and can extend at your own liberty. You are the one who does not know what he is talking about. I am well aware of the interfaces others wrote to extend Amibroker. It does not change the fact of the shortcomings above and also that its management of broker and data connectivity is very badly integrated. It suffices for keeping the app alive for a day or two but it cannot handle missing fills, erroneous executions, which you cannot avoid from time to time, very well at all. Let's stick to facts not fiction.

Quote from GeorgeClueney:

If you don't like a software's own language then there is their free AmiBroker C/C++ SDK available for download! If one doesn't like C/C++ there is another AmiBroker SDK for .NET.

BTW you clearly have no knowledge about the software so it's useless talking to a "knowitall" anyway. If you know better go to their Yahoo forum and try out your expertise there. ;-)

 

[hftvol]

if someone is not professional enough to thoroughly learn a programming language, if one is not willing to seriously invest in this endeavor then maybe this is the wrong professional for such person. Fact remains that you cannot make serious money with a half-baked platform that limits you at thousand different ends.

As NetTecture correctly pointed out there are retail platforms out there that are so badly coded up that they indeed do result in different backtest results over the same data sets, even without any errors on the user's end. Come on, you cannot deny the work that someone like him has put into for a year or so. Its an affront. I have spoken to him and he comes across as honest and knows what he is talking about. And I believe I also know what I am talking about. What ax I have to grind with retail platforms? I have nothing to sell to anyone here, nor do I derive benefit from belittling retail packages. I merely try to do newbies a service by not leading them into false sales promises by those retail vendors. They got the basics right (well some not even) but to make money in this competitive market you need to get more than just the basics right.

Quote from dom993:

Hftvol, you have very good arguments in favor of professional-grade platforms, however, you are missing the point ... most ETers are not professional, merely trying to get their feet wet, discover and learn, and there is no point for that purpose to spend the kind of money required by a professional platform, and corresponding historical data.

I will contend that most of the "random" results arising in backtesting with retail-grade platforms are self-inflicted, meaning result from programming errors in the strategy, rather than platform instability, and people would get same random results on a professional-grade platform.

There should be a lot less bugs & much better turnaround in fixing them with professional-grade platforms, than with retail ones. Certainly Ninja doesn't fare well at all on that front, and needs at times quite a bit of sofware development to workaround their bugs, and for some other things just needs to be carefully used so to stay away from those bugs.


J2ee, you have to be blind not to realize that NinjaScripts are plain C#, and that Microsoft .NET software components are available to be used in those "scripts". I know someone will come & say Ninja doesn't support .NET past 2.0, this I don't know, and to be honest I have not encountered any situation where it would make a difference for me.

 

[hftvol]

I never claimed so. Amibroker is maybe the best out of the whole pack of retail platforms and the most reasonably priced one for beginners but that is it. It also is not primed for serious systematic trading. If you cannot read between the lines and get when someone is being ironic then that is not my fault.

Quote from j2ee:

You explain why you think creating our own system is the best solution over here but you didn't explain why you think amibroker is a good choice in your opinion.

 

[hftvol]

See this is where you really do not understand the picture, so it seems: This industry, this market is maybe one of the hardest industries in the world to make money. If you are after easy money you are better off in tons of other industries. This is a highly competitive industry where hedge funds and banks plank down hundreds of millions of dollars to get things to work the way they want it to work. This system is not designed for an average "normal" programmer to succeed. Dude, I have worked for over 10 years as professional exotic options market maker (caps, floors, swaptions, and FI structured notes), traded proprietary books on the options and cash equity, and futures side, have traded for over 4 years exclusively foreign exchange products, have known C++ since university, deem myself an advanced C# programmer with intricate knowledge of the CLR and .Net platform and I myself need to work hard every single day, sit for hours in front of the screen, need to constantly re-innovate for what? To STAY COMPETITIVE. You nor anyone else makes money with pipe dreams. If you think you can beat me, or anyone else who spends much more than I could ever imagine to extract alpha in this industry, with a 500 dollar platform or with a little programming here and there then you are absurd. This is not career for a "normal" programmer. Sorry to deliver you this shocking news. If you want to succeed then you gotta work your ass off and work harder than you ever have. The rewards are there but they are not laying on the ground for lazy people to pick up. Period.

Quote from j2ee:

ok so you agree AFL is limited too.

I agree building own system is the best way but only if someone knows how to build a system very well. That is not even a normal programmer can do.

 

[hftvol]

good you mention Deltix. I wanted to list it earlier but the name did not come to mind. I actually talked to those guys because I think they have a very well made platform, I almost decided to go with them years ago rather than writing my own but in the end they could not deliver what I was after. And they are slightly overpriced in regards to their deliverables (their full platform starts at around a 100k USD, which is ok if the product supports it but I did not get the feel it did. Similarly for KDB/Kx, they charge an arm and a leg for their database and while I think its top notch I do not believe it should be priced at such ridiculous levels. IT departments with heads who do not have to pay out of their own pockets do not care as long as they have a juicy budget in good times, all they care about is their next promotion by having delivered a high performance database that trading desks and quants can access. Deltix is a good product I like it very much though I have not followed up with them in almost a year.

Quote from j2ee:

For your own system, what language do you choose and why? Why don't you use something like Deltix?

 

[hftvol]

makes me laugh, a good programmer can create a workable trading engine in less than a week? Yes I can also create a facebook like page in less than a week. I guess we have very different definitions regarding "workable" ;-)

Quote from Nab:

Yes, I developed from the scratch my own system.
(In Java. But please no discussions about all this no-knowledge BS like GC etc. . With the right JRE it *is* even used for real-time military grade applications like radars, avionics etc. ... ).

I don't want to start discussing elementary technical details, since there are already a lot of threads on this board about that ... But let me say what I mean under basic:

1. Your data frequency is low enough for standard db storage
(I don't want to start this discussion, but for most retail brokers it does not make much sense to go lower than 1s resolution as the execution anyway lags roughly 1/2-x secs)

2. Your system is *not* able to automatically solve eventual problems/discrepancies (for example a disconnect at an unfortunate moment) but rather requires user intervention (therefore just notifies an admin via mail/sms or whatever)

3. No GUI BS, but just a socket based command & control console

Such a basic engine is simple and does not take long ... but number of hours strongly depend on the skill/talent/knowledge of the programmer. A good one can do this in less than a week ...