Holy Grail Trading Software

[jaypaul]

Originally posted by buzzy2
I've used Matlab intensively, it has a number of limitations for really big projects, it's not evident at first, and it takes too long to explain. The biggest limitation is that there is no data-management included. But it's very useful for "small" research projects. Don't bother about using it real-time, it's not worth the trouble, even if it may look good in some small demos.

Matlab is not good at data management. You just call external C routines or DLLs for that. Matlab is not suited for processing data in real time. You use it to backtest a solid trading system offline, then convert the core Matlab functions to C and compile that into your real-time app.

The issue to me is not data management niceties or real-time code. It's designing a profitable trading system that also meets certain statistical requirements in it's trading behavior (liquidity, drawdown risk, per-equity screening/diversification and stability of these factors over time). This is an enormous challenge. Matlab provides the most important element of all, mathematical dexterity, where I can write high-level clear concise code which is easier to debug.

Originally posted by metooxx
Not to be too disrespectful, but you have no idea what you are talking about.

You could not link in 1 data feed in 6 months, let alone the rest of the problem.

We have a graveyard out back for programmers that have worked here and told us 2 weeks and were still working on a subset of the problem 2 years later.

Be realistic.

I wrote one interface to NYSE TAQ data, and another interface to TAL data. Together they took about 20 hours of programming and have served me for several months with perhaps about 3 hours of maintenance in that time. The TAQ interface uses no 3rd party code. It interfaces to the TAQ files directly. The TAL interface also archives T&S data, merging it into the TAQ database since TAL only keeps data for 7 days. Everything is C code which runs on the command line, producing a data file that Matlab can read. All of this could be further automated in a variety of ways.

Sure, you can call this hacked. But it was quick to develop, easy to use and reliable. The only third party tools are TAL's DLLs and Matlab itself.

Jay

 

[nitro]

quote:
--------------------------------------------------------------------------------
Originally posted by nitro

If this were its only problem, then it is no big deal to get a gc (garbage collector) for C/C++. If memory allocation was inneficient, it would again be trivial to get a high speed allocator "drop in" replacement for malloc/new.
nitro
--------------------------------------------------------------------------------



IMHO this shows a lack of experience in numerical/scientific and financial programming. malloc/new are low level issues. i am more concerned about the high level stuff, how are you going to represent financial instruments, how much memory for each, how much data you keep in memory and on hard disk etc. etc.

Huh? My post was in response to the poster that said that in the C/C++ code, there were a lot of memory management bugs and that the memory management was inneficient ?

nitro

 

[buzzy2]

Originally posted by nitro


Huh? My post was in response to the poster that said that in the C/C++ code, there were a lot of memory management bugs and that the memory management was inneficient ?

nitro

just kidding nitro. my point was: it's easy to find fault in others if you don't read/listen carefully.

i was talking about data types AND memory management. the guy in questions uses apparently nice, intuitive C++ classes to store financial data, but it wastes RAM like crazy. for some trivial tasks, 1GB memory is too little for that application...
the solution: redesign everything again, rewrite everything again, since the source of all the problems reside in some base classes, and the rest of the system inherits heavily from these classes. no way of bypassing the inheritance issue because a good design would require completely different behavior from the base classes.

 

[nitro]

...i was talking about data types AND memory management. the guy in questions uses apparently nice, intuitive C++ classes to store financial data, but it wastes RAM like crazy. for some trivial tasks, 1GB memory is too little for that application...
the solution: redesign everything again, rewrite everything again, since the source of all the problems reside in some base classes, and the rest of the system inherits heavily from these classes. no way of bypassing the inheritance issue because a good design would require completely different behavior from the base classes.

In that case, he does not understand C++ very well. In either case, I doubt that he would need much redesign, as it sounds like aggregation instead of Inheritance would solve the bloat problem. This is a lot of typing, but it would be almost mindless.

nitro

 

[jaypaul]

Originally posted by ArchAngel
Bullfighter -
If it was, jaypaul could do it in a week in Matlab and sell it for a million dollars...LOL

That's a great income! At that rate, I can catch Bill Gates within my lifetime.

Originally posted by buzzy2
But it's very useful for "small" research projects. Don't bother about using it real-time, it's not worth the trouble, even if it may look good in some small demos.

What constitutes a "small" research project? I do 3-year 5000 stock analysis on an ordinary PC with 768 MB ram. I work with 5000*5000 element matrices and other hideously large data structures. Why such large data structures? Because I am looking for statistically significant short-term patterns across 3-years/5K stocks. I've run several months worth of round-the-clock simulations, several thousand distinct simulations overall. It's been very educational.

What are the goals in developing trading software? Some choose to design elegant GUIs with object-oriented real-time code to possibly sell to short-term traders. I'm more interested in trying to understand the market and make smarter trading decisions, using the richest, highest-level tools possible. I won't ever sell my systems. These two goals actually seem mutually exclusive, for obvious reasons. Would you sell a profitable system? Can any algorithmic-design GUI match the dexterity of handwritten code? Can a rich multi-variant trading system ever run real-time? Are in-depth market analyses better suited for longer-term positions?

Any ideas or criticism?

Jay

 

[nitro]

...Matlab is not good at data management. You just call external C routines or DLLs for that. Matlab is not suited for processing data in real time. You use it to backtest a solid trading system offline, then convert the core Matlab functions to C and compile that into your real-time app. ...

Yes. I like Mathematica as well, though because Matlab is closer in spirit to C/C++, the learning curve is easier for programmer types.

Using a high level tool in this manner is smart use of time.

nitro

 

[metooxx]

Originally posted by jaypaul


Be realistic.

I wrote one interface to NYSE TAQ data, and another interface to TAL data. Together they took about 20 hours of programming and have served me for several months with perhaps about 3 hours of maintenance in that time. The TAQ interface uses no 3rd party code. It interfaces to the TAQ files directly. The TAL interface also archives T&S data, merging it into the TAQ database since TAL only keeps data for 7 days. Everything is C code which runs on the command line, producing a data file that Matlab can read. All of this could be further automated in a variety of ways.

Sure, you can call this hacked. But it was quick to develop, easy to use and reliable. The only third party tools are TAL's DLLs and Matlab itself.

Jay

God bless you if you did that, but I have been doing this for a long time, with a lot of different groups and I do not believe it unless you are talking about a minor subset; last sale on say NYSE.

Quite the contray to how some of my remarks seemed; I have no problem with hacked together solutions to get there first, we do that all the time, however you still need to spend the time on the basics, data acquisition, maintaining your databases, execution capability and it takes a very long time and is never done or completely.

Your technique falls apart on multiple systems trading multiple markets or a large number of symbols.

 

[metooxx]

Originally posted by jaypaul

What constitutes a "small" research project? I do 3-year 5000 stock analysis on an ordinary PC with 768 MB ram. I work with 5000*5000 element matrices and other hideously large data structures. Why such large data structures? Because I am looking for statistically significant short-term patterns across 3-years/5K stocks. I've run several months worth of round-the-clock simulations, several thousand distinct simulations overall. It's been very educational.
Jay

That is what I was saying; a small subset of the problem.

Try 125,000 option strikes with 1 to 5 exchanges that get added/deleted to monthly, however many equity symbols there are, maybe 50,000 with X number of ECNs or regional exchanges, through in 140 plus MM quoting and look @ the bid/ask in real time and now you got the problem. Forgot the futures markets and the foreign markets.

It cannot be done quickly, cheaply or on conventional PC set-up.

But, if you got a way, PM me and we will talk.

 

[saschabr]

Hello,

i just do not think it makes sense to invent the wheel several times...

sorry if i am too direct. it is just my opinion.

maybe you (app.-developers) check out www.smartquant.com

you will find that many other brillant coders have tried
what you want to do now - and they failed before, at least found
nothing really exciting.

i knew of a software firm here in germany (forgot it's name - sorry), they employed 3 really brillant math guys who developed
a neural net stragegy game - of course without any measurable success.

or check www.susq.com (susquenna or whatever it spells),
one of the nations biggest priv. MM firms, they employ
"innovative" technologies for a long time - and have they blown
away the market - of course not - and this is like things will go on.

 

[metooxx]

You don't think Susquehanna, Timber Hill or bunch of private trading firms you probably never heard of, haven't blown away the market with black box programs? Where have you been? They invented program trading and computer directed arbitrage.