OCaml.

[TigerBalm]

Some clarification is in order : "algo trading" is clearly an overused term. The 100K to 400K LOC servers I talked about were from my line of work : sell-side broker algo trading systems that do order management and DSA-style strategies a la http://www.ibb.ubs.com/marketing_campaigns/dsa/strategy.shtml. As an example of order management, think of what runs behind the scenes at IB's servers connecting to multiple exchanges, combining market depth to give you a consolidated order book / NBBO in real-time, executing your orders, etc;

I was not referring to the kind of strategy that is often discussed on ET (opening range breakout, stat arb etc.

This is my day job, and the technology really matters in controlling complexity, reducing bugs, and scaling up without a performance hit.

Apologies for not clarifying the context!

 

[TSGannGalt]

Quote from TigerBalm:

Some clarification is in order : "algo trading" is clearly an overused term. The 100K to 400K LOC servers I talked about were from my line of work : sell-side broker algo trading systems that do order management and DSA-style strategies a la http://www.ibb.ubs.com/marketing_campaigns/dsa/strategy.shtml. As an example of order management, think of what runs behind the scenes at IB's servers connecting to multiple exchanges, combining market depth to give you a consolidated order book / NBBO in real-time, executing your orders, etc;

I was not referring to the kind of strategy that is often discussed on ET (opening range breakout, stat arb etc.

This is my day job, and the technology really matters in controlling complexity, reducing bugs, and scaling up without a performance hit.

Apologies for not clarifying the context!

TigerBalm,

No worries. I understood it.

I used to trade in-house with LEH for a short time and have a 1.5st hand knowledge of what's going on behind the scenes at a sell-side firm. Your "day-job" inputs are very valuable.

Thank you for your post.

 

[comintel]

Quote from TigerBalm:

The big plus about Java and the .NET languages is the huge ready-to-use libraries behind the language. Though the Ocaml community is trying to address that concern (http://batteries.forge.ocamlcore.org/), I believe that it will take time before they catch up
a) with the sheer amount of stuff available on .NET
b) the sheer amount of people who do .NET / Java

If I came from a .NET background, I would use F# while learning functional programming concepts from the Haskell and Ocaml folks from discussion lists.

I am from a Java background, and I would migrate towards Clojure once I am satisfied the the FP paradigm is good stuff (Clojure goodness includes macros, STM, functional programming). I am learning about Monads from the very helpful and lucidly verbose Haskell folks, and once it looks like I have understood the core concepts well, I would translate to Clojure.

I approach the whole FP paradigm with cautious optimism. I know exactly what I am looking for but do not get enough time to evaluate it thoroughly:
- does the functional paradigm (Monadic / Arrow way of composing programs) help conquer complexity in the large? (a typical algo trading server, for example, is ~150000 to 400000 lines of code)
- without becoming an incomprehensible and unmaintainable mess? are programs easier to read? I'm not interested in performance at the cost of comprehension, especially in a world where many performance problems can be solved by throwing better hardware at them
- does QuickCheck work on Monadic programs, and can help automate QA for such large programs?

The issue I see with Clojure is that I have heard it said that the JVM is not well-suited to parallelism, so that you would not see the performance gains in a many-core environment that you can get with other functional environments.

 

[HockeyPlayer]

I do not have experience with Clojure, but it runs on a standard Java JVM, which typically handle concurrency very well. I write multi-threaded Java middleware professionally, and the concurrent performance that a modern JVM can provide is exceptional.

Quote from comintel:

The issue I see with Clojure is that I have heard it said that the JVM is not well-suited to parallelism, so that you would not see the performance gains in a many-core environment that you can get with other functional environments.

 

[comintel]

Quote from HockeyPlayer:

I do not have experience with Clojure, but it runs on a standard Java JVM, which typically handle concurrency very well. I write multi-threaded Java middleware professionally, and the concurrent performance that a modern JVM can provide is exceptional.

See
http://java.dzone.com/news/javaone-brian-goetz-concurrenc


"The tools we have in Java 5 are good enough to find coarse-grained parallelism (usually at the unit of a user request) and spread it over a small number of cores (2-8). However, these tools do not scale up to many-core boxes, which will become increasingly prevalent. The shared queues and other infrastructure used by executors and thread pools becomes a point of contention and reduce scalability."

Also it requires very skilled programmers to write thread-safe code in Java without horrible bugs and bottlenecks.

Most of these problems go away in functional languages that inherently make no use of shared (global) memory.

 

[HockeyPlayer]

Brian Goetz is great, his concurrency book is a must read. But I think he is arguing that the current libraries in Java don't scale well to multiple cores, which is why we need fork-join. The JVM itself scales pretty well to many cores, and a tool like Clojure would presumably be able to utilize the JVM as well as the fork-join framework.

Quote from comintel:

See
http://java.dzone.com/news/javaone-brian-goetz-concurrenc


"The tools we have in Java 5 are good enough to find coarse-grained parallelism (usually at the unit of a user request) and spread it over a small number of cores (2-8). However, these tools do not scale up to many-core boxes, which will become increasingly prevalent. The shared queues and other infrastructure used by executors and thread pools becomes a point of contention and reduce scalability."

Also it requires very skilled programmers to write thread-safe code in Java without horrible bugs and bottlenecks.

Most of these problems go away in functional languages that inherently make no use of shared (global) memory.

 

[comintel]

Quote from HockeyPlayer:

Brian Goetz is great, his concurrency book is a must read. But I think he is arguing that the current libraries in Java don't scale well to multiple cores, which is why we need fork-join. The JVM itself scales pretty well to many cores, and a tool like Clojure would presumably be able to utilize the JVM as well as the fork-join framework.

A new virtual machine design (as well as language design) is needed for best multi-core performance.

See

http://geek.susanpotter.net/2009/04/scala-vs-erlang-debate-part-2-geek-off.html

"The Erlang VM was built from the bottom up with lightweight processes (a.k.a. actors), therefore it is stackless and much better at lightweight concurrency. Erlang also has a built-in mechanism for load balancing these lightweight processes across all available CPU cores. This means Erlang has a massive advantage over JVM based functional programming languages at getting the most out of multi-core hardware, which is at the crux of the problem domain that software needs to address going forward.

Scala vs. Erlang: Performance

In terms of linear performance (sequential) Scala has been shown to beat Erlang in certain cases. However, concurrent performance consistently shows Erlang beating Scala, sometimes by wide margins on multi-core hardware. Not surprising considering what we learned in the concurrency sections above."

Benchmarks are showing functional Erlang trouncing JVM-based Scala in multi-core environments.

Now, I need to add that OCaml is not good at multi-core yet, but will be once the new parallel garbage collection is released.

Erlang is already there.

JVM-based Scala is not. Nor is Java itself. I think Clojure would be the same.

Here is another article:
"Multi-core may be bad for Java"

http://www.devwebsphere.com/devwebsphere/2006/11/multicore_may_b.html

 

[lolatency]

Quote from trader3cnd:

No offence you techies but if you cannot program a trading system in excel or basic and make money none of these fancy languages will help you.

Technology is a trap. Simple ideas make money. I know people who still take positions in commodity futures with quotes off daily newspapers and make tons of money. I know many others with 10 computers, multiple screens, and programming skills who constantly lose.

Oh yeah, 32-core computers for faster stupidity are here. That will help geeks to faster optimize their systems and send faster their losing trades to hungry MMs.

Give me a break...

<i>No offence you commuters but if you cannot ride a horse none of these fancy cars help you.

Technology is a trap. Simple ideas make money. I know people who still raise horses. I know many others with 10 cars who constantly commute poorly.

Oh yeah, V8 engine cars transit for faster stupidity are here. That will help geeks to drive faster and get more tickets.</i>


... We've heard it before.

 

[lolatency]

Quote from HockeyPlayer:

Brian Goetz is great, his concurrency book is a must read. But I think he is arguing that the current libraries in Java don't scale well to multiple cores, which is why we need fork-join. The JVM itself scales pretty well to many cores, and a tool like Clojure would presumably be able to utilize the JVM as well as the fork-join framework.

Fork-Join is idiotic. The whole point of multiple threads is to make use of the fact that you have the same virtual address space being utilized by different threads of execution. You either need this for performance or you don't, and you can use it correctly or you can't.

Look at the i7 and Intel's quickpath architecture. Why would these people go through all this trouble to bring you integrated memory controllers and point-to-point connections if you're going to create disjoint address spaces? Even with SHMs, you are still looking at larger context switch penalties.

What fanciful world are you people living in where these low-level details don't matter?