How to build an automated system

Has anybody tried hft_boy code on page 6? That's interprocess communication implemented using shared memory. Edit: file ipc.zip. I found the following:

1. Producer must be started first, o.w. consumer gives an error: "shm_open: No such file or directory". -- I want to be able to start the processes in any order.
2. If Producer restarts, then Consumer must be restarted as well. -- That's very inconvenient, if my producer goes down then I'd have to restart Consumer as well.
3. Consumer takes 100% of CPU time of one core. -- I think that's because Consumer is polling the buffer, can be resolved using semaphores?

Someone could comment how to resolve those?

THX
 
Quote from vincegata:

Has anybody tried hft_boy code on page 6? That's interprocess communication implemented using shared memory. Edit: file ipc.zip. I found the following:

1. Producer must be started first, o.w. consumer gives an error: "shm_open: No such file or directory". -- I want to be able to start the processes in any order.
2. If Producer restarts, then Consumer must be restarted as well. -- That's very inconvenient, if my producer goes down then I'd have to restart Consumer as well.
3. Consumer takes 100% of CPU time of one core. -- I think that's because Consumer is polling the buffer, can be resolved using semaphores?

Someone could comment how to resolve those?

THX
More...

well,

1: learn programming. producer not running try reopening once per second. finished.

2: program it.

3: no, and if zou want hft then zou want to avoid those ß thez are sync points that delay you.
 
I've worked as a quant mostly on the buy-side and am currently in the process of launching a fund with a few partners. We've been working on an ATS for a few months now and I'd be happy to field general questions on this topic.

That being said I think one huge issue which I don't see any optimal solution to would be reconciling your ATS and backtesting platform. A production ATS IMHO needs to be done in a proper programming language, no scripting languages. I also think any good scientist/engineer would say that your ATS and backtesting platform should exist within the same platform however you'll run into library constraints which will inevitably increase your dev time and decrease your flexibility. Now I'm a firm believer that your ability adapt quickly key in trading.

So that being said, we're currently doing backtesting in R and we have a production ATS in Java (which also has backtesting capability) and also working on something in C++ for more latency sensitive strategies. Report generation is done in the ATS and R backtests in order to verify results.
 
Thank you both for responding.

@slickpick - so I have a producer (Data Adapter) that reads the data from the Internet over the sockets when in live mode, or it reads the data from the database / text file in the backtest mode. So I am using the same code for live and backtesting. Producer is housed in a separate process and it sends the data to the consumer (Strategy Execution System) over the named pipes (FIFO). This set up works well in live mode (I am not after HFT) but it's slow during the backtesting when I run the app through millions of records. Hence, I am looking to replace FIFO with either shared memory or using callbacks aka Observer Pattern.

What do you use for your producer-consumer to communicate? I know some people on this forum use shared memory, others use messaging such as ZeroMQ, yet others use Observer.
 
Quote from vincegata:

Has anybody tried hft_boy code on page 6? That's interprocess communication implemented using shared memory. Edit: file ipc.zip. I found the following:

1. Producer must be started first, o.w. consumer gives an error: "shm_open: No such file or directory". -- I want to be able to start the processes in any order.
2. If Producer restarts, then Consumer must be restarted as well. -- That's very inconvenient, if my producer goes down then I'd have to restart Consumer as well.
3. Consumer takes 100% of CPU time of one core. -- I think that's because Consumer is polling the buffer, can be resolved using semaphores?

Someone could comment how to resolve those?

THX
More...

Hi,
Not around much anymore and was kind of disappointed earlier by the quality of discussion so I don't really want to continue the thread. But to answer your questions

1. I did this on purpose, as it doesn't make much sense to subscribe to nonexistent messages. If you really want to, try changing the flag on ipc_consumer.cpp:23 O_RDWR to O_RDWR | O_CREAT.
2. I think I wrote it that way to avoid opening the memory and reading dirty/old data. You can try removing the call to shm_unlink on ipc_producer:26.
3. Your thinking is correct, and yes you can use semaphores stop the polling. However at this point you give up the benefit of polling and you may as well use Unix domain sockets or some such which will probably have the same performance characteristics.

Regards
 
Quote from hft_boy:

Hi,
Not around much anymore and was kind of disappointed earlier by the quality of discussion so I don't really want to continue the thread. But to answer your questions

1. I did this on purpose, as it doesn't make much sense to subscribe to nonexistent messages. If you really want to, try changing the flag on ipc_consumer.cpp:23 O_RDWR to O_RDWR | O_CREAT.
2. I think I wrote it that way to avoid opening the memory and reading dirty/old data. You can try removing the call to shm_unlink on ipc_producer:26.
3. Your thinking is correct, and yes you can use semaphores stop the polling. However at this point you give up the benefit of polling and you may as well use Unix domain sockets or some such which will probably have the same performance characteristics.

Regards
More...

-- hft_boy, thanks for coming back.

So, you think shared memory with semaphores will roughly have the same performance as tcp sockets? - That's kind of slow.
 
Quote from vincegata:

-- hft_boy, thanks for coming back.

So, you think shared memory with semaphores will roughly have the same performance as tcp sockets? - That's kind of slow.
More...

Not really. Unix domain sockets don't have the same overhead as TCP. Since AFAIK (don't quote me on this) Unix domain sockets are basically just the same thing as this shared memory + semaphores contraption, they will have about the same performance. Try out http://www.bitmover.com/lmbench/ and you can do some benchmarks yourself.

If you are not latency sensitive then there is not really any point in sweating over some microseconds here and there, and you may as well use an established architecture / API.
 
Quote from hft_boy:

Not really. Unix domain sockets don't have the same overhead as TCP. Since AFAIK (don't quote me on this) Unix domain sockets are basically just the same thing as this shared memory + semaphores contraption, they will have about the same performance. Try out http://www.bitmover.com/lmbench/ and you can do some benchmarks yourself.

If you are not latency sensitive then there is not really any point in sweating over some microseconds here and there, and you may as well use an established architecture / API.
More...

I am not latency sensitive and I am currently using FIFO which is alright for me in live mode. However I want to use the the same code for backtesting and my backtesting running through millions lines of tick records it takes forever, hence I am looking to replace FIFO. THX for your help.
 
I don't know where hft_boy wants to take re-inventing the wheel but if he's so well versed in C++ this is an utter waste of time.

Learn Multicharts, and I hate to sound like the tool of the MC Universe, but that's the way it is. You'll get more done working in Multicharts than you ever will simply designing robotic programs yourself that aren't even close to as useful to trade in real time as Multicharts would be.

It would also give time to develop, and, no, what we have here is not "development" but ATS programming and that's not the same as automated system trading.

HFT_boy, go get a free trial of Multicharts and code in easylanguage C++ since you are so familiar with it you won't save much time comparatively speaking between developing your so called "automated system" than simply using MC to develop an actual "strategy."

I think there's miscommunication about "automated system" here. HFT_boy, this is an ATS. No one would consider that an "automated system." An automated system deals in strategies and probablisitic analysis, not whether to fire FIX protocol through various data ports or through route switches, let MC do that and learn to trade.

This is too ambitious a project I see many beginners with advanced programmatic ability do. Seriously, worry about your strategy, then come back. We don't need a thread for this if Multicharts or Ninjatrader will do the exact same thing for you, and what does brokerless mean? You must always use a broker, so whatever maintenance will be required even if you are successful is moot since you'll be updating the software to work with a broker eventually so you are not prepared for this type of commitment and neither is practically anyone on this board. It's like trying to save a grand when it'll take weeks to do whatever it is here you're trying to do which seems to just gain access then have a supposed "automated system" to route orders through.

Multicharts does this already, and so going through this part is not going to help your trading, which, when everyone else sees this thread, is concerned with your strategy for trading, not what pipe you plug your software into to send orders. These are two different definitions of "automated system" and we want the strategic not programmatic explanations from you, hft_boy.
 
You must log in or register to reply here.
Back
Top