Complex Event Processing/Event Stream Processing

[zzt]

If anyone has experience with Complex Event Processing(CEP)/Event Stream Processing(ESP), I would appreciate your input.

My question(s) is/are very general.

With regards to processing speed and scalability,what are the advantages of CEP/ESP engines vs dedicated C++ algorithms which process streaming data?

There is an awful lot of hype around the subject space of CEP/ESP, but very little data in terms of comparing it to dedicated event based algo's which process streaming price data.

For example, say i have a dedicated arb model, which subscribes to real time streaming data, builds its own time series arrays, processes the tick events and sends buy/sell/amend/cancel orders. In effect my data query is static, (i.e. same datafeed subscriptions, same algo) so i dont have need for a streaming data query language.

Where is the advantage of a CEP engine? Where would it fit in the data processing chain so that it would give an advantage? Would there be saleability advantages if I were running 15 arb models in parallel?

To my mind, any generic solution that attempts to solve all streaming data issues in one app. must have trade offs in terms of either data processing or speed vs dedicated data processing algorithms?

Thanks in Advance

 

[rosy2]

the advantage is the query language in esp software. you can create a new algorithm in a minute. I use esper which is a java api.

 

[zzt]

Thanks for the response rosy, but what i was getting at was i dont need that query language when i already know how i want to process the data in a real time environment. Or am i underestimating the power of the queries in these CEP engines?
I still cant see how a a sql based query language will give me the real time number crunching power of a C++ in memory process for example. I guess it would be much quicker to work with the data however.

I'm just having trouble seeing how this much hyped technology fits in to the whole process.

Thanks again

 

[psytrade]

Apama

 

[wave]

Depends on what you are trying to do. But yes, ESP/CSP seems to be the new buzz and have to have. Granted, I still use simple C to do everything ESP/CSP can do.

http://www.aleri.com/technology/complex_event_processing.html

 

[HoundDogOne]

Quote from zzt:


Would there be scaleability advantages if I were running 15 arb models in parallel?

To my mind, any generic solution that attempts to solve all streaming data issues in one app. must have trade offs in terms of either data processing or speed vs dedicated data processing algorithms?

Thanks in Advance

You are right.
Generic apps can be MUCH slower than custom.

But the FUTURE is "grid processing"...
Stringing together 10 or 100 generic boxes...
Into one virtual "supercomputer"...
Making processing bottlenecks obsolete.

The $1,000 product is Digipede.

http://www.digipede.net

The Open Source product is Alchemi.

http://www.alchemi.net

I'm going to implement one of these soon.

 

[spreadem]

Try Apama, http://www.progress.com/apama/index.ssp

 

[HoundDogOne]

Algorithmic Trading = "The New Technical Analysis"

New century... same scam.

 

[zzt]

HoundDogOne. Thanks for the links I will check it out.

I agree with you.... Lots of slick salesmen and no real info.
I went to a presentation on algo offerings at a major investment bank a few weeks back and the execution trader says something along the lines of "...well first we pull up the chart to see which way the stocks going, and then decide which algo to use..." we left soon after. Your right man. Same old sh..t!

 

[rosy2]

Quote from zzt:

Thanks for the response rosy, but what i was getting at was i dont need that query language when i already know how i want to process the data in a real time environment. Or am i underestimating the power of the queries in these CEP engines?
I still cant see how a a sql based query language will give me the real time number crunching power of a C++ in memory process for example. I guess it would be much quicker to work with the data however.

I'm just having trouble seeing how this much hyped technology fits in to the whole process.

Thanks again

other than the query language then I cannot see an advantage. btw the data streams through the queries so it is realtime. give an example of your algorithm and I would think it can be done in a cep environment.