Complex Event Processing/Event Stream Processing

[THE-BEAKER]

Quote from TraderMojo:

Unfortunately, your analogy is fundamentally flawed.

You seem to be mistakenly equating ESP/CEP product vendors with trading system vendors. Perhaps you are not aware that ESP/CEP products are just software components - not trading systems.

ESP/CEP products are just tools - one of many tools available to a software architect to build a particular solution. They have applications in many areas outside of trading.

Vendors of these products are just like vendors of other software products such as Microsoft. They sell their wares to consumers who might have a use for them. For example, Microsoft sells Office to consumers that need office productivity software.

You could ask, why does Microsoft sell Excel to banks when they could use Excel to make their own trading systems to make billions?

The point is that Excel is just a tool/technology that different people can put to use in different ways. Excel is used in may areas outside of trading. If you have used Excel then this should be clear to you.

Similarly, ESP/CEP products are just a tool/technology that different people can put to use in different ways.

Trading systems could not be built without making use of other software components or libraries. Using ESP/CEP in a trading system is just using another software component to build a system.

I hope I have cleared up the confusion. The comparison between trading system vendors and software component vendors is clearly nonsensical in this context.

point taken and understood

 

[onelot]

Quote from snaggydog:

It's obvious you are biased and you think CEP is only "in-memory database processing" as you posted earlier.

You actually have very little knowledge of what you are talking about and if you read the links with an open mind, and intellect engaged, you would quickly see that CEP is not simply "in-memory database processing".

PS: I have a feeling you really want me to teach you, but you are either too biased or too proud to ask.

OTOH, it is not my "job" to do your reading and homework for you, but it you want me to spoon feed it to you, please ask nicely and I will teach you. But, you should do your reading and homework first, and be able to answer basic questions about CEP, ESP, event-processing, EDA, messaging, CCL, rules engines, Bayesian and neural analytics, Rete, and a whole lot more.

Your knowledge is good, but only scratches the surface of CEP, ESP and event processing.

"CEP is a method of aggregating large amounts of information in a database that uses Event Query Language (EQL) to sift through the data for meaningful information."

This is a quote from David Luckham, he was of the originating researchers of CEP at Stanford and was cofounders of Rapide, the spinoff company of that research.

I'm not biased. I've done my homework. My conclusions match those who pioneered the field. If you want to contune to argue with someone, argue with him. email: dcl at anna dot stanford dot edu.

http://www.rfidjournal.com/article/articleview/1196/1/82/definitions_off

 

[snaggydog]

Quote from onelot:

I'm not biased. I've done my homework. My conclusions match those who pioneered the field. If you want to contune to argue with someone, argue with him. email: dcl at anna dot stanford dot edu.

http://www.rfidjournal.com/article/articleview/1196/1/82/definitions_off [/B]

I know David Luckham very well. The notion in the marketing article you mention does not represent David's technical views. David's views are represented here:

http://complexevents.com/?p=103

Where is says, in his direct quote:

What problem was CEP designed to solve? CEP was developed between 1989 and 1995 to analyze event-driven simulations of distributed system architectures.

If you had done your homework you would not be picking out a quote from a collaboration with a marketing guy from ESP vendor Apama.

More from the quote, which clearly shows that ESP is from the "database query" side of the house and CEP is more from real time messaging analytics:

While CEP was being developed there was a parallel research effort going on in real-time event data analysis. This started in the mid-1990s when the database community realized that databases were too slow to do real-time data analysis. They started researching the idea of running continuous queries on streams of incoming data. They used sliding time windows to speed up the queries. An answer to a query would be valid only over the events in the current time window, but as the window slid forward with time so also the answer was updated to include the new events and exclude the old ones. This research was called Data Streams Management (DSM) and led to the event streams processing world of today. The emphasis was on processing the data in lots of events in real-time.

Have you read Dr. Luckhams original paper on CEP? Would you like a copy?

 

[snaggydog]

Have you read Dr. Luckhams original paper on CEP? Would you like a copy?

Attached (hope it works!) is a copy of David Luckham's original paper on CEP.

You will note the opening quote in the original paper:

"Complex event processing is a new technology for extracting information from message-based systems." - David Luckham and Brian Frasca

Also, there is a guy who has put together a survey of papers that quote the original paper. You can see easily that CEP is not a database related technology from the referencing research papers.

REF: http://www.timbass.info/index.php?title=CEPinDS

Have you reviewed the CEP literature, or simply read a few marketing releases? The literature is clear on CEP. The database, data streaming vendors has created the confusion, which, unfortunately, has confused many in the market!

Cheers!

 

[QuantPlus]

Quote from THE-BEAKER:


the fact is that there are a very few , very complex strategies running that are way beyond what some so called programmers on this thread are doing.

these kind of systems are not for sale and will be used by the people making money out of us at the moment.

as for ' apama ' and some of the other off the shelf systems my arguement is the same old one. if its so good and profitable then why do they need to sell it.

the answer is its very basic and relies on old techniques that no longer work in the new environment.

i no longer try to compete with speed, execution and black boxes.
my evolvement has been this:

im right or wrong and avoid all the noise in the middle.

as for an off the shelf, black box, white box, grey box, algo techo bullshit system forget it.

they do not exist to make money for traders only money for the people who sell it.

99% agree

Here is a man who uses experience plus logic...
To filter out the silly NOISE all around him.

One can view the "black boxes"...
As a SEPARATE Zero Sum Game played ONLY amongst themselves...
With the usual result...
10% of the boxes win all the money 90% of the boxes lose...
While having negligible impact on the rest of the market.

Back in the real world...
A lot of money is still being made...
Applying CLASSIC arb and market making techniques...
Where execution is everything... and nobody talks.

And there is nothing more pathetic...
Than grown men who believe they can buy off-the-shelf "winning systems"...
Not matter how sexy the con.

 

[onelot]

Quote from snaggydog:

I'm 100% full of shit

You don't know David. If you did, you wouldn't be writing all this bullshit. If you do know him "very well", it wouldn't take much to end this by asking him to email me from the address I gave and have him tell me why I'm wrong. PM me for my address. I'm calling 100% bullshit on this one.

Nothing you wrote, nor any of David's articles/papers disproves anything I've said. If so, please show me with concrete coding examples, or using coding terminology as I have done.

How the hell do you think a "real-time messaging analytic" engine works? I'll tell you how: in-memory db. I've given a couple different examples of how they work technically. You haven't showed shit except some links and quotes which disprove nothing. You don't know dick about programming, becuase if you did, you wouldn't be arguing.

Get over yourself.

 

[TSGannGalt]

Quote from onelot:

You don't know David. If you did, you wouldn't be writing all this bullshit. If you do know him "very well", it wouldn't take much to end this by asking him to email me from the address I gave and have him tell me why I'm wrong. PM me for my address. I'm calling 100% bullshit on this one.

Nothing you wrote, nor any of David's articles/papers disproves anything I've said. If so, please show me with concrete coding examples, or using coding terminology as I have done.

How the hell do you think a "real-time messaging analytic" engine works? I'll tell you how: in-memory db. I've given a couple different examples of how they work technically. You haven't showed shit except some links and quotes which disprove nothing. You don't know dick about programming, becuase if you did, you wouldn't be arguing.

Get over yourself.

I'VE DONE MY F*CKIN' RESEARCH, TOO!!! IN TECHNICAL TERMS, ONELOT IS RIGHT, DIPSH*TS! EVERYTIME A SOCKET RECIEVES A MESSAGE IT GOES TO THE MEMORY, D*MB ASSES!!!

WOULD I CALL IT F*CKIN' IN-MEMORY DB??? HELL NO!!!

ANYWAYS, IT'LL BE HELLA FASTER IF I JUST STICK AN G*D DAMN ANALYTIC CODE IN THE MESSAGE CRACKER!!!! THAN CEP/ESP, FOOLS!!!

... or am I still missing things on this one...

 

[onelot]

Quote from TSGannGalt:

I'VE DONE MY F*CKIN' RESEARCH, TOO!!! IN TECHNICAL TERMS, ONELOT IS RIGHT, DIPSH*TS! EVERYTIME A SOCKET RECIEVES A MESSAGE IT GOES TO THE MEMORY, D*MB ASSES!!!

WOULD I CALL IT F*CKIN' IN-MEMORY DB??? HELL NO!!!

ANYWAYS, IT'LL BE HELLA FASTER IF I JUST STICK AN G*D DAMN ANALYTIC CODE IN THE MESSAGE CRACKER!!!! THAN CEP/ESP, FOOLS!!!

... or am I still missing things on this one...

You're still missing a bit, but you get the basic gist of it: processing data.

They're not talking about solving the problem of handling singular messages. If they were, sticking code in the message parser would solve that problem though, you're right.

What they're talking about is message streams. So, an example of an ESP stream would be time and sales for a given stock. If you just had the "ANALYTIC CODE IN THE MESSAGE CRACKER" (hehe), then you'd have no access to all the other prints that came before it. ESP processes and stores all the past events for x amount of time or x amount of prints or however it's being organized. The key here, is that it's being organized and stored. If it's going to be realtime, it's going to be done in-memory. I used the term "in-memory db" to generalize the concept of using hased arrays, tuples, or trees as in-memory storage containers and I don't think that was a poor term for it.

CEP is the same thing, except instead of just one stream you're now using multiple streams. So arbing one stock against its index, you might want to calculate some sort of fair value for the stock which you would analyze against the stock stream. The FV is now itself a stream composed of multiple streams, making it a CEP stream. In-order for real-time processing of the stream, it must be orgainized the same way as the ESP stream using some sort of in-memory processor.

Does that clarify things?

 

[onelot]

btw snaggy, every paper I read that you linked to, mentions using some sort of in-memory container for the cep implementation.

the very first one mentions using tuples. if you read them, you didn't understand them.

 

[TSGannGalt]

Quote from onelot:

You're still missing a bit, but you get the basic gist of it: processing data.

They're not talking about solving the problem of handling singular messages. If they were, sticking code in the message parser would solve that problem though, you're right.

What they're talking about is message streams. So, an example of an ESP stream would be time and sales for a given stock. If you just had the "ANALYTIC CODE IN THE MESSAGE CRACKER" (hehe), then you'd have no access to all the other prints that came before it. ESP processes and stores all the past events for x amount of time or x amount of prints or however it's being organized. The key here, is that it's being organized and stored. If it's going to be realtime, it's going to be done in-memory. I used the term "in-memory db" to generalize the concept of using hased arrays, tuples, or trees as in-memory storage containers and I don't think that was a poor term for it.

CEP is the same thing, except instead of just one stream you're now using multiple streams. So arbing one stock against its index, you might want to calculate some sort of fair value for the stock which you would analyze against the stock stream. The FV is now itself a stream composed of multiple streams, making it a CEP stream. In-order for real-time processing of the stream, it must be orgainized the same way as the ESP stream using some sort of in-memory processor.

Does that clarify things?

YOU CALLIN ME A F*CKIN' CRACKA!!!

... wait... I'm not white (I keep forgetting... )

Anyways, I'll bite (agree mostly)...

Here's a very excerpt of a simple Socket (.NET) code that handles messages:

public void OnDataReceived(IAsyncResult asyn)
{
SocketPacket theSockId = (SocketPacket)asyn.AsyncState;
int iRx = theSockId.thisSocket.EndReceive(asyn);
char[] chars = new char[iRx + 1];
Decoder d = Encoding.UTF8.GetDecoder();
int charLen = d.GetChars(theSockId.dataBuffer, 0, iRx, chars, 0);
String szData = new String(chars);

someAnalysis(szData);
...
...
}

OK... the socket recieves a message, then turns it to a string. Then that data goes to another class method for whatever analysis you want to do...

That said, from my understanding, CEP/ESP is worthless. someAnalysis can be dealing with the data however it wants whether it's multiple symbols or sources...