3.75million ticks per second

J

januson

Finally completed my tick compressor :)

I was interested in making it much faster than the fabled TickZoom where the inventor claimed that it could process 11 million ticks in 8 seconds on a quad core machine.

Well.. my algorithm compresses 3.75 million ticks per second in debug mode.

It compresses the ticks into 15 minutes bars.
It is developed in C# 4.0

=> 30 milliion ticks in 8 seconds :D

- and that is performed by a Dual Core:
Processor Intel(R) Core(TM) i7 CPU Q 720 @ 1.60GHz, 1600 Mhz, 2 Core(s), 4 Logical Processor(s)

Take a look at the screenshot, I actually tried to optimize to the best performing parallel processing threads, it's is around 49 :)

I will keep posting on this project.
 

Attachments

Quote from januson:

I was interested in making it much faster than the fabled TickZoom where the inventor claimed
More...

Heh... Whatever happened to that wingnut?

Well.. my algorithm compresses 3.75 million ticks per second in debug mode.
More...

Congrats.
 
by the time you GET the ticks, the big boys are already trading against them.

try working on a time machine


the_time_machine.bmp


heres my prototype
 
By a mistake I didn't mention, that it was for historical testing only :)

In real time, it's only able to process 1.5million ticks per second, well at least in theory. This theory is based at my own tick plant and tick pipeline into the tick engine.

Unfortunately I don't have access to a broker that is able to deliver real time data in such speed :mad:

Anyway... the speed is quite interesting and should give other developers a good hint about what can be accomplished by C#.

IMHO everything more than 50 million ticks per minute is more than enough for both back testing, forward testing (whatever that is ? *LOL*)) , and real time processing of tick data.
 
Define "Compress" please?

are you refering to 1.5m ticks of data hitting the socket and your app reads off the socket, decode, and process the data?

How many products/symbols is your feed? How big is one of your ticks? is it a single price change (i.e. bid price change = a tick, bidquant change = another tick, etc)?

If your canned data consist of only a single product and ticks are refer to the same price level, it can cause scewed results as you get much more cache hits than misses.
 
Um... looking around the chart, isn't the optimal number of threads around 7-8? The absolute best performance is pretty useless since after around 7-8, it's all basically just random variations?

Quote from januson:

Finally completed my tick compressor :)


Take a look at the screenshot, I actually tried to optimize to the best performing parallel processing threads, it's is around 49 :)

I will keep posting on this project.
More...
 
Quote from mickmak:

Define "Compress" please?

are you refering to 1.5m ticks of data hitting the socket and your app reads off the socket, decode, and process the data?

How many products/symbols is your feed? How big is one of your ticks? is it a single price change (i.e. bid price change = a tick, bidquant change = another tick, etc)?
More...


Hi mickmak

A compression is defined in my architecture as an aggregation over a frequency of some type. For instance, as we all know "time", a compressor then takes the frequency, i.e. 5 minutes and each bar then consists of the ticks that matches between bar start time and bar end time.

In my POC there is only one symbol, but more symbols will not affect the performance, max throughput will still be 1.5million/ second.

A tick is described as a trade, trade price can either go to the sell side og buy side.

The tick format is : datetime, price, size, please take a look at the attached tick sample.

Quote from mickmak:

If your canned data consist of only a single product and ticks are refer to the same price level, it can cause scewed results as you get much more cache hits than misses.
More...

Not sure exactly what you mean ? Please explain :)
 

Attachments

Quote from DontMissTheBus:

Um... looking around the chart, isn't the optimal number of threads around 7-8? The absolute best performance is pretty useless since after around 7-8, it's all basically just random variations?
More...

hmm.. yes... you're actually right about that :)

Thank you
 
You must log in or register to reply here.
Back
Top