Recent content by DevBrian

True, my bottleneck has always been the strategies themselves. And none of our strategies can process greater than probably 500k messages a second. Is this your experience as well? I ask, because it makes one question the usefulness of any data stream optimized to read faster than your...

After removing a bool check (if the background thread should keep processing) on the producer side, speed is above 10 million items per second. Constructor: _actionBlock = new ActionBlock<long>(l => { _handled++; }, new ExecutionDataflowBlockOptions() { SingleProducerConstrained = true })...

Thanks! I was under the impression I needed a BufferBlock for async behavior. Throughput is about 5 million messages per second by cutting that out. Not sure why, but setting the BoundedCapacity of the ActionBlock significantly hurts the performance. Checking InputCount before calling Post...

I believe my bottleneck is just the call stack. I'm currently using my own implementation of a consumer producer pattern. It succeeds in increasing the end to end throughput of the system, but it has limits as well. I've looked at the dataflow library, it's fast. But... Doing ...

Sorry, I meant to say 200k-300k messages per second. ps - if I'm repeating myself, ET isn't showing my posts, so I'm trying again.

I'm sorry, I meant to say 200k-300k.

I'm sorry. I did mean 200k-300k, not 200-300. I was able to strip things down, to where I could pump 5 million messages per second. Few things I did: - Minimized the number of private method calls. - Removed interface calls. - No locks/no multi-threading/no arithmetic - No maintaining...

I'm curious about your implementation. I too have written a "custom binary time series data storage for backtesting purpose". All in c#. I did a few optimization that reduced the storage size of the file, but it hurt the read speed. I'm content with this, because any of the strategies we...