Generating 5 second realtime bars from real time tick streams

S

sappjason

I'm an automated trader that relies heavily on Interactive Broker's 5 second real time bars, but they have recently become VERY unreliable. So, I've purchased a subscription to DTN's IQFeed and I'm trying to replicate the real time 5 second bar behavior.

Replicating the historical 5 second OHLC bar behavior (with VWAP) was easy, but I'm really scratching my head with how to generate 5 second OHLC bars in real time.

My server is in a co-location facility that has HUGE internet bandwidth (about 80 mbs), so I don't think I'm going to have a problem receiving the data. But, I'm worried about lagging data and building bars accordingly.

What I'm looking for in this post is to see if anyone is familiar with doing this type of thing....

Here are my thoughts/questions?
Basically, say it's 10:00 AM (on the nose) and I want to generate a 9:59:55 to 10:00:00 bar and I'm receiving tick data on 100 stocks. Some of the data that I may be receiving could be a couple seconds old with timestamps that are still 9:59:56 or 9:59:57. So, at 10:00:00, it's not really appropriate to generate a 9:59:55 to 10:00:00 bar yet as they would be incomplete. I have to wait until all of my 9:59:55 to 10:00:00 data has arrived. But, how do I know when it's all arrived?
My initial thoughts are to ALWAYS have a tick stream for SPY and QQQQ and use their exchange timestamps to figure out when it's possibly appropriate to generate all 5 second bars for that particular 5 second period.

Basically I would just be using the SPY and QQQQ tick streams as my timer. So, perhaps when I start seeing 10:00:01 data on both SPY and QQQQ, then it's possibly safe to go ahead and generate my 9:59:55 to 10:00:00 bars (for all 100 stocks that I'm watching). But I'm guessing that this will even sometimes miss data for my 9:59:55 to 10:00:00 bars, because there still may be some lagging ticks coming in for the 9:59:55 bar after 10:00:01 (I really don't know).
I'm just very confused as to what really makes sense and I'm looking for direction. So, does anyone have ideas or suggestions?


Thanks,

Jason
 
hey jason...

we support arbitrary second bars in tradelink.

you could use our implementation which would be like example below.

alternatively, since we're open source you can checkout our code and unit tests on the site http://tradelink.googlecode.com Click Source and browse, look in TradeLinkCommon project for BarList code and TradeLinkTests for unit tests.

Code: 
public class Strategy : ResponseTemplate
{
 public override void GotTick(Tick k)
 {
   blt.newTick(k);
 }
  void NewBar(string sym)
  {
       // fired when new bar is created
     blt[sym].RecentBar.Close
  }
 BarListTracker blt;
 public Strategy() 
 {
  blt = new BarListTracker(BarInterval.CustomInterval, 5);
  blt.NewBar += new DebugDelegate(NewBar);
}
}
 
Jason, I have that solution for my trading and I must say is pretty robust.
I know some guys that have that solution ready made, I subscribed the service from them. And they use DTN IQfeed streaming based on TCP/IP, not DDE nor RTD, TCP/IP is 20 times faster than RTD, not to say more. The rest, calculations etc are all done in multithreading C++. In my case I think the application performs more than 30 million calculations per second.
From what I see everyday, the application receives and processes several thousands of messages at the same time. And you can do any kind of indicator with it. You can manipulate the data every possible way. I receive at peaks, around 50k messages per second, at the end of the day I receive around 120 million messages per day following lots of symbols. After the data is received you can develop indicators like,
trades at bid, trades at ask, volumes at bid or ask, trades above ask or below bid, you can calculate all the orderflow at the bid-ask midpoint,... after the data is received, you can do whatever you want with it. Spreads, arbitrage strategies, anything related with order flow. They also have a way to read and chart hidden/undisclosed orders, on any market
These guys are delivering the most reliable and flexible platform on high frequency data

You can reach them at: info@excelstreamer.com and they will send you the docs.
 
Quote from sappjason:


Here are my thoughts/questions?
Basically, say it's 10:00 AM (on the nose) and I want to generate a 9:59:55 to 10:00:00 bar and I'm receiving tick data on 100 stocks. Some of the data that I may be receiving could be a couple seconds old with timestamps that are still 9:59:56 or 9:59:57. So, at 10:00:00, it's not really appropriate to generate a 9:59:55 to 10:00:00 bar yet as they would be incomplete. I have to wait until all of my 9:59:55 to 10:00:00 data has arrived. But, how do I know when it's all arrived?
More...

Hi Jason
Do you have analysis on how much of the data that comes in like that? Are we talking 2 tick a day or per minut? Or?
 
Ohh btw...

I can tell you how I would do it, when a tick is recieved, an event is raised, the event subscriber groups the tick data into the current if the timestamp fits, otherwise it call another async event called DelayedTickSynchronizer which taks care of updating appropriate bar and, the normal tick processor needs to recalculate all the bars anyway so everything just works fine :)
 
Quote from T-Algo:

You can reach them at: info@excelstreamer.com and they will send you the docs.
More...

They go to the trouble of creating a web site that only has a splash page and an email contact (same as mentioned above)?

Looks like the old after-you-have-reach-out-and-touch-some-sales-critter-for-information-you-are-bothered-for-life gambit.

Pathetic...
 
You must log in or register to reply here.
Back
Top