Staggering Data Lags with Investor/RT and DTN - Any and All Help Would Be Appreciated

Quote from lolatency:

It = your cable modem/dsl/whatever

non-hard realtime dissemination means that, well ... During normal markets, the upper bound on the time it takes the exchange to send a tick is more or less stable or some fixed amount, ... but in fast markets if the time it takes to deliver goes up as a function of the speed of quotes, your time to process and transmit grows... this happens to a lot of data feed providers if they don't write their code correctly ...

For all intents of purposes, the issue is not you or your computer. At best, you're asking for too much data; however, the most probable cause is that DTN is not good enough for today's fastest market... (but it may be good enough for the usual days)
Normally I like your posts, lolatency, but on this one you really don't know what you're talking about.

DTN/IQ Fees is a top-line data feed priced correctly for the retail market, recently upgraded their data feed to tick level and are always fast to pick-up the phone whenever there is a problem ... which is not even their fault, more often than not.

***

To the OP,

You're going to get all kinds of answers on the board, but the only people who can really solve your problem are the folks at DTN/IQ Feed and Investor IRT combined.

One other thing you can try is to get a higher quality security software than McAfee (I use ESET NOD32).

If that fails, and your trouble shooting doesn't reveal anything else as the problem, you are going to have to start testing out other charting software with the DTN/IQ Feed and/or testing out other data feeds with Investor IRT until you can find the right combo which is most optimal for your needs (I personally think there might be software/data feed incompatibility issues between DTN/IQ feed and Investor IRT, which is why I said you have to got on the phone to both parties).

Welcome to the world of trading. :)

Good Luck
 
Quote from lolatency:

I

For all intents of purposes, the issue is not you or your computer. At best, you're asking for too much data; however, the most probable cause is that DTN is not good enough for today's fastest market... (but it may be good enough for the usual days)

I have to agree with MandelbrotSet I use DTN IQfeed with MultiCharts to do more than the OP is and have no lag.
 
Quote from MandelbrotSet:

Normally I like your posts, lolatency, but on this one you really don't know what you're talking about.

DTN/IQ Fees is a top-line data feed priced correctly for the retail market, recently upgraded their data feed to tick level and are always fast to pick-up the phone whenever there is a problem ... which is not even their fault, more often than not.

Well, I never used DTN, so maybe you're right and they're great.

I'm just saying, though, that a lot of these retail feed providers are running with 50ms as a baseline latency and they do skip beats when the market really moves. On the backend of their implementation, they're dropping packets, doing retransmits, and then pushing that into another relay server to send information along again.
 
I've been having the same problem. I've experienced much longer lag time than you.
The easiest way I've found to get rid of the "backlog" of ticks is to simply stop the data feed for a few seconds and restart it.
That does not solve the problem, it just gets the entire system back to "real time" - if the data load is (still) great the lag will go from zero and increase.

Like you, I put TWS on a separate computer. I time the lag by comparing the timestamp on T&S chart with the actual time. Normally, I cannot see any lag at all - at times when it is very busy, I've seen lags well over 30 seconds if I allow it to go that long.

Needless to say, for scalping or probably any other reason one would want "real time" data - this situation is totally useless.

I've tried a lot of things to fix the problem including keeping only 5 active symbols, even displaying ONLY Time and Sales (no charts) and shutting down most all other processes.

My hardware is similar to yours except the CPU has 4 cores and I suspect that may be part of the problem as i/rt does NOT account for more than one CPU (thought the OS does spread the threads across multiple cpu's - I've played with that also.)

I've more or less concluded I cannot use this setup anymore. If the problem is IQFeed, it is because its sending too much data too quickly! So, it could be the bandwidth limitation of my inet connection (not likely for reasons I won't go into now) or it is the way i/rt processes the data - or something else.
I never saw the lag problem using iqfeed with "quote tracker" but I haven't used that for 4 months or so.

Subjectively it seems the problem either started or got much worse when IQFeed switched to new server(s)

The worse part of this is the lack of confidence I have in what I'm seeing on the screen. I only first noticed the problem trying to figure out why I could no longer successfully scalp a "hot" market - like near the end of RTH. (it cost me!)

Constantly comparing the clock and timestamps is taking way too much attention! I have to do it all the time as I cannot know, without looking, if there is or is not significant lag.

I hope someone finds a fix as I have much time invested in i/rt and am resisting giving it up. Whatever its problems it has tremendous functionality - however for me, i/rt isn't something I can use "real time."
 
are the op's assumptions about the IB feed correct? They aggregate data but is it altered in any way? Maybe the assumption is wrong and the answer is to use IB's data...
 
Quote from srv:

I've been having the same problem. I've experienced much longer lag time than you.
The easiest way I've found to get rid of the "backlog" of ticks is to simply stop the data feed for a few seconds and restart it.
That does not solve the problem, it just gets the entire system back to "real time" - if the data load is (still) great the lag will go from zero and increase.
I've noticed the same thing myself. Btw, you can force a restart of the data feed by Shift+Left Click on the Start IQ Feed button on the toolbar; this turns the procedure into a 1 step process. But, I'm wondering what this tells us about the nature of the underlying problem. One theory I have is that IRT uses a database and that it may represent a significant bottleneck in the flow of data from DTN to IRT. Thus, in a fast market, a huge queue of data from IQConnect may develop waiting to get into IRT's database, at which point the data becomes available to IRT's charts, quotepages, etc. By killing the feed and restarting it, either that queue is given time to get into the database or is simply wiped out by the kill/restart data feed process. Either way, when the data restarts, initially there's no queue and thus very little true lag (the difference between a tick's time stamp and the time the data makes it into IRT). With the passage of time, however, the lag would become huge in a fast moving market and, thus, lag grows progressively larger. Similarly, a huge queue of data could, in fast markets, be developing awaiting processing by IQConnect with the kill/restart data feed process having the same effect on it. If you or anyone here has any thoughts on this, I'd sure be interested in hearing them.

Like you, I put TWS on a separate computer.
Actually, I'm running TWS on the same computer as IRT.

I time the lag by comparing the timestamp on T&S chart with the actual time. Normally, I cannot see any lag at all - at times when it is very busy, I've seen lags well over 30 seconds if I allow it to go that long.
There's an easier way to time the lag in IRT. One of the available quotepage (QP) columns is called Lag; if you add it to a QP, you'll see the lag for all the symbols in your QP. It's calculation and use is explained here: http://www.linnsoft.com/new/index57.html. One thing you should be aware of here though is that at present if no new tick comes in for a symbol before the QP is recalculated, the Rec Time (explained at the link above) becomes your clock time and, since in IRT Lag = (Rec Time - Time, where Time = the time stamp of a given tick), Lag keeps growing and growing until a new tick comes in for that symbol. This is only a problem in slow moving markets and should be fixed in the next release of IRT so that Lag is always true lag (Rec Time - Time, with Rec Time for any given tick remaining fixed at its original actual value).

Needless to say, for scalping or probably any other reason one would want "real time" data - this situation is totally useless.
Couldn't agree with you more!

I've tried a lot of things to fix the problem including keeping only 5 active symbols, even displaying ONLY Time and Sales (no charts) and shutting down most all other processes.
I've tried cutting down my active symbols and it has some benefit, although Lag is still considerably larger in the first and last half-hours that at other times of the day.

My hardware is similar to yours except the CPU has 4 cores and I suspect that may be part of the problem as i/rt does NOT account for more than one CPU (thought the OS does spread the threads across multiple cpu's - I've played with that also.)
I'm not sure whether throwing more processing power at this problem will resolve these lags. While clearly more processing power should enable IQConnect to process data more quickly as it comes in, there may be some limitation as to how fast IQConnect can take up data for processing, which limitation may not be affected by more processing power. Similarly, if there's a bottleneck created by some limit on how fast IRT's database can take up data, then more processing power is unlikely to overcome it, no matter how much processing power we throw at the problem. I'd be interested in anyone's thoughts on the issue of whether an increase in processing power is likely to have a beneficial effect on the lags I'm experiencing.

Subjectively it seems the problem either started or got much worse when IQFeed switched to new server(s).
If anyone from DTN is monitoring this thread, please comment. If anyone else using DTN has noticed this, please say so.

The worse part of this is the lack of confidence I have in what I'm seeing on the screen. I only first noticed the problem trying to figure out why I could no longer successfully scalp a "hot" market - like near the end of RTH. (it cost me!)

I couldn't agree with you more on the lack of confidence issue. If you're not sure that what you're seeing on your screen is timely and real, it's impossible to trade effectively. Also, I noticed the problem in exactly the same way you did and it's cost me (a bunch) too!

I hope someone finds a fix as I have much time invested in i/rt and am resisting giving it up. Whatever its problems it has tremendous functionality - however for me, i/rt isn't something I can use "real time."
You're right, IRT does have tremendous functionality and I, too, am reluctant to give it up, especially in view of the time I've spent learning to use the program. Unfortunately, it's becoming something, at least when connected to DTN (I've never used IRT with any other data provider), that cannot be relied upon for real-time trading in fast markets unless and until someone comes up with a genuine and lasting solution to these terrible problems with lag. I continue to welcome comments from anyone here who can shed any additional light on what may be causing these problems and how they can best be resolved.
 
Quote from cwb1014:
Btw, you can force a restart of the data feed by Shift+Left Click on the Start IQ Feed button on the toolbar; this turns the procedure into a 1 step process.


Thanks, I'll try that.

Quote from cwb1014:
But, I'm wondering what this tells us about the nature of the underlying problem. One theory I have is that IRT uses a database and that it may represent a significant bottleneck in the flow of data from DTN to IRT.

Actually the database is inherently very efficient. (I used to work with it and still have license to the source code.)
Having said that I should add that I speak from experience that is 18+ years old!

In my opinion, i/rt should be completely rewritten but for reasons other than an inefficient database engine. Please don't tell anyone I said that! :D
 
Quote from cwb1014:

I don't know whether this is a software or a hardware problem or a combination of the two, but I'm hoping there are some experts here who can give me some guidance. I'm running Investor/RT (IRT) with DTN's IQFeed data feed. I am running them on a Windows XP Pro system with a 3.16 GHz Core2Duo and 4 GB RAM. I also run Interactive Broker's TWS on the same system. I trade stocks and lately have been noticing staggering lags between the time stamps (i.e., the times the trades were executed as reported by the exchanges) on the ticks I receive from DTN and the time they show up in IRT, especially in the first and last half-hours of the regular trading session.

As an example, in the last half-hour yesterday, I was only monitoring 23 symbols and yet my average lag for each instrument was roughly 30 seconds, making it impossible to trade effectively off of charts. Making this all the more troubling is that yesterday's trading volume was below average. During this time, IQFeed was taking up roughly 50% of my processor with IRT and IB taking up roughly 15% each. So that you'll know, at all other times during the session, my lags run in the vicinity of 1-3 seconds, with the processor loads substantially reduced (total load of these 3 components would be in the vicinity of 25 - 35%).

It seems clear that the problem here is generally a function of the vastly increased tick flows during the first and last half-hours, but what specifically the problem is I don't know. Is DTN's software (IQConnect, which takes the data in the packets it receives and converts them to a form usable by IRT) taking too long to do its job when tick rates increase dramatically or is IRT not capable of taking the data up fast enough from IQConnect at these times for it to be delivered on a timely basis to IRT's charts? Would a more powerful computer solve the problem and, if so, what kind of system would I need to get this down to manageable levels?

I'd also be interested in knowing whether anyone else here has experienced these types of lags while monitoring similar or greater numbers of symbols during these times. If so, how have you dealt with it? If not, what charting software are you using with what data feeds and what kind of lags are you experiencing during these and other times? Finally, if anyone here has any general suggestions as to how to resolve the problems I'm experiencing with these staggering lags, I sure would appreciate your letting me know.

Since some of you may be wondering why I don't just use IB's data feed, which is compatible with IRT, it's because I want to have access to IRT's volume breakdown features, which are very inaccurate with IB's feed, because IB's feed does not transmit every tick (it aggregates ticks at the same price) while DTN's does.

Many thanks in advance for any help that anyone may be able to offer here.

Best regards,

cwb1014

:cool:

Investor RT sucks, they will always tell you it's you connection, your computer. Just leave there are plenty of good charting software Co out their
 
Quote from srv:
Quote from cwb1014:
But, I'm wondering what this tells us about the nature of the underlying problem. One theory I have is that IRT uses a database and that it may represent a significant bottleneck in the flow of data from DTN to IRT.

Actually the database is inherently very efficient. (I used to work with it and still have license to the source code.)
Having said that I should add that I speak from experience that is 18+ years old!
I'm glad to learn that the DB is very efficient and that you're intimately familiar with it. Do you think that there may be a bottleneck being caused by any limitation in how fast IRT can take up the tick data that IQConnect is putting out? This would be really good to know in trying to determine what the best solutions to the problems here may be. If there is a bottleneck here and/or with the rate at which IQConnect can accept data directed to it, it seems to me that throwing more computational power at the problem just isn't going to work. Would you agree?

Thanks in advance for any insights you may be able to offer here.

cwb1014

:cool:
 
Quote from Roman Candle:

Investor RT sucks, they will always tell you it's you connection, your computer. Just leave there are plenty of good charting software Co out their

The large number of customers we have that use Irt leads me to argue otherwise. I have the unique position of hearing feedback directly from users of the many different commercial products that support IQFeed. Some of these customers come and go or switch between products. InvestorRT users on the other hand have been one of the fastest growing and loyal customer groups we have seen. As a company that provides not only data for 3rd party apps, but also a fully integrated client workstation, I am very impressed at the level of functionality provided by their software and the enhancements they have added over the past couple of years.

The level of cooperation between InvestorRT and IQFeed developers is also impressive. We have worked with InvestorRT for nearly 15 years as they were one of the first products to support our Satellite feed. It is great to have a partner who not only works closely with us, but to their users as well. While I can appreciate Cliff looking for outside input (and it appears good insight is coming from it), I am sure he will also be the first to tell people how great the support from both InvestorRT and DTN have been. I personally received as many as a dozen emails in a single day from Cliff. I do my best to respond personally to each of his questions when he knows he could be contacting our support group via our 800#. I know InvestorRT staff has received and responded with the same level of support.

DTN continues to work with InvestorRT to ensure we satisfy our customers needs. The overwhelming majority of our customers are happy and we will always be striving to increase the number of users that make up that "happy majority".

Jay Froscheiser
Director, Active Trader Products
DTN
 
Back
Top