Staggering Data Lags with Investor/RT and DTN - Any and All Help Would Be Appreciated

6yaNYCjm5m · Apr 13, 2009

Data is coming from the IQFeed. That is what we are talking about all the time. I have only used SierraChart to see it better, because it handles flat files vs. IRT through the database. There is no way IRT gets anything else but the same ticks/records. It might be stored differently (in a more complex way) within the database and that is a possible performance challenge, but as I've mentioned earlier, I really don't think they have one, no proof so far.
It's been a while since I was exporting data from IRT, can't remember details, just be sure that you get tick level data.
Don't get me wrong, but I will not be looking any more into this, because I am pretty sure the previous numbers are correct.

On the other side, regarding my questions about the VB analysis, thanks for your answers. I have to say I am more than surprised that you don't see anything wrong with bid/ask data for stocks. To see 20+ days of data where 95% of trades are on the bid side, is something I am having hard time believing. My conclusion was that it is a result of a non-electronic trading where almost everything ends up keyed in the system on the bid side, but of course I could be dead wrong about that.
I am also quite sure that we are looking into the same numbers (having the same app and datafeed) and the only thing I can say is be at least somewhat careful when making decisions based on bid/ask vol for those instruments.

As far as price spikes, I know how to filter them, I just don't like the environment that has so much wrong data. My question about âclean dataâ was not about the spikes but a possibility that you maybe doing something to the bid/ask volume in order to get proper bid/ask and delta profiles and hence do not see anything wrong with the VB analysis.

Anyway, wish you good trading and take care.

cwb1014 · Apr 13, 2009

Quote from 6yaNYCjm5m:
Data is coming from the IQFeed. That is what we are talking about all the time. I have only used SierraChart to see it better, because it handles flat files vs. IRT through the database. There is no way IRT gets anything else but the same ticks/records. It might be stored differently (in a more complex way) within the database and that is a possible performance challenge, but as I've mentioned earlier, I really don't think they have one, no proof so far.
It's been a while since I was exporting data from IRT, can't remember details, just be sure that you get tick level data.

OK, if your data is coming from IQFeed, then I'm virtually certain that that data will include not only trades but bid/ask updates as well; thus, each record processed is not an actual trade. This is important because in IRT I have the option of ignoring bid/ask updates, which I've chosen to do. Thus, the numbers I've cited include only actual trades.

Pardon my ignorance, but what is a "flat file"? Also, when you say "I really don't think they have one," what do the "they" and the "one" refer to? If the "they" is IRT and the "one" is a performance challenge, I'm pretty sure they do, because too many folks are able to see data from DTN in other software products without the lags I'm experiencing in IRT and are able to do so on less powerful computers than my own. Anyway, I'll be checking DTN's data in tomorrow's trading session, using DTN's diagnostic tools, to see if their data appears to be part of the problem here.

iqfeed · Apr 13, 2009

Quote from Eight:

Is there a data feed with exchange timestamps? I'd like to have that to measure everything else with. I think most resellers apply their own timestamps.

yes.. IQFeed's timestamp is the timestamp from the exchange. Hence the reason some traders get confused when they see timestamps out of sequence or try to use timestamps as a measure of latency. Some datafeeds try to "make their data look pretty" by either re-ordering trades in timestamp order or applying their own time (obviously it is easier if they just apply their own). In our case, we provide a true look at what the market is doing.

In IQFeed, we provide a timestamp to the nearest second. Our NxCore service includes an internal timestamp in addition to the Exchange timestamp. Here is a snippet from the NxCore documentation that will help explain:

"nxExgTimestamp is an NxTime structure containing the Timestamp sent by each Exchange for ExgQuotes, MarketMaker Quotes, and Trades. It usually has a resolution of 1 second. Each exchange's timestamp has it's own peculiarities. Some will occasionally jump back in time. Some exchange timestamps will appear to be a few seconds ahead or behind other exchange timestamps. The NxCore Feed Timestamp contained in NxCoreSystem.nxTime is available for comparison and reference with each ExgQuote, MarketMaker Quote, and Trade message. The NxCoreSystem.nxTime is synchronized to within 10's of milliseconds, whereas some Exchange Timestamps are notoriously off by seconds."

lindq · Apr 13, 2009

Quote from cwb1014:

I don't know whether this is a software or a hardware problem or a combination of the two, but I'm hoping there are some experts here who can give me some guidance.

Have you discussed your problem with Bill Linn at Linnsoft?

Also, I haven't seen any posts on IRT's Yahoo chat regarding this. Did you post there to the other IRT users?

And if you've gone to both of those sources, what has been the advice you've received?

srv · Apr 13, 2009

Quote from cwb1014:

OK, if your data is coming from IQFeed, then I'm virtually certain that that data will include not only trades but bid/ask updates as well; thus, each record processed is not an actual trade. This is important because in IRT I have the option of ignoring bid/ask updates, which I've chosen to do. Thus, the numbers I've cited include only actual trades.

Pardon my ignorance, but what is a "flat file"? Also, when you say "I really don't think they have one," what do the "they" and the "one" refer to? If the "they" is IRT and the "one" is a performance challenge, I'm pretty sure they do, because too many folks are able to see data from DTN in other software products without the lags I'm experiencing in IRT and are able to do so on less powerful computers than my own. Anyway, I'll be checking DTN's data in tomorrow's trading session, using DTN's diagnostic tools, to see if their data appears to be part of the problem here.

If you haven't yet tried adjusting the "accumulate N Ticks Before Writing to Disk" - you might find it worthwhile adjusting those fields as high as possible given the amount of RAM available in your system.

I did some diagnostic work on the Lag Problem today. The volume today was too low to reproduce the serious problem I've had in the past.
But right around 3:30PM (Central) I did see a delay of 3 or 4 seconds (where it is otherwise zero, rounded) in i/rt while quotetracker running off the same DTN IQFEED did not show any lag.

The delay characteristics looked similar to what I saw when the Lag was significant.
(I have video capture of all this and would like to post some of it if anyone is interested - problem is the files are large, the only one that shows increasing lag on i/rt, followed by a short pause then followed by resumption of no (or very low) lag -- that file is 18MB. I'm sure there is a way to edit an .avi file and will do so if anyone thinks it would be helpful.)

I minimized , what I will call the "tick buffer size" to 100 and saw fairly rapid cycling as I described above.
I normally keep it at 2000 which is the largest size on the drop down menu -

I think I will put it all together for the good folks at Linnsoft so they can deal with it - after I or someone else confirms what I saw today.

I don't think I'm going to bother with it tomorrow.

Roman Candle · Apr 13, 2009

Quote from srv:

If you haven't yet tried adjusting the "accumulate N Ticks Before Writing to Disk" - you might find it worthwhile adjusting those fields as high as possible given the amount of RAM available in your system.

I did some diagnostic work on the Lag Problem today. The volume today was too low to reproduce the serious problem I've had in the past.
But right around 3:30PM (Central) I did see a delay of 3 or 4 seconds (where it is otherwise zero, rounded) in i/rt while quotetracker running off the same DTN IQFEED did not show any lag.

The delay characteristics looked similar to what I saw when the Lag was significant.
(I have video capture of all this and would like to post some of it if anyone is interested - problem is the files are large, the only one that shows increasing lag on i/rt, followed by a short pause then followed by resumption of no (or very low) lag -- that file is 18MB. I'm sure there is a way to edit an .avi file and will do so if anyone thinks it would be helpful.)

I minimized , what I will call the "tick buffer size" to 100 and saw fairly rapid cycling as I described above.
I normally keep it at 2000 which is the largest size on the drop down menu -

I think I will put it all together for the good folks at Linnsoft so they can deal with it - after I or someone else confirms what I saw today.

I don't think I'm going to bother with it tomorrow.

I went thru the same things youâre going thru, lots of quirky things happening. Itâs not the data feed, itâs not your computer. Itâs Investor RT. I think it all has to do with the fact it can run on Windows and Macintosh. I never gained confidence in the software and got tired of questioning the data, I recommend you move on if you want to trade intraday in real-time.

I spent more time fooling with the support group and the data base manager that I had no time to view charts. I kept going to Stock Charts,.com.

There are just to many other good choices. Ensign,Quotetracker,Tradestation,Esignal, and if you want the very best CQG.

Roman Candle · Apr 13, 2009

Quote from srv:

If you haven't yet tried adjusting the "accumulate N Ticks Before Writing to Disk" - you might find it worthwhile adjusting those fields as high as possible given the amount of RAM available in your system.

I did some diagnostic work on the Lag Problem today. The volume today was too low to reproduce the serious problem I've had in the past.
But right around 3:30PM (Central) I did see a delay of 3 or 4 seconds (where it is otherwise zero, rounded) in i/rt while quotetracker running off the same DTN IQFEED did not show any lag.

The delay characteristics looked similar to what I saw when the Lag was significant.
(I have video capture of all this and would like to post some of it if anyone is interested - problem is the files are large, the only one that shows increasing lag on i/rt, followed by a short pause then followed by resumption of no (or very low) lag -- that file is 18MB. I'm sure there is a way to edit an .avi file and will do so if anyone thinks it would be helpful.)

I minimized , what I will call the "tick buffer size" to 100 and saw fairly rapid cycling as I described above.
I normally keep it at 2000 which is the largest size on the drop down menu -

I think I will put it all together for the good folks at Linnsoft so they can deal with it - after I or someone else confirms what I saw today.

I don't think I'm going to bother with it tomorrow.

Are you trading? Or are you farting around with the software?

cwb1014 · Apr 13, 2009

Quote from lindq:

Have you discussed your problem with Bill Linn at Linnsoft?

Also, I haven't seen any posts on IRT's Yahoo chat regarding this. Did you post there to the other IRT users?

And if you've gone to both of those sources, what has been the advice you've received?

I have discussed this extensively with Bill Linn at Linnsoft and he is presently hard at work on getting this issue resolved. Also, Jay Froscheiser, Director, Active Trader Products and Sales, and Loren Foster, Senior Technical Representative, from DTN are lending great support to the cause as well.

So that everyone here will know, it is now clear that IRT's database was not and is not in any way involved in the problems that have arisen here. As it turns out, the IRT database is not actually engaged in the reception or processing of real-time data by the program itself.

lindq · Apr 13, 2009

Quote from cwb1014:

So that everyone here will know, it is now clear that IRT's database was not and is not in any way involved in the problems that have arisen here. As it turns out, the IRT database is not actually engaged in the reception or processing of real-time data by the program itself.

Thank you for reporting that. I have great respect for the people at Linnsoft and would hate to see their software get a bad rap for an issue that lies elsewhere.

But as you know it's possible with IRT to load up quote pages with many custom indicators and technical indicators all calculating tick-by-tick on a big watch list, so that's the first place that I'd look for things slowing down unless you have a Cray CX1 in your basement. But I'm sure you've already considered that.

Roman Candle · Apr 14, 2009

Quote from cwb1014:

I have discussed this extensively with Bill Linn at Linnsoft and he is presently hard at work on getting this issue resolved. Also, Jay Froscheiser, Director, Active Trader Products and Sales, and Loren Foster, Senior Technical Representative, from DTN are lending great support to the cause as well.

So that everyone here will know, it is now clear that IRT's database was not and is not in any way involved in the problems that have arisen here. As it turns out, the IRT database is not actually engaged in the reception or processing of real-time data by the program itself.

Are you a lawyer? IRTs data base is not the problem, hum. So its something else then you need to say that if you going to defend one of Elite Traders sponsors.