Staggering Data Lags with Investor/RT and DTN - Any and All Help Would Be Appreciated

[6yaNYCjm5m]

Hi,

Reserving the right to be wrong, I doubt that such staggering delay has something to do with "usual' latency issues (measured in ms). Some posters are way more educated in subject than myself, hence my reservations, but I believe something is wrong with your environment. Your hardware is definitely not an issue, don't waste more money. Before you take away Duke resources from some serious research, why don't you try simple cross-reference test. Test another charting app (e.g. SierraChart, excellent choice) with IQFeed and get another feed (e.g. eSignal, should be in the same retail league with IQFeed) and test it with IRT. That's going to immediately pinpoint your problem: charting app or datafeed or if every combo has the same issue, then you start drilling upstream to your OS/network setup and your IP provider.

I am using IB TWS, IQFeed, MarketDelta (it's IRT with special features, I guess you know that) and SierraChart (2 instances), all on one box. Often I have OandaFX running as well. My box is twice less power than yours and it never spikes to more that 50% CPU, averaging 30-40%. I have not noticed any lag but I admit I do not watch the timestamps down to secs and milisecs. My visual observation between the IB feed and IQFeed so far did not show any diffs. If there is occasional spike, I don't know how human could possibly react to that. Saying that, I don't want to argue about trading styles and approaches (scalping, auto-systems, or whatever...). Another difference with my environment is that I do not trade stocks and do not have 23 symbols, but still I have 10-ish rather high volume symbols in the database and have 8 charts open (4 symbols). To sum it up, if there is something so radically wrong with those apps (30 sec delay) way too many people will be screaming in their faces.


I would like to second srv's post about the database. I was for 20+ years IT professional (mostly database design and architecture, but also dirt-down programming ). In the mid 80's I was looking into the dbVista as my development platform, but ended up married for life with Oracle. I was surprised when I saw dbVista in IRT when I started using it several years ago. I personally do not like when charting app uses database, they have to fight too many issues for relatively few benefits. Probably the biggest one is performance, second being data consistency (with quotes being notoriously dirty data). Once when I saw that IRT is running fast enough (for me) I've stopped paying attention to performance, but the second issue is constant battle. Often there are messages of corrupted database objects and data. Haven't seen any data loss or significant setup/configuration loss, and the key is, IMHO, to keep as little as possible in terms of data, charts and other objects, i.e. only things you use every day.


At the end here is my REAL question to you. I don't like to de-rail the thread from the original issue but I hope you will answer with you opinion and experience. You have mention that the main reason for IRT is volume breakdown analysis (love it myself). You have also noted (correctly) that IB feed is useless for that analysis because it is snapshot based, hence producing incorrect results. I trade only electronic futures and with good tick feed VB analysis works like a charm. Couple of months ago I wanted to take a look at some stocks and was perplexed how dirty were the quotes. The SPY has more price spikes than porcupine. When I put the VB on, every trade (well 98%) was on the bid side. There was a lot of selling lately , but that's just wrong data. Granted I was not that well educated regarding the stocks, I started looking further and saw that SPY, for example, is listed on Pacific Exchange, which I don't know how much of electronic trading they have. Then I went to NASDAQ, thinking it's all electronic. It looked somewhat more realistic, but still far cry from anything in the electronic futures world (regardless of exchange). So my questions are:
- Do you see the same issue with VB analysis on stocks?
- If yes, how you can use it, because IMHO, it's useless?
- If not, what do you do to get the clean data?
- Are stocks (some or most of them) traded on other (several) exchanges than the one where they are listed?
- If yes, is that the reason why NASDAQ listed stocks, that should be 100% electronic, get aggregated quotes from other pit-traded exchanges, hence dirty data?

I've seen trades $1 away from the current bid/ask on SPY, which is by all measures huge. This was through three different data feeds. I don't know which one is worse: if it WAS a real trade so far away from current bid/ask (on such volume) or if it is a bad quote. Same thing is happening with volume, wild spikes as well. That instrument should be comparable to miniSP in terms of volume. It has 100+ mil daily vol, but I believe that 100 stocks is considered 1 lot and that's comparable to 1+ mil of daily vol of the miniSP. I can not recollect seeing ever any such dirty data in electronic futures (excluding obvious system problems).

I would appreciate any opinion and explanation.
Thanks.

By the way, you should really deinstall that anti-virus crap. Tighten up your firewall in your router and have only business apps on your rig. Don't browse, download and email from there and you are safe. Get yourself some small box or whatever notebook for browsing pleasures, put it on the second router and separate IP space btw. the business rig and browsing box. There you can put whatever anti-virus... For all practical purposes this is usually the biggest performance killer and (unfortunately) one of the biggest scams among apps. Good Lord, it's like wearing condom 24/7 ... Even better, get yourself a Mac for personal use... Wait a second, you can actually run all your stuff on Mac, what the hell I was thinking... You are lucky man. And in case you really need a stupid Win, you take Vmware Fusion for $80-something and run multiple virtual Win-boxes, one of them you dedicate for communications with the outside world and in case you pick up something nasty you just delete the whole thing and copy/paste your clean seed virtual-box and off it goes again...

 

[cwb1014]

Quote from srv:I meant to say that the problem is NOT that they are using the wrong tool or an inefficient tool. A database can be implemented using a database ENGINE, such as dbVista efficiently or otherwise depending on the knowledge, skill, and experience of the programmer.

What I'm wondering is whether it is likely, or even possible, that the database itself presents a bottleneck when the volume of trading explodes as it does around the open and close of the regular stock trading session. In other words, can the flow of data attempting to get into and/or out of it get to so high a rate that the database simply cannot input and/or output that data on a timely basis, thus accounting for a very large part of the lags we're encountering? If so, it strikes me that adding more computing power, a solution frequently suggested by the developers (neither of whom, by their own admission, is anything resembling an expert in computer hardware), will accomplish little or nothing by way of solving the problem here. While it presumably would enable IQConnect to process its inbound data and get it out the door faster, what's the sense of that if IRT's database has already reached its limit in terms of how fast it can take the data up? Seems to me, from a common sense standpoint, that in this scenario a faster processor would accomplish little by way of reducing lag and instead would, at any given point in time, simply create a larger pool of data awaiting entry into the database.

I have no idea whether what I'm suggesting here is actually happening (which is why I'm asking, btw), but if it is, it seems to me to present a real obstacle to getting the problem resolved via an increase in computational power. I'd be very interested in your thoughts as well as those of any others here who may be knowledgeable in this area. Many thanks in advance for any additional insights that anyone may be able to offer here...

cwb1014

 

[cwb1014]

Quote from 6yaNYCjm5m:

I am using IB TWS, IQFeed, MarketDelta (it's IRT with special features, I guess you know that) and SierraChart (2 instances), all on one box. Often I have OandaFX running as well. My box is twice less power than yours and it never spikes to more that 50% CPU, averaging 30-40%. I have not noticed any lag but I admit I do not watch the timestamps down to secs and milisecs. My visual observation between the IB feed and IQFeed so far did not show any diffs. If there is occasional spike, I don't know how human could possibly react to that. Saying that, I don't want to argue about trading styles and approaches (scalping, auto-systems, or whatever...). Another difference with my environment is that I do not trade stocks and do not have 23 symbols, but still I have 10-ish rather high volume symbols in the database and have 8 charts open (4 symbols). To sum it up, if there is something so radically wrong with those apps (30 sec delay) way too many people will be screaming in their faces.

This is interesting information, but let's begin the process of determining whether we may be comparing apples and oranges. I have no idea how many ticks per time interval are trading in the 10-ish instruments you follow, because I don't trade futures and don't subscribe to a futures feed. However, the following is the average ticks per second for the 3 most active stocks (by share volume) among the 23 I was monitoring when this problem with staggering lags developed, broken down by time period (all times are EDST). I'd be interested in knowing what the comparable data is for the 3 most active instruments among the 10-ish you follow:

1 - 2 pm: 10.65 ticks per second (tps)
2 - 3 pm: 16.35 tps
3 - 4 pm: 33.14 tps
3:30 - 4 pm: 46.77 tps (this is the period when lag gets truly ridiculous)

 

[Landis82]

Quote from cwb1014:

This is interesting information, but let's begin the process of determining whether we may be comparing apples and oranges. I have no idea how many ticks per time interval are trading in the 10-ish instruments you follow, because I don't trade futures and don't subscribe to a futures feed.

Comparing "apples and oranges" is an excellent point. The amount of updates that a data-feed has to deal with when confronted by the tremendous activity in stocks like Citi, Wells Fargo, BofA - - - not too mention in the SPY would clearly outweigh anything in the futures markets.

In my humble opinion, the comparison isn't even close.

 

[cwb1014]

Quote from Landis82:

Comparing "apples and oranges" is an excellent point. The amount of updates that a data-feed has to deal with when confronted by the tremendous activity in stocks like Citi, Wells Fargo, BofA - - - not too mention in the SPY would clearly outweigh anything in the futures markets.

In my humble opinion, the comparison isn't even close.

I think you're right. The most actively traded stock I traded on Thursday (FAS) traded 321.53 million shares. Meanwhile, the S&P500 Emini, which I gather is the most actively traded futures contract, traded only 1.78 million contracts. While I certainly remain interested in knowing the actual tick rates for "the 10-ish rather high volume symbols" 6yaNYCjm5m has in his database, I'm guessing they are well below those I cited above. Also, 6yaNYCjm5m, when you say you have them "in the database," I'm assuming you mean you're monitoring them in IRT in real-time. Please let me know whether that assumption is correct.

 

[6yaNYCjm5m]

Unfortunately, I have a family medical emergency and I can not go into some more detailed statistical data. I have pulled quickly several instruments for comparison for the period of one week:

1)
Received 1,269,976 records from 03/04/2009 to 09/04/2009 4:14:59 PM (6.7 days)
and wrote 1,269,976 records for @ES# - 2009-04-12 01:49:58
File size: 48.4 MB

2)
Received 3,453,554 records from 03/04/2009 4:15:03 AM to 09/04/2009 7:59:58 PM (6.7 days)
and wrote 3,453,554 records for SPY - 2009-04-12 02:04:48
File size: 131.7 MB

3)
Received 1,314,644 records from 03/04/2009 6:15:28 AM to 09/04/2009 7:59:52 PM (6.6 days)
and wrote 1,314,644 records for FAS - 2009-04-12 02:07:18
File size: 50.1 MB

4)
Received 342,324 records from 03/04/2009 2:00:10 AM to 09/04/2009 4:03:32 PM (6.6 days)
and wrote 342,324 records for EX# - 2009-04-12 02:08:58
File size: 13.0 MB

5)
Received 156,462 records from 03/04/2009 2:01:04 AM to 09/04/2009 4:03:27 PM (6.6 days)
and wrote 156,462 records for BD# - 2009-04-12 02:10:25
File size: 5.9 MB

While it is clear that stocks tend to have more data, no way it can be said that comparison can not be done. The above numbers are records=ticks=trades. Load across the timeframes should be very much the same between the ES and stocks.
If you consolidate instruments to the best possible common denominators, you can definitely compare the system load and requirements.
I will repeat myself, but here is the math. You can not use the stock volume for tech comparisons. Nobody trades 1 or 10 stocks (unless you do Berkshire ). So, 1 lot = 100 stocks (usually) and that constitute 1 trade which is 1 tick. In your example, that gives you FAS volume of 3.22 mil lots which is not outrageously different from ES volume of 1.78 mil lots. And that was extreme daily value for FAS, I guess. Closest comparison is between ES and SPY, where SPY goes daily by 3+ mil lots and ES by 1+ mil lots.
Of course not every trade is exactly 1 lot in futures (or 100 shares in stock) but that's probably statistically irrelevant for ES vs. SPY load estimate. Good example for that case is DJ EURO STOXX 50 (symbol EX), which has same or bigger volume then ES but evidently 4 times less trades, hence conclusion more lots per trade are executed.
Admittedly, rest of the instruments in my case are having far less data, although I was monitoring the SPY and 5-ish stocks (GS,EXXON,RIMM,APPLE) for couple of weeks in January and did not see any performance impact whatsoever. Also, when I said “have in database” the answer is yes, they are all monitored real-time in MD/IRT.

Based on those numbers, your dataload would have to be 100-fold if not 1000-fold bigger to justify such lag (30 secs), and it is clearly not.

I believe you are wasting your time by analysing things the way you do it now. Again, do the simple cross app/datafeed comparison and if that doesn't reveal the culprit, check you OS and network setup and then finish with your ISP. And I am dead serious about getting rid of anti-virus crap.

Hope this help... Is there any chance you provide me your answer / opinion on the volume breakdown issue from my previous post?

Best regards.

 

[cwb1014]

Quote from 6yaNYCjm5m:
Unfortunately, I have a family medical emergency and I can not go into some more detailed statistical data.

Sorry to hear about your family medical emergency. Hope everything turns out well.

Quote from 6yaNYCjm5m:
I have pulled quickly several instruments for comparison for the period of one week:
1)
Received 1,269,976 records from 03/04/2009 to 09/04/2009 4:14:59 PM (6.7 days)
and wrote 1,269,976 records for @ES# - 2009-04-12 01:49:58
File size: 48.4 MB

While it is clear that stocks tend to have more data, no way it can be said that comparison can not be done.

I'm not saying that the comparison cannot be done; of course it can. What I am saying is that you have to be very careful when doing it to be sure that you are comparing instruments with a comparable number of trades or, at least, adjust the comparison to account for differing numbers of trades. When you have a chance, please let me know where you got the data you're citing as illustrated in item 1), above, and then I'll pull the same data for the instruments I was monitoring, assuming it's available in IRT. At that point, we'll know exactly what we're comparing here and that's important.

 

[cwb1014]

Quote from 6yaNYCjm5m:
I trade only electronic futures and with good tick feed VB analysis works like a charm. Couple of months ago I wanted to take a look at some stocks and was perplexed how dirty were the quotes. The SPY has more price spikes than porcupine. When I put the VB on, every trade (well 98%) was on the bid side. There was a lot of selling lately , but that's just wrong data. Granted I was not that well educated regarding the stocks, I started looking further and saw that SPY, for example, is listed on Pacific Exchange, which I don't know how much of electronic trading they have. Then I went to NASDAQ, thinking it's all electronic. It looked somewhat more realistic, but still far cry from anything in the electronic futures world (regardless of exchange). So my questions are:
1) Do you see the same issue with VB analysis on stocks?
2) If yes, how you can use it, because IMHO, it's useless?
3) If not, what do you do to get the clean data?
4) Are stocks (some or most of them) traded on other (several) exchanges than the one where they are listed?
5) If yes, is that the reason why NASDAQ listed stocks, that should be 100% electronic, get aggregated quotes from other pit-traded exchanges, hence dirty data?

[second paragraph]: I've seen trades $1 away from the current bid/ask on SPY, which is by all measures huge. This was through three different data feeds. I don't know which one is worse: if it WAS a real trade so far away from current bid/ask (on such volume) or if it is a bad quote. Same thing is happening with volume, wild spikes as well. That instrument should be comparable to miniSP in terms of volume. It has 100+ mil daily vol, but I believe that 100 stocks is considered 1 lot and that's comparable to 1+ mil of daily vol of the miniSP. I can not recollect seeing ever any such dirty data in electronic futures (excluding obvious system problems).

To answer your specific questions above, which I've now numbered, here you go:
1) No
2) Not applicable
3) I filter my data using IRT's data filtering protocols, described at http://www.linnsoft.com/qa/a/22.htm and discussed elsewhere on the site
4) Yes
5) I have no idea. However, if you start a thread along these lines in the Order Execution section here (http://www.elitetrader.com/vb/forumdisplay.php?s=&forumid=30), you may get the answers you're looking for from folks who can offer authoritative insights into this issue and the ones you raise in the second paragraph I've quoted above. I've denoted this paragraph "[second paragraph]:" to avoid any confusion here.

Hope this helps.

 

[mtt]

I use IQFeed with Ensign Windows and Quotetracker. In the past several months, I have been experiencing some lag with the quotes in Quotetracker especially during heavy trading times. Quotes start to slow down or even stop at times. My computer system is similar to yours but with a C2Q processor. I don't know what the problem is but it seems to be some kind of bottlenecking occurring in the QT software. I don't think IQFeed is the problem because my Ensign Windows is not affected. The only thing I know is that since Feb 2009, Quotetracker has been trying to repair some issues with the IQFeed. Check the beta log for details. I don't have experience with Investor/RT but I do recommend Ensign Windows. You can do a trial for several days before you purchase a subscription.

 

[cwb1014]

Quote from 6yaNYCjm5m:
And I am dead serious about getting rid of anti-virus crap.

Believe me, I'm going to shut my real-time virus scanning down completely before tomorrow's trading session and see what happens. Thanks for the advice. I actually checked this out with a computer wizard buddy of mine, and he agrees with you wholeheartedly, pointing out that if one's experiencing data lag with any program, the last thing you want running is real-time virus scanning.

I'll also be testing IQConnect with its diagnostic functions to see to what extent, if any, it's contributing to the data lags. Based on what's being said throughout this thread, my best guess is that it's not the real problem here, but you never can tell until you do the testing.