IQFeed / DTN.IQ data quality

W

wsws

If I'm informed correctly, IQFeed / DTN.IQ provides unfiltered data.

While this may bring some advantages because no trade is lost, several users have complained about artificial "spikes" in the data. Last summer some users even said that there were such a high number of spikes that IQFeed's historical data was almost useless for charting, in comparison with eSignal.

But I also remember that an IQFeed representative mentioned some months ago that IQFeed was planning to implement filters (activatable by the users) in order to allow users to filter out unwanted trades outside the bid/ask range. So maybe the situation has improved in the meantime.


My question is: How is the IQFeed data quality now? Is IQFeed's historical data good/accurate enough to be used for charting or backtesting of trading systems?
 
For futures, it is not bad. Stocks, which i rarely look at, i always see bad ticks hampering the charts.

I think for the price, it is reliable. Note that: For the price.
I always log in, stays up all day.

Here is the BIGGEST problem: If you log in, sometimes you get different bars (minute bars).
For example, 9:34 ES bar, may be high of 1055. Then, you log in later, its 1059. Then log in later, its 1055 again.
I think this is because they have diff. servers on their side that maintain this info. If the server misses a few ticks, this causes this.

But, again, for the price its not bad. I also use their API - i channel it to my own app. Their API is not good, its fair.
 
IQFeed works fine in normal market conditions. However, I have noticed that as soon as we go into a "fast" market, IQFeed begins to lag.

My guess is that they are very close to having a terrific feed, but for the price point, it does not make sense for them to upgrade their Internet pipes or their servers.

One nice thing about them is they don't ever disconnect you. I have my software constantly on with them and I don't miss a tick. Realtick or IB on the other hand will disconnect you.

I use them as backup.

nitro
Quote from wsws:

If I'm informed correctly, IQFeed / DTN.IQ provides unfiltered data.

While this may bring some advantages because no trade is lost, several users have complained about artificial "spikes" in the data. Last summer some users even said that there were such a high number of spikes that IQFeed's historical data was almost useless for charting, in comparison with eSignal.

But I also remember that an IQFeed representative mentioned some months ago that IQFeed was planning to implement filters (activatable by the users) in order to allow users to filter out unwanted trades outside the bid/ask range. So maybe the situation has improved in the meantime.


My question is: How is the IQFeed data quality now? Is IQFeed's historical data good/accurate enough to be used for charting or backtesting of trading systems?
More...
 
Thanks for the answers, guys.

I tested IQFeed with the following result:

Due to the high symbol limit the feed is good for tracking a larger number of stocks with a couple of quotelists. I didn't notice any lag when comparing IQFeed's quote updates with IB's feed.

Basically, IQFeed could be a terrific feed. The only problem is the data quality issue which hopefully gets addressed by DTN.

I've attached three screenshots of charts that show the typical "spikes"/bad ticks in the data. To avoid misunderstandings: Not in EVERY chart spikes occur as often as in the chart of INTC, I just chose INTC as a typical example to illustrate the problem.


Data with artificial spikes/bad ticks (in the backfill data) is useless for charting methods such as Point&Figure charting. And needless to say that it is not suitable either for trading system applications that critically depend on clean historical data, for example because the historical high/low range of individual bars is used in the trading system.

Note: As regards current streaming quotes, a good trading system has to be able do deal with spikes/bad ticks. It can easily filter them out by comparing the current tick data with the current bid ask range or the following tick etc. But bad ticks in the historical backfill data are a problem because there is no way for the client application to filter them out.



My understanding is that there can be three causes of spikes:

(1) occasionally there are trades which are executed above or below the current bid/ask range. For example block trades. These trades are considered valid and must not be filtered out (if considered valid trades by the exchanges).

(2) mistakes during the order entry can lead to mistrades. Typically (if an institutional instead of retail trader made the mistake) the mistrades get later busted by the exchange. There is no way to filter out mistrades immediately in the quote stream, but when trades are busted by the exchanges, the historical backfill data needs to be corrected.

Question in this regard: Would it be possible for DTN to correct the historical data after trades get busted? Do data vendors get notified of busted trades by the exchanges? At least in the case of end of day-data (e.g. as available through Yahoo), the data is adjusted, ie. does NOT include any trades which were busted.

(3) erroneous tick data both in the current quote stream and/or the historical backfill data is caused by software errors (protocol errors, tranmission errors, software bugs etc.).



As DTN's vice president of operations and products has posted in this forum previously (user id: iqfeed), I'd appreciate any update on the implementation of optional filters.


With improved data quality, IQFeed could be a terrific feed suitable for point&figure charting and trading system applications. If I was working for DTN, I would put the data quality issue on top of the programming queue.
 
I have not noticed any spikes at all with the ES and IQFeed , though from your charts - there does appear to be a severe problem there. I can confirm that DTN does lag regularly when the market becomes very very fast , but I'd imagine nearly every feed suffers from this.
 
I found the statements posted by Jay Froscheiser in October 2003 again, vice president DTN Market Access, LLC.


original source:
http://www.elitetrader.com/vb/showthread.php?s=&postid=343218#post343218

(Questions by user droth and answers by IQFeed)


- data quality (I)
I understood from your website, that IQFeed realtime data is unfiltered. So I have to perform my own bad tick filtering. I also read in this thread, that there are negotiations about modifying this behaviour. Could you please update the current status ?


We are still working on changes to our historical server architecture which will allow us to create a filtering mechanism. I don't have a hard date for completion, but we are shooting for the end of the year.


- data quality (II)
Is the historical data aggregated from the tick data ? Will it also include bad ticks, or is there some filter procedure applyed ?


Daily data is not aggregated from the tick data, but minute data is. If a customer informs us that there is a bad tick, it will get corrected, and that correction will follow through to all intervals.

++++++++

prox:
I have not noticed any spikes at all with the ES and IQFeed , though from your charts - there does appear to be a severe problem there.
More...


As regards the market indices (and probably the futures too; I checked only two futures since futures are not my primary interest), there does not appear to be the spike/bad tick problem.

The problem of frequent spikes/bad ticks occurs with regard to stocks, and as the example of INTC demonstrates, not only in the case of illiquid, rarely traded stocks.

But the comments by DTN's Jay F. quoted above give some hope that DNT is working on the issue. This is also why I posted here. I do not know to what extent DTN is already aware of the problem. Certainly the market indices/futures are less affected than stocks where the problem is really severe.


I think that IQFeed would have the potential to be an excellent feed, should DTN manage to solve the problem of very frequent bad ticks (with regard to historical backfill minute data for stocks).
 
There is also another thread on this issue, started 08-07-03

http://www.elitetrader.com/vb/showthread.php?s=&threadid=20834&perpage=6&pagenumber=1


Again, it appears that primarily the historical minute data for stocks is affected by the bad tick problem, whereas for futures/indices the data is ok.


The mentioned thread contains a couple of interesting comments:

iqfeed ( 08-07-03 10:13 PM):
Also, we have a new history server in development that will help clean things up. As everyone knows, it is a fine line between providing exactly what the exchange sends (which isn't always pretty), and creating accurate filtering algorithms. We think we will have a good solution when released, but it is in the early stages now.

Jay Froscheiser
DTN Market Access, LLC.



Question in this regard by me (wsws):

What about mistrades that are originally sent by the exchange as valid trades but get later busted? In this case the historical minute data needs to be adjusted, or alternatively somehow marked to allow client programs to ignore the mistrades.


nitro (08-07-03 10:17 PM):
Please be careful not to shove this algorithm down our throats. IMHO, there should be an option to set filtering on/off. You should be able to set filtering on/off in REALTIME without restarting the IQFeed software (this way, when a number is coming out, you can turn it off without having to restart your software and therefore lose quotes, and then turn it back on when the market quiets down.) There should be seperate filters for equity and futures so that you can set one or both or none, and be able to tell the system what threshold to use. Equties "tradethrough" all the time, whereas futures rarely do (not on FED days though.)

The idea is to be as flexible as possible.


iqfeed 08-07-03 10:43 PM:
Absolutely.. In fact the current plan is to just mark the trades as questionable, and let the client determine what they want to do with them. This way, the client can do whatever it wants with the data. Either color code the tick (showing that it is suspect), or automatically throw it out, or leave it in. We don't want to be liable for our customers missing data due to a "bad filter".


comment by wsws:
this sounds promising, but the approach has the following problem. If historical tick data is downloaded by client applications, it is a viable approach to filter out ticks that were marked as questionable trades.

But when historical data is downloaded in the form of minute data, the individual minute data bars were already formed on the server side by aggregating individual ticks. Therefore it's not simply possibly to mark an entire minute bar as invalid if it contains a bad tick (unless for every questionable bar an alternative is sent by the server; which may be complicated and not viable if the minute bar contains several questionable ticks and thus there are several alternatives).
 
You must log in or register to reply here.
Back
Top