IQFeed / DTN.IQ data quality
- Thread starter wsws
- Start date
Historical minute data for futures is also filled with spikes. It would
be totally useless for bactesting. I don't use it for backtesting, but
for backfill up to 5-10 days back and I have to erase the spikes
individually with my charting program each time I have to backfill.
You would think they could put in their own spike filters... ?
Using IQFeed's historical time&sales tick data, I investigated some of the spikes shown in the chart examples posted previously:
Conclusions:
(1) If the spikes have really happened as mistrades, it is actually impressive that IQFeed delivers all ticks including the mistrades (even though that's not what the user wants when backfilling historical minute bars instead of historical tick data).
(2) For the download of historical tick data it would be a good and viable approach not to filter bad ticks out but to simply mark them as questionable, as already suggested. Reason: The processing of downloaded ticks is done by the client application, which can freely decide what to do with questionable ticks.
(3) For the backfill of historical minute bars calculated by aggregating tick data, with the tick aggregation conducted by the IQFeed servers, it leads to the unwanted spikes in the chart when the tick aggregation also includes ticks that are known to be bad ticks.
How can bad ticks be safely detected? According to iqfeed (Jay F.), only minute backfill data is formed by aggregation of tick data. Daily data, on the other hand, is NOT aggregated from tick data.
If at the end of the trading day, the IQFeed quote servers have an end-of-day bar (daily bar) for the trading day, which can be considered authoritative (as it is not formed by tick aggregation), a simple solution to safely detect bad ticks is as follows:
Every trade (tick with volume >0) that occurred above the high or below the low of the end-of-day quote must have been a mistrade and hence got busted, otherwise it would have been reflected by the high/low range of the end-of-day bar.
This should safely eliminate the majority of spikes, with the filtering being correct in the sense that no valid trade got filtered out. The worst and most disturbing spikes would be safely eliminated, greatly improving the quality of the backfill minute data.
However, the filtering approach would not be complete, as it cannot detect mistrades that occured within the high-low range of the end-of-day bar. An interesting question is here whether the data feed vendors get notified by the exchanges after trades got busted, which would allow the data feed vendors to filter out all bad ticks at the end of the trading day.
It would be a great improvement if DTN could implement the proposed filtering approach for the generation of backfill minute data.
JDSU:
probably a mistake during order entry: 0.34 instead of 4.34
low of authoritative EoD bar for 03/12/04 is 4.25
=> trade must have been busted (as it is not reflected by EoD bar) and can be filtered out at the end of the trading day
03/12/04,11:55,4.339,100,4.33,4.34,21591883,,,Trade,
03/12/04,11:55,4.34,800,4.33,4.34<<,21591783,,,Trade,
03/12/04,11:55,4.33,500,4.33<<,4.34,21590983,,,Trade,
03/12/04,11:55,0.34,300,4.33,4.34,21590483,,,Trade, spike!
03/12/04,11:55,4.33,200,4.33<<,4.34,21590183,,,Trade,
03/12/04,11:55,4.34,200,4.33,4.34<<,21589983,,,Trade,
03/12/04,11:55,4.34,100,4.33,4.34<<,21589783,,,Trade,
INTC:
An example for a couple of trades below current bid price.
low of authoritative EoD bar for 03/08/04 is 27.62
=> as trades are below this low, they must have been busted and can be filtered out at the end of the trading day
03/08/04,14:06,27.9700,1700,27.9700<<,27.9800,61070682,,,Trade,
03/08/04,14:06,27.9700,250,27.9700<<,27.9800,61068982,,,Trade,
03/08/04,14:06,27.9700,200,27.9700<<,27.9800,61068732,,,Trade,
03/08/04,14:06,27.9700,200,27.9700<<,27.9800,61068532,,,Trade,
03/08/04,14:06,27.9700,200,27.9700<<,27.9800,61068332,,,Trade,
03/08/04,14:06,27.9700,1500,27.9700<<,27.9800,61068132,,,Trade,
03/08/04,14:06,27.9700,100,27.9700<<,27.9800,61066632,,,Trade,
03/08/04,14:06,26.9600,161,27.9700,27.9800,61066532,,,Trade, spike!
03/08/04,14:06,26.9600,200,27.9700,27.9800,61066371,,,Trade, spike!
03/08/04,14:06,26.9600,202,27.9700,27.9800,61066171,,,Trade, spike!
03/08/04,14:06,27.9700,200,27.9700<<,27.9800,61065969,,,Trade,
03/08/04,14:06,27.9700,200,27.9700<<,27.9800,61065769,,,Trade,
03/08/04,14:06,27.9700,100,27.9700<<,27.9800,61065569,,,Trade,
03/08/04,14:06,27.9700,1300,27.9700<<,27.9800,61065469,,,Trade,
03/08/04,14:06,27.9700,301,27.9700<<,27.9800,61064169,,,Trade,
03/08/04,14:06,27.9700,501,27.9700<<,27.9800,61063868,,,Trade,
03/08/04,14:06,26.9600,100,27.9700,27.9800,61063367,,,Trade, spike!
03/08/04,14:06,27.9700,130,27.9700<<,27.9800,61063267,,,Trade,
KLAC:
Trade @ 54.46 above high of authoritative EoD bar (54.03)
=> trade must have been busted and can be filtered out at the end of the trading day
03/05/04,12:11,53.4300,400,53.4200,53.4300<<,6771260,,,Trade,
03/05/04,12:11,53.4300,100,53.4300<<,53.4500,6770860,,,Trade,
03/05/04,12:11,54.4600,10000,53.4300,53.4600,6770760,,,Trade, spike!
03/05/04,12:11,53.4400,100,53.4300,53.4700,6760760,,,Trade,
03/05/04,12:11,53.4500,400,53.4400,53.4700,6760660,,,Trade,
again a trade above high of authoritative EoD bar (54.2) for
03/08/04
=> trade must have been busted and can be filtered out
03/08/04,12:51,53.3000,100,53.3000<<,53.3200,3606995,,,Trade,
03/08/04,12:51,53.3200,100,53.3100,53.3200<<,3606895,,,Trade,
03/08/04,12:51,53.3200,400,53.3100,53.3200<<,3606795,,,Trade,
03/08/04,12:51,56.2500,10000,53.3100,53.3300,3606395,,,Trade, spike!
03/08/04,12:51,53.3300,122,53.3000,53.3300<<,3596395,,,Trade,
03/08/04,12:51,53.3300,100,53.3000,53.3300<<,3596273,,,Trade,
Conclusions:
(1) If the spikes have really happened as mistrades, it is actually impressive that IQFeed delivers all ticks including the mistrades (even though that's not what the user wants when backfilling historical minute bars instead of historical tick data).
(2) For the download of historical tick data it would be a good and viable approach not to filter bad ticks out but to simply mark them as questionable, as already suggested. Reason: The processing of downloaded ticks is done by the client application, which can freely decide what to do with questionable ticks.
(3) For the backfill of historical minute bars calculated by aggregating tick data, with the tick aggregation conducted by the IQFeed servers, it leads to the unwanted spikes in the chart when the tick aggregation also includes ticks that are known to be bad ticks.
How can bad ticks be safely detected? According to iqfeed (Jay F.), only minute backfill data is formed by aggregation of tick data. Daily data, on the other hand, is NOT aggregated from tick data.
If at the end of the trading day, the IQFeed quote servers have an end-of-day bar (daily bar) for the trading day, which can be considered authoritative (as it is not formed by tick aggregation), a simple solution to safely detect bad ticks is as follows:
Every trade (tick with volume >0) that occurred above the high or below the low of the end-of-day quote must have been a mistrade and hence got busted, otherwise it would have been reflected by the high/low range of the end-of-day bar.
This should safely eliminate the majority of spikes, with the filtering being correct in the sense that no valid trade got filtered out. The worst and most disturbing spikes would be safely eliminated, greatly improving the quality of the backfill minute data.
However, the filtering approach would not be complete, as it cannot detect mistrades that occured within the high-low range of the end-of-day bar. An interesting question is here whether the data feed vendors get notified by the exchanges after trades got busted, which would allow the data feed vendors to filter out all bad ticks at the end of the trading day.
It would be a great improvement if DTN could implement the proposed filtering approach for the generation of backfill minute data.
JDSU:
probably a mistake during order entry: 0.34 instead of 4.34
low of authoritative EoD bar for 03/12/04 is 4.25
=> trade must have been busted (as it is not reflected by EoD bar) and can be filtered out at the end of the trading day
03/12/04,11:55,4.339,100,4.33,4.34,21591883,,,Trade,
03/12/04,11:55,4.34,800,4.33,4.34<<,21591783,,,Trade,
03/12/04,11:55,4.33,500,4.33<<,4.34,21590983,,,Trade,
03/12/04,11:55,0.34,300,4.33,4.34,21590483,,,Trade, spike!
03/12/04,11:55,4.33,200,4.33<<,4.34,21590183,,,Trade,
03/12/04,11:55,4.34,200,4.33,4.34<<,21589983,,,Trade,
03/12/04,11:55,4.34,100,4.33,4.34<<,21589783,,,Trade,
INTC:
An example for a couple of trades below current bid price.
low of authoritative EoD bar for 03/08/04 is 27.62
=> as trades are below this low, they must have been busted and can be filtered out at the end of the trading day
03/08/04,14:06,27.9700,1700,27.9700<<,27.9800,61070682,,,Trade,
03/08/04,14:06,27.9700,250,27.9700<<,27.9800,61068982,,,Trade,
03/08/04,14:06,27.9700,200,27.9700<<,27.9800,61068732,,,Trade,
03/08/04,14:06,27.9700,200,27.9700<<,27.9800,61068532,,,Trade,
03/08/04,14:06,27.9700,200,27.9700<<,27.9800,61068332,,,Trade,
03/08/04,14:06,27.9700,1500,27.9700<<,27.9800,61068132,,,Trade,
03/08/04,14:06,27.9700,100,27.9700<<,27.9800,61066632,,,Trade,
03/08/04,14:06,26.9600,161,27.9700,27.9800,61066532,,,Trade, spike!
03/08/04,14:06,26.9600,200,27.9700,27.9800,61066371,,,Trade, spike!
03/08/04,14:06,26.9600,202,27.9700,27.9800,61066171,,,Trade, spike!
03/08/04,14:06,27.9700,200,27.9700<<,27.9800,61065969,,,Trade,
03/08/04,14:06,27.9700,200,27.9700<<,27.9800,61065769,,,Trade,
03/08/04,14:06,27.9700,100,27.9700<<,27.9800,61065569,,,Trade,
03/08/04,14:06,27.9700,1300,27.9700<<,27.9800,61065469,,,Trade,
03/08/04,14:06,27.9700,301,27.9700<<,27.9800,61064169,,,Trade,
03/08/04,14:06,27.9700,501,27.9700<<,27.9800,61063868,,,Trade,
03/08/04,14:06,26.9600,100,27.9700,27.9800,61063367,,,Trade, spike!
03/08/04,14:06,27.9700,130,27.9700<<,27.9800,61063267,,,Trade,
KLAC:
Trade @ 54.46 above high of authoritative EoD bar (54.03)
=> trade must have been busted and can be filtered out at the end of the trading day
03/05/04,12:11,53.4300,400,53.4200,53.4300<<,6771260,,,Trade,
03/05/04,12:11,53.4300,100,53.4300<<,53.4500,6770860,,,Trade,
03/05/04,12:11,54.4600,10000,53.4300,53.4600,6770760,,,Trade, spike!
03/05/04,12:11,53.4400,100,53.4300,53.4700,6760760,,,Trade,
03/05/04,12:11,53.4500,400,53.4400,53.4700,6760660,,,Trade,
again a trade above high of authoritative EoD bar (54.2) for
03/08/04
=> trade must have been busted and can be filtered out
03/08/04,12:51,53.3000,100,53.3000<<,53.3200,3606995,,,Trade,
03/08/04,12:51,53.3200,100,53.3100,53.3200<<,3606895,,,Trade,
03/08/04,12:51,53.3200,400,53.3100,53.3200<<,3606795,,,Trade,
03/08/04,12:51,56.2500,10000,53.3100,53.3300,3606395,,,Trade, spike!
03/08/04,12:51,53.3300,122,53.3000,53.3300<<,3596395,,,Trade,
03/08/04,12:51,53.3300,100,53.3000,53.3300<<,3596273,,,Trade,
example: e-cbot Error Trade Policy
http://www.cbot.com/cbot/docs/45972.pdf
November 6, 2003, Effective November 24, 2003
3 Trade Price Outside of the "No Bust Range"
If the price of the asserted error trade is more than the specified number of ticks from the
reference price, e-cbot Operations will send a broadcast message to the user community
indicating that the trade has been called into question. If the asserted error trade is outside of the
specified tick range and involves only two parties, e-cbot Operations will attempt to contact the
parties to the transaction.
If both parties agree to bust or re-price the transaction, e-cbot
Operations shall send a broadcast message to the user community and an alert to the quote
vendor network indicating that the trade was busted or re-priced.
Do other exchanges send such kind of alerts to the quote vendor network, too?
http://www.cbot.com/cbot/docs/45972.pdf
November 6, 2003, Effective November 24, 2003
3 Trade Price Outside of the "No Bust Range"
If the price of the asserted error trade is more than the specified number of ticks from the
reference price, e-cbot Operations will send a broadcast message to the user community
indicating that the trade has been called into question. If the asserted error trade is outside of the
specified tick range and involves only two parties, e-cbot Operations will attempt to contact the
parties to the transaction.
If both parties agree to bust or re-price the transaction, e-cbot
Operations shall send a broadcast message to the user community and an alert to the quote
vendor network indicating that the trade was busted or re-priced.
Do other exchanges send such kind of alerts to the quote vendor network, too?