No, It's not documented what the source is. A possible clue is that the trades come together with 4 condition codes. By googling for the code names I found they are very similar or identical to the CTA (consolidated tape association) format described at www.ctaplan.com
Trade reporting lag?
- Thread starter cruisecontrol
- Start date
Ok, cool. Then I did the right thing creating a bug report and I will try to lobby for them to improve it to the point where not even one instance is out of order
Sorry to not answer about the timestamps but I wasn't sure. I've now confirmed that it is the exchange time by comparing with other data.
i can share my CSV data for a couple of stocks if you would like. However, I have a method to filter backward looking datasets. Are you noticing this a lot with polygon. I have seen this issue "Real Time" with other data cheap (Retail) and expensive (Prop Shop / Quant Firm) vendors as well.
One thing I want to mention ... Let's say vendor experiences a gap (they lost a couple seconds worth of data). A good vendor will request a retranmission of the data. Once they receive it (a couple of seconds later), they will send it through with the original timestamps. If you trade real-time, this is totally unacceptable unless they mark it somehow so you can drop it. The reason they keep the original timestamp is because many users of data display it in their Time and Sales displays or use it to calculate VWAP/TWAP at the end of the day. So they want the data to appear in-line when sorted by time. I don't know about this particular vendor, just my insight. Do they have anything to indicate that a trade is a possible out-of-sequence one?
qlai / cruisecontrol - Question for you guys.
During backtesting mode, I was able to sequence the data as per my desire.
However for Real Time data collection, my trades are sometimes out of sync (most times by 1 to 500 milliseconds). I am creating my "Trading Indicators" based on 1 second Bars (which are updated based on Tick by Tick)
Ex:
Data for 20180619 09:20:56.333 could arrive before 20180619 09:20:56.234
So for example, if I process only Trades such that:
CurrentTradeTime >= LastTradeTime, the ProcessTradeInStrategy
--->
I miss approx 5% of the trades dataset for a given symbol.
On the other hand, a cleaner way would be to "Sequence" The Trades (like in backtesting mode)
requires usage of Queues and Firing only at the end of 1 second (too much usage of timers and makes just market data processing fairly complicated)
I was wondering what kind of methodologies / solutions / software patterns might be useful in such scenario.
Thank you
During backtesting mode, I was able to sequence the data as per my desire.
However for Real Time data collection, my trades are sometimes out of sync (most times by 1 to 500 milliseconds). I am creating my "Trading Indicators" based on 1 second Bars (which are updated based on Tick by Tick)
Ex:
Data for 20180619 09:20:56.333 could arrive before 20180619 09:20:56.234
So for example, if I process only Trades such that:
CurrentTradeTime >= LastTradeTime, the ProcessTradeInStrategy
--->
I miss approx 5% of the trades dataset for a given symbol.
On the other hand, a cleaner way would be to "Sequence" The Trades (like in backtesting mode)
requires usage of Queues and Firing only at the end of 1 second (too much usage of timers and makes just market data processing fairly complicated)
I was wondering what kind of methodologies / solutions / software patterns might be useful in such scenario.
Thank you
K, this is just my $0.02 ... You should not use exchange timestamps. You should timestamp each message based on your local time and the rest of the code should use that nice clean time. Shield your code from idiosyncrasies of exchange/vendor time sync.
As far as firing timed events, I only trigger events based on the local timestamp from the message (which I recorded myself as described above) ... I never use the system timers. Why? Because I made it a requirement to be able to replay the data back at the end of the day and achieve the same functionality as in real-time. So my timers only use timestamps from the messages to fool them into thinking that current time is whatever I want it to be. Helps a lot with troubleshooting your strategy and testing modifications. It's a good idea to embed your own heartbeat messages, let's say each second, into the data stream as to not have to wait for a trade message to roll/trigger.
P.S. When you say that exchange timestamps are off by 1 to 500 ms, need to be sure you are comparing apples to apples. Let's say you are looking at AAPL trades, the times should only increase. Let's say you look at IBM trades, the times should always increase. But if you are looking at both AAPL and IBM, there is no such guarantee as the data comes from two different aggregators (SIPs).
As far as firing timed events, I only trigger events based on the local timestamp from the message (which I recorded myself as described above) ... I never use the system timers. Why? Because I made it a requirement to be able to replay the data back at the end of the day and achieve the same functionality as in real-time. So my timers only use timestamps from the messages to fool them into thinking that current time is whatever I want it to be. Helps a lot with troubleshooting your strategy and testing modifications. It's a good idea to embed your own heartbeat messages, let's say each second, into the data stream as to not have to wait for a trade message to roll/trigger.
P.S. When you say that exchange timestamps are off by 1 to 500 ms, need to be sure you are comparing apples to apples. Let's say you are looking at AAPL trades, the times should only increase. Let's say you look at IBM trades, the times should always increase. But if you are looking at both AAPL and IBM, there is no such guarantee as the data comes from two different aggregators (SIPs).
So I've been communicating with Polygon and they have confirmed to me that the OOS trades are due to a bug in their system which they are in the process of fixing.
Thank you qlai, your 2 cents are worth a lot more to me.
". You should not use exchange timestamps. You should timestamp each message based on your local time and the rest of the code should use that nice clean time. Shield your code from idiosyncrasies of exchange/vendor time sync." - This thinking solved many of my problems.
I will replay using the "Trading System Record Time" instead of "Exchange Time" - this should allow to create record / replay as per my live system. Using Exchange Time is not "Accurate Replay" and thank you for shedding light upon that.
WRT "P.S. When you say that exchange timestamps are off by 1 to 500 ms, need to be sure you are comparing apples to apples. Let's say you are looking at AAPL trades, the times should only increase. Let's say you look at IBM trades, the times should always increase. But if you are looking at both AAPL and IBM, there is no such guarantee as the data comes from two different aggregators (SIPs)."
I agree, I am isolating each symbol and comparing it to its previous last trade time of that symbol. However the issue was happening due to Race conditions amongst threads which has been resolved by your "2 cent" Solution
Good luck with trading and blessing you with loads of money $
". You should not use exchange timestamps. You should timestamp each message based on your local time and the rest of the code should use that nice clean time. Shield your code from idiosyncrasies of exchange/vendor time sync." - This thinking solved many of my problems.
I will replay using the "Trading System Record Time" instead of "Exchange Time" - this should allow to create record / replay as per my live system. Using Exchange Time is not "Accurate Replay" and thank you for shedding light upon that.
WRT "P.S. When you say that exchange timestamps are off by 1 to 500 ms, need to be sure you are comparing apples to apples. Let's say you are looking at AAPL trades, the times should only increase. Let's say you look at IBM trades, the times should always increase. But if you are looking at both AAPL and IBM, there is no such guarantee as the data comes from two different aggregators (SIPs)."
I agree, I am isolating each symbol and comparing it to its previous last trade time of that symbol. However the issue was happening due to Race conditions amongst threads which has been resolved by your "2 cent" Solution
Good luck with trading and blessing you with loads of money $
LOL. Thank you. Anytime.
cruisecontrol,
Just wondering have you tried listening to:
Level 1 and Trades data for all ARCA, NSDQ, NYSE symbols from Polygon and had any issues other than OOS trades.
Would they be able to support anything like this? Pricing is great
rb7 - What is the pricing of IQfeed for all NYSE, NSDQ, ARCA Symbols (Approx. 4000 symbols)?
Last time, i had checked they require a $300 API fee just to test out the API and Iqfeed was limited to 500 Symbols. Does this mean i would have to get 8 connections to / subscription or 4000 symbols