Reliable way to get market close of all US stocks using PolygonIO

Z

Zuiquan

Hello,

I thought getting the most recent market close for all US stock tickers would be a simple task.
But, I probably do it the wrong way.

I need to get this info whenever during a day (even during afterHours or after it, during week-end)
If there is no trade for a symbol that day during market hours, I need the most recent (eligible trade) price prior to the market close.
By eligible trade, I mean a trade with eligible Last Price conditions, and coming from the primary exchange of the symbol.

NOTE: I would also be interested in having similarly the close of the previous 'session'
(meaning: if we are in market hours -> get the last price at preMarket close, if after 8pm EST -> get the last price at afterHours close)

How do you guys do to get that simple feature?

Regards
 
  1. Why do you care so much about "eligible" trades?
  2. Did you read the API docs?
  3. If you're a Polygon customer why don't you ask support instead of posting here?
  4. I suspect you're going to have to download all the tick data and find the trades you want. AFAIK the polygon APIs have no concept of primary exchange.
 
1. What I want is to somehow replicate what I can see on google finance in premarket, market, afterHours.
(https://www.google.com/finance/quote/AAPL:NASDAQ)

So, seeing the price curve, the market close price, the previous session close price,...

From what I understood, I get many trades coming from the polygon feed, and not all are to be taken into account to establish a symbol's price.
I could see 'weird' jumps in prices when not taking that into account.
I am not an expert.
Hence my question here.

2. I read as much of the docs I could find about the API.
Not everything is defined clearly.
They have a few pages about the conditions.
(https://polygon.io/glossary/us/stocks/conditions-indicators)
When I use the conditions, I still do not match what I can see on various platforms.

3. I am in contact with the tech support already, and they suggested me to also take into account the exchange id.
For example, AAPL has primary exchange XNAS and I receive trades from
* exchange - equities - XASE - NYSE American (AMEX)
* exchange - equities - XNAS - NASDAQ OMX BX
* TRF - equities - XCIS - National Stock Exchange
* TRF - equities - FINR - FINRA
* exchange - equities - EDGA - Cboe EDGA
* exchange - equities - EDGX - Cboe EDGX
* exchange - equities - XCHI - Chicago Stock Exchange, Inc
* exchange - equities - XNYS - New York Stock Exchange
* exchange - equities - ARCX - NYSE Arca
* exchange - equities - XNGS - Nasdaq
* exchange - equities - IEXG - IEX
* exchange - equities - PHLX - Nasdaq PSX
* exchange - equities - BATY - Cboe BYX
* exchange - equities - BATS - Cboe BZX
* exchange - equities - MPRL - MIAX Pearl
* exchange - equities - MEMX - Members Exchange
And, the price is sometimes quite different on two successives trades from different exchanges.
Still, when I restrict tot XNAS, I still do not manage to match what I can see on various platforms (Google, Yahoo, tradingview)

I chose to not depict here all I attempted because I do not see the point in describing something that does not work.

However, asking people here, on 'Data sets and Feeds' section, seemed to me appropriate as there might be someone here who have done successfully something similar, and having the answer here could also be useful to more Polygon customers than just myself.

4. PolygonIO offers an API request to get the 'primary exchange' id associated to a symbol.
Then, each trade (in history or feed) has an exchange id field.
But, it is true that I did not find a query where I could filter based on the exchange and/or conditions.
For example, the 'Last trade for a symbol' is not useful to me because it gives the last trade whatever the condition, exchange,..
So indeed, so far, I had to parse everything.
Which is long when having to process more than 26k symbols.

But as I said, I am quite new to the field and to PolygonIO, so I may miss something obvious.

Do you download everything, process it, keep a local database to avoid having to perform queries again?

Regards
 
Zuiquan said:
1. What I want is to somehow replicate what I can see on google finance
More...

When I use the conditions, I still do not match what I can see on various platforms.
More...

OK, I understand where you are coming from :) Being new in the field, you assume that there is one correct way of doing things and that "platforms" like google do things in the correct way. Erase these thoughts from your mind. The reality is platforms are shit. They are all buggy and all work slightly differently. The best you can do is understand the raw data and use it in the way that makes sense for your needs.

Do you download everything, process it, keep a local database to avoid having to perform queries again?
More...

Yes this is exactly what I do.
 
Thank you for sharing your experience.

Do you think restricting to the primary exchange has a meaning?
Because, I do not really grasp why I have trades from NYSE for a symbol on NASDAQ...
 
Aside the exchange questionning, there are the conditions to take into account or not.

I found those two links about the conditions and the kind of data I have in the feed/history.
http://nasdaqtrader.com/content/technicalsupport/specifications/utp/utdfspecification.pdf
https://www.ctaplan.com/publicdocs/...er-update/CTS_BINARY_OUTPUT_SPECIFICATION.pdf

PolygonIO is not using the same protocol, of course.
So, the packet description is not that relevant.
But it seems PolygonIO data contains the same data organized differently.

I can also find in the PDF the conditions table and glossary also given on PolygonIO website.

I would tend to rely on the table given here to decide what condition to take to update the LastPrice and OHLC, as those conditions are a bit cryptic to me today.
Plus, sticking to the same conditions used by PolygonIO would enable be to rely on some of their API queries.

So aside the exchange choice, it is more on the proper architecture to reliably get the data, update them back when my server have rebooted,...

In case, I need to catch up on data (because my server was down), it would have been good PolygonIO allows in one query to download all the trades between two timestamps.
But, it seems I will need to loop over the symbol, thus making >20k requests!

Maybe I could start from using 'Snapshot - All Tickers' or 'Grouped Daily (Bars)' (https://polygon.io/docs/get_v2_snapshot_locale_us_markets_stocks_tickers_anchor)
(https://polygon.io/docs/get_v2_aggs_grouped_locale_us_market_stocks__date__anchor)
queries to quickly get the current state, while I download the missing data in background

How do you handle reliability?

Do you have one server constantly feeding/downloading and storing to your disk,
while having another running the processing?

The idea being to be able to restart the processing system without loosing trades during that time and with necessity to catch up (time-consuming),
and so allowing the processing system to be up-and-running as fast as possible.

Regards
 
cruisecontrol said:
What is the use case for all of this? Are you actually trading?
More...

No, it is not to trade buy/sell directly

The aim is to make some live analysis to help knowing which symbol is currently of interest among the many symbols.
So, it uses the live feed to make a regular screener, use some pre-processed past data, does some quick computations, and allow the user to display/plot one or more symbol-of-interest current activity.
The analysis must be done in a few seconds at worst.

The live part being really work-in-progress. I have to start and stop it quite often.
And, I can especially miss a change of session (nomarket,premarket,market,afterHours).
So, my queries are made to get back the close prices at the most recent change of session, and close price of the most recent market close.
 
You must log in or register to reply here.
Back
Top