Whole-market options snapshot data from ORATS

thecoder

thecoder

Experience Report

I today started to test one of the special offers by ORATS for Elitetrader users,
ie. the offers on this page: https://info.orats.com/elitetrader
(On that page are also links for downloading sample data)

Mine is 15-minutes delayed snapshot data of the whole US options market.
15-min-delayed is enough for my uses cases.
ORATS has also offers for non-delayed data, check their plans.

I used this webpage for ordering a 14-day trial for $29:
https://orats.chargifypay.com/subscribe/shnh5nktjp4s

Today I already downloaded snapshots of 14:30 and 15:50 ET using normal https URL with an individual/secret "token" one gets from ORATS.

Beware: I was told that with this budget/economic plan the download interval should not be shorter than 30 minutes, otherwise there can be a "surcharge" fee.

Alternatively one can also specify an Amazon AWS S3 bucket where the data will be copied to by ORATS. I could not test this yet b/c my AWS space is not set up yet. AWS is new for me and also seems to be a little bit complicated to understand as it has dozens of options...

The snapshot files are compressed gzip files, of size about 130 MB compressed.
The download takes about 5 seconds on my Gigabit home Internet link
(I also have some faster links via VPS etc., but 5 seconds is fast enough for me).
The unpacked CSV file is about 380 MB (using the command "zcat fname >fname.csv").

These are the columns in the CSV:
Code: 
ticker,
tradeDate,
expirDate,
dte,
strike,
stockPrice,
callVolume,
callOpenInterest,
callBidSize,
callAskSize,
putVolume,
putOpenInterest,
putBidSize,
putAskSize,
callBidPrice,
callValue,
callAskPrice,
putBidPrice,
putValue,
putAskPrice,
callBidIv,
callMidIv,
callAskIv,
smvVol,
putBidIv,
putMidIv,
putAskIv,
residualRate,
delta,
gamma,
theta,
vega,
rho,
phi,
driftlessTheta,
callSmvVol,
putSmvVol,
extSmvVol,
extCallValue,
extPutValue,
spotPrice,
quoteDate, 
updatedAt,
snapShotEstTime,
snapShotDate,
expiryTod

I hadn't time yet to analyze the data itself, but this of course will come soon.

See/search also postings of @Matt_ORATS here, for example this one:
https://www.elitetrader.com/et/thre...r-for-single-names.362757/page-2#post-5487409

See also their Snapshot API:
https://docs.orats.io/datav2-snapshot-api-guide/data.html#strikes
 
Last edited:
Some basic quick stats:

The above mentioned snapshot file belonging to today's 15:50 time, ie. 202111171550.csv,
has 722765 lines with strike data (ie. all the data of a strike is on the same CSV line)
belonging to 5496 underlying tickers.
The data is sorted in ascending order, I think on these first 5 columns:
ticker,tradeDate,expirDate,dte,strike
 
Last edited:
I wonder why some IVs are zero, whereas Bid and Ask for calculating the IV are present.
For example in snapshot 202111171550 the three IVs for Put for this line are missing: TAL,2021-11-17,2021-11-19,3,17.5,...
Or is something else is required as well, but which is missing?
Code: 
$ grep "TAL,2021-11-17,2021-11-19,3,17.5" 202111171550.csv
TAL,2021-11-17,2021-11-19,3,17.5,4.53,0,8045,0,1449,2,6113,641,863,0,0,0.05,12.9,13.002853296307832,13.1,0,0,0,1.526,0,0,0,1.3299672409799808,0,0,0,0.00002,0,0,0,1.526163191472839,1.526163191472839,1.5259778256839143,0,12.970000000000006,4.53,2021-11-17T20:59:13Z,2021-11-17T20:59:57Z,1600,2021-11-17T21:00:02Z,pm
 
1) The data as such (ie. timely snapshots of whole-market data) is good, but I miss the following fields (columns) as they are not part of the 1-min-data set (cf. the fields/columns table in the original posting):
Code: 
- last trade price
- last trade size
- last trade time
@Matt_ORATS told me that these fields are part of the 2-minute subscription plan.
Unfortunately, for my use case, the 2-min-data is a much different data set, and it also gets expensive to subscribe both.
I suggested to append these 3 numbers to the 1-min-data as new columns, but unfortunately it seems not doable at the moment :-(

2) I was told that the fields/columns of the 1-min data set are documented in this blog entry:
https://blog.orats.com/smoothing-options-implied-volatilities-using-orats-smv-system
[but it either is a little bit outdated or it covers also other data sets as many of the column names are different]

3) Another important observation: it seems that one has to get the data on the same day as there is no possibility to download the data of previous days!
Ie. "backfill" seems not possible.
It seems this limitation applies only when using standard download method. When instead using AWS cloud space then the data will be copied to there, so will be available always in own AWS cloud data space. Btw, I don't have AWS yet, so I couldn't test this feature yet.
[This is just my own experience/observation and preliminary interpretation as I haven't got a confirmation yet due to weekend]

.
 
Last edited:
Regarding my above mentioned problems with signing-up for Amazon AWS cloud space (S3 etc.):

This is the answer I got from Amazon support as reply to the support ticket I had opened
b/c of the bank card problem when I tried to sign up for Amazon AWS cloud space (S3 etc.).
Therein I also had asked whether payment via PayPal is possible instead of bank card.
(mail is of course sanitized by me with XXX to protect privacy)

Btw. they write below
Even though the $1 has been deducted from your account, it has not reached us. Your bank will refund these two transactions automatically.
More...

Since I tried it twice, they deducted $2 from my very bank card they say has a problem :)
They say by this $1 method they check the validity of the data of the customer... Blah..blah...
Btw, I used this card also when subscribing to ORATS data; it worked flawlessly.
And: I haven't got the $2 back yet... :) Very funny, I wonder if that trick (or fraud? :)) is the real reason how Jeff Bezos has become the richest (or 2nd richest) person on the planet... :)

There are 3 transactions from Amazon Web Services (or Aws Emea) logged in the activity log of my bank card acct: the first one is about checking the card, and the latter two have deducted $1 each. The 1st one has also this comment: "This was Amazon Web Services checking your card is active. Any charges it made will be returned on or before 24/11."

From: "no-reply-aws@amazon.com" <no-reply-aws@amazon.com>
To: XXXXXXXX@XXXXXXXX
Subject: RE:[CASE XXXXXXXX81] Bank card not accepted
Date: Thu, 18 Nov 2021 13:22:15 +0000

Hi XXXXXXXX,

I hope this email finds you well.

I had reached out to the relevant team reaching this issue and I was advised to provide you with the following statement:

=========================

AWS identified an issue preventing customers in some regions from using Visa cards during sign up or adding a new card for payment. We are investigating the issue and will post an update once it is resolved. Meanwhile, please retry with a non-Visa card.

=========================

The team also advise they have placed a banner on your Billing Console stating the same: https://console.aws.amazon.com/billing/home#/paymentmethods

Even though the $1 has been deducted from your account, it has not reached us. Your bank will refund these two transactions automatically.

Regrettably, we do not accept PayPal as a payment method.We only accept major debit/credits card.We will post an update once the issue has been resolved.

I apologies for any inconvenience this issue may cause.

We value your feedback. Please share your experience by rating this correspondence using the AWS Support Center link at the end of this correspondence. Each correspondence can also be rated by selecting the stars in top right corner of each correspondence within the AWS Support Center.

Best regards,
XXXXXXX XXXXX
Amazon Web Services

===============================================================
More...
 
Last edited:
The topic of this thread is, as said, my testing of the 15-minute delayed whole US options market data offered by ORATS for especially for Elitetrader users (ie. a special offer for ET users as advertised on that page https://info.orats.com/elitetrader ). On that page are multiple offers, I just am interested in the 15-min delayed data.

Unfortunately some drawbacks:

When not using AWS cloud space, ie. when using normal download method, then the 15-min delayed plan has only the possibility to download the data exactly at the time they are produced and published, ie. specifying "tradeDate=YYYYMMDDHHMM" in the URL just gets simply ignored and instead one gets the latest available 15-min-delayed-data file.
So, you have to be very exact at what times you issue the download requests.
This is a job for scheduled automatic downloads, ie. trying to do it manually is IMO problematic.
As said this limitation is the case when not using AWS. With AWS one just self-defines the download times and submits ORATS a short file containing these times so that their system can automatically copy the desired data to client's own AWS cloud space.

@Matt_ORATS, told me that for an extra $100 one can get the feature to overcome the above limitation with API usage. Ie. then the total price becomes $99 + $100 = $199/month. Of course the limitation of not making requests in shorter intervals than 15 minutes still persist, otherwise another fee of $50 to $299 per month occurs depending on the time interval (minutes) the subsequent requests are made. Citing Matt on this: "There is a Data API surcharge if you download every: 1min/$299, 5/$100, 15/$50, 1hr/free. If you use the API every 30 minutes, you do not have a surcharge."
Ie. I interpret this so: if one downloads in intervals > 15 then there is no such surcharge to pay.
Hmm. is that really > 15 minutes or is it rather >= 15 minutes, for the delayed 15-minute data?

[Matt, please correct me if I got you wrong somewhere as this all seems so complicated to understand for my little brain :)
Almost a week now and I still don't know the full specifics of the subscription I'm testing, as I get the specifics like some bread crumbs when I specifically ask for it. There seems to be no product document available that explains the product full and in detail. ORATS, you clearly have to improve on that :-)]
 
Last edited:
Btw, on the above said page with special offers for ET users, ie. at https://info.orats.com/elitetrader
they have also advertised these two subscription services:
Code: 
Data API Special $49 Month
  Unlimited symbols
  1,000 API Calls Per Month
  1 user
  For individual use

Data API Basic $99 Month
  20,000 API Calls Per Month

But, it is unfortunately unclear, at least to me, what these packages include in detail.
Is it just the permission to use the API and make that much API calls per month,
but that besides that, one needs also a data package subscription to be able to access via these very API packages?
Matt, please explain if you read this. Thx.

Ie. can I use this API subscription to get 15-min-delayed snapshot files? :)
Would be nice, b/c would be cheaper: $49 vs. $99 for the other subscription package (cf. original posting here).
But, I'm afraid I surely must somehow have misinterpreted these offers :-(
 
Last edited:
Regarding my above said problem with Amazon AWS S3 cloud space registration: today got this mail from their support, and tried again as suggested, and surprise surprise: NADA! Same shitty error happens, and has again deducted $1 from my bank card, and gave the same error as in the previous 2 attempts... Are they Idis @ Work? :)
From: "no-reply-aws@amazon.com" <no-reply-aws@amazon.com>
To: XXXXXXXX@XXXXXXXX
Subject: RE:[CASE XXXXXXXX81] Bank card not accepted
Date: Tue, 23 Nov 2021 13:47:22 +0000

Hello there,

As you know, recently we experienced an issue which prevented customers in certain regions from using Visa credit cards during sign up or when adding a new card to their account for payment. And there was at the moment no workaround to solve the issue.

As of November 20th, 6:19 AM PST, the issue has been resolved and customers are able to use any supported credit cards for payment. So in this case you can retry to use your credit card again now that the issue is resolved to go ahead and start using the resources.

I do want to thank you for your patience with this issue and for your collaboration with us!

We value your feedback. Please share your experience by rating this correspondence using the AWS Support Center link at the end of this correspondence. Each correspondence can also be rated by selecting the stars in top right corner of each correspondence within the AWS Support Center.

Best regards,
XXXXXXXX XXXXXXXX
Amazon Web Services
More...
 
Last edited:
The above said Amazon problem seems to have its root cause in the fact that my bank card has a security code against fraud (called CVV etc., there are many many names for it; see the wiki page below), but Amazon seems not to like such security codes, and tells me to talk to my bank to remove that security code! Unbelievable!!! See mail below!
https://en.wikipedia.org/wiki/Card_security_code

Amazon called me today by phone and then also sent me the following email:

From: "no-reply-aws@amazon.com" <no-reply-aws@amazon.com>
To: XXXXXXXX@XXXXXXXX
Subject: RE:[CASE XXXXXXXX81] Bank card not accepted
Date: Tue, 23 Nov 2021 16:09:04 +0000

Hello there,

As discussed in the call, we apologize for the inconvenience with the payment method registry process.

In this case you can definitely contact your bank to see if a CVV restriction lift is possible for this transaction, we appreciate your feedback and we'll take it in count in the future to improve our processes.

Here is some information that might be useful for this specific case:

https://aws.amazon.com/premiumsupport/knowledge-center/credit-card-declined/

https://aws.amazon.com/premiumsupport/knowledge-center/change-default-payment-method/

Again, thank you for your patience and collaboration.

We value your feedback. Please share your experience by rating this correspondence using the AWS Support Center link at the end of this correspondence. Each correspondence can also be rated by selecting the stars in top right corner of each correspondence within the AWS Support Center.

Best regards,
XXXXXXXX XXXXXXXX
Amazon Web Services
More...
 
Last edited:
You must log in or register to reply here.
Back
Top