Developing "Spartan"

As time has passed, I've been progressively including more and more functionality into Spartan. Initially, I was only tracking forex but have started tracking stocks and client sentiment data as well.

I've found it invaluable to have my own source of data. In the long term, I'll continue to do this no matter what. Why? It's fun.

Things I want to add:

- Tracking option chains: This is probably doable under the current architecture, just treat price of each contract as a standalone instrument, similarly with greeks (or maybe those would be backed out algorithmically using BS or whatever else)

- Earnings data: estimated, initial, updated
 
Just keeping this here as a note for myself, eventually will start storing it and see relations to daily returns:

Code: 
curl -q -o - https://api.stocktwits.com/api/2/streams/trending.json 2> /dev/null | jq '.messages | .[] | .symbols | .[] | .symbol' | sort | uniq -c | sort -nr
 
First week where IB didn't randomly go down and stay down for hours. Though it did of course randomly stop publishing quotes as you can see in the "number of quotes per 10 minute period. Sometimes I wonder what they do over there.

upload_2019-7-6_22-24-12.png
 
I want to start storing option data because there is a definite handicap for me without the ability to play with historical data. Obviously, I'm not going to store tick data. I don't think it makes sense to store every strike, since I won't be looking at holding options for longer than a few weeks.

The interface I'd like to have for beginning to track this data would be to create a new row in the instruments table, something like:

Code: 
insert into instrument(symbol,type) values('SPY','OPTION')

Then Spartan would automagically begin tracking contracts and the underlying for up to 6 months in future with a 10% interval on either side. 10% because in my data, SPY moves a median of 5% for any given 6 month period in the last decade so this should be OK for my immediate purposes.

In terms of expiration, it looks like there are ~6+5 contracts for each strike (50 strikes for SPY) over a 6 month period if you include weeklies. I would also track some greeks as they come in from the data feed.

So this probably works out to hourly bars for 11*50 new contracts just to track SPY. Currently, Spartan tracks only 31 instruments and has a database of about 300MB. So this means that after 6 months, I can expect my DB to be between 5-8G in size at the current rate. Should be fine. But I can't back it up locally every hour, in that case.

In comparison, AMD/INTC have ~10 strikes to track so over a 6 month period, they would add a relatively smaller amount.

In terms of the table layout, I'd probably have a child table, something simple like this specifying the contracts:

* instrument_id
* parent_instrument_id (i.e., the underlying)
* expiry_date
* strike
* is_call

Then I can re-use the rest of the mechanisms for pulling and storing the data. I'd likely have a separate table for option-specific data attributes like the greeks even though I could probably re-use the mechanism used for storing prices.
 
Compounding

One of the things that is very important to me is to compound on work that I've done before. As I mentioned elsewhere on the site I have begun to use a desktop wiki (moinmoin) to track my research.

I've been experimenting with a trading journal on the wiki and one of the steps in that I've got in prep work for the day is a check list, where one of the items is to check the economic calendar. Previously, I had a link to investing.com's calendar but I wanted it to be more efficient than a link, I wanted to embed it to avoid having to visit a new site. Unfortunately, investing.com does not allow embedding in an iframe but I realized I've actually been tracking economic calendar data for a while now. A little bit of code later, and it's available inline in my journal template. This updates dynamically every time I visit the page and the entire purpose is to keep me efficient. The ugly, hacky code is below.

upload_2019-7-18_10-29-1.png



Code: 
import sqlite3
import pandas as pd
import os.path

dbpath = os.path.expanduser("~/src/noobymcnoob/spartan/run/db.sqlite")

def highlight_todays_date(df):
  if df['diff'] < 0:
  return ['background-color:lightgrey; opacity: 0.25']*len(df)
  if df['diff'] >= 1:
  return ['opacity: 0.25']*len(df)
  else:
  return ['']*len(df)

def execute(macro,args):
  currencies = '(select distinct currency from economic_event)'
  if args is not None and args != "":
  currencies = args.split(',')
  currencies = ["'" + c + "'" for c in currencies]
  currencies = "("+",".join(currencies)+")"
  conn = sqlite3.connect(dbpath)
  try:
  df = pd.read_sql("select julianday(datetime)-julianday('now','localtime') as diff,date(datetime) as date,time(datetime) as time,currency,type,impact,description,forecast,actual from economic_event where currency in %(currencies)s and datetime(datetime) BETWEEN datetime('now','-24 hours') AND datetime('now','24 hours') order by datetime asc" % dict(currencies=currencies),
  conn)
  html = ",".join(df.currency.unique()) + df.style.hide_index().hide_columns(['diff']).apply(highlight_todays_date,axis=1).render()
  return html
  finally:
  conn.close()

if __name__ == "__main__":
  print(execute(None,None))
 
nooby_mcnoob said:
I want to start storing option data because there is a definite handicap for me without the ability to play with historical data. Obviously, I'm not going to store tick data. I don't think it makes sense to store every strike, since I won't be looking at holding options for longer than a few weeks.

The interface I'd like to have for beginning to track this data would be to create a new row in the instruments table, something like:

Code: 
insert into instrument(symbol,type) values('SPY','OPTION')

Then Spartan would automagically begin tracking contracts and the underlying for up to 6 months in future with a 10% interval on either side. 10% because in my data, SPY moves a median of 5% for any given 6 month period in the last decade so this should be OK for my immediate purposes.

In terms of expiration, it looks like there are ~6+5 contracts for each strike (50 strikes for SPY) over a 6 month period if you include weeklies. I would also track some greeks as they come in from the data feed.

So this probably works out to hourly bars for 11*50 new contracts just to track SPY. Currently, Spartan tracks only 31 instruments and has a database of about 300MB. So this means that after 6 months, I can expect my DB to be between 5-8G in size at the current rate. Should be fine. But I can't back it up locally every hour, in that case.

In comparison, AMD/INTC have ~10 strikes to track so over a 6 month period, they would add a relatively smaller amount.

In terms of the table layout, I'd probably have a child table, something simple like this specifying the contracts:

* instrument_id
* parent_instrument_id (i.e., the underlying)
* expiry_date
* strike
* is_call

Then I can re-use the rest of the mechanisms for pulling and storing the data. I'd likely have a separate table for option-specific data attributes like the greeks even though I could probably re-use the mechanism used for storing prices.
More...

This was much easier than I expected, less than 100 lines of code. Every morning, Spartan will pull down all strikes expiring within the next 10 days that are within 1 ATR on either side of the last closing price.

Next, I am going to be keeping 2 weeks of tick data around for all instruments. That will be fun I'm sure!

Currently tracked options (edit looks like it was still updating):

upload_2019-7-20_9-45-29.png
 
Last edited:
You must log in or register to reply here.
Back
Top