Developing "Spartan"

[nooby_mcnoob]

I've begun to track ticks for all instruments, including options. This seems to be letting me get a true "feel" for the market in that I can gather intraday swings pretty accurately. For example, I rode the 300 SPY PUT today for a 30% gain, and did the same yesterday and the day before (with a different instrument). I am not putting too much money on this at the moment because I'm still trying to automate the process. Right now it's very manual.

Interesting chart of SPY midpoint vs bidask spread. No pattern here...

 

[HobbyTrading]

Interesting chart of SPY midpoint vs bidask spread. No pattern here...

Interesting. I did something similar quite a while back. I had the impression that for multiple instruments (single stocks, ETFs) the bid/ask spread started wide at the beginning of the day and tightened during the day. So I automated that data collection. And indeed: during the first 30~60 minutes of the day is the spread larger than the hours after that. Which was one of the reasons why I stopped trading near market open, and shifted to trading later during the day.

 

[nooby_mcnoob]

Interesting. I did something similar quite a while back. I had the impression that for multiple instruments (single stocks, ETFs) the bid/ask spread started wide at the beginning of the day and tightened during the day. So I automated that data collection. And indeed: during the first 30~60 minutes of the day is the spread larger than the hours after that. Which was one of the reasons why I stopped trading near market open, and shifted to trading later during the day.

Good point. There is an obvious pre-market/early day pattern. I don't mind trading during high bid/ask spread times as long as I don't have to get out right away.

 

[nooby_mcnoob]

Currently, Spartan tracks only 31 instruments and has a database of about 300MB. So this means that after 6 months, I can expect my DB to be between 5-8G in size at the current rate. Should be fine. But I can't back it up locally every hour, in that case.

This estimate was almost bang on, except I'm tracking many more instruments than I initially planned and so the database grows at 800MB/day. I plan to keep around 2 weeks of tick data for research purposes, so this works out to a database size of around 10G.

 

[nooby_mcnoob]

Been working for a couple of weeks on a new strategy that requires real-time tick data to work. Tested it manually, was ready to take it live today (on paper trading) and....... apparently I am gathering too much tick data.

I have a server that collects tick data for 60 instruments, and starting up a strategy that needs real-time data (aside from the data gathering) results in going over the 100 line limit for my account.

So this means I have to either reduce the number of instruments I track (nope, it really gives me a feel for everything) or bring up the use of 0mq. I'm opting for the latter, I expect it'll take till the end of this week as I have a couple of days to myself this week.

 

[nooby_mcnoob]

Been working for a couple of weeks on a new strategy that requires real-time tick data to work. Tested it manually, was ready to take it live today (on paper trading) and....... apparently I am gathering too much tick data.

I have a server that collects tick data for 60 instruments, and starting up a strategy that needs real-time data (aside from the data gathering) results in going over the 100 line limit for my account.

So this means I have to either reduce the number of instruments I track (nope, it really gives me a feel for everything) or bring up the use of 0mq. I'm opting for the latter, I expect it'll take till the end of this week as I have a couple of days to myself this week.

Took less time than I thought, only a couple of hours. 0mq isn't really that great programatically. If you really don't want spaghetti code, you have to use something like protocol buffers and that isn't a dependency I want/need right now. Overall, not happy with the code.

Pasting in a gist in case someone else wants to use it or point out problems. Note that it is incomplete:

 

[nooby_mcnoob]

Made a couple of updates to the real-time tick relay and I am now much happier with it. Basically, I changed it from being stateful to being stateless so either the server or client(s) can restart without losing too much data.

I also discovered that... The reason I was having the issue was that I didn't unsubscribe from market data properly.

Whoo what a dumbass.

Not sure what I'll do going forward, the old code can be swapped out but I think I'll leave it like this. It will let me simulate stuff as well.

 

[nooby_mcnoob]

I've been measuring latency compared to TWS and it looks like with all the garbage (IB -> TWS -> Python -> SSH -> Python -> Web) there is about a half second delay. I don't trade tick data but I'm pretty happy with that delay. The most obvious change I'd probably do is move everything to the server, for now I do it the above way just to avoid having to make changes to the server during the day.

 

[nooby_mcnoob]

Good luck, sir. 1 recommendation....don't tie yourself to QT. I made that mistake on my first iteration of a trading platform. I really like Qt, but I strongly recommend you use something browser based (bokeh) if you have any desire to host your software in a data center and monitor it remotely.

Very good point. Very little of the code is using qt so far, I think it would be possible to replace with a web frontend if needed later.

Just FYI, I did this the past week. Only a few hours of work. You were right about not sticking to Qt. The core is good, but when you need to do anything non-trivial (date picker?!!!), it gets ugly quick. Don't love web programming, but I figured out a way to write 99% of the logic in Python through Jupyter widgets.

 

[nooby_mcnoob]

Had to vertically partition the (SQLite) database into realtime and (relatively) static because updating the static tables would often lock the realtime tables. Took a LONG ASS TIME to figure out the magic incantation and even then, it's a huge hack. However, it works in that the realtime tables are lazily attached to the process, preventing locking the tables for realtime data. The chart of the number of locking errors tells the story (arrow indicates when I deployed the new code):

As for trading through my framework, things are looking pretty darn good. Right now, I have automated some trades I used to do manually and although I still confirm everything through a click, it will be very easy to flip it to fully automated. I just need more experience with it in various environments first.