I like the simplicity of your setup. I currently have about 100 GB of data w/ Dropbox. I'm not too worried about $150 on the remote machine, but whether it is solving a problem for me.
The question still remains whether the desktop setup is stable enough. In the past, I've had problems with machines dying when they've been on 24/7 which is really the main reason I chose to use someone else's infrastructure as they can avoid world ending hardware failures better than I can. However, the local machines that died weren't really high quality parts. I have very high quality parts in the machines I build today.
Yes, I think we get rid of the remote machine assuming no serious problems by end of this year.
Thanks for the chat buddy
Two issues here, the data storage and the cloud.
I've discussed the cloud stuff before, but briefly if machines breaking is your issue then you can stay local by buying another computer. I am also using headless NUC type units, and I have 3 of them that are high enough spec to run my system which cost about $500 each new, though I bought two of them secondhand (there are a few slower machines knocking around the house that I keep meaning to use for various hobby projects and never get round to). I use one for development, the other two are live and backup, and I swap them round regularly. In the last 6 years I've had one machine failure, so I really wouldn't trust one machine entirely.
Similarly I've had issues with backups, so I've gone to town with a RAID NAS drive on which everything backups every night, plus a USB drive that backs that up, plus everything is on at least 2 machines anyway. For belt and braces I should probably set up some offline storage like dropbox, and that's on my to do list. That also stores all our household data, and I can write the cost of all this stuff off against tax which I couldn't do if I was purely a trader.
Of course this doesn't help you if your internet fails, or your power fails... and that I guess is the appeal. $1000 of local hardware does buy you quite a lot of cloud computing time. I think local hardware is still cheaper but time I've done the maths it's got closer and closer depending on how many years you amortise your hardware over.
I'm considering containerising my new system when it eventually goes on line (that's basically when pysystemtrade is production ready), which will seriously reduce the up front hassle of moving everything to the cloud, so this is still something I might well do in the future. I would still want local copies of everything for persistence reaasons, and so I could spin it up locally if I wanted to, so I'd need at least one machine to do this on. It is also also pretty cool having a stack of computers though...
As for data storage, sqllite has done well for me and I think it's acceptable for low frequency trading, but I have had occasional issues with files becoming corrupted (writing when a process fails? it doesn't have the concept of a record lock, just uses the OS file lock). I'm planning to move everything to mongoDb / Arctic which is how the production side of pysystemtrade is set up. Using it for the last few years it's just a nicer solution once you get the nosql idea in your head, and a lot quicker. I'm a bit reluctant to rely on third party libraries (even if it's AHL!) but I could easily write a native mongoDb pandas read/write client if I had to in about 5 minutes.
The 'black box' nature of how stuff is stored in mongoDb slightly worries me however - you can do dump a backup file but if you can't recover from that file for whatever reason, you're stuffed. So I'm also planning to write backup files in .csv format so I can always manually recover from a corruption issue.
GAT
