Search results

Well, let me 'splan to you what's goin one here. I have gotten to the point when an old way of doing things (namely, storing data in text files) is getting a bit creaky. Storing data in a DB allows for more flexibility at the expense of added work for me and my junior. I was not sure which exact...

Thanks, useful stuff! Exactly. It's also likely that as the time goes, the needs will evolve.

Out of curiosity, how does the cost compare to kdb+? I got a chance to think about it today and discuss it a little. At this stage of the game, we don't have enough dense data to worry about "a proper solution", it's more about slightly more optimized reads/writes for backtests and such...

- If I go with a stand-alone software product, i might as well go with a proper time series database (kdb or maybe try that arctic thing)? - I already have an instance of Mongo storing things like security details, so Arctic makes sense from continuity of support perspective, no? - I am sure...

“Cargo cult” - have not heard this one since college :) Valid points. - Well, if HDF5 is a standard, why would I not want to sacrifice a little performance for flexibility? There is an active code base to use it from python, R and c++, and, if needed, I can find people (e.g. consultants) that...

I store both T/Q ticks (since I don’t play UHF games, no book updates thank God) and resample to 1 second and 1 min. Why would Bcolz be better? Is it faster?

It's on my radar, but I don't know if I can supporting yet another stand-alone product when I can get away with a simpler solution. What made you pick this vs say any other time series databases (arctic looks pretty impressive, I'd say)? Yeah, that's for sure - upon some thought, I am leaning...

Makes sense. The added advantage of figuring out an open source solution is that should I decide to move to another firm (or should it be be decided for me, for some reason :() I can quickly re-deploy the same technology stack. So far, given how small the team is and how little resources we are...

Probably talking a 100GB of data at the very very most if I do add other strategies I am planning to add. Yup, unfortunately It is "novel" in a sense that it's high-effort from modeling perspective. In any case, when I say "latency-sensitive" I don't mean UHF, I simply mean strategies that...

I guess I have to do some real cost/benefit analysis before I pick a direction. The inputs are as follows: - very small team, one techy young monkey and one useless old fart that can barely boot up his MacBook; so we can’t support heavy technology stack - small number of assets that do require...

Pretty much everything I do with this data is case 2 in one or another form.

Relational databases are not very good at handing large amounts of columnar data at high speeds (at least, most are not - apparently, there is a brand new TSDB that is optimized for it). They are very good for other things, though.

This looks very good, actually. PS. Bcolz looks very straight forward, this looks more "enterprize ready". Need to think about the various aspects, for a tiny team

Now we are talking value added! Thanks a bunch! I've heard of bcolz but never used it

Oh, no, that was not the question - I got the data already (it's a mess). I need the database engine to store it all, the text files are starting to feel a bit heavy :)

He means I should find out what engine they are using and piggyback on their due diligence. Smart idea, since they have probably put more time into researching the platform components than I ever could. Yeah, I looked at them a while ago. Very expensive for what they are trying to sell. Plus...

Well, it's just one guess - the curve is eerily smooth for anything that's not a proper high frequency strategy. There is a number of possible issues that could cause this (I'll go through some ideas shortly), but rehashing them here would be a bit too public. PS. Just tried sending you a PM -...

Are you sure there is no lookahead bias of some sort in there? How many trades is that?

I think you’re going all right :)

Hmm. They are using some form of python interface, right, even for tick data? Have you used it? Ps. They don’t seem to publicly mention what they use to store the data, unfortunately