Quote from Graham1:
Is the .tck file it is reading from a flat file or a database file ? I presume most users would want their data collected into some kind of database to make ongoing data updates easier and simulation runs involving subsets of the data more efficient.
Graham, thanks for asking. I know how you feel. I assumed the same thing originally and, as a database speed guru, I thought something would work.
Problem is that when reading 10Million ticks, if the database takes more then
1 microsecond to read each tick, then it will take foreever to load them.
At 10 microseconds, it takes 100 seconds or almost 2 minutes to load the file.
I found that even the fastest possible with OODBMS was about 30 to 50 microseconds.
Now, that's extremely fast compared to RDBMS but it's not fast enough for my taste at reading ticks.
Problem with those is they have overhead since they know how to do alot more than handle ticks.
I haven't completely give up on OODBMS. Especially if we use one that's open source.
Question is, will it be easier to just change on OODBMS to turn off all the functionality we don't need or just make our own data organization.
Noone has come up with any other query we need to do other than Symbol and time range.
So if we simply make a separate folder for each Symbol. And drop tick files in there which each have a "header" with the offset to every 10,000 tick boundary, that will be very fast and easy to work with since you just drop files there whether they cover 1 year, 6 months, 3 months, etc.
That's my thought at present and I'm open to well tested ideas in that area.
Sincerely,
Wayne
