SqlLite?

[Volker Butzlaff]

@Januson,

my little freeware is a (non realtime) analyzing tool. The advantage is in programming productivity. I can avoid conventional programming code (arrays, ifs, sorts etc.) using the existing SQL Toolbox commands. And using the in-memory option these analysis is nearly as fast as normal programmed code.

Sample:

slSel "Select pattern, count(*) From ZenReport " & _
"Where pattern Like 'white%'" & _
"Group by pattern ", 1

(* the table zenReport is build before reading the ASCII input price file and writing the content in a SQLite table)

This statement (slSel) alone avoids loop, if conditions and sort/grouping algorithms in a high level programing language like C/C++ (or in my case PowerBasic).


But you are right:
realtime performance isn't normally the reason for using SQL. There are far faster and simpler DB alternatives for extrem fast indexed or sequential data processing - e.g. the good old B+Tree components like thisone:
http://www.powerbasic.com/products/powertree/

bye,
Volker

 

[PocketChange]

What performance metrics are you considering?

Writing ticks from your data feed handler into an in Memory DB serving a few local servers to process or distribute the feed in real time into other BI systems?

Takes less time to capture and store tick data to an in memory DB than logging ticks out to CSV or any binary disk format.
The single file format isn't the most space efficient but can be used without any further import/mapping etc. Makes a nice application file format without any other dependencies.

From my point of view this format is better for logging tick data because it can be used instantly for synchronized instrument queries. Even the very fast KDB has issues when you want to do any multi instrument sequentially synchronized queries. ie. Build and analyze in real time an indicator using YM and the top 80% of its underlying components.

 

[tradelink]

"Takes less time to capture and store tick data to an in memory XXXXX than logging ticks out to CSV or any binary disk format. "

Any data source will generally write faster if you do it in memory versus the same source on disk.

This would be true for sql, nosql, flat file, etc.

 

[PocketChange]

Sure... the point i was trying to make was the inmemory data is instantly available for queries and use without any additional manipulation.

Hardware and Ram are cheap... one can easily operate intraday entirely in memory while running a background thread to backup the DB to disk. Worse case you may lose a minute of tick data on a forced reboot.

Quote from tradelink:

"Takes less time to capture and store tick data to an in memory XXXXX than logging ticks out to CSV or any binary disk format. "

Any data source will generally write faster if you do it in memory versus the same source on disk.

This would be true for sql, nosql, flat file, etc.