The technology has changed considerably.
Your stats may be accurate for single individual drives but using a program like fancycache can yield a 1000x performance boost.
http://www.romexsoftware.com/en-us/fancy-cache/index.html
Infiniband / Striping / hybrid arrays of SSD + disks / clustering etc are all consumer cost reachable technologies today.
1 GB per second sustained read/writes is doable today and with clustering over infiniband 5gb/sec is attainable while remaining affordable.
Even some of the journaled posts on index settings and speed may not accurately reflect the hardware impact.
Try these simple changes for a 1000x boost
For DB Use covering indexes\ 64k page size (cuts read i/o in half and better cache hit rates)
For Disk use Fancycache setup with write only cache (1gb 512k blocks write deferred 2 sec) this buffers writes to achieve near SATA 3 spec speeds. Deferring the writes for 2 secs increases cache read hits and reduces hammering of i/o write operations... less fragmented sequentially writes of random data... which in turn optimizes read i/o.
CAUTION: Write deferred caching can cause serious data corruption on power failure/system lock ups.
Your stats may be accurate for single individual drives but using a program like fancycache can yield a 1000x performance boost.
http://www.romexsoftware.com/en-us/fancy-cache/index.html
Infiniband / Striping / hybrid arrays of SSD + disks / clustering etc are all consumer cost reachable technologies today.
1 GB per second sustained read/writes is doable today and with clustering over infiniband 5gb/sec is attainable while remaining affordable.
Even some of the journaled posts on index settings and speed may not accurately reflect the hardware impact.
Try these simple changes for a 1000x boost
For DB Use covering indexes\ 64k page size (cuts read i/o in half and better cache hit rates)
For Disk use Fancycache setup with write only cache (1gb 512k blocks write deferred 2 sec) this buffers writes to achieve near SATA 3 spec speeds. Deferring the writes for 2 secs increases cache read hits and reduces hammering of i/o write operations... less fragmented sequentially writes of random data... which in turn optimizes read i/o.
CAUTION: Write deferred caching can cause serious data corruption on power failure/system lock ups.
Quote from inflector:
I've got an eclectic background. Started programming in high school over 20 years ago writing futures trading systems. Had a bit of fame in my early twenties as a trader and then left for 15 years to start a few software companies.
One of them sold an embedded database which was the number one product on the Macintosh. I worked on the internals, disk access, etc. as well as the query optimization.
There is a huge difference in read time between a database and a binary file unless the database has been specifically optimized for large binary data storage (known as BLOBs in the business).
The reason is simple, even in a database with an efficient caching mechanism large data sets generally involve multiple reads from the disk because the data is split up into chuncks. Every separate read will take a while because on average it will require 1/2 of a rotation of the disk before the data comes under the read heads so the read can start.
Unlike almost every other aspect of computing, disk speeds have not followed Moore's Law. Disks are maybe 30 to 100 times faster than they were 20 years ago while computers are 10,000 times faster.
Even a 10,000 RPM disk takes 6 milliseconds to rotate. So you only get 167 rotations per second. That's a lot of time when computers are doing billions of instructions per second.
For tick data analysis the speed of reading the data is the determining factor for the speed of testing unless you have very inefficient code or are doing esoteric analysis.
So I suggest storing information about your data in a database but storing the physical data on the disk in raw binary files.
You can get acceptable performance from a database if you know what you are doing, however, you will always pay a performance penalty.
- Curtis