Why use a database?

prophet · Oct 15, 2004

Quote from linuxtrader:
Well, the hallmark of a system headed for the dumpster is starting out without knowing what you intend to build, how long it will be in service and the points with repsect to system load or problem space where it becomes useless or invalid.

âKnowing what you intend to buildâ?? Then we must be masters of all markets, with our crystal balls able to foresee the design of a profitable system before any testing or simulation is actually done.

Trading systems require just as much discipline as any other software system if you expect to build anything of long term value.

Discipline balanced with flexibility and adaptability. Markets change. Systems must adapt.

Building high quality software that is extensible and high performance is just not that difficult, nor is it very expensive if you know what you are doing.

Agreed!

I guess we will agree to disagree.
Have a great weekend ....

Same too you and Sparohok. Have a great weekend and successful trading in the future!

prophet · Oct 15, 2004

Quote from Sparohok:

You're right, incompetence or lack of knowledge or lack of dedication can lead to bad strategies. I was just trying to point out the converse, bad strategies do not mean incompetant trader. Even the best quant trader will have many bad ideas which is why we need to test our ideas extensively and creatively, while constantly being cautious of overfitting.
Martin

I agree completely.

John Goodfellow · Oct 16, 2004

hi everybody!

I'm working on a similar problem-creating a tick by tick database for market analysis.

In your experience, and what is the best data feed for this?

MY SELECTION CRITERIA:
1. I am price-sensitive.
2. I would be happy to start with limited range of stocks - i.e. 300 active issues form NYSE.
3. I prefer Internet delivery rather than dedicated line.

Please tell me what the data feeds you use and what pros and cons you've experienced.

How do you get data form the network into your DB/ Flat file?

Thank you all for replying.

John Goodfellow · Oct 16, 2004

Oops!

Forgot to mention:
I need streaming real-time stuff, not end-of-day or1 minute or snapshots.

Sparohok · Oct 16, 2004

I'm in the same spot, I'm just starting to capture tick data. So I hardly have all the answers. But I've heard multiple people recommend IQFeed for reliability and data quality. However they have a $300 fee to get the API which is really irritating. For now I am just downloading data through QuoteTracker's HTTP server using a Python script and storing it in a PostgreSQL database.

In my experience, broker quote APIs do not provide sufficiently reliable data, few symbols, and no backfill for gaps.

Martin

prophet · Oct 17, 2004

Quote from Sparohok:

In my experience, broker quote APIs do not provide sufficiently reliable data, few symbols, and no backfill for gaps.

There are exceptions on some of those points. IB's real time data is excellent in terms of accuracy, reliability, speed and cost. You'll have to provide your own backfill.

nitro · Feb 3, 2005

Quote from inflector:

...There is a huge difference in read time between a database and a binary file unless the database has been specifically optimized for large binary data storage (known as BLOBs in the business).

The reason is simple, even in a database with an efficient caching mechanism large data sets generally involve multiple reads from the disk because the data is split up into chuncks. Every separate read will take a while because on average it will require 1/2 of a rotation of the disk before the data comes under the read heads so the read can start.
- Curtis

I have actually done some interesting tests on some of this. It turns out that a RAIDed SATA solution worked best on "BLOBs" like movies. RAIDed SCSI still rules on the row type operations (or column operations done now by some dbs) of data stored in relational tables.

nitro

CoolTrader · Feb 3, 2005

Quote from nitro:

I have actually done some interesting tests on some of this. It turns out that a RAIDed SATA solution worked best on "BLOBs" like movies. RAIDed SCSI still rules on the row type operations (or column operations done now by some dbs) of data stored in relational tables.

nitro

I prefer a small db, the entire db can be read into memory.

Joel Reymont · Jul 27, 2005

I'm building "KDB for the rest of us" and the core will be open source. Visit http://wagerlabs.com/uptick if you are interested and click on "forum" in the blog headline to join the discussion group. I'm looking to process, say, 1000 ticks per second but my main emphasis will be on similarity search in time series data, i.e. being able to select a portion of a chart and find stocks that match the pattern.

nitro · Jul 27, 2005

Outstanding site. Thanks for the link and good luck with your project. If it works I would be interested...

nitro

Quote from Joel Reymont:

I'm building "KDB for the rest of us" and the core will be open source. Visit http://wagerlabs.com/uptick if you are interested and click on "forum" in the blog headline to join the discussion group. I'm looking to process, say, 1000 ticks per second but my main emphasis will be on similarity search in time series data, i.e. being able to select a portion of a chart and find stocks that match the pattern.