Search results

We tested SQLite too. It is very good for small databases who can fit in the memory. It very popular because it is free and with very small footprint. This is the main reason it is fast. If you have more data and more columns they would not fit in the memory because SQLite is still a...

SSD disks right now are reported to deteriorate with time and the access time to increase after usage. It is still not clear why but this info is not popularized. They still expensive and indeed storage logic is half of the story. If you format your disk properly - with right sector size and...

Column oriented solves only the reading from the disk problem - to read only the needed data and read them faster because of better compression. This works only if you have an algorithm for faster decompression, better if it is a parallel decompression. Yes they have to be properly grouped...

It is no so simple as that when you are trying to do an actual work. The concept of column oriented is only a part of the picture. It solves the problem of accessing only needed columns. But a pure column oriented DB is a simplification. Actually the data has to be column oriented but...

Yes but you can store as one column in a separate file. That way if you want a sliding window on C(Closing price) you load the file with just C(Closing Price) not all columns stored in this file. It compresses better because you have similar values and thus load faster. It requires less...

In fact internally the database is reading all the columns, then using only the columns you need in the query. Because the database is stored on the disk and reading from the disk is the slowest operation, this is why it is so slow. If you don't believe you can make some benchmarks and see...

Yes - this is possible with SQL but it is slow because of these joins. The main reason COTS databases are slow is because they are designed for online transaction processing - e.g. row oriented. If you have a table with HOLCV in 5 different columns every time you perform for example a sliding...

If we work for example with EOD data, sliding window is when you want for example to calculate a regression on 360 days back. 360 days is your window. First you calculate the 360 days window regression from current day, store the result together with the current day, then slide the 360 days...

I have been through this. Had to test a strategy and needed a lot of data (EOD) which is 20,212 trading days since 1928 for DJIA. MS SQL do not have regression, sliding windows, etc. so I coded in in VB.NET and used MS SQL for data storage. The algorithm generates ~1,000 columns so it turns out...

Trading Technologies claims it accounts for more "50 percent of the combined electronic volume on the world’s six leading derivatives exchanges". And a patent charging you on every transaction back and forth.

Where you get these data from?

Another problem with web based stock trading is this: "A relatively unknown data-stealing Trojan horse program that has claimed more than a quarter-million victims in the span of a few months aptly illustrates the sophistication of modern malware and the importance of a multi-layered approach...

If this is the best software available what can I say ;) I agree about the server congestion. I was commenting on the desktop client only. I also had a very bad real experience with server congestion - I made my research and prepared for the trade - an options trade. I was calculating my...

I am using Schwab StreetSmart Pro Desktop on MS Vista and also tested the web Java based StreeSmart Pro. The desktop version is fast and stable, web version freezes and is very unstable. Both versions have congestion server based problem in moments when the stock is traded with high...

Yes, this in one advantage no much people are knowing about - 40/60. Also DJX is the first DJ related ETF options - from 1997 and have a good liquidity. I think these are what most good hedge funds are trading by one simple reason - they need a lot of liquidity. Also it is very hard to...

Yes, also the Dow Jones related ETFs have a liquid options. So we have about 10 Instruments at a time. The mix changes but my observations are these.

I suppose IB are outsourcing too? ;)

This is an opportunity to have the key but not the key itself. A robust and stable trading platform with this huge amount of data is very hard to have even if you develop it itself and control the things. Also what trades you decide to perform depends from the historical research you do. I...

If you are looking for EOD - CSI have the best data quality.

;) Yes I am thinking it would be an interesting thread. You can start it. Probably the topic should be more general - something like "tick by tick data and sampling"? This is a great observation. The problem I see is nobody knows for sure how exactly this is handled. Now, even with...