new software tool development for technical Analysis of Quot data

A

aqtrader

Hi,

I have been looking around but never found a good technical analysis software package that can satisfy my need. I am a software engineer. I decided write a tool myself. I intend to make this tool as an open source software, so everyone will enjoy for free.

This is one of my hobby projects.

Firstly, what this tool can do? The goal is
1. easy stock/equity data technical analysis - many other famous software or programming language claimed to be able to do this such as MatLab, R language, etc, but not designed for stock data in the first place, so very difficult to use;
2. support EOD (end of day) and intra-day stock data analysis;
3. support fundamentals analysis such as PE, growth, etc;
4. easy to define indicators that use any length of historical data, and/or intraday data plus some performance numbers - I need a new language for this.
5. easy to filter out trash - kind of data mining for useful information;
6. able to handle "big" data fast - not at the really big level for petabytes but good at gigabytes level for handling 5,000 symbols for 20 years stock data in a regular good laptop or a desktop computer running Linux. Sorry, I am not a Windows guy.
7. etc - open for suggestions

How do I implement it?
Ok, I will not start from scratch. I already have a lot of software pieces but not good enough. Basically, I will do:
1. aqdb: define a new database scheme/format called aqdb
2. aqlang: define a new computer language - specialized stock data analysis functional language called aqlang
3. aq: write software program to manipulate aqdb and aqlang
4. git-hub: share it as an open-source project when I am done.

What do I expect from ET community?
I post here because I think someone here may give some good suggestions and save my time although most of ETers may never understand what I am doing here.

I will try to do my best to post my progress in this thread. I do not consider this tool as "trading software", so not post this at "Trading Software" category, because this software is not really going to tell you any kind of trading edges or trading strategies. Finding a trading edge or strategy is your job to use or not to use some software tools or any tools whatever.

Thanks for reading this.

- aqtrader
 
ok eod data can be had from yahoo
intradya us stocks/fx from freestockcharts.com
also free data from dukascopy
mjt4 can export data and that is free
also mt5
so between all those ,there is plenty to work on
once you have the dat ,then write the software
 
Quote from dentist007:

ok eod data can be had from yahoo
intradya us stocks/fx from freestockcharts.com
also free data from dukascopy
mjt4 can export data and that is free
also mt5
so between all those ,there is plenty to work on
once you have the dat ,then write the software
More...

Thanks for telling me more data sources. I have been downloading eod data from Yahoo and collected some 1 min intra-day data from other sources. The quality of raw eod data from Yahoo or any other places has a big issue of bogus low/high due to out-of-band trading or other reason that will defeat back-testing if not handled properly. This issue is out of topic on my TA tool development.
 
you can get a free NinjTrader program with, I think, free data feed, program wouldn't
be much use without a feed: http://www.ninjatrader.com/

MetaTrader 4 is free, primarily a forex charting/order entry program with realtime feed
some brokers do have futures, some have CFDs and some stocks
1 5 15 60 240 D W M price data is available. the price data from each timeframe can
also be downloaded in ASCII format: Menu/File/Save of the chart currently displayed
personally at a development stage I wouldn't care about which instruments I was
using - stocks, futures or fx pairs, that could be refined later, main thing is having a
dependable free data source
MT4 is used by fx brokers world wide but different country's regulations permit or not
instruments other than fx to be traded
try Oanda's MT4, in Canada or the UK it'll include cfds but not in the US:
http://fxtrade.oanda.com/trade-forex/metatrader/
 
Quote from Wallace:

you can get a free NinjTrader program with, I think, free data feed, program wouldn't
be much use without a feed: http://www.ninjatrader.com/

http://fxtrade.oanda.com/trade-forex/metatrader/
More...
Agree. a program is useless in trading without quality data feed. Well, I will take a look at NiniTrader program. But for TA, either offline or online in real-time mode, I want to have a tool that is very flexible. The goal is not to replace any of trading systems or programs, but to assist other trading programs and curious people who want to figure out more from huge historical or current data, for example, to answer some curious TA questions about market behaviors that is hard to find by other programs. Thanks for your input.
 
Quote from bashatrader:

Sound to me like a saturated market you are looking at with diminishing returns.
More...

Read my post again. You will see the goal is not for any return. I do not care someone gains or losses in whatever market. You do not need any tools to find your holy grail if you are smart enough. The gaol is to make an open source free software tool that does some TA in a better way or a new way.

Anyway, I am still on this tool. Slowly but steadily making some progress. I do this only in weekend.

Here are what I am considering about aqdb database design

Ignore the following if you are not interested in details.

Major implementation considerations

  • Speed performance - use multi-core vector CPU
    I have a decent workstation Dell T3600 Intel Xeon CPU E5-1650 @ 3.2 GHz RAM 16.0 GB 6/12 CPUs
  • Memory usage - in memory database
    Rough analysis:
    120 GB memory is required for 5,000 syms of 20 years 1 minute data
    assuming each data point is encoded to consume only 24 bytes.
    600 MB memory is required for 5,000 syms of 20 years EOD data

Scope

  • EOD and 1-min data: do not cover tick data
  • Generic tables as value tables: earning table, dividend table, etc
  • Generic profile information as value groups, etc

Major technical differentiator

Timed vector: Vectors/Arrays are always inherently tied to a date/time vector, so in aqlang, a vector is actually a timed vector. This builtin fundamental feature enables aqtools specially optimized for considerable conveniences and optimizations to execute certain applications on stock data analysis.

Data quality issues

  1. Adjustments: Dividend adjustment, especially some big payouts
  2. Split - stock split and reverse split adjustment
  3. Out-of-band fill - (bogus high and low) adjustment
  4. Historical syms - syms that no longer existing due to acquisition, merge, or bankruptcy.
    [/list=1]
 
I am in the middle of coding adjustment of historical quote. I am trying to understand what is the proper way to deal with stock split and dividend payout. It seems not really simple as it looks. I think it is ok to treat stock split proportionallly for previous prices and volumes. But I am having difficulty to deal with dividends in adjusted prices. Look at Yahoo Financial, the impact of a dividend payout gradually phases out on previous historical prices (so we do not see negative prices if the accumulated dividend amount is more than the stock prices). I do not understand what rationale is for doing that way. I can do the same like converting a dividend payout to a stock split ratio so as to deal with dividend in the same way as for stock split. Anyone has a good idea? Thanks in advance.
Why do I care about the impact of dividend on prices movement? It is because there are so many cases a company pays really sizeable dividend and you see the prices drops huge accordingly but not really get de-valued.
 
You must log in or register to reply here.
Back
Top