crossposting: do you have a tip to get the tickers from indexes

Ivano

Ivano

In Alpaca i used to have an API call to get the Nasdaq tickers with IB API do not have them and if I call a ticker from the NASDAQ site that is made for options/Warrant or is unlisted i get an error that block all the batch i am loading, which is really frustrating because need to implement resume strategies, parsing the dataframes in python and manually get the daily list. Do you have better recommendation to automate the process so that can easily automate the delta.
https://www.elitetrader.com/et/threads/best-nasdaq-nyse-ticker-list-that-is-ib-api-friendly.377540/
 
To help automate symbol retrieval, you could try something like:
Code: 
curl --show-error --insecure --silent --location --max-redirs 10 --max-time 30 'ftp://ftp.nasdaqtrader.com/symboldirectory/nasdaqtraded.txt' > nasdaqtraded.csv
You would need to parse out undesired fields and symbols. For example symbols for warrants have more than 4 characters and end in W or .W

See also
https://www.nasdaqtrader.com/trader.aspx?id=symboldirdefs#nasdaq
https://www.nasdaqtrader.com/conten...taproducts/nasdaqfifthcharactersuffixlist.pdf
https://curl.se/docs/manpage.html
 
ph1l said:
You would need to parse out undesired fields and symbols. For example symbols for warrants have more than 4 characters and end in W or .W
More...
I made it, pigs gonna get slaughtered!

Did not know about the ftp, thank you. Will look next days in future for delta updates, although maybe are present also AMEX data(which i could easily subscribe)

Also I did not think tickers had between 1 and 4 characters LOL, would have been much easier, i ended up implementing business logic checking with regular expressions warrant, Warrant, Warrants in the name/description field.

Boosted a lot my confidence being able to do it, still I think python is ugly, but implemented my custom conventions to architecture some clean code.

Most challenging task has been checking for consistency when I stopped the connection and make retry afterwards to not incur in data limitations. Worked on concurrency, batches and nice workarounds.

I stopped candles in *.CSV, but i see a postgree SQL DB with some satellite star strategy as the way to move further to craft my custom indicators, should be better than NOSQL for my plans

Looking forward to work my ideas,strategies, filters, backtests, and add options and fundamentals data and possibly update my m1 mac pro to compete with fellow Jim and Ken.
 
Last edited:
Anybody know a list that shows all the currently available options expiration dates, incl. LEAPS, for all listed US tickers w/ options?
Something like:
Code: 
tickerABC,expDate1
tickerABC,expDate2
tickerABC,expDate3...
...
tickerXYZ,expDate1
tickerXYZ,expDate2
tickerXYZ,expDate3...

Update: found it! One can extract it from this csv file (delimiter is the pipe sign '|'): ftp://ftp.nasdaqtrader.com/symboldirectory/options.txt
I hope it contains all listed US tickers, this not tested yet.
 
Last edited:
I have scraped a dataframe before of the Russell 3000 tickers and then had a column of true/false if that ticker is part of a major index. Then it is trivial to create some functions to return whatever index you want.
From doing this though I realize I don't really care about the bottom 2000 tickers and just look at IWM for that. Then I do care about the Russell 1000 and what ticker is part of what major index. More interesting is what is part of the Russell 1000 but not in the S&P 500.
I just don't see the point of SQL here. A more robust solution to a problem you don't really have.
I just think it is hard to beat a query on dataframes with a datetime index if the analysis is going to end up in a dataframe anyway.
I haven't worked with IB API though. Anything I have used has been trivial to get all symbols but I could see all symbols on IB being a bit much to say the least. Such a huge percentage of that though is going to be illiquid instruments that you will never bother with.
Based on the thread though I am still not sure what you are looking for exactly.

Ivano said:
thank you I already implemented in the past this solution scrape+yahoo following a tutorial from youtube(i guess) but wanted something more complex
More...
 
metalztrader said:
I have scraped a dataframe before of the Russell 3000 tickers and then had a column of true/false if that ticker is part of a major index. Then it is trivial to create some functions to return whatever index you want.
From doing this though I realize I don't really care about the bottom 2000 tickers and just look at IWM for that. Then I do care about the Russell 1000 and what ticker is part of what major index. More interesting is what is part of the Russell 1000 but not in the S&P 500.
I just don't see the point of SQL here. A more robust solution to a problem you don't really have.
I just think it is hard to beat a query on dataframes with a datetime index if the analysis is going to end up in a dataframe anyway.
I haven't worked with IB API though. Anything I have used has been trivial to get all symbols but I could see all symbols on IB being a bit much to say the least. Such a huge percentage of that though is going to be illiquid instruments that you will never bother with.
Based on the thread though I am still not sure what you are looking for exactly.
More...

good moment to take care for IWV, a boolean or a 1-0 equivalent is the way to go for databases or whatever is your upstream source of data
I do not know how much you are familiar with algorithms and data structures, every use case depends, arrays, dataframes, list, push/pop, graphs, SQL,no SQL, LinkedList. Is all about big O

a dataframe vs a database are two different kind of beasts. Let's say you have ohlcv and then you want 30 indicators, and fundamental data, and you need to operate so called joins to merge two or more tables and want also to store performance, would you still more comfortable with datasets?


I would not be so comfortable if you need to establish relationships, if you need proven ways for querying and analysis on data aggregations, scalability if you have REALLY large volumes of data, optimization mechanisms, consistency of the data, atomicity, integrity.

If you need simple operations, and one is not able or do not see profitable to optimize SQL queries then storing CSV files can be a better solution, especially in terms of storage space(as you will avoid metadata)

There is also the streaming option, but for a retail use case, no HFT, does not make any sense imo.
 
Last edited:
You must log in or register to reply here.
Back
Top