i'll do it for $1098 and have it done in a day
. to the OP, search for screeners. Not sure if there is any software because this is trivial if you have the data
looking for software
- Thread starter Roddog101
- Start date
. to the OP, search for screeners. Not sure if there is any software because this is trivial if you have the dataideas are a dime a dozen. Now to test your idea you need someone else.
Python is interesting language... its the slicing and data handing thats a bit hard to cogitate
however, a lot of what you listed could actually be done on the TOS platform..
there you can write your own studies, and those studies can be used to scan for stock
Quantiacs is what i am trying to write for now, as they have better terms than Quantopian
but i am a member there too... (recently out of work applications engineer)
surprisingly most dont know that TD has an applications API area too
this is quite a bit more complicated, but their API developer area lets one even programmatically trade... and gives access to backdata, and other stuff..
here is a new spreadsheet thing i am tinkering with
but this has no history data, it only has live data
the next sheet is for options trading off this current sheet..
the first column is the tickers you can add, the rest of the data comes from think or swims RTD...
there are also sites that let you get back data..
and if you can scrape data, as i have, you can build a database of history data for free
while not perfect, i was able to build a pretty good one.. over 10,000 tickers and about 8 years of data from investopedia and yahoo.. and not using their now difunct API...
you can also get a limited amount of data from various companies that let you query for a certain amount per minute..
where there is a will there is a way, a large enough hammer can crack anything...
however, a lot of what you listed could actually be done on the TOS platform..
there you can write your own studies, and those studies can be used to scan for stock
Quantiacs is what i am trying to write for now, as they have better terms than Quantopian
but i am a member there too... (recently out of work applications engineer)
surprisingly most dont know that TD has an applications API area too
this is quite a bit more complicated, but their API developer area lets one even programmatically trade... and gives access to backdata, and other stuff..
here is a new spreadsheet thing i am tinkering with
but this has no history data, it only has live data
the next sheet is for options trading off this current sheet..
the first column is the tickers you can add, the rest of the data comes from think or swims RTD...
there are also sites that let you get back data..
and if you can scrape data, as i have, you can build a database of history data for free
while not perfect, i was able to build a pretty good one.. over 10,000 tickers and about 8 years of data from investopedia and yahoo.. and not using their now difunct API...
you can also get a limited amount of data from various companies that let you query for a certain amount per minute..
where there is a will there is a way, a large enough hammer can crack anything...
Good thread with interesting suggestions. Let us know if you decide on something, OP.
Is this a study you want to perform on say all NYSE stocks? If so, the initial obstacle seems to be the screening part, i.e., finding stocks that fullfill your listed conditions. I'm not sure if standard screening software can check for N days down in a row?
Anyway, assuming you've found a list of stocks that fullfills these conditions, you can pretty easily perform this kind of analysis in Excel. That's probably the best way to go for someone without a lot of programming experience.
Quick solution:
1. Download daily data in *.txt or *.csv format for said stocks. This can be done in NinjaTrader, actually. There may be easier ways. I don't know. Personally, I use NT and also QCollector with IQ Feed.
2. Load this data into Excel using PowerQuery. You can add custom calculations either in PQ or directly in Excel. It's obviously easy to calculate daily change or N-day change.
3. Filter and analyze the data in Excel. You'd most likely need some custom VBA code in order to efficiently do this in Excel. I have a ton. Some I've coded myself and some I've had help coding.
It will probably take you some time to get a handle on this and create efficient templates, but once you do, you have it.
If you want to do this with a ton of stocks, Excel may not be suitable, but if you're happy just looking anything between 1-20 stocks at a time, I think Excel can do the job.
Is this a study you want to perform on say all NYSE stocks? If so, the initial obstacle seems to be the screening part, i.e., finding stocks that fullfill your listed conditions. I'm not sure if standard screening software can check for N days down in a row?
Anyway, assuming you've found a list of stocks that fullfills these conditions, you can pretty easily perform this kind of analysis in Excel. That's probably the best way to go for someone without a lot of programming experience.
Quick solution:
1. Download daily data in *.txt or *.csv format for said stocks. This can be done in NinjaTrader, actually. There may be easier ways. I don't know. Personally, I use NT and also QCollector with IQ Feed.
2. Load this data into Excel using PowerQuery. You can add custom calculations either in PQ or directly in Excel. It's obviously easy to calculate daily change or N-day change.
3. Filter and analyze the data in Excel. You'd most likely need some custom VBA code in order to efficiently do this in Excel. I have a ton. Some I've coded myself and some I've had help coding.
It will probably take you some time to get a handle on this and create efficient templates, but once you do, you have it.
If you want to do this with a ton of stocks, Excel may not be suitable, but if you're happy just looking anything between 1-20 stocks at a time, I think Excel can do the job.
Finally, I'll just add that not knowing how to code is a major detriment in this business. If I knew what I know now when I first started, I would actually have spent 1-2 years learning to code first before proceeding to learn trading.
The time I've spent dealing with programmers and the $$$ I've spent on programming services is considerable. I would have been far better off if I initially learned to code instead. And the harsh truth is that 1-2 years in this business really ain't that much...
The time I've spent dealing with programmers and the $$$ I've spent on programming services is considerable. I would have been far better off if I initially learned to code instead. And the harsh truth is that 1-2 years in this business really ain't that much...
There is a huge difference between learning to be a software engineer vs learning data science.
Data science is pretty easy. I personally hate code and want to write as little as I can. I am interested in results and analysis, not code.
Learning to use pandas in a jupyter notebook is not much harder than using any other software. Trying to find workarounds for trivial data science problems when we already have great data science tools makes no sense.
Just learn the data science tools and be done with it.
Data science is pretty easy. I personally hate code and want to write as little as I can. I am interested in results and analysis, not code.
Learning to use pandas in a jupyter notebook is not much harder than using any other software. Trying to find workarounds for trivial data science problems when we already have great data science tools makes no sense.
Just learn the data science tools and be done with it.
I was looking over the request again.. more to the specifics..
As with the stuff i am putting together, the hardest part (for a app engineer) is data aquisition (and potentially quality)
I ALSO looked over the answers from others, and the truth is Quantiacs or Quantopian may not do as they are not complete datasets or even near complete. Quantiacs is mostly futures as their customers want the size and liquidity that such can bring compared to the stock market (as far as i can tell from what work i have done). Quantopian on the other hand, restricts their data set to a subset of companies and other items to have certain qualities and so is also not as complete.
The data set I have contains 11,114 symbols
and it starts about 2006 and goes to 2019 (w 2020 pending)
though i have not had time to write code to go back and check for missing parts.
AND it has the caveat that it would be missing companies that went out of business before the date i initially created the symbol list and so had no symbol for me to pull data up on. there are a few other things about it, like figuring out how the sources handled splits so that this method can be known in any software one would write against it (given i am out of work, i dont have lots of time to work on anything that may not lead to earnings for my wife and family so things tend to sit due to priorities).
The data is OPEN, HIGH, LOW, ADJUSTED CLOSE, VOLUME, DIVIDENDS AND SPLITS (in separate tables).
Most of the open APIs that used to provide this information are closed..
Google API has stopped providing the data (see below)
Investopedia has also closed the door as its old link gives 404 error
This was part of the reason i stopped (temporarily?) collecting data..
Yahoo has made it harder (impossible?), to get it from their pages in a easy programatic way
however, if you wanted to look at one company, they still provide.
https://finance.yahoo.com/quote/AAPL/history?p=AAPL
but this is AJAX and only populates the whole table if you page down
and is 2019 to 2020.. (maybe more, i haven't really looked)
the download is this link
https://query1.finance.yahoo.com/v7...eriod2=1581870306&interval=1d&events=history&crumb=wDEQ8G8kzwU
But notice that last (bold part). That is a digital crumb that basically prevents easy access to the download data for the period you put in... I have not bothered to even try to get around it, or work with it... (yet?)
NASDAQ provides historical quotes (Date Close/Last Volume Open High Low) on their site on a company by company basisbut this is just NASDAQ not other areas
There are now MANY web businesses that offer historical data
they provide free accounts that are limited in the bandwidth, and so, if you want more, you have to pay more...
Some are limited by the number of daily requests, some are limited by the number of requests per minute or hour (which is the same thing in different ways of putting it)
As an example (i am NOT endorsing!!!!!!!!)
https://www.worldtradingdata.com/
they allow 250 historical requests a day
(which would take me 44 days to update my database given its ticker size)
Their plus account is $32 a month and lets you do 250,000
So i would be able to update my DB in a day (historically)
The advantage to this kind of company is that you get good data, you get what you need.
depending on company you may or may not pay a lot. Many of them also provide more granularity data than just the daily data (like what i have)..
but also provide intra-day as well..
Wife needs me now, so i will have to finish this later..
however, just because you have an idea of looking up things, doesnt mean that it will be fruitful... I go through lots of them that i think may look good, only to find out in back testing that over time and accross 300 companies (on either quantopian, quantiacs, or TD (limited)) that it does not do anything which i thought it would
in fact.. its very hard to beat two ema averages and a few crossover rules
which is why Quantiacs and Quantopian are now trying to shift their work to using the 1600 data points that morningstar provides including social data (like stock twits post rates... (given i am on stock twits i shudder to think they would use that data with the rampant lying bears thinking they can influence the market there))
IF one is curious why i have this data for myself or am trying it...
i have a Titan X card and can do Machine learning coding using the card..
which is why i am self teaching Python... (but given Bronx Ccience, 30 years of professional programming with last 15 in medical research, and many languages, this isn't that hard... the hardest part is grasping how they work with Numpy array data vs the more traditional ways).
Too bad i am not diverse (and young) enough to be employed any more...
As with the stuff i am putting together, the hardest part (for a app engineer) is data aquisition (and potentially quality)
I ALSO looked over the answers from others, and the truth is Quantiacs or Quantopian may not do as they are not complete datasets or even near complete. Quantiacs is mostly futures as their customers want the size and liquidity that such can bring compared to the stock market (as far as i can tell from what work i have done). Quantopian on the other hand, restricts their data set to a subset of companies and other items to have certain qualities and so is also not as complete.
The data set I have contains 11,114 symbols
and it starts about 2006 and goes to 2019 (w 2020 pending)
though i have not had time to write code to go back and check for missing parts.
AND it has the caveat that it would be missing companies that went out of business before the date i initially created the symbol list and so had no symbol for me to pull data up on. there are a few other things about it, like figuring out how the sources handled splits so that this method can be known in any software one would write against it (given i am out of work, i dont have lots of time to work on anything that may not lead to earnings for my wife and family so things tend to sit due to priorities).
The data is OPEN, HIGH, LOW, ADJUSTED CLOSE, VOLUME, DIVIDENDS AND SPLITS (in separate tables).
Most of the open APIs that used to provide this information are closed..
Google API has stopped providing the data (see below)
Investopedia has also closed the door as its old link gives 404 error
This was part of the reason i stopped (temporarily?) collecting data..
As you may have heard from the Google Developers Blog, Google is doing an API spring cleaning. One of the APIs affected is the Google Finance API (both the Portfolio API and the Finance Gadgets and Tools API), which will be shut down on October 20, 2012.
This was not an easy decision to take. We work with a large number of data providers and need to respect our relationships with them. As a result, we had to keep the API restricted to end users, which prevented a meaningful ecosystem from growing around the API. We also realized that we could serve more people better by integrating the data into other Google products rather than requiring them to write code to access the data. For example, check out the GoogleFinance() function in Google Spreadsheets which replicates some of the API's functionality without requiring you to write code.
Thank you for being loyal users of the Google Finance API over the last few years.
This was not an easy decision to take. We work with a large number of data providers and need to respect our relationships with them. As a result, we had to keep the API restricted to end users, which prevented a meaningful ecosystem from growing around the API. We also realized that we could serve more people better by integrating the data into other Google products rather than requiring them to write code to access the data. For example, check out the GoogleFinance() function in Google Spreadsheets which replicates some of the API's functionality without requiring you to write code.
Thank you for being loyal users of the Google Finance API over the last few years.
Yahoo has made it harder (impossible?), to get it from their pages in a easy programatic way
however, if you wanted to look at one company, they still provide.
https://finance.yahoo.com/quote/AAPL/history?p=AAPL
but this is AJAX and only populates the whole table if you page down
and is 2019 to 2020.. (maybe more, i haven't really looked)
the download is this link
https://query1.finance.yahoo.com/v7...eriod2=1581870306&interval=1d&events=history&crumb=wDEQ8G8kzwU
But notice that last (bold part). That is a digital crumb that basically prevents easy access to the download data for the period you put in... I have not bothered to even try to get around it, or work with it... (yet?)
NASDAQ provides historical quotes (Date Close/Last Volume Open High Low) on their site on a company by company basisbut this is just NASDAQ not other areas
There are now MANY web businesses that offer historical data
they provide free accounts that are limited in the bandwidth, and so, if you want more, you have to pay more...
Some are limited by the number of daily requests, some are limited by the number of requests per minute or hour (which is the same thing in different ways of putting it)
As an example (i am NOT endorsing!!!!!!!!)
https://www.worldtradingdata.com/
they allow 250 historical requests a day
(which would take me 44 days to update my database given its ticker size)
Their plus account is $32 a month and lets you do 250,000
So i would be able to update my DB in a day (historically)
The advantage to this kind of company is that you get good data, you get what you need.
depending on company you may or may not pay a lot. Many of them also provide more granularity data than just the daily data (like what i have)..
but also provide intra-day as well..
Wife needs me now, so i will have to finish this later..
however, just because you have an idea of looking up things, doesnt mean that it will be fruitful... I go through lots of them that i think may look good, only to find out in back testing that over time and accross 300 companies (on either quantopian, quantiacs, or TD (limited)) that it does not do anything which i thought it would
in fact.. its very hard to beat two ema averages and a few crossover rules
which is why Quantiacs and Quantopian are now trying to shift their work to using the 1600 data points that morningstar provides including social data (like stock twits post rates... (given i am on stock twits i shudder to think they would use that data with the rampant lying bears thinking they can influence the market there))
IF one is curious why i have this data for myself or am trying it...
i have a Titan X card and can do Machine learning coding using the card..
which is why i am self teaching Python... (but given Bronx Ccience, 30 years of professional programming with last 15 in medical research, and many languages, this isn't that hard... the hardest part is grasping how they work with Numpy array data vs the more traditional ways).
Too bad i am not diverse (and young) enough to be employed any more...
Check out Kinetick. They offer free EOD data. Not sure if it's applicable for your use if you need to download data for a massive basket of stocks.
No. But that's just part of the process.
If you never ask questions and answer them, you're surely not learning either.
Kinetick does not provide the kind of data the original person posting the question needs...
As to my data... well, I have a API developer account with TD, which can give me data and information (and even allow computer trading)...
however the database i have i built myself collecting EOD data going back to 2006 and have that stored in a MSSQL database..
Education is Bronx Science, Barnard Baruch college...
Been writing software since before desktops were a big thing..
remembering fondly the days when a 300 baud modem was a big deal
started on mainframes..
Last job was in medical research for a famous medical school / hospital
just pointing out that i have good creds for giving advice on many things
I am a kind of eclectic sort that has very wide experience in many fields not just programming, and data science... I used to be a fashion celebrity photographer too.. portrait artist, graphic arts. Did some work in advertising on Madison ave, as a yute... I get around a lot and not much stops me when it comes to doing things...
however, if you look at the first post, what they want to do is not that hard...
IF you have the data already available in a format that can be queried over and over
and it was this aspect of such a project i was trying to point out is the hardest part
even if you paid Kinetick to get their historical data, you would still have to bring it all in and store it so that you could do the kinds of queries needed...
my database has 11,000 stocks in it.. spanning 2006 to 2019..
software now has to be rewritten to get data from another source
i was thinking of using the TD ameritrade API to round it out
but i am not the person who made the original request
i am just me...
As to my data... well, I have a API developer account with TD, which can give me data and information (and even allow computer trading)...
however the database i have i built myself collecting EOD data going back to 2006 and have that stored in a MSSQL database..
Education is Bronx Science, Barnard Baruch college...
Been writing software since before desktops were a big thing..
remembering fondly the days when a 300 baud modem was a big deal
started on mainframes..
Last job was in medical research for a famous medical school / hospital
just pointing out that i have good creds for giving advice on many things
I am a kind of eclectic sort that has very wide experience in many fields not just programming, and data science... I used to be a fashion celebrity photographer too.. portrait artist, graphic arts. Did some work in advertising on Madison ave, as a yute... I get around a lot and not much stops me when it comes to doing things...
however, if you look at the first post, what they want to do is not that hard...
IF you have the data already available in a format that can be queried over and over
and it was this aspect of such a project i was trying to point out is the hardest part
even if you paid Kinetick to get their historical data, you would still have to bring it all in and store it so that you could do the kinds of queries needed...
my database has 11,000 stocks in it.. spanning 2006 to 2019..
software now has to be rewritten to get data from another source
i was thinking of using the TD ameritrade API to round it out
but i am not the person who made the original request
i am just me...
Ok..
been looking for sources of data...
hoping to help the poster (myself as well) and any others that wish to do things on their own.
supposedly..IF you know PYTHON..
> pip install yfinance
I have not been able to test it (yet?), but you can get all the info needed (for a programmer)
https://github.com/ranaroussi/yfinance
The NEXT best place i found was Google sheets..
While google has deprecated its api a long while back
the googlesheets function still works (i tested it!)
you log into google sheets and open a spread sheet
put this in a cell: =GOOGLEFINANCE("GOOG","price")
and you will get a piece of price data
put this in a cell:
=GOOGLEFINANCE("NASDAQ:GOOG","price",DATE(2014,1,1),DATE(2014,12,31),"DAILY")
and it will populate a list in the sheet with close prices..
here is (i think) the best free source i could find..
https://www.alphavantage.co/
Composed of a tight-knit community of researchers, engineers, and business professionals, Alpha Vantage Inc. is a leading provider of free APIs for realtime and historical data on stocks, forex (FX), and digital/crypto currencies. Our success is driven by rigorous research, cutting edge technology, and a disciplined focus on democratizing access to data.
example json data
https://www.alphavantage.co/query?function=TIME_SERIES_DAILY&symbol=MSFT&apikey=demo
I think i will use them to finish up my personal database as it would not require much work for that... (so this research helped me as well as helping others here!!!!!!!!!!!!)
the rest of them cost money..
some a lot, some a little for a lot..
A poster suggested (to me?) to try Kinetick..
they are quite expensive.. $65 per month!
you will get socked for upgrade fees..
https://kinetick.com/Subscribe
so i dont suggest them..
MANY places give free daily EOD numbers
[so once you build a db, you can maintain it if you run code every day]
i do suggest a site (there are others) like
https://www.worldtradingdata.com/
They work by REST API calls..
you query, and they return JSON (or CSV) data you can use
They do historical data, intraday data, forex and real time
[though if you have TD, you can get Real time with RTD in Excel]
they also provide lists so you can download their entire database of tickers
[they said so]
Their free account lets you do 250 API requests a day...
and less for things like intraday..
there are about a half dozen to a dozen that give this free 250 request to 500 requests
Basic is $8 for 5,000 api calls
Standard is $16 for 50,000 api calls
Plus is $32 for 250,000 api calls
Then advanced pricing
500K is $64
750K is $96
1M is $128
i do not know how much historical data you get in one API
a day? a year? several years? it would make a difference as to how many calls to get what you want
my database has 11,000 tickers... and covers 13 years of data
so has around 37 million days of data...
one year for 11,000 is about 28,741,400 points of data...
i would only have to update from 2019 to current..
which would not require a large cash amount
probably use the standard for the 11,000 eod amounts
and then use whats left over to fill up whats missing
so about 200 for one year to do that.. then my cost would go down
this link will give you an idea of the data.. which is more than i have saved!
https://api.worldtradingdata.com/api/v1/stock?symbol=SNAP,TWTR,VOD.L&api_token=demo
https://iexcloud.io/pricing/ is another decent one.
they include a calculator to figure out how much to pay to get what you need
https://iexcloud.io/calculator/
[about 40 dollars for me... ]
they are about $9 for 1 million accesses a month.. and 5 million core messages
not to shabby
been looking for sources of data...
hoping to help the poster (myself as well) and any others that wish to do things on their own.
supposedly..IF you know PYTHON..
> pip install yfinance
I have not been able to test it (yet?), but you can get all the info needed (for a programmer)
https://github.com/ranaroussi/yfinance
The NEXT best place i found was Google sheets..
While google has deprecated its api a long while back
the googlesheets function still works (i tested it!)
you log into google sheets and open a spread sheet
put this in a cell: =GOOGLEFINANCE("GOOG","price")
and you will get a piece of price data
put this in a cell:
=GOOGLEFINANCE("NASDAQ:GOOG","price",DATE(2014,1,1),DATE(2014,12,31),"DAILY")
and it will populate a list in the sheet with close prices..
Usage restrictions: The data is not for financial industry professional use or use by other professionals at non-financial firms (including government entities). Professional use may be subject to additional licensing fees from a third-party data provider.
here is (i think) the best free source i could find..
https://www.alphavantage.co/
Composed of a tight-knit community of researchers, engineers, and business professionals, Alpha Vantage Inc. is a leading provider of free APIs for realtime and historical data on stocks, forex (FX), and digital/crypto currencies. Our success is driven by rigorous research, cutting edge technology, and a disciplined focus on democratizing access to data.
example json data
https://www.alphavantage.co/query?function=TIME_SERIES_DAILY&symbol=MSFT&apikey=demo
I think i will use them to finish up my personal database as it would not require much work for that... (so this research helped me as well as helping others here!!!!!!!!!!!!)
the rest of them cost money..
some a lot, some a little for a lot..
A poster suggested (to me?) to try Kinetick..
they are quite expensive.. $65 per month!
you will get socked for upgrade fees..
https://kinetick.com/Subscribe
so i dont suggest them..
MANY places give free daily EOD numbers
[so once you build a db, you can maintain it if you run code every day]
i do suggest a site (there are others) like
https://www.worldtradingdata.com/
They work by REST API calls..
you query, and they return JSON (or CSV) data you can use
They do historical data, intraday data, forex and real time
[though if you have TD, you can get Real time with RTD in Excel]
they also provide lists so you can download their entire database of tickers
[they said so]
Their free account lets you do 250 API requests a day...
and less for things like intraday..
there are about a half dozen to a dozen that give this free 250 request to 500 requests
Basic is $8 for 5,000 api calls
Standard is $16 for 50,000 api calls
Plus is $32 for 250,000 api calls
Then advanced pricing
500K is $64
750K is $96
1M is $128
i do not know how much historical data you get in one API
a day? a year? several years? it would make a difference as to how many calls to get what you want
my database has 11,000 tickers... and covers 13 years of data
so has around 37 million days of data...
one year for 11,000 is about 28,741,400 points of data...
i would only have to update from 2019 to current..
which would not require a large cash amount
probably use the standard for the 11,000 eod amounts
and then use whats left over to fill up whats missing
so about 200 for one year to do that.. then my cost would go down
this link will give you an idea of the data.. which is more than i have saved!
https://api.worldtradingdata.com/api/v1/stock?symbol=SNAP,TWTR,VOD.L&api_token=demo
https://iexcloud.io/pricing/ is another decent one.
they include a calculator to figure out how much to pay to get what you need
https://iexcloud.io/calculator/
[about 40 dollars for me... ]
they are about $9 for 1 million accesses a month.. and 5 million core messages
not to shabby