Data base setup and polygon.io API, cost?

D

Daniel.a

Hi,
As a project owner and non dev, I have been quoted for getting help of setting up a database storing pricedata for my personal trading use, and some fundamentals for symbols, index constituents, listed, delisted, adjusted, unadjusted etc.
And then a API websocket, for feeding it with data from polygon.io.

I would be interested hearing what the existing developers here would charge for this in order to give me a rough idea what is reasonable to pay.

a, setting up the database
b, creating the api for polygon.io

Cheers
 
It really depends on what kind of developers you're talking to and where they live.

For guys who are very good and have no trouble getting a job as a developer + have good english, it is probably not worth taking anything less than 10k for even a few days project. If they size it up as couple of months you might be looking at 40-60k.

If you get someone from a country with very low cost of living and is struggling to get a job, your lower end might be couple of thousands. Not saying there is something wrong with that. You might get something out of it. Will take longer than you expect, there will be lots of iterations and you might need to change a developer few times. But total cost combined will be less. So it will come down to how much your time is worth, as there will be way more hand holding and guidance required.

Either way - look for someone who has done something similar before. They will know what questions to ask and are more likely to deliver what you want.

Good luck!

Val
 
ValeryN said:
It really depends on what kind of developers you're talking to and where they live.

For guys who are very good and have no trouble getting a job as a developer + have good english, it is probably not worth taking anything less than 10k for even a few days project. If they size it up as couple of months you might be looking at 40-60k.

If you get someone from a country with very low cost of living and is struggling to get a job, your lower end might be couple of thousands. Not saying there is something wrong with that. You might get something out of it. Will take longer than you expect, there will be lots of iterations and you might need to change a developer few times. But total cost combined will be less. So it will come down to how much your time is worth, as there will be way more hand holding and guidance required.

Either way - look for someone who has done something similar before. They will know what questions to ask and are more likely to deliver what you want.

Good luck!

Val
More...

Thank you ValeryN,

I really appreciate your reply.

Euro dev in this case, but i have no preference of dev origin, really as long as english is fluent.
Skillset matters more.

I have to say though since its a private project, without a CTO, skillset difficult to judge.

Anyway, i am surprised about your price ranges, an API and a sql or similar database,
i thought it was a rather small job like a 2-3 days, for a good developer?.

- Anyone that have an idea where to get point in time data for index constituents for equities?
It seems that this data is not often supplied from pricefeed supplier and i would need a second source for that to populate my database.
Anyone have ideas?
Its rather important when backtesting equities so solutions must be out there.

Cheers
 
Daniel.a said:
Thank you ValeryN,

I really appreciate your reply.

Euro dev in this case, but i have no preference of dev origin, really as long as english is fluent.
Skillset matters more.

I have to say though since its a private project, without a CTO, skillset difficult to judge.

Anyway, i am surprised about your price ranges, an API and a sql or similar database,
i thought it was a rather small job like a 2-3 days, for a good developer?.

- Anyone that have an idea where to get point in time data for index constituents for equities?
It seems that this data is not often supplied from pricefeed supplier and i would need a second source for that to populate my database.
Anyone have ideas?
Its rather important when backtesting equities so solutions must be out there.

Cheers
More...

Having implemented this exact thing, I can tell you 2-3 days will get data pushing into a database but the primary challenge here is ensuring performance, consistency, and fault tolerance with the data import which can be time-consuming.

Before providing a quote, a good developer would have you get into specifics about the nature of your backtesting and anticipate the necessary optimizations and potential pitfalls. Backtests are often I/O bound, meaning the overall performance is limited by how quickly the DB can crank out data. A good solution here might outperform a mediocre one by orders of magnitude, which is a big consideration if you're doing a lot of tweaking and re-testing. Sometimes throwing hardware at a mediocre solution is more economical, especially if you have idle infrastructure.

Price would also heavily depend on the number and complexity of the specific Polygon endpoints you're interested in, but I agree $10k wouldn't be an unreasonable starting point for quality custom development. Developers with strong DBA competency and finance domain knowledge tend to be very busy and expensive.

I'd be shocked if this hasn't been implemented hundreds of times over already, so you might have some luck tracking down a pre-existing solution for a fraction of that. Acquiring and customizing something that's already battle-tested and production-ready could certainly spare you time, cost, and headaches.

For your index constituents, have you checked Siblis Research?
 
PersonaInconsolata said:
Having implemented this exact thing, I can tell you 2-3 days will get data pushing into a database but the primary challenge here is ensuring performance, consistency, and fault tolerance with the data import which can be time-consuming.

Before providing a quote, a good developer would have you get into specifics about the nature of your backtesting and anticipate the necessary optimizations and potential pitfalls. Backtests are often I/O bound, meaning the overall performance is limited by how quickly the DB can crank out data. A good solution here might outperform a mediocre one by orders of magnitude, which is a big consideration if you're doing a lot of tweaking and re-testing. Sometimes throwing hardware at a mediocre solution is more economical, especially if you have idle infrastructure.

Price would also heavily depend on the number and complexity of the specific Polygon endpoints you're interested in, but I agree $10k wouldn't be an unreasonable starting point for quality custom development. Developers with strong DBA competency and finance domain knowledge tend to be very busy and expensive.

I'd be shocked if this hasn't been implemented hundreds of times over already, so you might have some luck tracking down a pre-existing solution for a fraction of that. Acquiring and customizing something that's already battle-tested and production-ready could certainly spare you time, cost, and headaches.

For your index constituents, have you checked Siblis Research?
More...

Thank you PersonaInconsolata,

I agree with spec, my brief description was just very broad and high level, but as you say, one step further and it needs to be detailed as much as possible.

Its just very difficult providing a spec when as a non developer you dont know the weak points, and parts that needs extra attention.
However you have given me a good start.

As you say, this must have been implemented thousands times x, and it would make sense if as you say this does not already exist as a pre existing solution.
I dont know though in which end to start looking?
Would you or anyone have any tips of where one could find a good pre existing solution that is of great quality ?

Cheers
 
Daniel.a said:
Thank you ValeryN,

I really appreciate your reply.

Euro dev in this case, but i have no preference of dev origin, really as long as english is fluent.
Skillset matters more.

I have to say though since its a private project, without a CTO, skillset difficult to judge.

Anyway, i am surprised about your price ranges, an API and a sql or similar database,
i thought it was a rather small job like a 2-3 days, for a good developer?.

- Anyone that have an idea where to get point in time data for index constituents for equities?
It seems that this data is not often supplied from pricefeed supplier and i would need a second source for that to populate my database.
Anyone have ideas?
Its rather important when backtesting equities so solutions must be out there.

Cheers
More...

No idea how long it will take. Just sharing time-bound price ranges.

It is quite typical that getting one simple scenario to work doesn't take long, but lots of intricacies come up around error handling come up or data sources not working as expected or you will change your mind once you see first results.

Sometimes it helps to look for any 3rd party software that supports the data feed you are thinking about. Just as a starting point of what you could use as a platform. That might lead you to discover some sort of aggregation solution with potentially Drag&Drop configuration that you can customize yourself and hire a developer to do the initial setup. IQFeed been on a market longer that Polygon, so checking who supports them is another way to look for what sorts of software could be used as a platform.

Having said that, I personally gave up on a 3rd party products for my own trading. With the exception of a great backtesting tool I'm using these days instead of my own and AmiBroker in the past. And limitations of that tool actually shaped my trading significantly. That might look like a bad thing, but after weighting pros and cons it is a good choice for me. I am also not comfortable using anything I can't get quality data over multiple market cycles. Btw, Norgate has current and historical membership data. Incredibly valuable as what worked on today's index stocks is not even representative of the past performance.

Val
 
ValeryN said:
No idea how long it will take. Just sharing time-bound price ranges.

It is quite typical that getting one simple scenario to work doesn't take long, but lots of intricacies come up around error handling come up or data sources not working as expected or you will change your mind once you see first results.

Sometimes it helps to look for any 3rd party software that supports the data feed you are thinking about. Just as a starting point of what you could use as a platform. That might lead you to discover some sort of aggregation solution with potentially Drag&Drop configuration that you can customize yourself and hire a developer to do the initial setup. IQFeed been on a market longer that Polygon, so checking who supports them is another way to look for what sorts of software could be used as a platform.

Having said that, I personally gave up on a 3rd party products for my own trading. With the exception of a great backtesting tool I'm using these days instead of my own and AmiBroker in the past. And limitations of that tool actually shaped my trading significantly. That might look like a bad thing, but after weighting pros and cons it is a good choice for me. I am also not comfortable using anything I can't get quality data over multiple market cycles. Btw, Norgate has current and historical membership data. Incredibly valuable as what worked on today's index stocks is not even representative of the past performance.

Val
More...

Hi Val,

I agree, with you, would be much better to use software that supports natively the datafeed i want.

My issue is that i dont trade using EOD data, i trade daily bars, calculated as N time before session end, so i send orders most often just before session close. This creates headache since most software works better by using eod data. Then norgate would be great. However for me i cant use that since Norgate is only EOD.

I investigated your software you are using, i am following your thread btw. Great stuff.

However it does not have a trading engine, so thats out of the question for me, cant use it. And coding up a proper trading engine is not a small task, either as a developer or project owner, so i would rather avoid that route.

Regarding data, i have been using IQ for years, futures and for equities, but when it comes to equities, one have to deal with listed, delisted, sectors, factor adjustments for dividends, splits, chosing total return series or not etc etc etc, and not to mention indec constituents point in time... you are of course already aware of all this, but solving it when not using EOD data, is a rather complex project :).

So finding data that support the above, is not easy, and connect this with a good software that can handle portfolios and do proper backtesting... its a project that will take some time to solve..
Meanwhile i am stuck with MultiCharts, it works ok, but not optimal.

Thanks for your input
 
Daniel.a said:
I investigated your software you are using, i am following your thread btw. Great stuff.
More...

Thanks!

Re IQFeed - I meant to use most popular data services like that as a way of discovery of who supports them and that maybe leading you to find some platform that can do more of what you want.

Depending on your budget, you might use Norgate just for the memberships :) As they have historical constituency + current. But only current should be available publicly. Might take a bit of time to cleanup and maintain though. I really appreciate them as they are the ones doing all the data cleaning.

As someone who created own execution engine I certainly support your decision of not writing one :) It would be a long journey.

Just curious - have you hired someone to do a MultiCharts execution setup for you? How long did that take?

Val
 
Hi Val,

Yes i have thought of using Norgate despite their only EOD setup, to get historic and current, plus their cleaning... i am able to use EOD for general backtesting, however since my executioin is close but not real close price, its not an accurate test, but still useable for general reports.
So Norgate i will get for anyway.

I can imagine, writing your own execution engine is a pain, however on the plus side it does exactly as you want :).

Multicharts is very good for research and execution of single strategies on charts, futures... but dealing with large portfolio of equities its a pain.. no i have done all setup on my own on Multicharts since many years, hired some developers though for more complexed portfolio management, but new platform would be in C++ or C# which i would need dev for, but thats ok, since it will be used for deployed working live strategies only and incubation ones.

Cheers
 
Database Questions if any experienced developers kindly wants to share some tips.

I am asking from the point of view, that its shall be mainly used for storing equities data, montly, weekly, daily and intra down to 1min, thousands of symbols, US nasdaq and nyse.

1, Is it a good idea to use same database and divide it into data from two suppliers?
For example you have data from Norgate, and then you also have data from IQ.

2, is it advisable to store raw non adjusted data only, and then its expected that one adjust it from the software that shall use it each time? for example for splits and dividends.

Or is it advisable to store all separate already adjusted in database after fetched from data supplier, as raw, adjusted for splits only, adjusted for dividends only and adjusted for both. And for example daily bars ending in custom time.

3, Is it consider ok to store mixed data from different suppliers into one database for different bar sizes, for example daily bars uses data from Norgate, then intraday data is fetched from IQ.

4, What should one think about when i ask someone to set this up for me, because i want to make it possible for me in the future to add new data sources, fundamentals or other factors for symbols, and also start to fetch bid ask unless this requires something vastly different or complicates things.
 
You must log in or register to reply here.
Back
Top