Most reliable data source with NinjaTrader

[mason macgregorson]

First of all, data quality depends on the edge you want to test. If you want to test intraday, you'll most likely need tick data and these are expensive. Minute data does not work because it depends on the data collector how the minute is defined and which data is included in a period.

EoD doesn't matter that much, margin of error can be larger.

You most likely need to setup your own database and compare it to the data you bought from an external source. If you have more ticks than external datasource the data you bought might be filtered or snapshotted.

Competitive data will always be collected from a direct exchange connection, never from a broker or data provider, especially intraday. It's a ton of work and really expensive, which is why everything on intraday basis is dominated by larger shops. You have no chance doing that as retail with retail platforms.

If your edge is 20cts on average and you obtained data from IB which is a snapshot of SIP, you can asume that you have no edge at all because you only tested on 2/3s of the ticks that actually happened on exchange.

right, so go do it and tell us the answer. Otherwise blah blah blah

 

[MrMuppet]

right, so go do it and tell us the answer. Otherwise blah blah blah

I'm already doing it but why would I give you the answer for free you lazy fk?

 

[mason macgregorson]

I'm already doing it but why would I give you the answer for free you lazy fk?

says you with no information and no data; could call you a lazy fk as well if I was easily prone to vulgarity and obscenities.

 

[MrMuppet]

says you with no information and no data; could call you a lazy fk as well if I was easily prone to vulgarity and obscenities.

perhaps you cannot read or are just not capable of digesting information properly, but IMHO I gave away more information about how to properly orchestrate data collection and storage than anybody else in this thread.

I mean I could add, that you could use MongoDB and feedhandlers from Github as a starter before you get into TSDBs like OpenTSDB or Prometheus, but I felt like that doesn't do a lot for people who entertain the use of Ninjatrader + IB feed for their business operations.

You on the other hand are just one of the "show me proof" rubbernecks who contribute absolutely nothing and want someone else to do the grunt work. No, dude. I did the grunt work and I'm making money from it. If you want to, too...sit down and do it yourself

 

[mason macgregorson]

perhaps you cannot read or are just not capable of digesting information properly, but IMHO I gave away more information about how to properly orchestrate data collection and storage than anybody else in this thread.

I mean I could add, that you could use MongoDB and feedhandlers from Github as a starter before you get into TSDBs like OpenTSDB or Prometheus, but I felt like that doesn't do a lot for people who entertain the use of Ninjatrader + IB feed for their business operations.

You on the other hand are just one of the "show me proof" rubbernecks who contribute absolutely nothing and want someone else to do the grunt work. No, dude. I did the grunt work and I'm making money from it. If you want to, too...sit down and do it yourself

Perhaps you cannot remember or just not capable of reading, but my original comment was: "reliability cannot be defined much less measured in a comparable manner". Show us all how you have defined reliability, measured reliability and then addressed comparable values. And while we are still young.

 

[MrMuppet]

Perhaps you cannot remember or just not capable of reading, but my original comment was: "reliability cannot be defined much less measured in a comparable manner". Show us all how you have defined reliability, measured reliability and then addressed comparable values. And while we are still young.

You most likely need to setup your own database and compare it to the data you bought from an external source. If you have more ticks than external datasource the data you bought might be filtered or snapshotted.

Perhaps you're already not young anymore or you've gotten bad eyesight...or you probably aren't that smart in which case I'll explain it for simpletons:

There is only one true datasource which is tick/book - data directly from the exchange itself. Per definition, there cannot be anything more reliable than this datafeed, because the exchange is where the trade happens...unless you mess up your database and feedhandler. And I'm saying "from the exchange" because the feed you get from your broker is NOT a direct exchange connection datafeed. It's filtered, slowed down, snapshotted or you get the SIP feed at best. I hope that's obvious.

If you pay for 3rd party datafeeds, the only source you can compare it to regarding reliability is above mentioned tick data dirctly from the exchange.

 

[mason macgregorson]

Perhaps you're already not young anymore or you've gotten bad eyesight...or you probably aren't that smart in which case I'll explain it for simpletons:

There is only one true datasource which is tick/book - data directly from the exchange itself. Per definition, there cannot be anything more reliable than this datafeed, because the exchange is where the trade happens...unless you mess up your database and feedhandler. And I'm saying "from the exchange" because the feed you get from your broker is NOT a direct exchange connection datafeed. It's filtered, slowed down, snapshotted or you get the SIP feed at best. I hope that's obvious.

If you pay for 3rd party datafeeds, the only source you can compare it to regarding reliability is above mentioned tick data dirctly from the exchange.

Age and eyesight aside, and your juvenile need to ridicule, I see no definition of reliability, no units of measurement and no comparable data. If you have nothing offer for public consumption, then we are finished.

 

[MrMuppet]

Age and eyesight aside, and your juvenile need to ridicule, I see no definition of reliability, no units of measurement and no comparable data. If you have nothing offer for public consumption, then we are finished.

lol...I ridicule, because you ARE ridiculous. Units of measurement is number of ticks, but I already wrote that. And how many times do I have to tell you that comparable data is data collected directly from the exchange.

You have a datafeed and all it does is feeding data. Either you get all the correct data or you don't...and to figure that out you compare it to the ultimate source. What's so difficult do understand?

Either you just want to argue or you have been falling on your head as a child.

 

[mason macgregorson]

lol...I ridicule, because you ARE ridiculous. Units of measurement is number of ticks, but I already wrote that. And how many times do I have to tell you that comparable data is data collected directly from the exchange.

You have a datafeed and all it does is feeding data. Either you get all the correct data or you don't...and to figure that out you compare it to the ultimate source. What's so difficult do understand?

Either you just want to argue or you have been falling on your head as a child.

When it comes to falling on your head as a child, you would be most qualified to know.

And since you do not understand reliability, let me help: Reliability is a measure of the frequency of failures as a function of time. Reliability has a major impact on accuracy, costs and on the continuity of service. Every product has a failure rate, λ which is the number of units failing per unit time. Measure λ? My post started with it.

 

[MrMuppet]

When it comes to falling on your head as a child, you would be most qualified to know.

And since you do not understand reliability, let me help: Reliability is a measure of the frequency of failures as a function of time. Reliability has a major impact on accuracy, costs and on the continuity of service. Every product has a failure rate, λ which is the number of units failing per unit time. Measure λ? My post started with it.

dude pls...do you expect an ANSI norm for datafeeds? If you want to theorize, at least do it right.

Your pompous lambda (I don't know who do you want to impress with it) can be exactly measured by the amount of ticks that you receive from the third party data feed vs. the ticks you receive directly from the exchange. Even uptime is accounted for since...well you don't receive ticks during downtime.

1000 ticks on exchange/your datafeed delivers 800 ticks -> not so reliable
1000 ticks on exchange/your datafeed delivers 900 ticks -> a bit more reliable

So if you still want to call your failure rate lambda, you could say that in the first case your lambda is 20% and in the second case 10%.

There are only two qualifiers for a datafeed: completeness and speed. Speed is not the question here, although you could compare timestamps and include it in your quest for reliability. And if you now come up with technicals like jitter and lag, these are NOT qualifiers for a datafeed but for connectivity.



So please think before you post again, the hole you dug yourself into is already deep enough