Why running on Linux in the first place if your data feed, execution, and trading system not natively support that?
linux - datafeed
- Thread starter Tcl
- Start date
In my case, my real time architecture is 100% Linux (custom c++ and Rithmic) . The WINE and NxCore solution exists for me only when processing historical data. I haven't found better reasonably priced historical data than what NxCore provides
Am not familiar with Rithmic, on what architecture and technology is the API based on? Sockets? REST?
I'm just getting started with it, and I'm actually not sure whats "under the hood" yet..its proably in the API docs..but I'm lazy. My guess is the API calls are hiding POSIX Sockets (its cross platform.. which is why POSIX is my suspicion). I've never used REST, but google has led me to believe it is not as high performance as other comm protocols...and Rithmic advertises as a low latency, high performance feed/execution platform.
Sorry, I meant which technology their API exposes.I doubt it is Posix...but then I might be too pessimistic. Truth is I don't know their API.
But I can say that I advice against running on Linux unless someone's entire setup runs natively on Linux (not addressing you here specififally). I am not a fan of Wine and the like at all.
But I can say that I advice against running on Linux unless someone's entire setup runs natively on Linux (not addressing you here specififally). I am not a fan of Wine and the like at all.
@Tcl Within the past years I, as part of a team, developed and implemented several trading system and market data feed solutions for different clients under linux, which definitely has been a hassle sometimes.
But yes, I think the best way to do receive market data is colo with a proper connection under linux.
Anyway...this is mostly not affordable as individual.
Beside my work as software developer I'm interested in the markets personally, so I have been trying some different solutions.
My initial requirements:
- Receive TAQ for approx 5000 symbols or if possible all NASDAQ and NYSE stocks (UTDF, UQDF)
- Not ultra low latency, so I don't mind being some millis behind
- Data should be unfiltered and somewhat reliable
To make it short:
Nanex with WINE works perfect but I don't want to pay that much for single server personal usage.
I tried IQFeed, but symbol limit is crap and spreading around multiple IQFeed instances under WINE wasn't funny at all.
I'm currently using ActiveTick and are quite happy with the real time data quality watching around 18000 symbols (including OTC) at once.
Ok, there is some erroneous data in their historical offer but I filter and don't care. I found that the data quality is better from mid-end 2013.
You probably can't expect a complete appropriate symbol change handling from their side and should track and work with symbol changes accordingly.
But in case you have a day or so spare time to work with it there are some ways to do it.
I love C and wrote a small wrapper for their "old" C++ API.
Things to consider:
If you try to kill/exit the feed through their API functions/methods, you are better off letting your process die and restart again.
Some bugs in the API lead to strange behaviour. (it seems to come from improper lock handling with boost, but I have no fun to dig into this)
But you know your ways to efficient process management under linux.
Real time and historical concurrently is a no go, so you if you want to request historical while having real time data without large delay its not possible in a single connection. But this isn't the way to go either.
100K TAQ limit per historical request is also somewhat boring.
Within two months of usage I had one major (1 day) and one minor (premarket, 3 hours) real time data disruption.
To be honest, for personal purposes, I'm quite happy with my ActiveTick + server for 210 EUR/mo configuration.
If I have some time left, I'm going to write a small test to compare their feed with the propriety one we do use at our current project.
Cheers,
Marc
But yes, I think the best way to do receive market data is colo with a proper connection under linux.
Anyway...this is mostly not affordable as individual.
Beside my work as software developer I'm interested in the markets personally, so I have been trying some different solutions.
My initial requirements:
- Receive TAQ for approx 5000 symbols or if possible all NASDAQ and NYSE stocks (UTDF, UQDF)
- Not ultra low latency, so I don't mind being some millis behind
- Data should be unfiltered and somewhat reliable
To make it short:
Nanex with WINE works perfect but I don't want to pay that much for single server personal usage.
I tried IQFeed, but symbol limit is crap and spreading around multiple IQFeed instances under WINE wasn't funny at all.
I'm currently using ActiveTick and are quite happy with the real time data quality watching around 18000 symbols (including OTC) at once.
Ok, there is some erroneous data in their historical offer but I filter and don't care. I found that the data quality is better from mid-end 2013.
You probably can't expect a complete appropriate symbol change handling from their side and should track and work with symbol changes accordingly.
But in case you have a day or so spare time to work with it there are some ways to do it.
I love C and wrote a small wrapper for their "old" C++ API.
Things to consider:
If you try to kill/exit the feed through their API functions/methods, you are better off letting your process die and restart again.
Some bugs in the API lead to strange behaviour. (it seems to come from improper lock handling with boost, but I have no fun to dig into this)
But you know your ways to efficient process management under linux.
Real time and historical concurrently is a no go, so you if you want to request historical while having real time data without large delay its not possible in a single connection. But this isn't the way to go either.
100K TAQ limit per historical request is also somewhat boring.
Within two months of usage I had one major (1 day) and one minor (premarket, 3 hours) real time data disruption.
To be honest, for personal purposes, I'm quite happy with my ActiveTick + server for 210 EUR/mo configuration.
If I have some time left, I'm going to write a small test to compare their feed with the propriety one we do use at our current project.
Cheers,
Marc
thanks for sharing your experience.
As an aside, don't you think it is causing trouble when not being able to run a market data feed for weeks/months without a hitch? Whenever I hear anyone on the retail side trying to run market data feeds and broker APIs on the Linux side, I come across problems and issues. I run on a Windows box and have not had a single disconnect or outage for months...
As an aside, don't you think it is causing trouble when not being able to run a market data feed for weeks/months without a hitch? Whenever I hear anyone on the retail side trying to run market data feeds and broker APIs on the Linux side, I come across problems and issues. I run on a Windows box and have not had a single disconnect or outage for months...
@volpunter
You are absolutely right, the feed should run forever without problems but this doesn't prevent you from implementing proper error/failure handling, no matter on which OS you run your system.
I don't want to get into pros and cons of Linux or Windows here and a good written piece of code might run on both forever but when you develop software its irreversible to come into situations where you have to work with a "poorly" written or not suitable piece of code.
Actually I do think that most of the retail solutions are poor pieces of code and this is one of the reasons why these are not suitable under Linux. Because if these had been engineered carefully, everything would just work fine.
I had to think about running IB TWS API with OpenJDK, horrible.
Anyway, the lock/mutex problem with the ActiveTick API might be the same under Windows ... I don't now, and I don't care.
For propriety solutions, which I do have to develop most of the time, this might not acceptable, but for my individual use I don't care about the API bug. Its simply handled by catching a signal, clean-up and restart ... 99% below 1 sec until I get the ACK of being reconnected (~25ms cleanup and process restart only) which has to be implemented by a careful programmer anyway. I assume this could be acceptable to Tcl as well.
By the way, we never had to deal with a customer who wanted a Windows solution.
You are absolutely right, the feed should run forever without problems but this doesn't prevent you from implementing proper error/failure handling, no matter on which OS you run your system.
I don't want to get into pros and cons of Linux or Windows here and a good written piece of code might run on both forever but when you develop software its irreversible to come into situations where you have to work with a "poorly" written or not suitable piece of code.
Actually I do think that most of the retail solutions are poor pieces of code and this is one of the reasons why these are not suitable under Linux. Because if these had been engineered carefully, everything would just work fine.
I had to think about running IB TWS API with OpenJDK, horrible.
Anyway, the lock/mutex problem with the ActiveTick API might be the same under Windows ... I don't now, and I don't care.
For propriety solutions, which I do have to develop most of the time, this might not acceptable, but for my individual use I don't care about the API bug. Its simply handled by catching a signal, clean-up and restart ... 99% below 1 sec until I get the ACK of being reconnected (~25ms cleanup and process restart only) which has to be implemented by a careful programmer anyway. I assume this could be acceptable to Tcl as well.
By the way, we never had to deal with a customer who wanted a Windows solution.
That surprises me given you said earlier that you are not after UHFT nor HFT. But maybe I misunderstood and you referred to your personal development work and now talk more about your job. I think we can all agree that Windows Server 2008 and 2012 are rock solid, as rock solid as any Linux Distro. For any latency requirements that can get away outside the sub millisecond arena I would always favor a Windows based solution. But that is just me.
@MarcG Can you share information with an issue you ran into with boost/locking using ActiveTick API? You can email it directly to support@activetick.com, and we will forward it on to IT to take a look at.