...Basically, the target is to buy historical tick feed and keep on adding to it on a daily basis using a real-time feed provider...
Quote from thstart:
What is the benefit of analyzing real time data? If you go through all of this and come with a profitable strategy how it will be used in practice?
If the real time data are in the milliseconds range you have to be closer physically to exchange to get the benefits from real time data strategy.
A real time strategy is profitable in a long run if you are a broker/dealer and have the benefits of sending the order flow to the exchange. Then the speed matters a lot.
Actually, I am intending to analyze tick data and somewhat quasi real-time data. Real Time data is supposed to be added to the historical data store, as it comes in, rather than be accessed on slow internet speeds it can be accessed by multiple simulations at LAN speeds.Quote from thstart:What is the benefit of analyzing real time data?
I agree, but then thats what DMA technology is for. Too early for me yet to even dream of it, but I am targeting a DMA platform, where my order go directly to the execution exchange on a leased line, of course under a sub-id issued by the broker.If you go through all of this and come with a profitable strategy how it will be used in practice?
If the real time data are in the milliseconds range you have to be closer physically to exchange to get the benefits from real time data strategy.
Actually, I am targeting running multiple simulations in parallel with actual trading. Don't really know where I will land up.A real time strategy is profitable in a long run if you are a broker/dealer and have the benefits of sending the order flow to the exchange. Then the speed matters a lot.
Quote from tickzoom:
If you are not close to the exchange you don't really have a "real time", but more like a "delayed" time.1. ...Monitoring a real time feed..
2. ... let an order server handle the trades...
This makes sense because/if the server is located closer to the exchange. But from other side it is not the wisest because somebody is seeing your orders.
3. Others want to respond withing milliseconds to certain events rather than wait all the wait till end of day.
They cannot do that fast enough if far away from exchanges. For example if they are located in California.
There are more reasons but these are the most common reason for monitoring a real time data feed.
For monitoring it is OK. But if you cannot respond fast enough it is useless. Millisecond race game is for the big companies having a servers on the exchanges.
Their strategy has nothing to do with
traditional technical analysis, etc.
If you are not a broker dealer you can not have an advantage. For example pinging the market is one strategy. You ping the market to see if there is an appropriate order. If you are broker/dealer you can CANCEL your order within 1 sec if not appropriate. Then go down until you recognize an institutional algo computer against your computer. The you can go below his buying orders to get all orders from the market thus mini-cornering the market for several seconds. Then sell higher to him with a little profit. Plus getting your commission for the order flow.
Now, if you are not broker/dealer with all his leverages and advantages how you can do that? Meaning you do not have the leverage being broker/dealer you cannot have the advantage of milliseconds executions.
It is just a perception that you have an advantage.
If you are not one of them better stick to EOD analysis and have the peace of mind.
Also to handle and analyse the millisecond level amount of data is not easy but this is a different topic.
To me, this server is just a data feed server and further back-testing infrastructure would be separate, most probably a grid of inexpensive machines on linux or other license cost free OS, so that new servers can be added to the grid on need, at hardware & maintainence costs only.
I would be really interested to know how do the data feed providers store & serve out their data. That's all I intend with this sort of a server. Just like a proxy server for feed but with full history so no data except real time data need be queried from data provider, all data provided from local data store to multiple clients on the local LAN at LAN speeds.
Sanjay. [/B]
Quote from Pippi436:
If you feel like spending money, have a look at KDB+ or Vhayu. Serveral institutions (as well as data providers) use these to store and distribute historical as well as realtime timeseries data.
Quote from thstart:
If you are not close to the exchange you don't really have a "real time", but more like a "delayed" time.
Quote from thstart:
This makes sense because/if the server is located closer to the exchange. But from other side it is not the wisest because somebody is seeing your orders.
Quote from thstart:
They cannot do that fast enough if far away from exchanges. For example if they are located in California.
Well, I agree with you but other traders disagree depending on their needs. Even if their data lag is 100 milliseconds, some traders find the slippage acceptable at least to avoid the expense of a server at a data center.
Most traders are trading based on hour bars or even daily bars. Usually turning around the data, trading decision, order submission and confirmation within 1 second is excellent for those traders.
Now if someone is doing high frequency trading or MM then they need to be colocated in the data center to get their round trip to trade time down as you say.
Quote from thstart:
For monitoring it is OK. But if you cannot respond fast enough it is useless. Millisecond race game is for the big companies having a servers on the exchanges.
No offense, but your mixing two different issues. Most traders aren't trying to do high frequency, sub second trading. That's not the reason for the trading speed. To understand this the requirements must be clear.
Here is the requirement:
The trading server must receive data, make decision, submit the order, and get confirmation as fast technology allows to reduce or eliminate slippage between the tick that triggered the trade and the actual fill price.
That way users can create any kind of sophisticated orders or logic that never appear at the broker or exchange, historically test them, and then get reliable and similar results when real time trading due to consistently low or zero slippage.
Quote from thstart:
Their strategy has nothing to do with traditional technical analysis, etc.
If you are not a broker dealer you can not have an advantage. For example pinging the market is one strategy. You ping the market to see if there is an appropriate order. If you are broker/dealer you can CANCEL your order within 1 sec if not appropriate. Then go down until you recognize an institutional algo computer against your computer. The you can go below his buying orders to get all orders from the market thus mini-cornering the market for several seconds. Then sell higher to him with a little profit. Plus getting your commission for the order flow.
Now, if you are not broker/dealer with all his leverages and advantages how you can do that? Meaning you do not have the leverage being broker/dealer you cannot have the advantage of milliseconds executions.
You really know your stuff! But again, the O.P. nor most traders are interested in that level of institutional trading.
Regardless, all traders want very quick and accurate execution of trades to reduce slippage. When you keep the trades off the brokers books then you must process the market or at the money limit orders rapidly.
With TZ at a datacenter near the exchange (but not colocated), we find that the network lag for real time data is only 60 ms. Then TZ turns around and fire a trade decision in less than 1 ms and usually within 50 ns (nano seconds) so the 60 ms network lag happens again to xmit the order. Then the order actually processes at the exchange within 300 or 500 sm but, of course, there's another 60 ms lag for the broker to send confirmation of the trade. So the total round trip from exchange, decision, trading, and confirmation is from 500 to 1000 ms depending on exchange load.
The results of slippage study is 30% of the time zero slippage or trade at the same price as the tick bid/ask. 30% of the time you get favorable slippage and 30% of the time unfavorable slippage.
The final result or total slippage was very near zero since the positive and negative slippage canceled each other.
If a trader is doing anything more frequent than EOD analysis than near zero slippage makes a major improvement in trading results.
Quote from thstart:
It is just a perception that you have an advantage.
I'm not sure if anyone was looking for an advantage. It's just a necessity for accuracy. In other words, who in their right mind wants slow order fills and large slippage?
Quote from thstart:
If you are not one of them better stick to EOD analysis and have the peace of mind.
On that point, we agree completely. EODers need to stick to EOD at least until the lag in settlement prices and overnight trading sessions starts to hurt. Then another approach is necessary.
Quote from thstart:
Also to handle and analyze the millisecond level amount of data is not easy but this is a different topic.
You're correct, it's very hard if someone wants to build this entire processing themselves. But it's very easy with TickZOOM because you're using tools built, tested, and proven to be suited to that purpose
Sincerely,
Wayne
Quote from sanjay_arora:
Actually, I am targeting running multiple simulations in parallel with actual trading. Don't really know where I will land up.
Quote from sanjay_arora:
Hypothetically, assume you have 100 trading strategies, built over say 25 years and finetuned over the years. My assumption is that a few of them would be profitable in one time frame, few in another, few would be good to day trade, few for swing trade & few for scalping but at any point of time most of them would be unprofitable.
Another assumption is that for short or intermediate periods in time, some of the profitable strategies would become loss-making & visa versa.
Quote from sanjay_arora:
I see GRID based inexpensive computers running all strategies available to you in various timeframes and a master algo keeping track, what is becoming profitable and what is becoming unprofitable...and starting & stopping them as required or rather alerting the trader team that a strategy that had been unprofitable since last n years, has suddenly become profitable.
Quote from sanjay_arora:
Pick your permutation. All I am doing is creating a local feed & simulation/monitoring system.
Quote from tickzoom:
So your servers are located in the basements around exchanges?...We run in real time with exchange...
Unless you host a customers' accounts on your server this is inevitable.The central issue isn't simply the speed or lag of the data but the total turn around of exchange data, decision, order, and confirmation to reduce slippage between the tick that triggered the trade and the actual fill price.
To clarify - I mean such high frequencies makes sense for order execution, when the server is closer to exchange, there is a web service in place to host customers' settings, etc....No offense, but your mixing two different issues...
Form other side to make historical analysis, screening, back testing on millisecond data doesn't make much sense if you cannot apply all of this to to an actual order execution.
Now, the question is if you cannot make a reasonable decision for placing a real time buy or sell order, based on a historical analysis, how do you place your order then, based on what?
My point is - only "anybodies" how you name them can take advantage of all this high frequency stuff. Because what they are doing is very simple and based on advantages they get from being "they". All this milliseconds race don't make sense to them too because practically this is computer against computer race, a human simply cannot be involved much in the real time process, only off line, tweaking algorithms, etc.
This is a part of the problem what we have seen to happen recently - when the fastest and closest to the source computer discovers an order first, the other computers which are constantly searching "catching" this and bidding and outbidding in sub second speed and quitting before 1 sec, nobody else knowing of this, but the price up at no volume. I you follow these fluctuations at millisecond level whatever you do they are faster.
The trading server must receive data, make decision, submit the order, and get confirmation as fast technology allows to reduce or eliminate slippage between the tick that triggered the trade and the actual fill price.
...receive data, make decision, submit the order, and get confirmation...
The point is there is somebody else with a faster technology and you cannot catch them fast enough. Unless you are broker/dealer with their resources it is simply not possible to keep up with the speeds.
You really know your stuff! But again, the O.P. nor most traders are interested in that level of institutional trading.
Even if they are interested they are not ALLOWED to participate.
Regardless, all traders want very quick and accurate execution of trades to reduce slippage. When you keep the trades off the brokers books then you must process the market or at the money limit orders rapidly.
This part - keeping trades off the brokers' book this would be useful.
With TZ at a datacenter near the exchange (but not colocated)...from 500 to 1000 ms depending on exchange load...
.5-1 sec is a lot of delay, I know why - this is not a problem in your system. It simply a combination of regulations (who has the right for <1 sec rule), and law of physics.
If a trader is doing anything more frequent than EOD analysis than near zero slippage makes a major improvement in trading results.
I also wish this to be possible but for me - even you get full advantage from situation or get another strategy. With high frequency you lose the war before to begin it. Opposed to - you have to win the war before starting it (Sin Tzu).
I'm not sure if anyone was looking for an advantage. It's just a necessity for accuracy. In other words, who in their right mind wants slow order fills and large slippage?
If the trading strategy depends from millisecond early or later execution, this is not wise. You cannot control anything in that time span whatever you do.
On that point, we agree completely. EODers need to stick to EOD at least until the lag in settlement prices and overnight trading sessions starts to hurt. Then another approach is necessary.
I believe you have to do EOD to have a little long term prospective. Then an actual trading with a time span of 3-5 days would be profitable if you make a good research, trend following etc.
But if you depend from milliseconds, seconds, even minutes in execution there is not a point to do that unless you are not a broker/dealer with a lot of leverage to extract the maximum potential of your position.
You're correct, it's very hard if someone wants to build this entire processing themselves. But it's very easy with TickZOOM because you're using tools built, tested, and proven to be suited to that purpose
I believe you made a lot of effort to create your system.
Quote from thstart:
My point is - only "anybodies" how you name them can take advantage of all this high frequency stuff. Because what they are doing is very simple and based on advantages they get from being "they". All this milliseconds race don't make sense to them too because practically this is computer against computer race, a human simply cannot be involved much in the real time process, only off line, tweaking algorithms, etc.