Ninja Zen no more unfiltered data?

gastropod · Mar 6, 2010

Quote from PocketChange:

Windows performance timers are accurate for measuring processes to microseconds. Using NTP to sync System Time with the exchange or Internet Time Servers should get you within +/- 50ms.

Should be fine if your just collecting data to run simulations.
You may need to adjust or throttle your processes if you plan to trade live off the data stream.

This thread is confusing real time feed and order processing versus sequential data feed logging. From your replies seems like you are focused on the latter.

Two things are going on here...

Please see -> http://en.wikipedia.org/wiki/Preemption_(computing)#Time_slice

Winblows can measure microseconds, IF SOMETHING IS TIMED IN A TIME SLICE. The problem is that Winblows does 10 ms time slices. Thus the AVERAGE "hunk of data" (aka your ticks) will have an added extra 10 ms on a Winblows box. If both you and your data provider are using Winblows, then you will get your and the data providers added 10 ms average time slice...thus you will have 20 ms average latency on your ticks. The previous discussion is for XP/2000/2003/NT. I do not know about Vista/7. (It can be "tuned," but most shops don't have the in house expertise to tinker with the registry settings to put the time slice to 1 ms...and changing that setting may screw up a bunch of things on a Winblows box.)

For Linux (and many UNIX) shops and users...
If you are using a 2.4 kernel...upgrade...you are still using a 10 ms time slice. If you are using a 2.6 kernel...than (IIRC) you have a 1 ms time slice out of the box.

That is why linux shops (generally) have less latency than your Winblows shops.

To agree with a previous poster (and to disagree) - yes, Winblows does not run NTP well. A linux box runs it fine. I have seen many Linux boxes running within 1 microsecond of UTC. I am sure there are those here who have seen linux boxes within nanoseconds of UTC.

-gastropod

disclaimer: yes, I work with data on Linux boxes.

maxima120 · Mar 7, 2010

The problem is that Winblows does 10 ms time slices. Thus the AVERAGE "hunk of data" (aka your ticks) will have an added extra 10 ms

you talking rubbish mate..

Cazza La Randa · Mar 7, 2010

Quote from AMT4SWA:

Have not looked at this thread in a few days so I did not see your questions.

Real simple though.....the MAIN reason I have been very interested in TradeVec is the fact that they will be adding CUMULATIVE DELTA volume tools in March. So with TradeVec, I could then have my trade platform and charting all in one package. The second MAIN reason I really like TradeVec is the TWO FEED capability......I could have DTN.IQ feed for my charts (for CLEAN bid/ask data needs) and the TT Fix Adapter feed for my order execution. They do not yet have the capability to connect to DTN.IQ but they are moving forward to add this data vendor in the months ahead.

I am also a good friend of Paul at Vcap Futures who helped with the initial development feedback for the TradeVec platform. Paul is the one who first introduced me to the TradeVec platform and I really like the overall direction this trading software is going. BTW, I have found Paul at Vcap Futures to be one of those few very sharp and excellent at customer service. That is very rare in the trading brokerage world these days and I very much value his information......hope that helps!

Bid Ask Cumulative delta volume is already available through Ninja Trader

AMT4SWA · Mar 7, 2010

Quote from Cazza La Randa:

Bid Ask Cumulative delta volume is already available through Ninja Trader

Yes.....I was using it as back up charting (in addition to my Investor RT set ups) with the GomCD Cumulative Delta study off the NT forums. Since the CME data changes which started October 5th of 2009, I no longer use NT for any Cumulative Delta work. Broker supplied feeds like Zenfire/ Rithmic, TT Fix Adapter, etc no longer work for proper robust BID/ASK differential data use in NT......these feeds have too many intraday data drops. At this time, DTN.IQ feed is about the only feed I have seen used with NT that seems to work OK for BID/ASK differential data work.

maxima120 · Mar 7, 2010

Broker supplied feeds like Zenfire/ Rithmic, TT Fix Adapter, etc no longer work for proper robust

What about Transact/Infinity? I heard they are improving their feed...

PocketChange · Mar 7, 2010

Try Excel with the VBA code below to precisely time processes. Windows performance timers should resolve to millionths of a second depending on your hardware.

Code:

'just a small helper-module for precise timing
Option Explicit
Private Declare Function QueryPerformanceFrequency& Lib "kernel32" (x@)
Private Declare Function QueryPerformanceCounter& Lib "kernel32" (x@)

Sub processtimer()
Dim Ctr1 As Currency, Ctr2 As Currency, Freq As Currency, Overhead As Currency, i As Long 
QueryPerformanceFrequency Freq
QueryPerformanceCounter Ctr1
QueryPerformanceCounter Ctr2
Overhead = Ctr2 - Ctr1 ' determine API overhead
QueryPerformanceCounter Ctr1 ' timer start

'Place Code to time here  

QueryPerformanceCounter Ctr2 'timer stop
Debug.Print (Ctr2 - Ctr1 - Overhead) / Freq
End Sub

Out of the box windows does not support NTP.

Quote from gastropod:

Two things are going on here...

Please see -> http://en.wikipedia.org/wiki/Preemption_(computing)#Time_slice

Winblows can measure microseconds, IF SOMETHING IS TIMED IN A TIME SLICE. The problem is that Winblows does 10 ms time slices. Thus the AVERAGE "hunk of data" (aka your ticks) will have an added extra 10 ms on a Winblows box. If both you and your data provider are using Winblows, then you will get your and the data providers added 10 ms average time slice...thus you will have 20 ms average latency on your ticks. The previous discussion is for XP/2000/2003/NT. I do not know about Vista/7. (It can be "tuned," but most shops don't have the in house expertise to tinker with the registry settings to put the time slice to 1 ms...and changing that setting may screw up a bunch of things on a Winblows box.)

For Linux (and many UNIX) shops and users...
If you are using a 2.4 kernel...upgrade...you are still using a 10 ms time slice. If you are using a 2.6 kernel...than (IIRC) you have a 1 ms time slice out of the box.

That is why linux shops (generally) have less latency than your Winblows shops.

To agree with a previous poster (and to disagree) - yes, Winblows does not run NTP well. A linux box runs it fine. I have seen many Linux boxes running within 1 microsecond of UTC. I am sure there are those here who have seen linux boxes within nanoseconds of UTC.

-gastropod

disclaimer: yes, I work with data on Linux boxes.

maxima120 · Mar 7, 2010

You made mistake.. the dude is going now to open full blown discussion about linux, open source, bush-idiot, ufo in nevada etc.....

gastropod · Mar 7, 2010

Quote from maxima120:

You made mistake.. the dude is going now to open full blown discussion about linux, open source, bush-idiot, ufo in nevada etc.....

Yet another clueless Winblows idiot! What I wrote is correct. With a preemptive multitasking OS time is broken up into slices. Each "task/process" is given a slice to run in. When the time for the task to run is over the process is "switched out" and another process has control of the CPU. Think of it this way...

<pre>
Begin time slice 1
Your trading app runs here...
a packet comes in - we'll call it "tick A"
tick A is processed in microseconds
your trading app does some other processing
End time slice 1
Begin time slice 2
Your solitaire game has control of the CPU
Another tick packet comes in - we'll call it "tick B" - "tick B" is "buffered" in RAM
You make another move in solitaire
End time slice 2
Begin time slice 3
Your trading app runs again
"tick B" is processed
etc., etc.
End time slice 3
</pre>

Will your ticks that come in during your trading apps time slice process in microseconds...yes. The problem is - about half of your ticks will come in during a time slice that your trading app is NOT running on the CPU. Ticks - like "tick B" will have to wait until your trading app has the CPU again to get processed. They are buffered in memory until they can be processed. ON AVERAGE - your ticks will wait one time slice to be processed. On a Winblows box running without optimization - that time slice is 10 ms. On a linux 2.6 kernel box - the time slice is 1 ms - out of the box - no added configuration necessary. Those are the facts! The time slice is added latency regardless of OS. The difference is that Winblows uses much longer time slices.

Conclusion: Out of the box - linux introduces MUCH less latency to ticks than a Winblows box.

-gastropod

gastropod · Mar 7, 2010

Quote from maxima120:

Go and fuck yourself cluesfull fagot

Funny - a guy who bashes Bush and then calls somebody a "fagot"??? But I thought you liberals were the champions of gay causes and the like...now you bash gays? Typical liberal! Straight liberals only associate with gays for the "gay" vote. In reality the straight liberals hate the gays and that is odd, because so many gays consider themselves liberals - go figure!

If you really had a brain - you would have looked past my calling you an idiot and you would have seen that I am giving you useful information.

-gastropod

gastropod · Mar 7, 2010

Quote from PocketChange:

Try Excel with the VBA code below to precisely time processes. Windows performance timers should resolve to millionths of a second depending on your hardware.

Code:

'just a small helper-module for precise timing Option Explicit Private Declare Function QueryPerformanceFrequency& Lib "kernel32" (x@) Private Declare Function QueryPerformanceCounter& Lib "kernel32" (x@) Sub processtimer() Dim Ctr1 As Currency, Ctr2 As Currency, Freq As Currency, Overhead As Currency, i As Long QueryPerformanceFrequency Freq QueryPerformanceCounter Ctr1 QueryPerformanceCounter Ctr2 Overhead = Ctr2 - Ctr1 ' determine API overhead QueryPerformanceCounter Ctr1 ' timer start 'Place Code to time here QueryPerformanceCounter Ctr2 'timer stop Debug.Print (Ctr2 - Ctr1 - Overhead) / Freq End Sub

Out of the box windows does not support NTP.

Hello PocketChange, I hope you do not think I am arguing with you on this thread. My discussion/rant on time slices may or may not concern what you are working on.

I do not know whether you care about the added time slice time. I believe your calculations will show you your "total" time for processing in microseconds. My whole beef is that calculations may not show (which is OK, if you don't care about them showing up and latency is not a problem for you) latency, which (if you care) is included in packet processing on any OS.

The scenario is this...

Begin time slice - you get packets - you process packets - near the end of first time slice - you get "tick C" - you begin processing "tick C" and have added a "time stamp" to it - oh oh - time slice over - wait 1 time slice for "other" program to run - now trading app runs again - "tick C" finishes processing

Does your calculation include the time of the time slice or does it just show the time of processing? Some people (who watch the latencies) care about the latency time and others (those who don't care about latencies) don't care about the added time slice.

Sorry about my lack of good communication here, but I will try an example to explain what I am saying...

The world record for the 100 meter dash is around 10 seconds. How about if I "sprint" 5 meters in 0.5 seconds...take a one hour break...come back and "sprint" another 5 meters in 0.5 seconds...repeating this sprint/break pattern for a total of 20 of my "sprints." In the end, yes, I have "run" 100 meters in about 10 seconds...but, did the 19 hours of breaks I took in between count???? My total calculation is a "10 second 100 meter run" - but my "elapsed" time is close to 1 day!

Just a thought or two

-gastropod