TickZOOM Decision. Open Source and FREE!

[greaterreturn]

One additional thought about that strategy/indicator discussion...

It will be a good habit to break up different indicators and strategies into separate parts.

Want to know why?

Part of my plan to make TickZOOM faster will be to make more of your logic run in "parallel".

Let's say you have strategy A which includes indicators B, C, and D.

Well TickZOOM will automatically work out those dependencies and it can calculate the B, C, and D formulas at the same time on different CPUs.

Once those results are done, then it can calculate your strategy A.

If you, instead, glum all that code into strategy A without B,C,andD. you lose out on that performance gain.

Now that is future.

But won't that be cool? Just to separate out the code and WHAM it runs much faster?

Wayne

 

[greaterreturn]

Quote from maxpi:

You are on the right track Wayne, little doubt about that. Handling ticks almost down at the firmware level makes imminent sense, I used to write firmware at times, I was always amazed at how long things took in higher level environments... making absolutely sure that backtests and real operations are the same is paramount. I do that in Ninjatrader and Openquant and Tradestation before them. I have to build my own bars in arrays and filter out bad ticks in my strategy code. Hard to program because I then have to write my own indicators to run on the arrays but worth it, I've proven that for myself, it sidesteps any and all ambiguities and many bugs in the environment.

Maxpi, just want to also thank you for these comments. It is nice to know someone else besides me has been around the block a few times and understands all this stuff.

I'm really hoping people like you will see the value in TickZOOM so that we can band together and make a powerful system.

I only humbly ask, if you and others feel I'm qualified to act as the steward to make sure everything everyone contributes gets integrated properly and thoroughly tested. (Since we have money riding on this thing.)

But if that happens, and we're all pooling our efforts we can have a really powerful automated platform.

By the way, maxpi? Is TickZOOM missing anything major that you need? Like what broker do you use? etc.

Sincerely,
Wayne

 

[Fractals 'R Us]

That all sounds good to me Wayne. You are focusing on the design, it's the only way to get good software, lots of effort up front in the design phase and the rest will fall into place nicely.

I'm with IB currently.

I'll provide beta tester feedback.

 

[janus007]

Quote from greaterreturn:

It's just as easy, if not easier, to do any of this type of analysis inside TickZOOM itself. What I do is write out comma separated values for statistics and load them in into a spreadsheet.

Yes, excellent and simple.

I personally see zero purpose in putting all the ticks over into a database like BDB. Maybe that's a matter of taste but I believe in doing things the easy way.

I could only remotely see copying bar data to a database for that kind of analysis but ticks don't give you bars directly. And you'll be hard pressed to find anything faster than TickZOOM at converting ticks to bars for your analysis.

I agree, it would be bars/ indicator values that would carry the interesting information.
This could be nice when a strategy needs decision making by datamining, but I guess one could develop a higher database integration in the future and if needed. Users can allways find 1000 things the system can't, instead of looking what's built for

You're missing the point. The TickZOOM engine cleans the data in real time both during back testing and real time trading.

It would be very naive to clean the data on disk because then you don't know for certain how TickZOOM will behave with real data in production.

Why on earth would anybody clean the data before storing it? In my world it would be the most stupid thing to do, as TICKZOOM handles and cleans data on-the-fly is completely perfect.

What? Well you are doomed to fail at auto trading live then, aren't you? What do you mean "no way"? Systems scrub data in real time all over the world. TickZOOM does it too--automatically.



Okay, you're very smart. Answer this:

If you supposedly "clean" all your data in the historical dataset and THEN you realize you found a better way of cleaning or discover it has an error or problem in the cleaning algorithm, how will you undo it/redo it?

The answer is simple. Never clean your historical data set. Keep it dirty and always run your tick filter against it any time you want to do statistical analysis, etc.

Then as the tick filtering algorithm evolves and improves. You can re-run it and make sure it still works.

Strange discussion, but apparantly few people knows how robust data cleansing/ conforming should be done the most effective and flexible way.

Certainly, use clean historical data but clean it "on the fly" and never physically modify the data as it was received from exchanges/providers. You need that original copy for testing.

That's because in LIVE trading you will get all those dirty ticks and you need to be confident TickZOOM cleans them appropriately or you adjust the algorithm, etc.

Of cause

Apparently you don't know what "max bars back" is. That's not related to ticks at all. It relates to "bars" and thus called "max bars back". 100,000 isn't arbitrary but a default.

Most people never need more than 200 or 300 max bars back. So 100,000 (when they know what it means) will blow their minds.

100.000 is a stupid high number which did blow my mind

 

[greaterreturn]

Quote from janus007:

I agree, it would be bars/ indicator values that would carry the interesting information.
This could be nice when a strategy needs decision making by datamining, but I guess one could develop a higher database integration in the future and if needed. Users can allways find 1000 things the system can't, instead of looking what's built for

Excellent point. That is possible and also the beauty of open source let's aLL innovate!

Why on earth would anybody clean the data before storing it? In my world it would be the most stupid thing to do, as TICKZOOM handles and cleans data on-the-fly is completely perfect.

Hey! are you saying I'm stupid because I did it that way myself till I realized better? (Just kidding )

Strange discussion, but apparantly few people knows how robust data cleansing/ conforming should be done the most effective and flexible way.

Of cause

100.000 is a stupid high number which did blow my mind

I want to make a correction to this statement. While documenting I noticed TickZOOM currently has 100,000 as the default only for TickSeries. In other words, that's the default max bars back for single ticks.

That's because for calculating "rolling" bars, it needs plenty of previous ticks to calculate.

The rest default to 1,000 at the moment. Using 100,000 on all of them would probably be a waste of memory. Especially on Year bars? Who needs to look back 100,000 years? If you do, I want a picture of you on the wiki!

So my idea is to have graduated default max bars back. For tick bar, starts at 100,000 for single tick bars and divides by the number of ticks in a bar. So a 100 tick bar has 1000 max bars back.

For time based bars, we can start with 50,000 for second bars and graduate down to 100 m.b.b. for year bars.

BTW, TickZOOM converts all bar durations to seconds so we need a little formula that sets the default based on the number of seconds.

Any mathematicians out there have a suggestion?

For the volume, change, and range bars, they can all be 10,000 default.

Sincerely,
Wayne

 

[greaterreturn]

Someone just mentioned in a P.M. that one of the problems with AmiBroker is that it has only one developer and the list of requested enhancements is a mile long.

Does anyone want that to happen to TickZOOM? Making it open source will help some. But we must look to other solutions also. Open source developers work at their own pace and only on what they want to work on. Not necessarily what users in general want.

Sincerely,
Wayne

 

[Fractals 'R Us]

Quote from greaterreturn:

Someone just mentioned in a P.M. that one of the problems with AmiBroker is that it has only one developer and the list of requested enhancements is a mile long.

Does anyone want that to happen to TickZOOM? Making it open source will help some. But we must look to other solutions also. Open source developers work at their own pace and only on what they want to work on. Not necessarily what users in general want.

Sincerely,
Wayne

Uhh... have you written to Santa? jk

Open source projects are sort of known for lists of unattended to issues. You have to go commercial. Accept donations, charge, maybe not for the core but for add-ons, strategies, etc... or get some grant money.

I think this was mentioned before... offer a core version that has some functionality and demonstrates the speed and if people like it they will pay [a good bit probably] for charting add-ons, indicators, strategies, etc... or allow free use of everything but charge to allow connection to a brokerage as does Ninjatrader.

One thing that I don't like about Ninja is that it has to check it's license server before I can trade with it... that runs counter to my security measures, I have a whitelisting firewall, the less url's I allow the more secure I am. I don't want my intellectual property stolen.

 

[ML_QUANT]

I have studied and tried EL, OQ and now actually using NT. Your Open Source IS what has been missing with all of them!
I wish you all the luck in the world your undertaking is humongous and hope others would step in to help too.
Maybe eventually add taking on your site might generate some income too.
As for the size of data bases, I am presently writing live tick data to txt files, one for each day/symbol and then using low level functions to read just the freshly added data using a timer! Not the fastest in the world but saves a ton of memory and can record files as large as the HDD available!
Your Vid. is very nicely done and WMP didn't open it then I used VLC, which is free and it was just fine no glitches.
Looking forward to your source and future Vid. clips.

 

[ML_QUANT]

Quote from OpenQuant:

PS. If someone is interested in looking on OpenQuant demo videos, demonstrating backtesting with 1M+ ticks per second, including market depth events, and switching between backtesting and live trading with one mouse click, it's here

http://www.smartquant.com/doc.php

The source code, supported under 7/24, is also available, though it's not free

Cheers,
Anton

I think SQ is a good $20k+!!!, so TZ if ever completely developed should be a blessing...

 

[ML_QUANT]

The Zenfire feed is very quickly becoming the feed of choice for the serious advanced traders due fast and unfiltered data.
I also use IB which is fine as a secondary source but definitely not a Primary.