Open Source Black Box Trade Platform in C#?

[greaterreturn]

I dont' think I mentioned market replay. TickZoom allows you play though ticks in a back test while showing you on a chart. You can choose the speed of the replay. I find it very educational to watch my strategy run at say 5 to 10 X speed.

Very often issues pop out which don't catch my eye on a static back testing chart.

If you ever run automated trading systems, you'll find out the EVERY trade counts. In a back test of 1000 trades it's easy to miss a problem trade here or there. But when running live with real money, you never miss a single trade.

So it helps to get a feel for how your strategy runs during a market replay.

Wayne

 

[greaterreturn]

Quote from Pippi436:

Looking forward to that.

That whole project sounds quite monumental. How long have you been working on it?

Well, it's several years, if you include:

1. Trying to build an automated system using other platforms and learning their limitations.

2. Using platforms like NinjaTrader and NeoTicker that allowed some C#. But ultimately fell short due to the close source policy.

3. Actually building TickZoom.

#3 has been at least 10 months of near full time work. (Weekends and evenings.)

Sincerely,
Wayne

 

[greaterreturn]

Interestingly, the most challenging part was getting the black box server portion working reliably with MB Trading.

Specifically, there are nightly interruptions in the connection to MB Trading at midnight. And periodically there can be short interruptions in data feed.

Overal MB Trading servers are extremely reliable however it was frustratingly difficult to figure out an algorithm to reliably reconnect in every situation.

Why? MB Trading discourages the public from running black box systems, hands-free.

They don't want to be blamed if the trader makes a mistake in their rules and loses a bunch of money.

So they make it possible but decline to provide instructions or info on how to do it.

That alone seemed like the hardest part to figure out and get it working.

Oh. But it works beautifully now. Whether you unplug the cable, reboot the server, if MB quote server goes out briefly, or whatever, it reconnects quickly and happily.

Anyway, thanks for recognizing the hard work that went into this, pippi.

Sincerely,
Wayne

 

[greaterreturn]

An administrator agreed to edit my original post in this thread. And he put the change in bold.

The change was to clarify the business model. I found some better business intelligence on how the open source model works.

The idea is to build a two part community that benefits each other. One part gets the source code entirely for free and contributes (or not) by time in testing, reporting or fixing bugs, adding documentation, and enhancements.

The other community is made up of actual paying customers who also have access to the source code free of charge but choose to pay for services like a packaged build with easy installation, technical support with an SLA, and security or reliability updates.

That's the model followed most of the successful open source projects.

Sincerely,
Wayne

 

[greaterreturn]

Another point of research was how to build the community.

Before beta testing the following needs to be setup:

1. Website Wiki with documentation and a project roadmap.

2. Source code in ZIP and also a public Subversion repository to make it easy for committers to submit new code or fixes.

3. A defect tracking system.

4. And a forum for questions and discussion.

After that is all setup we can begin beta testing.

Sincerely,
Wayne

 

[greaterreturn]

Quote from TSGannGalt:

Here's a few things.

1. Are you looking for a Testing Platform or all-in-one platform?

"looking for"? If I understood your question (maybe not). TickZoom has both historical testing, live trading, and a quote server to collect data. Here's more detail on each.

TickZoom 3 parts.

A. The main program with a GUI interface with charts for back testing or live automated trading.

B. Windows Service called TickZoom Quote Server which connects to data server and collects historical data with bid/ask, volume, and DOM. It's a separate program so it can stay running always and avoid gaps in historical data. I don't see how you could effectively combine this. It could be a separate tool some people don't need right away.

C. A Windows Service called TickZoom Order Server. It runs the same strategies for live trading but without any GUI so it can run 24 hours a day and trading based on your rules. It connects to the Quote Server to get a live data feed and place trades. Again I don't see how you could combine this. However, the main GUI can run strategies live while updating the charts in real time. So again, maybe this is a separate piece retail customers don't need up front.

This may be just a question of which components get packaged for download and installation together.

If you want more retail / small people. Keep them together. You'll get more people but most of the feedback will be full of whiners and gimme gimme mentality.

Understood. Still, those people provide value. Their ideas and suggestions (or whining as you call it) is a tremendously valuable service in that they find and report issues and offer ideas for enhancements. Also a percentage of those retail users may contribute documentation or corrections to documentation that are valuable.

If you want more professionals coming in. Then seperate them. If you want execution stuff and all the other perks, then implement FIX and interfaces relating to them. Broker APIs are just going to be a pain in the arse.

I agree completely. I checked on FIX at MB Trading and they have high requirements or high costs to use it. So I didn't need it initially. But I'm happy to implement it soon so we can attract those type of customers. My guess is that those people who want FIX will also wish to pay for support and other services to become paying customers.

On the other hand, only supporting FIX would prevent many retail people from using TickZoom due to the barriers to getting FIX access with most brokers. (am I wrong?)

The key is to have them seperated while using the same class conventions. For example:

"public void BuyAtMarket (symbol, quantity) {...}"
will work at:
"public void BuyAtMarket (symbol, quantity) {...}"
in the execution platform.

Those are great suggestions. But, of course, that's already included in TickZoom. In fact, it works in such a way that your custom strategy and indicator code doesn't need anything different to trade historical versus live.

That's critically important in my view. Don't you want the EXACT same code tested thoroughly to also be same to run in live trading?

One of the commercial platforms I tried to use shocked me in that the code I wrote to back test couldn't be used in live trading. I had to modify it significantly for it to work live.

But how could I test that code to make sure my modifications didn't create a problem?

You see, it's a critical point that the strategies and indicators be exactly the same without any if/else between test and live trading.

In TickZoom, that's configuration of which you choose to run or your login id and password--whether a live or demo trading account.

Take a class with the model inside and changing the top:
"using BlackBox.OrderEntry.Test;"
to
"using BlackBox.OrderEntry.Live;"
... making it work...

what is better is if you can code the core features and set a custom assembly like "Debug" or "Release" to "Test" and "Live":

#if LIVE
using ....
#elif TEST
using ....
#elif DEMO
using ....
#endif

In another words, I have 2+ projects accessing a 3rd Model Lib. where it would do different stuff...

Exactly. Exactly. That's how this system works. Only it's the reverse of how you describe it.

Think of Model / View / Controller pattern (you're obviously a programmer).

Consider the your customer strategies and indicators to be the "model".

Now the tick engine is the "controller". It may interact with the model in a test mode, live mode, replay data mode, or optimizing parameters.

Finally, the charting and HTML statistics output are the "View" into the model. That view also changes. If running on a server, it just writes out a chart every minute or two to a file and generates HTML with all the trade statistics so you can open and see what's going on.

But on the testing platform, it gives a different view in the form of dynamic charts with zoom/pan/scrolling, etc.

2. Mind as well help out the C# QuantLib guy...

http://www.qlnet.org/

Hunh? You lost me. Do you mean you help him? Or me help him? FYI, you can use QuantLib with TickZoom together if you want. Especially if you're already familiar with QuantLib.

Maybe we'll see about packaging them together if they agree and it makes sense.

I also include .NET port of the Drools a business rules engine. I don't use it much now but it does seem to simplify the rules on complex trading models.

It needs some more investigation. Maybe some advanced users will appreciate that feature.

3. I have to continue to emphasize this... How far do you want to go? Do you want things for non-programmers or do you want things for professional developers?

I think the answer is both. After reading this, let me know if you agree or not.

That's probably the only way to construct a healthy community with enough market share to continue to thrive. I built it to be relatively easy to convert EasyLanguage to TickZoom strategies and indicators.

I think this is more important to straighten out rather than all the discussions made so far.

I GREATLY appreciate you delineating this question.

FYI, I plan to write an official "roadmap" saying where the project is headed, define who are the target users of the platform, etc. So you're helping greatly to frame it.

I'll expect you and others to give feedback until we have something that make sense.

Again, any contribution of ideas or advice on the Roadmap is EXTREMELY welcome.

Now here's my current thought on the direction in terms of retail vs professional.

Those who are programmers and want to tinker with the code (like me) will be the life blood of the project and due to the nature of open source make it more powerful over time than any other platform.

Those people deserve to use it for free and add whatever features make sense in the platform and get free support, etc. The product will driven largely but this community of developers. I expect to just be one of many. But it looks like I'll also play the "administrator" role in getting the original tools setup and drafting and publishing the roadmap or direction of the project.

Separately, the project will need paying customers to fund full time developers and support.

Perhaps that will be the non-programmers if they want additional training or support. Perhaps we sell video training or books on how to use the system, etc.

The primary focus in the beginning will be building a healthy community of programmers, testers, and documenters who contribute to the community good.

You for example are making a big contribution by asking this question. Most of this thread will end up on the TickZoom Wiki when I have ready.

The best intelligence on how to run an open source project like this can be found at :

http://wiki.pentaho.com/display/BEEKEEPER/The+Beekeeper

That site explains the "beekeeper business model" that is behind all the majority of successful open source projects in the world today.

Please work back and forth with on this. Let's bang out a roadmap that makes sense to build a thriving community.

Sincerely,
Wayne

 

[good one]

Well, I just have to say you are the man Wayne.
This sounds fantastic.
As far as programmers/non programmers go..I'm a non programmer with one basic class in Pascal who has struggled through to learn C# to program ninja trader..
Obviously, I'm biased but I just think things that try to "help" non programmers program, really just ends up adding bloat and wasting everyone's time.
I remember reading a quote on this board searching for autotrading stuff that "retail software tries to do everything and in doing so does nothing well" or something like that.
I think a good model is being able to do anything you could want autotrading wise, with the ability to chart and discretionary trade as a secondary feature.

As far as a robust community, I think far more than indicators and what not, datafeeds are the spark that creates the community. To me that is ultimately what will decide getting the critical mass of users/pro developers/hackers/people just trying out the software to make the project viable over time.
I do think Ninja also came up with a brilliant strategy as far as giving a free version that is usefull but slightly crippled. I'm sure there are far more payed license now than if they had made people pay upfront.

anyway, can't wait to see what happens with this.

 

[Big]

As far as storing Time series with an open source technology that works with C#, I would personally go with db4o (http://www.db4o.com/) as it has proven many times to be very reliable, high performance, easy to use, support queries and is object oriented (ie you just save objects in it).

 

[greaterreturn]

Quote from Big:

As far as storing Time series with an open source technology that works with C#, I would personally go with db4o (http://www.db4o.com/) as it has proven many times to be very reliable, high performance, easy to use, support queries and is object oriented (ie you just save objects in it).

Wow. Big. That sounds like a great suggestion to consider. It so happens, I used an OODBMS in a past life (C++) called ObjectStore. That was on a real-time financial system and, as you say, simply storing objects produced screamingly fast performance.

In fact, that's what TickZoom does for speed but it doesn't have a query language for selecting tick ranges, etc. So it seems like a great suggestion to incorporate an OODBMS.

Since db40 is open source also, that makes it a potentially great partner.

Right now in TickZoom when you want to say zoom in and run tests only on one week of tick data in my database, it still takes 10 seconds to sequentially read to the right point. As the data grows that will become unacceptable.

Here's a potential challenge to resolve with OODBMS.

One of the big issues I have found with writing and reading objects and especially ticks, are versioning of objects. Earlier, as the Tick class evolved with additional fields, like volume, and later the DOM. It orphaned past stored data.

So tick storage was refactored to support backwards version compatibility. That is, any new version of tick format still reads the old version and just extrapolates or zeroes out any new fields unavailable in that version.

We'll have to figure out how to do something similar in db40. To do it in TickZoom, I had to drop serialization and write and read the raw bytes.

The way it works now is when a programmer needs to add a field to the tick class, you increase the version number field. Then you add into the ReadData() and WriteData() methods to read and write those additional bytes only if the data version is >= the current version. Then you add what to use for the data for lower versions.

In that way, it's always possible to read older versions of data.

If you already know or have solved this issue please share your ideas.

Wayne

 

[greaterreturn]

Quote from jdeezero05:
Well, I just have to say you are the man Wayne.
This sounds fantastic.
As far as programmers/non programmers go..I'm a non programmer with one basic class in Pascal who has struggled through to learn C# to program ninja trader..
Obviously, I'm biased but I just think things that try to "help" non programmers program, really just ends up adding bloat and wasting everyone's time.

Excellent point, jdeezero5. And I agree. Interestingly, I find that if you focus on making it easier for programmers, you actually help non-programmers at the same time. Allow me to explain how that works.

Originally, the philosophy I used in building TickZoom was only to make my life easier for myself as a skilled programmer.

But guess what one of the first things I wanted to do as programmer was?

It was make some of my EasyLanguage code work on TickZoom!!

So I started by copy pasting the code and trying to add features of EasyLangue into C# work so you have the best of both worlds.

Here's some features I've added.

TickZoom offers the data object in reverse like EasyLanguage. That means that Bars.High[0] means the "last" object in the array like EasyLanguage instead of the first.

And referencing Bars.High[5] means the high 5 bars ago [in the default time frame for this strategy/indicator.]

Okay. So far so good. We haven't hurt but only helped our programmers.

Next I found it irritating in C# to have to always check the size of the array before using it. I previously had very ugly code like this all over the place.

if( Hours.High.Count > 5 && Days.Close.Count > 2 && Minutes.Close.Count > 10) {

use those bars.

}

And WORSE of all, I was constantly trying out different "look back" lengths and forgetting to change the checks above to match. So I got C# Exceptions frequently after waiting for the project to build, startup and load the tick data. (That all takes about 45 seconds.) BAM, all I got was an exception.

That happened too many times. Too many wasted minutes. YUCK.

So guess what happened? I automated that whole process entirely.

NOW, in TickZoom you can simply refer directly to Hours.High[5] (just like E.L.) without ever checking the bounds!

TickZoom internally catches that and does just like EasyLanguage, it records that fact and waits until there are 5 (or whatever) bars before calling your method again.

There's one last way to make it easier for programmers, especially if they're familiar at all with EasyLanguage.

And that is to immitate the indicator and function names to be similar to EasyLanguage.

That just makes it easier if you take something like Pivot Points and want to use it. You can just paste the code into a TickZoom indicator and do a few things with search and replace:

Switch the variable assignments from the := in EasyLanguage to the c# version.

Since EasyLanguage always assumes a single time frame, you can search and replace High[x] and with Bars.High[x] or Bars.Low[x] which refers to the default bars for the strategy.

In other words, with just search and replace I was in about 15 minutes able to take an EasyLanguage script and convert it.

In short, programmers (like me) are people too. So we all appreciate making things easy to program.

It seems that, in turn, makes it easier for non-programmers.

Beside what else about EasyLanguage makes it easy? I think I've got EasyLanguage beat now. That because in EasyLanguage it's extremely hard to implement code exit strategies or money management code.

In TickZoom, you simply make a separate strategy for each of those functions. So your strategy and just focus on the trading model and rules. Then the ExitManager can handle any exits you configure or you can extend it to add your own custom exits.

Next in the chain is MoneyManager, it has standard money management exits like DailyMaxProfit or DailyMaxLoss, etc. Or you can extend it and add you own custom money management.

Frankly, I'm using TickZoom with C# because it's far easier to do what I need to do for an automated trading system than EasyLanguage.

Oh, there are a few other ways, that TickZoom, makes it easier to program strategies and indicators.

Unlike E.L., TickZoom does allow any number of timeframes in a single indicator or strategy. There again, it's possible to goof in using a timeframe you didn't setup or request. TickZoom figures out exactly which one automatically and gives you a clean message with what you need to do to fix it and the file and line # where it occurred.

That was a replacement for the meaningless "NullReferenceException" of C#.

All these things rapidly speed the development so you spend less time stepping through code figuring out what's wrong and more time testing ideas to make money.

Sincerely,
Wayne