Open Source Black Box Trade Platform in C#?

[greaterreturn]

Quote from coolraz:

Wayne,

I think your idea is very interesting. There is currently something similar out there called TradeLink Suite

http://code.google.com/p/tradelink/

which has historical backtesting, live trading and works on tickdata (can create bars etc). It has the base of what you are trying to achieve, but does not have some features that you mentioned that I think would be great (such as backtesting a portfolio of customized strategies at the same time)


It's written in C# with brokerservers for interacting with IB's TWS API and Anvil (Assent) API in C++.

I have been using it for backtesting and some live trading and it works quite well. The developer is INCREDIBLE at supporting it and very helpful.
My only issues is that there are not many people using it for IB and futures trading and that there's no plans to use FIX, which is something I would like to see.

I think it might be worthwile for you to perhaps check it out, or even consider combining forces with the TradeLink developer to make an even better result.

Anyway, BEST OF LUCK to you in your project! You are doing something that will benefit a lot of people (and hopefully you in the process! : )

Thanks for the info. I'll check it out. At the moment I'm setting up the forums, web site, and stuff.

Amazing I didn't find TradeLink before. Is it relatively recent?

QUESTION: I'm just curious of you posted your suggestions of TradeLink to the TradeLink forum?

Note, I just reviewed the object graph of TradeLink.

There's one major difference in the design of order and trade handling with TickZoom.

It's a completely different philosophy. some of this may be new to many of you.

Someone on EliteTrader gave me this idea many months ago.

Basically, to achieve the mixed portfolios on a single instrument you have to handle order processing differently and much simpler.

Please think this through as it's a big and important TickZoom concept.

Replace the idea in your mind of a customer strategy placing orders with limit or market, stops, etc.

That is impossible to combine across multiple strategies on the same instrument. In fact it's highly disadvantageous in other ways as you'll see as I progress.

Instead think of each strategy simply generating a single steady signal of how many lots to be long or short and zero for flat.

Now if I have a trend and a reversion strategy. One may be generating a 1 lot signal (meaning long) and the other generates a -1 signal (meaning short).

Now a the portfolio strategy (built-in but you can extend it) can sum the two signals together. The sum result is to go flat. So it tells the broker interface to go flat.

On the other hand if they both said long, then it would go long 2 contracts. Of course, you might prefer to make it so if they agree you only go 1 contract long or short, etc.

Now, the beauty of this paradigm gets even better.

Now you can "CHAIN" strategies together to test adding on different options.

For example, every strategy by default has an ExitManager and a MoneyManager strategy.

So visualize the 1, 0, -1 signal from the strategy feeding into the exit manager.

It now handles all the stops or target profits exits you like. It has built-in defaults. But all it does is watch the bid/ask prices to see when a stop was hit LOGICALLY.

In then modifies the signal going to the MoneyManager by making it go flat to 0 (zero) if a stop is hit.

Now the built in moneymanager generates all the trade statistices. ( I need to rename it to StatisticsManager. And add a separate one specifically for MoneyManagement that comes before statistics.)

It watches all the changes in the signal and records them as entries or exits.

NOTE: I understands both signal trades and combo trades. In other words, if you it goes from 0 (flat) to 1, that starts a trade and a combo trade.

If the signal then goes to 2, then it closes on trade and starts another but keeps the combo trade open and keeps the average puchase price.

So in the end it generates statistics for all the lone trades and all the combo trades separately.

That's really powerful for analysis if you're scaling in or out.

Now the broker interface simply gets a signal of the number of contracts to do long or short.

That TREMENDOUSLY simplifies the broker interfaces.

All it has to do is take the total contracts currently ordered (but not yet filled perhaps) plus the current position size and compare that the the latest signal to adjust the order.

It makes it very simple to handle all the complexities of partial fills, etc that usually trip of automated traders.

At the moment, I don't have any stops registered with the brokers. Most skilled traders dislike having them at the broker for various reasons.

I personally intend to implement a "fail safe" or "emergency" stop parameter which is separately configurable and always entered with every trade.

I mentioned the MoneyManager earlier. I don't have this logic built in, but you can see it's simple to take the stastical current equity of the account and use a percentage or other fancy formula to again alter the signal and increase or decrease contracts before it hits the broker.

In fact this makes it possible to chain and connect any number of strategies together.

This, is the practical scalable way to implement portfolios of strategies.

That would be a major design change, I think, with TradeLink.

However, if it already interfaces with several brokers, maybe someone can leverage that code to add interfaces to TickZoom.

Sincerely,
Wayne

 

[good one]

Quote from greaterreturn:

By the way, I looked a the FIX interface a while back too. It's not hard. Just in my case MB Trading, my broker makes it hard to use unless you do a ton of trades or pay a big monthly payment.

While I totally respect what TSGannGalt posted and believe its totally true, like my counter response to what he posted, I think you ultimately have to first develope a conceptual framework of what problem this project is trying to solve and then move forward from that mindset.
The bounds of the problem are on one side you have the retail moron using daily data trying to point and click their way to a trading system, on the opposite end of the problem you have system developers at a quant hedge fund who would love for someone to grunt through the process of developing their ATS framework for them, then giving it away, then they just have to "trim the bushes with a feedwacker instead of cutting the lawn themselves"...
The market for this product sits in the middle of this. FIX is far skewed to the quant hedge fund side...
"What the world needs now is more institutional software like I need a hole in my head"..

On a tangent, as far as data storage...HDF5 has to be considered..
www.hdfgroup.org/HDF5/
An easy speculation to make here is that data storage requirements will be vastly greater 10 years from now than in the present..so to me it makes sense to address that problem now, even if it sacrifices a bit of present speed, because you could easily be sacrificing exponentially more speed by "optimizing" to the present..

 

[greaterreturn]

Quote from jdeezero05:

While I totally respect what TSGannGalt posted and believe its totally true, like my counter response to what he posted, I think you ultimately have to first develope a conceptual framework of what problem this project is trying to solve and then move forward from that mindset.
The bounds of the problem are on one side you have the retail moron using daily data trying to point and click their way to a trading system, on the opposite end of the problem you have system developers at a quant hedge fund who would love for someone to grunt through the process of developing their ATS framework for them, then giving it away, then they just have to "trim the bushes with a feedwacker instead of cutting the lawn themselves"...
The market for this product sits in the middle of this. FIX is far skewed to the quant hedge fund side...
"What the world needs now is more institutional software like I need a hole in my head"..

Right, I'm focused right now on some improvement in my own ATS so that is make money more consistently rather than trying build a product for others to use. That said, I ALWAYS had an eye to building a community of developers since it seemed such a waste that I had to start from scratch.

On a tangent, as far as data storage...HDF5 has to be considered..
www.hdfgroup.org/HDF5/
An easy speculation to make here is that data storage requirements will be vastly greater 10 years from now than in the present..so to me it makes sense to address that problem now, even if it sacrifices a bit of present speed, because you could easily be sacrificing exponentially more speed by "optimizing" to the present..

You make a valid point. The simplistic storage I use is extremely fast but now it only holds one file at a time.

It would need significantly more logic to handle years worth of data that will logically require mutliple files. Plus I currently put a separate symbal in each file.

Read my next post about organizing this please...

Sincerely,
Wayne

 

[greaterreturn]

Someone mentioned TradeLink. I see several superficial but important issues with that project.

First of all they don't have the tradelink.com or tradelink.org domain names.

Second, when you google tradelink, you get tons of other companies or products. It's very hard to get anything related to their platform until you add several more words. I do find the google code site under TradeLink Suite.

Third, eventually this has to become commercial to succeed long term (as we described). So it's important, marketing wise, to give a good professional presence. Much like a resume.

In contrast, if you google the word tickzoom. Guess what you get first? This forum listing. It's a unique and new term. So it will make a distinctive product name.

Also, I have done the following:

1. I establish a hosted wiki with the same wiki software used by Wikipedia and a server at the domain name tickzoom.org. With a day or two, you'll get to it by visiting tickzoom.org

2. Registered a TickZoom forum with nabble.com. It's cool because it let's you reply to forum posts just by replying to your email rather than having to click a link. That will make it much quicker and easier for me (and others) to reply to posts or questions on the forum. If you use gmail, it will organize the threads for you. Very cool.

3. You only have to go to the forum to create a new post. That will make it

4. I signed up a TickZoom project on Google Code using Subversion as the repository.

5. We can use the Google Code defect tracking system there. It seems to integrate well with the subversion.

That source code and bug tracking can be more essoteric for us programmers.

The public will enjoy the wiki. (So will programmers initially.) as Iwill post examples and screen shot, in answer to FAQ.

Sincerely,
Wayne

 

[greaterreturn]

Now jdeezero, what will work best as to the database issue or others is for us to all enter them into the Google issue tracking system as enhancement requests.

Then we can prioritize. I'm sure you'll agree once you get in there that there are some that maybe higher priority immediately.

Plus we need to think how to best use resources.

That is, maybe certain issues are better for me to fix because I can do them faster.

But perhaps database research, benchmarking, and integration someone else can do?

I say that because that code is isolated and should be relatively easy to integrate. After all, all you need as input in symbol to load, the start and end date/time and perhaps the start and end time of day.

Does that make sense?

I'll endeavor to get the code up there and some install instructions and examples this weekend.

We'll try to have the whole environment as clean and ready as possible before the holidays.

FYI, with regards to ticks. One of the big hassles I had to work out was the datetime class in C#. Since the server I collect data on is in another timezone. I managed to write the datetimes out in a Universal time so anyone reading can have it convert to their local time.

Also, the DateTime in C# is very bad for performance. It allocates a new DataTime for every operation you do on it. Horrible. Dead slow. So I had to implement a customize dataTime class ( I mean struct) for greater speed. That way it computes time in place rather than allocate a new object.

I call it DateTimeInPlace

Sincerely,
Wayne

 

[TSGannGalt]

Quote from jdeezero05:

While I totally respect what TSGannGalt posted and believe its totally true, like my counter response to what he posted, I think you ultimately have to first develope a conceptual framework of what problem this project is trying to solve and then move forward from that mindset.
The bounds of the problem are on one side you have the retail moron using daily data trying to point and click their way to a trading system, on the opposite end of the problem you have system developers at a quant hedge fund who would love for someone to grunt through the process of developing their ATS framework for them, then giving it away, then they just have to "trim the bushes with a feedwacker instead of cutting the lawn themselves"...
The market for this product sits in the middle of this. FIX is far skewed to the quant hedge fund side...
"What the world needs now is more institutional software like I need a hole in my head"..

On a tangent, as far as data storage...HDF5 has to be considered..
www.hdfgroup.org/HDF5/
An easy speculation to make here is that data storage requirements will be vastly greater 10 years from now than in the present..so to me it makes sense to address that problem now, even if it sacrifices a bit of present speed, because you could easily be sacrificing exponentially more speed by "optimizing" to the present..

Hrmmm... I actually agree in an odd way. (Funds won't be leeching like what you mention. And FIX is easy. I'm too lazy now to learn another broker's API, that's why I learned it...)

First off, I'll never use TickZoom.

Obviously, I'm perceived more on the Quant side than retail even though I'm self-taught developer who started out at a ghetto prop. shop. As my ability grew, so did my surrounding software starting with TS 4.0, Wealth-Lab and now I have everything written in C#. I'll prolly never use TickZoom, because my apps are custom and a direct reflection of MY trading and system development skills. My execution platform is based on QuickFIX and I've become well-versed enough to send whatever kind of msgs I want and can. My lib is designed to fit how my brain works. It's a waste of time trying to use something I can already do.

...

So who really needs this...

Perhaps, it's for people who's still trading on a retail platform looking for some direction towards finding their own kind of environment to trade under. I've never touched NinjaTrader but from what I've been reading, it seems like you should be making an "arch-nemesis" - type application that is open source.

You can't make everyone happy. Wayne, develop what would make yourself happy, having it.

 

[greaterreturn]

Quote from TSGannGalt:

Hrmmm... I actually agree in an odd way. (Funds won't be leeching like what you mention. And FIX is easy. I'm too lazy now to learn another broker's API, that's why I learned it...)

First off, I'll never use TickZoom.

Obviously, I'm perceived more on the Quant side than retail even though I'm self-taught developer who started out at a ghetto prop. shop. As my ability grew, so did my surrounding software starting with TS 4.0, Wealth-Lab and now I have everything written in C#. I'll prolly never use TickZoom, because my apps are custom and a direct reflection of MY trading and system development skills. My execution platform is based on QuickFIX and I've become well-versed enough to send whatever kind of msgs I want and can. My lib is designed to fit how my brain works. It's a waste of time trying to use something I can already do.

...

So who really needs this...

Perhaps, it's for people who's still trading on a retail platform looking for some direction towards finding their own kind of environment to trade under. I've never touched NinjaTrader but from what I've been reading, it seems like you should be making an "arch-nemesis" - type application that is open source.

You can't make everyone happy. Wayne, develop what would make yourself happy, having it.

TSGannGalt,

Clearly, you're offering ideas to help this succeed and I appreciate it. I think all of you are right. Obviously several others have said FIX is a go for them.

I read about open source that first you must have a fairly solid product before trying generate revenue. So let's get some other people using, testing, documenting, reporting bugs, etc. Eventually it will be somewhere that an institution will be interested.

But institutions don't want to deal with open source projects of developers who do whatever they want. Institions and funds want some reliable full time developers and support staff. They wan SLA support contracts, etc.

So I have that in mind for the future. Right now, I'm more concerned about collaborating with other programmers and building up my own trading account. *smile*

In fact, maybe a company or organization will fund some developers or take over the commercial side of the project.

I'm willing to see this through the long haul but would much prefer to be just a user/developer on the project than administering it and dealing with the commercial aspects.

Perhaps, if it gets going enough some VC money and management might want to get involved.

I would only want to make sure that legally we're all protected with the licenses so that whatever we contribute stay always open source and free.

It's alright with me, just like jdeezero that someone commercializes since that benefits the community by having paid developers. The key of course is to let the community version remain driven by the community.

The commercial version can be driven by financial interests but should never interfere or break stuff the community wants or needs.

As far as FIX, I will be very happy if someone adds FIX. Maybe I should consider other brokers.

About NT. I think TickZoom will be in a category far separate. For example, TickZoom can't handle portfolios of strategies on one instrument or baskets of instruments for money management.

It can't even touch TickZoom for performance in handling tick data. (They promise to change that in the future but I'm not holding my breath based on discussions I had with them. I feel they'll need to hire a performance and tuning expert to get there.)

Hey. Maybe they should've hired me?

Wayne

 

[greaterreturn]

Folks,

The code is almost ready to release but your help is needed!

It's necessary to decide once and for all on the software license
to use for the open source because it may be impossible to change
or at least very aggravating to everyone if it gets changed later.
Some projects have failed due to changing their license after the fact.

Please see if you agree that the TickZoom license must meet
several requirements:

1. It must always remain totally free for all who actively contribute
to the project by documenting, fixing or enhancing the code.

2. As a market strategy, it needs to also be free for any research
and development for any one as an indefinite free trial period. (Just
like NinjaTrader--I agree that's smart.)

3. But those who use the software in production without actively
contributing to the project must still contribute and not "freeload".
But they can do so monetarily. Similar to NinjaTrader there can be
license fees to run in production.

4. Money from paying customers must feed back into improving the open
source system. NOTE: Documents say that it's very important to never
have a "dual" code base so the open source community never gets
cheated by the commercial side holding out any code. As a developer
and contributor myself, I concur with this.

FIRST, do you agree/disagree with the above requirements?

SECOND, here's the problem...

The open source intel says that dual license setup of GPL and Commercial
(as I was thinking) only works in certain industries. That is, it works
when the open source software ordinarily gets incorporated with other
software that gets resold.

However, that model seems to fail in this type of industry where
TickZoom will be an end product. Not just a component. The GPL license
allows and even encourages people and companies to just use the software
without giving back to the community in any way unless they redistribute
it. Then they must submit back any changes they made. Some users will
voluntarily support financially but industry reports show that voluntary
alone doesn't usually keep projects afloat.

So we need a solution that meets all the requirements above.

I'm open to any ideas. But I do have a proposal.

What if we use an open source license that states exactly the same 4
points above (with more legalease of course).

Logistically, it would mean that the software would need a license key
(not a stupid dongle) to operate on a live or demo account. But,
of course, programmers could disable that in the code legally and in
good conscience if they're actively contributing to the project.

Other users who aren't programmers but do contribute as in testing or
documenting will receive a free license key for production use.

Bottom line, if we make it clear legally from the beginning that what
we're all working on has value and those who use it must contribute
back with money or effort, this project will last a very long time.

Does anyone have a better/different suggestion?

Please, we cannot move forward till this is decided. And I won't
feel comfortable until most of us in this thread have chimed in about
what you support. I want everyone comfortable with this up front.

I personally feel the idea above protects us programmers so that what
we contribute will continue to be free to ourselves and other programmers
to encourage collaboration and collective value and also that our hard
work will get paid back by the commercial side adding to the software,
fixing defects, doing upgrades, integrating components and especially
adding any new technology that makes tick testing go faster, etc.

We, of course, want it free to other programmers so they contribute
back design ideas, fixes, and enhancements.

It's obvious that there will be a big commercial interest in this down
the road especially because a super fast, open source tick engine will
be a unique and high quality solution. Plus, all the ideas you have
posted here like adding FIX, an OODBS, clustered server architecture,
etc. Will make this a very powerful system.

I hope you agree that no one, especially for-profit businesses,
should have use of it unless they contribute back to the community in
some way shape or form.

Sincerely,
Wayne

 

[greaterreturn]

Quote from janus007:

hi greaterreturn

Extremly interesting thread, it seems that many of your ideas is taken out of my scratch-board

I think an open source project should follow some simple rules for instance versioning, testing and so forth.
Are you using TDD in the development?

I've been working as a programmer for 10years, now I'm primarily only working in C# and SQLServer.
- Could you maybe post some code snippets and maybe some class documentation i.e. for the Bars class? I'm just curious to see the level of programming

Again thank you for this post, it just proves that there's a lot of commercial crap outthere that needs some "thinking out of the box"
Keep up the good work!

Cheers
Janus

Sorry, Janus. It seems I missed your post earlier.

Yes, I'm using Subversion for version and I have NUnit unit tests.

There's not complete at the moment but I have them for all the statistics and bars calculations because that stuff must be accurate.

So I hand tested and verified it all in the beginning and now just run those test to make sure it's still right when I refactor stuff.

OMG there isnothing so sweet and automated testing.

I do TDD about 50% but when working on a trading system sometimes it's more efficient to whip it up the chart to SEE it first before writing the test.

Anyway, I'm committed to TDD in theory and believe it's crucial to a system with money at stack.

Sincerely,
Wayne

P.S. I'll post code in due time. We need to solve the licensing question first.

 

[greaterreturn]

Big,

Wow. I missed your post also.

31 MB / 485218 means 63 bytes per tick???? OMG that's huge.

I will verify the exact bytes from TickZoom tonight after work but, it's at least more than 3 or 4 X smaller per tick. Like I said, it optimizes, or you could say, compresses the data at the byte level. NOTE: That saves both on file size, memory, and loading time.

The smaller the tick bytes, the faster all the way around. Every SINGLE byte counts when you deal with huge volumes of data.

Of course, you don't mind 39 seconds for 485218 ticks but what about when you have 7.5GB of data that takes 602,821 seconds or 167 hours to load? That's only to load. Not to mention running the test.

I do all my testing right now on 5 years of USD/JPY ticks from gain data (so not as dense as my own collected ticks) at 159 Million ticks.

They load into TickZoom in 15 seconds.

With db40 that would take, 9243 seconds or 2.5 hours.

By the way, let's compare to TickZoom at 10 years of data.

485,218 ticks X 260 days a year = 126Million ticks per year. Well that takes about 15 seconds or less for TickZoom to load. Now, multiply that by 10 years and you 150 seconds.

How does 150 seconds compare to 167 hours?

Are you, really, sure you don't mind the db40 load time?

Wayne

Quote from Big:

Ok, little update on my storage thing here.

So far, I have only one day of tick data. That amounts to 485218 ticks. This saves into a 31 MB Db4o database.
That means roughly 600 MB per month, so around the 7.5 GB per year, or 75GB for 10 years of data. Not bad (this is only for one instrument).

Saving those ticks into the db took around 39 seconds (I don't mind).

Retrieving one hour of data (mid day) gives me 3059 ticks.
With a nice looking "Native Query" looking like this :
IList<Tick> result = db2.Query<Tick>(delegate(Tick tick)
{
return tick.Date > startQuery.Ticks
&& tick.Date < endQuery.Ticks;
});
It took around 30 seconds to return them (indexed or not) which is unacceptable.
I suspect some kind of "table scan".

with a not so good looking "SODA" query looking like this :
IQuery query = db2.Query();
query.Constrain(typeof(Tick));
query.Descend("_date").Constrain(startQuery.Ticks).Greater();
query.Descend("_date").Constrain(endQuery.Ticks).Smaller();
IObjectSet os = query.Execute();

We are looking at 15 ms to return the data, which is much more like it.

I am told LINQ queries (but running only on .Net 3.5, I'll get to it at some point) compile usually into SODA queries, so there could be some good looking, performance efficient queries around the corner.

I'll play more later.