TickZOOM Decision. Open Source and FREE!

Status
Not open for further replies.
Hi Wayne, I have been following this thread but haven't posted anything yet because I wanted to digest all of this and not add to the clutter.

I am very interested in supporting this and have several programmers that I work with that would be as well. You are doing a great job so far, and the GPLv3 is the right way to go.

We prefer to work in the Linux/GNU-Toolchain/C++ environment because of reasons already stated in the previous thread (so wont restate them). Some other big advantages to the Linux/C++ environment are that:

1. there are several hundred million of lines of high quality open source code to draw from. You don't have that in a Microsoft environment. You can mix and match as you like form the best of breed in many areas. In the MS environment you end up coding many things from scratch. That is why you don't see any high profile major open source projects written in C# or an MS environment.

2. Overwhelmingly the majority of the programmers that heavily contribute to open source projects prefer this environment. Programmers that are only proficient in the MS environment tend to be focused on closed source projects at the smaller application and user interface level. There are exceptions but I have found that if you want to get involvement of the most active open source contributors that will do real testing, code review, implementation and coding then the best environment will be GNU/C++ for you core system. The system should be designed in a modular way so that all of your subsystems can be done in the language and environment of choice for the individual contributors. The main system/engine could run on Linux or Windows, I am just talking about the preferred development environment being Linux/GNU/C++ or you could use Cygwin for windows. So you can use this environment on Linux systems or Windows systems. Using MS/C# you are locked into Windows only and you cant run on Linux or Macs and also you loose out on all the heavy open source contributors that will not invest their time into a closed source MS environment. Mono does not run well enough for that to be a serious option for high performance systems like this.
 
Cliff,

All your assessments are well founded.

You make several points. Let me state all those I agree with and then where we differ and why.

1. Especially, I agree 100% of supporting GNU/Linux. I'll see all your reasons and raise you one. Let's add the cost of servers to run TickZOOM for live get expensive with Windows but cheap (comparatively) for GNU/Linux.

2. I completely agree with making TickZOOM modular (and it already is) but this idea is new to support other modulus written in any language. The most interesting modules will be the custom formulas, strategies, indicators which by the way are all seen as just a formula interface to the engine.

It seems that interface could be easily made to interopate with C++ or any other language. In .NET they could use VB, etc. Right out of the box.

If you want to get started before we port to Java or C++, ave you considered Mono? C# runs on Mono in Linux and it could call out to your modules in C++.

3. Now about porting to C++. Without elaborating since we can talk in circles on which is better, I think Java is the priority to go for Linux. However, I have some ideas to help you.

A. Java can call out to C++ ...

B. Maybe someone else will port it to C++ as a fork, etc. There could be some disadvantages to doing a fork type port.

However, if someone wanted to do a port by working with TickZOOM then could continue supporting all three languages in sync for testing.

That is, I'm willing to support C++ but not, at least for while, do the work of porting it. (Well, money talks of course. But for free, I prefer Java.)

Now if someone else did a port for us to support, they would have to make sure they did a straight port so the class names, method names, etc remained exactly the same as much as possible and not add or modify anything else besides port it.

After a stable, regression tested version exists it can then be kept in sync with all the languages.

A fork of course that was different would be an option but we couldn't accept supporting something significantly different.

The key to supporting multiple languages is that they are kept virtually identical in functionality. In that case most fixes or enhancements in one make sense also in the others without much additional work.

Just some musings.

Wayne

QUOTE]Quote from cliff5200:

Hi Wayne, I have been following this thread but haven't posted anything yet because I wanted to digest all of this and not add to the clutter.

I am very interested in supporting this and have several programmers that I work with that would be as well. You are doing a great job so far, and the GPLv3 is the right way to go.

We prefer to work in the Linux/GNU-Toolchain/C++ environment because of reasons already stated in the previous thread (so wont restate them). Some other big advantages to the Linux/C++ environment are that:

1. there are several hundred million of lines of high quality open source code to draw from. You don't have that in a Microsoft environment. You can mix and match as you like form the best of breed in many areas. In the MS environment you end up coding many things from scratch. That is why you don't see any high profile major open source projects written in C# or an MS environment.

2. Overwhelmingly the majority of the programmers that heavily contribute to open source projects prefer this environment. Programmers that are only proficient in the MS environment tend to be focused on closed source projects at the smaller application and user interface level. There are exceptions but I have found that if you want to get involvement of the most active open source contributors that will do real testing, code review, implementation and coding then the best environment will be GNU/C++ for you core system. The system should be designed in a modular way so that all of your subsystems can be done in the language and environment of choice for the individual contributors. The main system/engine could run on Linux or Windows, I am just talking about the preferred development environment being Linux/GNU/C++ or you could use Cygwin for windows. So you can use this environment on Linux systems or Windows systems. Using MS/C# you are locked into Windows only and you cant run on Linux or Macs and also you loose out on all the heavy open source contributors that will not invest their time into a closed source MS environment. Mono does not run well enough for that to be a serious option for high performance systems like this. [/QUOTE]
 
Yes, I agree with all your points. Java is great but my thoughts are that for the main engine you want the performance of C++.

The main thing as far as environments go is to have the engine and core of TickZoom not be locked into an MS environment. Some additional thoughts on Linux/C++:

1. Linux supports the best multi CPU and clustering so later when you need high horsepower you can do it. With Linux it is in the grasp of individuals to put togather a cluster of 10 machines for about $7000 or less. You cant do this with windows. All of the top supercomputers run Linux for a reason.

2. Microsoft environment lets you implement fast and gives you bells and whistles (visual environment, .net, etc). They do this to bait you in and get you going on their environment. They don't give you these things because they like you so much or care about your long term benefit. Their long term goal is vendor lock-in and causing you to be dependent on them and in finding ways to take your money once you are "locked in". The more successful you are and the bigger your system requirements get the more you are "Locked-in" and the more you pay. And I am not just talking about the money, you pay when you don't have the source code to all of their code and you cant fix and change things. You pay when most of the open source people wont work on the project. Microsoft environment and Windows are fine for a GUI front end, charting, etc, etc. You just don't want the back end of the system locked to MS.

3. For a serious autotrading system many people will want to run it in a cloud or hosted environment such as Amazon EC2 Elastic Compute Cloud or Joyent Accelerator. I sure dont want to do serious autotrading on my home machine and then have Comcast or ATT go down in the middle of the trading day! If you are going to put the autotrading system on a cloud or hosted envirornment then you want Linux.

4. So, bottom line: You want the engine on Linux. It could also run on Windows too, but just no vendor lock-in. Mono is not far enough developed to support this type of project and not well enough accepted IMHO. So that rules out C#.

5. Java could be viable. There are some speed concerns but you could always take the few blocks of the code that need to be super high speed and optimize them in C++ as needed. So the vast majority of code could be Java.

6. Java would be great to use for programming the studies and trading systems!

7. If you prefer Java over C++ then port the system to Java and then let the other supporting coders optimize a few parts of the code in C++ if there are speed issues.

8. In a modular system people could then program front ends, charting, back testing control, etc on Windows with C#,,, that is fine! Other people could use Java and Eclipse or Java and whatever. Just not the engine locked on Windows, that all!
 
Just so you know, the GPU discussion was about using the GPU as a processing unit, not for graphics. Look up NVIDIA's CUDA and OpenCL. The point being, graphics processors are MUCH faster than CPUs, can compute in parallel, and have many many cores. Therefore, if you can write bytecode for the GPU, you get a huge speedup...
 
Quote from cliff5200:
4. So, bottom line: You want the engine on Linux. It could also run on Windows too, but just no vendor lock-in. Mono is not far enough developed to support this type of project and not well enough accepted IMHO. So that rules out C#.
More...

You should take another look at Mono. You can look at the <a href="http://mono-project.com/Roadmap">Roadmap</a> to see where it currently stands. It is pretty impressive...and pretty far along. You can also check out some speed comparisons <a href="http://shootout.alioth.debian.org/gp4/benchmark.php?test=all&lang=csharp&lang2=java">here</a>.
 
Corey.

Okay. Thanks for the specifics. You were modest enough originally to mention that you weren't so technical so you won't mind me clarifying a couple things.

First, GPU stands for "Graphics Processing Unit". That's because it's hardward (can't be change) to process common graphics routines in parrallel very fast.

So everything you say is true except GPU's can only effectively be used for graphics.

Keep the research and suggestions coming.

Sincerely,
Wayne

Quote from Corey:

Just so you know, the GPU discussion was about using the GPU as a processing unit, not for graphics. Look up NVIDIA's CUDA and OpenCL. The point being, graphics processors are MUCH faster than CPUs, can compute in parallel, and have many many cores. Therefore, if you can write bytecode for the GPU, you get a huge speedup...
More...
 
Cliff,

We agree on Linux. I personally don't want to run my black box on Windows anymore (I only did due to the broker having only COM interface.) So let's close the Linux discussion.

As far as C++ vs. Java. Please answer one question, when you compare performance of the 2, did you personally write the same code in each language, compile and compare the machine code generated by the compiler? Have you also benchmarked that exact code to compare the speed on the same CPU, etc environment?

If not, then you're only repeating hearsay. I have done what I just said and was shocked to see that the compiler generated the same code.

There is only one significant different about why Java falls in comparison to C++ in fact. Reality. Not hearsay.

That is garbage collection and dynamic memory management.

However, I also built real time systems in C++ and it was impossible to call the "new" operator or access the heap during critical code there also.

Sorry, if I sound antagonistic, I'm not, I just keep hearing C++ is faster. But there's no true difference in my real, hard experience (not theories or hearsay.)

Java USED to be dog slow when it ran interpretively on bytecode but now it always compiles to machine code just like C++.

The only difference is you don't have manually compile on each machine.

In short, we can put this to bed too because I will port to Java first. C++ will come much later, if ever.

But again, I will support a C++ build if someone else cares enough to port it.

However, if they do, it just occurred to me to run a benchmark and compare C++ to Java. Prove that it's not any faster.

One this is for sure and certain, the Java version (just like C#) will be orders of magnitude X faster than most other ever commercial products.

Furthermore it is scalable so the software will run no matter how many CPU's or Memory or network cards you want to install.

Even if C++ were 10% faster, would it be worth the developer time and effort to port to C++ rather then just add another CPU?

Developer is not free as you well know.

Sincerely,
Wayne


Quote from cliff5200:

Yes, I agree with all your points. Java is great but my thoughts are that for the main engine you want the performance of C++.

The main thing as far as environments go is to have the engine and core of TickZoom not be locked into an MS environment. Some additional thoughts on Linux/C++:

1. Linux supports the best multi CPU and clustering so later when you need high horsepower you can do it. With Linux it is in the grasp of individuals to put togather a cluster of 10 machines for about $7000 or less. You cant do this with windows. All of the top supercomputers run Linux for a reason.

2. Microsoft environment lets you implement fast and gives you bells and whistles (visual environment, .net, etc). They do this to bait you in and get you going on their environment. They don't give you these things because they like you so much or care about your long term benefit. Their long term goal is vendor lock-in and causing you to be dependent on them and in finding ways to take your money once you are "locked in". The more successful you are and the bigger your system requirements get the more you are "Locked-in" and the more you pay. And I am not just talking about the money, you pay when you don't have the source code to all of their code and you cant fix and change things. You pay when most of the open source people wont work on the project. Microsoft environment and Windows are fine for a GUI front end, charting, etc, etc. You just don't want the back end of the system locked to MS.

3. For a serious autotrading system many people will want to run it in a cloud or hosted environment such as Amazon EC2 Elastic Compute Cloud or Joyent Accelerator. I sure dont want to do serious autotrading on my home machine and then have Comcast or ATT go down in the middle of the trading day! If you are going to put the autotrading system on a cloud or hosted envirornment then you want Linux.

4. So, bottom line: You want the engine on Linux. It could also run on Windows too, but just no vendor lock-in. Mono is not far enough developed to support this type of project and not well enough accepted IMHO. So that rules out C#.

5. Java could be viable. There are some speed concerns but you could always take the few blocks of the code that need to be super high speed and optimize them in C++ as needed. So the vast majority of code could be Java.

6. Java would be great to use for programming the studies and trading systems!

7. If you prefer Java over C++ then port the system to Java and then let the other supporting coders optimize a few parts of the code in C++ if there are speed issues.

8. In a modular system people could then program front ends, charting, back testing control, etc on Windows with C#,,, that is fine! Other people could use Java and Eclipse or Java and whatever. Just not the engine locked on Windows, that all!
More...
 
Maybe I misstated something when I mentioned Jacket.

I know that there is work being done to process Financial computations on NVIDIA GPUs via CUDA that yield orders of magnitude of speed increases over other methods. I believe CUDA is an offshoot of C but am not sure.

http://www.nvidia.com/object/cuda_home.html#state=home

The 'Jacket' product is one of these items that allows Matlab to utilize the GPUs processing functionality. It is described on the link above.

If this functionality is feasible, wouldn't this be more beneficial than setting up a whole cluster of machines???? I think CUDA can be utilized on both Linux & Windows machines.

Regards,
Eric
 
Cliff,

You're taking a completely different take which is actually making me consider C++ for the engine also.

You're speaking from the community.

Personally, as a developer, I obey user requirements. In the case of TickZOOM there's many users. Question is: What's the right way for the majority of the users.

It may be that C++ is what users of financial system most expect.

I recall a programmer who worked at a prop shop IT department who said that most of their systems were UNIX/C++.

Therefore. Hey. I just noticed the DB40 support C++.

I always was okay with porting to C++. I just thought Java should be first.

Is there some kind of market information, statistics to show that C++ is most used language the profession financial sector or something to that effect?

Maybe Java hasnt' taken over that industry like it has the web industry.

IN FACT, Nobody builds websites in Java that I heard of. And I've been working on websites for years now.

So maybe my view of the market is very web.

I'm not impressed as you say be technical or speed comparisons between Java and C++.

I AM however impressed if the user base or market is larger.

AND you're listing of actual open source C++ projects for financial analysis hits me where it counts.

Question is, have those been ported to Java?

If not, then you might be waking me up to a fact that the financial sector has stuck with C++ while the web site industry moved on the Java.

When I referred to millions of lines of Java code I (stupidly) had in mind all the tools for building websites, like spring, struts, etc. etc.

But those are useless in this scenario.

Okay, Cliff you're winning the C++ Java discission. I'm not convinced yet but almost.

If you can sell me on the "market" direction especially with some more facts like you mention below. I'll switch to C++ for Linux as the first priority.

Sincerely,
Wayne

Quote from cliff5200:

Also, on Linux you can use other free open source software mostly written in C++ (or code chunks from this software) such as Scilab (http://www.scilab.org/), QuantLib (http://sourceforge.net/projects/quantlib/), MonetDB (http://en.wikipedia.org/wiki/MonetDB), Scicos (http://www.scicos.org/), Octive (http://en.wikipedia.org/wiki/GNU_Octave), The list goes on and on.
More...
 
Status
Not open for further replies.
Back
Top