Just another trading platform - But this time different!

[Steven.Davis]

Quote from LeeD:

I understand you solved the problem. So, it's not a pressing issue.

In case it helps... I'm sure you know that in Visual Studio 2005 and higher a C++ project has compilation options on the per-file basis. Each file can be compiled as C++, C++/CLI, or C. Then the linker makes sure all of these 3 types work together. Have you tried if the libgretl compiles as C?

I will give your idea a try. Half of the problem is getting the code to compile, but there is a second big hurdle. When Gretl was a console app, it looks like the core calculation code was in libgretl. when they went GUI, the separate was lost. When I look at the data structures, they have a mixture of core data, current model structure, and GUI state.

I was following the strategy of starting with a couple of source code files which had good stuff in them. Change them to get them to compile (things like void * casting, function pointer casting, namespaces, ....) Then look to see what was missing in the linkage. Add in a couple of files with missing function calls. Comment out stuff that, hopefully, isn't needed. And repeat.

If it were just a contained function like a numerical method, I am sure I could pull it all together, but thing needs its complex modeling structure. For example, if on an earlier run, you find that a certain variable has a very small weight in some factor, then you can setup by configuration to optimize the system with that factor constrained to be zero. Ok, cool, but sometimes that coefficient happens to fall to be the thing by which you are renormalizing the vector. Thus more logic to look for a better choice.

Then with GUI stuff getting pulled in, it was just too much for a side project on a resume/demo site.

Thank you for the suggestion though. I will try it if I give it another crack. Being able to compute cointegration all in C++ without OLE transitioning would make such a huge performance difference.

Another day, thanks though.

 

[mhtrader]

LeeD,

I read the whole thing and I got a clear understanding of what happened.

Now that I know you are technically savvy( I don't know by how much ), I think you should be smart enough to don't allow the loop to go away( that produces brain damage on the .NET fanatics ). If you have constants in the loop and at the same time functions are inlined, most C++ compilers ( and believe it or not some compiler for a NON-C++ language for trading platforms I implemented in the past, not bragging, just reporting the facts... ) will do something like this when you have constant expressions hanging around in your code:

1 instruction:

mov eax, K

Where K = Iteration Numbers * X, and X is the number you are adding. The same stateless constant expressions are "evaluated" at compile time for most C++ compilers. This optimization happens: in loops, in boolean expressions( with booleans, strings and numbers ) and numeric expressions.

By saying that if you want to benchmark 2 loops, one in C# and one in C++, you have to be smart enough to don't allow C++ to take the loop away. That is what I posted in my original benchmark: "too fast" means 1 instruction => mov eax, K

What he finds is inline assembler is substantially faster than assembler in a separate module (though it is the same code) and pure C++ takes zero execution time no matter what.

You know what happened to that lawyer, don't you? He posted the debug version of the function because the C++ function was optimized away due to the the use of constant expressions. In a real program there are not many constant loops.( constant loops generate other "phenomenon" of optimization... too long a list to explain here... like unwinding, parallelization, etc ). Most calculation are based on looping thru bars and this are not constant loops.

Again, I think you are aware of the C++ optimized compiler wonders. But google it.. I think it will amaze you. By the way you are not going to find this in any college book( I own most of the compiler books that are been published since the 80's. ) nor you will learn it in MIT(unless they started to teach it 1 year ago).

It is not possible in the near future that .NET( jittered IL ) in the current form will ever challenge any C++ optimized compiler.

How else can I interpret this other than:
1) You platform doesn't support trading strategies in C# ("No Java and No .NET")
2) It is a major selling point ("this is the real deal!!")

Where is C++ mentioned there? You are jumping to conclusions.

BTW, from your confrontational tone I detect you interpreted my post as a personal attack.

I posted a benchmark without mentioning any product or anyone's name. WITH THE CODE and the compiler setting change.

You are welcome to do it like that: post the code for everybody to test it and provide all settings of your compiler... a la peer-review.

That's the only way the truth will come out. The .NET fanatics will NEVER submit themselves to a peer-review exchange.

There is nothing personal LeeD. Just be transparent and truthful. I can assure you this will produce you more than "if you can't convince them, confuse them"

Thanks,
~mhtrader~

 

[mhtrader]

Quote from Steven.Davis:
I was surprised that you thought that compile-time optimizers were smart enough to convert loops into multiplication.

Steve, they are converted. Here is a example of 2 loops, NOT ONE BUT 2 UNRELATED LOOPS, gone... only a simple assembler instruction. Return value for integer functions are stored in EAX in _cdecl calls( actually in most calls ). As you can image and LeeD you got it right at first....not sure what change you made in your code that makes you change your mind...

C++ code:

int func( int param1 )
{
	int x = 0;

	unsigned int uTicks = timeGetTime();

	for( int i=0;i< 30000000; i++) 	
       {
		x = x + 80;
	}

	for( int i=0;i< 30000000; i++) 
	{
		x = x - 60;
	}

	unsigned int uTimePassed = timeGetTime() - uTicks;
	
	std::cout << "Milliseconds: " << uTimePassed << "\n";

	return x;
}

The code above add 80*30000000 and subtract 60*3000000 to x and x start value is 0. This is the same as (80-60)*30000000= 600000000 and in hex is 23C34600h

Now take a look at the optimized C++ code in assembler mixed with the original code. The loops are gone, they don't generate any code and the return value is calculated at compile time and store in EAX to be returned:

mov eax,23C34600h


The optimized code:

int func( int param1 )
{
01351010 push        ecx  
01351011 push        esi  
	int x = 0;

	unsigned int uTicks = timeGetTime();
01351012 mov         esi,dword ptr ds:[013520F0h]  
01351018 push        edi  
01351019 call        esi  
0135101B mov         edi,eax  

	for( int i=0;i< 30000000; i++) <= NO CODE GENERATED FOR LOOP 1

	{
		x = x + 80;
	}

	for( int i=0;i< 30000000; i++) <= NO CODE GENERATED FOR LOOP 1

	{
		x = x - 60;
	}

	unsigned int uTimePassed = timeGetTime() - uTicks;
0135101D call        esi  
0135101F sub         eax,edi  
	
	std::cout << "Milliseconds: " << uTimePassed << "\n";
01351021 push        1352144h  
01351026 push        eax  
01351027 mov         eax,dword ptr ds:[01352080h]  
0135102C push        1352148h  
01351031 push        eax  
01351032 call        013510E0  
01351037 add         esp,8  
0135103A mov         ecx,eax  
0135103C call        dword ptr ds:[01352070h]  
01351042 push        eax  
01351043 call        013510E0  
01351048 add         esp,8  

	return x;
}
0135104B pop         edi  
0135104C MOV         EAX,23C34600h  <= THIS IS THE CODE PRODUCED BY THE LOOPS
01351051 pop         esi  
01351052 pop         ecx  
01351053 ret

Of course the output will be Milliseconds:0

Compiler used: VC++ Express 2010. Compiled in release win32 with function inline disabled. The rest of the settings are the default settings.

Thanks,
~mhtrader~

 

[LeeD]

Quote from mhtrader:

Another of your smears:

Quote from LeeD:

However, observe how adamantly Mhtrader is defending the point that not letting users implement trading strategies in their favourite C# is a major selling point of his platform. I find watching it rather enjoyable and would hate to break his groove.

I have never mention what language the end user will use to write the strategies. I invite you to go thru the 12 pages to find that... you are not been noble.

Quote from LeeD:

In the opening post of this thread you wrote:

Quote from mhtrader:

-User written programs are not script languages. They are COMPILED for the CPU where it is running. No Java and No .NET, this is the real deal!!

How else can I interpret this other than:
1) You platform doesn't support trading strategies in C# ("No Java and No .NET")
2) It is a major selling point ("this is the real deal!!")

Quote from mhtrader:

Where is C++ mentioned there? You are jumping to conclusions.

I carefully quoted every relevant part of the discussion. I don't see how C++ is relevant either. You are the first to mention C++.

On my side, I just fail to comprehend how refusal to implement a feature that many find useful (such as C# support) can be considered a major selling point.

(It can be a "real deal" for people who otherwise like the platfom but have to drop it from their shortlist because of lack of .NET support.)

 

[mhtrader]

Quote from LeeD:

I have never mention what language the end user will use to write the strategies. I invite you to go thru the 12 pages to find that... you are not been noble.

How else can I interpret this other than:
1) You platform doesn't support trading strategies in C# ("No Java and No .NET")
2) It is a major selling point ("this is the real deal!!")
I carefully quoted every relevant part of the discussion. I don't see how C++ is relevant either. You are the first to mention C++.

On my side, I just fail to comprehend how refusal to implement a feature that many find useful (such as C# support) can be considered a major selling point.

I'm gonna write in bullet points style( can i say bullet points? or is this language aggressive? )

.NET and mostly anything else will make it for a good and cheap UI. I will venture to say .NET is great for UI... e.g charts. Flash too [ I have some "insider" information that the fastest chart in the market is a C++ chart and it beats the rest of the competition big time ]

.NET and mostly anything that is not C++ used for plumbing( data engine, execution environment, server or client ) it is doomed to failure.

User written programs to be applied to thousand of symbols( history or realtime ) shouldn't be done .NET. The performance will kill you. Unless you don't care about performance.

"Plumbing code" for 3 charts, 10 indicators and couple of strategies can be done in javascript. And in that case you can use E-Trade or even a flash-based thing.

Medium size stuff can work perfect with a .NET based language. I don't know where to draw the line for medium size stuff. But I can tell you is something "small".

Plumbing code to scan a big list of symbols on ticks will be doomed to failed if done in .NET if real-time response is needed.

Massive calculations can't be done .NET if performance and speed is important.

Now I see 2 main camps of users here:

Camp 1: "Platform 1 is in .NET(P1.NET) and Platform 2 is native 64bits(P2.Native). I don't understand very well the difference, but P2 is faster. I don't know is the difference in speed is significant for what I do or not. I can hire for cheap someone to make strategies for P1 and it can cost me a little more to hire someone to make strategies for P2. So I think I'm gonna stick to P1 in .NET

Camp 2: "I don't understand very well the difference, but P2 is faster. I don't know is the difference in speed is significant for what I do or not, I may not even need that speed, but I don't care... it is faster and it is what I want. I can hire for cheap someone to make strategies for P1 and it can cost me a little more to hire someone to make strategies for P2 and again I don't care, I want the fastest platform.

I met a lot of people in Camp 2 in the past. They don't care about the language, they want it fast and they want it now . But I believe that other of people will fall in Camp 1 as well.

The question is for the users ( and not for the people that are pushing their C# product or Visual Basic product )... as a user on what camp are you?

Thanks,
~mhtrader~

 

[fullautotrading]

Quote from mhtrader:


...
there is no "loop condition optimization" <= another esoteric phrase

00B01010 8B F7 mov esi,edi
00B01012 89 45 FC mov dword ptr [ebp-4],eax
00B01015 69 F6 80 C3 C9 01 imul esi,esi,1C9C380h

...
LeeD next time try to fact-check before you post or you may be spreading a lie again.
...

I have never mention what language the end user will use to write the strategies.

...

It seems to me that the one who would needs to check better is actually you:

imul esi,esi,1C9C380h <==> x = 30000000*param1; (essentially)

(1C9C380h = 30000000 decimal)

http://books.google.it/books?id=9Pe...DUQ6AEwAzgK#v=onepage&q=imul assembly&f=false

If we consider that an entire loop has been removed and replaced with 1 single multiplication, in a trivial local "optimization" (which would never occur in more realistic programs: it does not even occur later when the addition is replaced with the subtraction), if we account for the time spent on JIT-compilation, cashing, etc. (in contrast with AOT compilation), and that a more accurate timing method (stopwatch), comparable with timeGetTime(), would be appropriate, we would see that there are no significant differences. It's just compiled at a different time (JIT-compiled, which could be avoided with ngen.ex, but nobody bothers). Both are compiled to native code (it could even be the same native code.)

cf:
http://en.wikipedia.org/wiki/AOT_compiler
"AOT compilers can perform complex and advanced code optimizations which in most cases of JITing will be considered much too costly. On the other hand AOT can't usually perform some optimizations possible in JIT, like <b>runtime profile-guided optimizations</b>, pseudo-constant propagation or indirect/virtual function inlining."

In practice, the possibility of profiling the program and all the other advanced features will provide a substantial edge for meaningful projects.
This contributes to explaining why there are millions of programmers using the newer languages and why a considerable effort has been made to create them. And the doubt that, perhaps, you might not be exceedingly smarter than all of these people, should perhaps make an appearance in your mind.

Having made this premise, i am *very <b>happy</b>* that people develop automated trading systems and robotical platforms in c++.
Actually, I <b> encourage</b> anyone who wants to create a complex, and especially <b>profitable</b>, trading robot, to use c++. Please, leave alone c# and vb.net because they are way too slow !! Focus on simple loop optimization, memory management, etc. and <b>don't</b> bother too much building up a conceptual trading model.

Why do you expect a poor user should write strategies ? You are the author and the one who knows best. You are probably smarter than any future user and it should be fair that you provide the strategies too, or at least a profitable basis.
Now it's your time to really give substance to you point (not with a couple of meaningless loop, but) publishing the results of you trading activity with these automated tools. ;-)

Tom

 

[mhtrader]

I'm gonna answer the bottom part first: I'm not a talented trader. I hate to lose money and therefore this makes me busy almost 24 hours a day. It feels like 3 fulltime jobs. About writing strategies, I'm not good at it either.... I always try to stick to the existing stuff.. I'm a big fan a Bollinger bands.

But before considering strategies... I always dream to have a program that scan the whole market and will report to me any breakout(up or down) on any condition....price,volume,trades, etc at any interval I want, 5 seconds breakout, 10 ticks breakout......something tight that you can take action immediately... Something that gives me in that list that dormant company that had almost 0 volume for weeks and all of the sudden it starts to move in any direction at 9:30:01 AM.

Then you can trade the whole list( I'm too chicken for that ), or apply strategies on it. I don't doubt that a product like this exists as a proprietary software for big firms, but I have never seem a commercial product like that.

Up there you can see one of the motivation of why we started to work on the platform.


Now about the first part of the email, fullautotrading, I suppose this is a confusion and not a ill comment

Quote from fullautotrading:

It seems to me that the one who would needs to check better is actually you:

imul esi,esi,1C9C380h <==> x = 30000000*param1; (essentially)

Do you ever wonder why I put 2 loops one on top of the other? Not 1, but 2... And by the way the same reason I added param1 to the function, is the same reason why I disabled inline function optimization, is the same reason why I added 2 loops, is the same reason why I have return x in the function func[ to keep the variable alive during the data flow analysis of the compiler] and the same reason I have return x in the main. That list mentioned above forces compiler to: respect x in main because it is returned to the outside world.... then respect the values assign to x by calling func, then not inlining func because of the setting..., then param1 in func makes the addition and subtractions in the loop non-constant so the compiler cannot do the calculation of expressions at compile time... the value of local x in func is alive outside the function, so any assignment to x most be respected, so the loops cannot be taken away completely. All this thing sound crazy.... but I'm basically telling you what the compiler is doing, because I have written that code/data analysis before for other compilers( again not bragging, just reporting the facts ).

Now, why Microsoft compiler converted loop 1 in a multiplication and loop 2 in a real loop is out of my understanding. It could be an issue of how deep the compiler can go due to memory issues??? Honestly I have no clue... and I found the 2 loops thing by trial and error.

Back to the loop thing...one loop was optimized with the multiplication, but with 2 loops I found that the second loop was going to be there, the second wasn't optimized and it was in from of your eyes.

I don't understand why you didn't include that loop it in your comments? Because it will make your comment pointless I guess? Never mind... I gonna assume it was an honest omission.. may I use your own phrase...? ..."it seems to me that the one who would needs to check better is actually you"

Now let's take a look at the whole context:

LOOP 1:
for( int i=0;i< 30000000; i++) 
00B01010 mov         esi,edi   //EDI is the value of param1
00B01012 mov         dword ptr [ebp-4],eax   //save EAX in the tack, EAX is the return of timeGetTime
00B01015 imul        esi,esi,1C9C380h   //LOOP 1: Whole loop went away as you mentioned,
    { 
        x = x + param1; 
    } 
At this point ESI contains the result of the multiplication:
The real reason I put 2 loops is because LOOP 1 was taken away:

LOOP 2:
Here is a loop on 4 instructions, but in reality the LOOP is execute on 3 instructions
    for( int i=0;i< 30000000; i++) 
00B0101B    mov         eax,1C9C380h   //MOV eax, 30000000
    { 
        x = x - param1; 
looplabel:
00B01020  sub         esi,edi   //SUB ESI,[param1]
00B01022 dec         eax   // dec eax
00B01023 jne         00B01020(looplabel) Jump no zero/no equal 2 lines up to  00B01020  sub esi,edi
    }

loop condition optimization <= This is an esoteric phrase. The CONDITION OF A FOR-LOOP, for me and for most people I know is the second argument of the for...loop statement. Optimization will happen on the three expressions of the loop and on the whole loop statement and will spread out to the scope/statements block where it is, and from there up to the outer block and up and up to the function level and from there up and up to the program level. Most of this type of optimization aren't in the code generation of the compiler, but right after parsing and syntax check where all kind of stuff is analyze, from the live-time of a variable to its usefulness. I'm pretty sure you can find information about it in wikipedia.

Thanks,
~mhtrader~

 

[fullautotrading]

Quote from mhtrader:

I'm gonna answer the bottom part first: I'm not a talented trader. I hate to lose money and therefore this makes me busy almost 24 hours a day. It feels like 3 fulltime jobs. About writing strategies, I'm not good at it either.... I always try to stick to the existing stuff.. I'm a big fan a Bollinger bands.
...
Up there you can see one of the motivation of why we started to work on the platform.

Now about the first part of the email, fullautotrading, I suppose this is a confusion and not a ill comment

...

Back to the loop thing...one loop was optimized with the multiplication, but with 2 loops I found that the second loop was going to be there, the second wasn't optimized and it was in from of your eyes.

I don't understand why you didn't include that loop it in your comments? Because it will make your comment pointless I guess? Never mind... I gonna assume it was an honest omission.. may I use your own phrase...? ..."it seems to me that the one who would needs to check better is actually you"

LOL! nope, it's still you! ) In fact, i did mention the second loop with subtraction, because that was just part of my point. Read more carefully:
<i>"a trivial local <b>"optimization"</b> (which would never occur in more realistic programs: <b>it does not even occur later when the addition is replaced with the subtraction</b>)"</i>

i was saying that as soon as you use the subtraction instead of the addition the compiler is not able to make that trivial "optimization", and it's pretty obvious why. (But who needs such kind of fool-proof "optimizations" ?)

If you want to believe that a c++ project will perform better than one in C# or VB.net, you are free to do so. As I noted, stuff that is JIT compiled run just as native as stuff that is AOT compiled, so it's a basically a clueless discussion, but some high level features may be missing or lacking (profiling, memory management, etc.), Anyway, as I said, I am happy to hear about trading platform made in c++. ;-)

Tom

 

[fullautotrading]

Quote from mhtrader:

I'm gonna answer the bottom part first: I'm not a talented trader. I hate to lose money and therefore this makes me busy almost 24 hours a day. It feels like 3 fulltime jobs. About writing strategies, I'm not good at it either.... I always try to stick to the existing stuff.. I'm a big fan a Bollinger bands.

I wish to give you some constructive advice. It's unrealistical to think that any user will find useful something that not even the creator uses actively.

In my experience, it's impossible to separate the implementation aspects from the strategy aspects. It's not sufficient to know how to code to make a platform which, for instance, an hedge fund or a community of traders will be happy to use. If it were so, all programmers would do this activity. It takes much, much more... and your users must be inspired by your <b>enthusiam for trading and strategies</b> ...
If yourself are not able to use profitably your tool, there might be no user that will either. Just because there is a <b>profound design problem</b> at the basis of a <b>profitable</B> system, and if the design is flawed, there is no way to create anything profitable, for any user. Take programs like WL, for instance. Once the design is conceptually flawed, there is no way to come out from the empasse. The program may become a "curve fitter", a creator of illusions ...

Tom

 

[mhtrader]

Tom,

If you want to believe that a c++ project will perform better than one in C# or VB.net, you are free to do so. As I noted, stuff that is JIT compiled run just as native as stuff that is AOT compiled

AOT and JIT are brothers and they can't compete with the C++ optimized compiler. I don't doubt that most AOT/JIT share the same source code for most of what they do.

Native C++ optimized code will be always faster than native jittered .NET code. Because the jittered code is bloated with stuff and it not optimized. The optimization you just saw there in the examples I posted will never be done by any jitter. And if these optimization weren't practical, as I suppose some people may argue... why they put them in the compiler? to have fun? Jittered code is also bloated with function calls into the CLR that you never intended to have when you wrote your code... your .NET program also have threads that you never ask for it.... Even if you take the function calls out... Jittered code looks more like a debug version of a C++ compiled program.

Never ever compare C++ Optimized code to code generated by any jitter from IL.

Thanks,
~mhtrader~