Trading System Development

[volpunter]

you are an idiot. Microsoft can afford to sell 200 dollar OSs because they sold hundreds of millions of licenses (around 700 to be more precise for Windows 7). While Multicharts sold maybe thousand or a few more. If Microsoft offered the same features in Windows 7 and sold only a few thousand licenses each license would cost many tens of thousands of dollars. Hardly comparable I would say.

And here is credible proof that your software sucks big time at even the most basic tasks (google for other features at which it fails, too):

http://www.multicharts.com/discussion/viewtopic.php?f=1&t=47360&sid=3689f5dd410f7362bea937da35e7c1ea

http://www.multicharts.com/discussion/viewtopic.php?f=1&t=46824

(credits for the links go to another user who PMed me)

If they cannot even get the basics right how much faith do you put into the credibility of their overall features and especially your backtest results?

I am done with this topic, its simply ridiculous and outright bizarre of you to pretend Multicharts is a sufficient tool beyond some "play time" to get familiarized with the absolute rudimentaries of system development and algorithmic trading. Spare us with your hoax claims.

P.S.: Maybe you are not utilizing more than one thread at a given time if you believe you cannot load/process and generate results at an average 5 million ticks/second. By the way, I never claimed it takes one second from loading raw data from database until finishing the whole process. I said that the system can process 5-7 million ticks per second, which means that you start measuring from the precise moment you kick off the process until the last tick is processed and leaves the "pipeline", you then measure how many ticks in total were processed and divide by the number seconds it took.

I digress, this is totally the wrong way of thinking for people starting automated trading.

For example
windows is a 200 dollar software package you can find at wallmart, yet it is still the best state of the art software that an average person can install on their pc for everyday purposes like
browsing, office work and gaming.


Make use of readily available tool, don't bother reinventing the wheel for an extra 0.1% edge.
(unless your HFT or stat arbing).





Sure loading, but you said



Load 7 million ticks and do all that in a second on a commodity quad core ?

no way.

 

[IAS_LLC]

By the way why do you repeat the "cache thrashing" issue? I do not think it has anything to do with overall performance. I create a new struct for each deserialized tick/quote. Deserialization, time stamp sorting are all highly parallelized processes, and so are performance calculations and reporting. The strategies on purpose are not running in parallel because I need exact synchronicity for portfolio back testing purposes.

Maybe cache thrashing is the wrong terminology (although it is essentially the same), perhaps false sharing is the better phrase (http://www.drdobbs.com/parallel/eliminate-false-sharing/217500206) . Basically two processes are using the same chunk of cache due to shared memory, when one process reads/writes from the cache, it forces the other process to pull the data from memory instead of cache which results in additional clock cycles (this can be avoided by adding buffers on the side of shared memory segments). I suspect this is the problem I'm having because I have made no explicit protections against it (yet), and my CPU utilization is only around 70% for each thread, which means I am memory constrained. I plan on running a profiler on my code at some point, but that is low on my todo list right now.

 

[AdrianHagh81]

you are an idiot. Microsoft can afford to sell 200 dollar OSs because they sold hundreds of millions of licenses (around 700 to be more precise for Windows 7). While Multicharts sold maybe thousand or a few more. If Microsoft offered the same features in Windows 7 and sold only a few thousand licenses each license would cost many tens of thousands of dollars. Hardly comparable I would say.

Yawn

And here is credible proof that your software sucks big time at even the most basic tasks (google for other features at which it fails, too):

http://www.multicharts.com/discussion/viewtopic.php?f=1&t=47360&sid=3689f5dd410f7362bea937da35e7c1ea

I admit, severely nagging issue

http://www.multicharts.com/discussion/viewtopic.php?f=1&t=46824

Fixed in August 2014.

(credits for the links go to another user who PMed me)

If they cannot even get the basics right how much faith do you put into the credibility of their overall features and especially your backtest results?

I am done with this topic, its simply ridiculous and outright bizarre of you to pretend Multicharts is a sufficient tool beyond some "play time" to get familiarized with the absolute rudimentaries of system development and algorithmic trading. Spare us with your hoax claims.

The engineers at MC and Ninjatrader aren't just sitting idle.
It's just a matter of time before they come up with a solution that is good enough for
99% of requirements.

It's a bit like how people used to laugh at x86 hardware and use SPARCS and PowerTM
for serious stuff.
In the end, the commodity solutions became good enough, and all the servers are running it.

Back in the day to do serious 3D was the domain of SGI.
Now what's the best 3d platform available, it's NVIDIA. the cards that
is on everybody's laptop and PC.

It's futile in the long run to focus on the Software Engineering aspect of trading.

These are not hard problems, they'll be solved and packaged then comoditized.

P.S.: Maybe you are not utilizing more than one thread at a given time if you believe you cannot load/process and generate results at an average 5 million ticks/second. By the way, I never claimed it takes one second from loading raw data from database until finishing the whole process. I said that the system can process 5-7 million ticks per second, which means that you start measuring from the precise moment you kick off the process until the last tick is processed and leaves the "pipeline", you then measure how many ticks in total were processed and divide by the number seconds it took.

Did You design that for distributed processing across multiple machines yet ?

 

[AdrianHagh81]

Maybe cache thrashing is the wrong terminology (although it is essentially the same), perhaps false sharing is the better phrase (http://www.drdobbs.com/parallel/eliminate-false-sharing/217500206) . Basically two processes are using the same chunk of cache due to shared memory, when one process reads/writes from the cache, it forces the other process to pull the data from memory instead of cache which results in additional clock cycles (this can be avoided by adding buffers on the side of shared memory segments). I suspect this is the problem I'm having because I have made no explicit protections against it (yet), and my CPU utilization is only around 70% for each thread, which means I am memory constrained. I plan on running a profiler on my code at some point, but that is low on my todo list right now.

Have you tried changing your variables scope ?

malloc vs mspace ?

 

[volpunter]

* none of the currency conversion issues is fixed as of now. Stop making up lies when you are even confronted with the truth.

* 99% requirements? You are truly insane buddy. That would then of course put all the highly paid quants, traders and strategists at banks and hedge funds out of business. After all your little 400 dollar crystal ball forecasting and trading suite can do it all.

I do not need to distribute across machines, the process is way fast enough on a single machine.

You are on my ignore going forward, not only do you put out lies when you are confronted with hard evidence but you keep on putting up bullshit (latest blabber about nvidia and 3d after you moved on from your stupid windows 200 dollar comparison).

Newbies, use MultiCharts, NinjatraderOrWhateverItsCalled to learn the ropes but please please do not commit real funding to live trade on such platforms at least not unless you have in detail verified that each and every feature your strategy depends on works as intended and promised.

Yawn



I admit, severely nagging issue


Fixed in August 2014.






The engineers at MC and Ninjatrader aren't just sitting idle.
It's just a matter of time before they come up with a solution that is good enough for
99% of requirements.

It's a bit like how people used to laugh at x86 hardware and use SPARCS and PowerTM
for serious stuff.
In the end, the commodity solutions became good enough, and all the servers are running it.

Back in the day to do serious 3D was the domain of SGI.
Now what's the best 3d platform available, it's NVIDIA. the cards that
is on everybody's laptop and PC.

It's futile in the long run to focus on the Software Engineering aspect of trading.

These are not hard problems, they'll be solved and packaged then comoditized.



Did You design that for distributed processing across multiple machines yet ?

 

[IAS_LLC]

Have you tried changing your variables scope ?

malloc vs mspace ?

What do you mean by changing scope? Both processes are completely separate. One is a data feed handler and one is trading system, and both are C++ class types that live in different main() functions. I am using QT Shared memory for the IPC mechanism (I chose QT because it is platform independent), that is the only interface between the two applications

 

[IAS_LLC]

What do you mean by changing scope? Both processes are completely separate. One is a data feed handler and one is trading system, and both are C++ class types that live in different main() functions. I am using QT Shared memory for the IPC mechanism (I chose QT because it is platform independent), that is the only interface between the two applications

They are class variables as well, so they do not go out of scope.

 

[AdrianHagh81]

* none of the currency conversion issues is fixed as of now. Stop making up lies when you are even confronted with the truth.

Well first you claimed the features doesn't exist, but it does exist albeit with some minor
problems.

* 99% requirements? You are truly insane buddy. That would then of course put all the highly paid quants, traders and strategists at banks and hedge funds out of business. After all your little 400 dollar crystal ball forecasting and trading suite can do it all.

Yes, that is the eventuality of it, just the software guys though.
the alpha generator remains employed.

I do not need to distribute across machines, the process is way fast enough on a single machine.

Hm running on a single quad core box is state of the art ?
oh please.
I've wrote banking apps that needs to be distributed across 5 IBM AS/400.
And a solar cell simulation cruncher that needs to go into HP superdome.

You are on my ignore going forward, not only do you put out lies when you are confronted with hard evidence but you keep on putting up bullshit (latest blabber about nvidia and 3d after you moved on from your stupid windows 200 dollar comparison).

Newbies, use MultiCharts, NinjatraderOrWhateverItsCalled to learn the ropes but please please do not commit real funding to live trade on such platforms at least not unless you have in detail verified that each and every feature your strategy depends on works as intended and promised.

Look, I'm just pointing out that the level of Software Engineering involved in trading
is not particularly high compared to other industries or Academia.

 

[AdrianHagh81]

What do you mean by changing scope? Both processes are completely separate. One is a data feed handler and one is trading system, and both are C++ class types that live in different main() functions. I am using QT Shared memory for the IPC mechanism (I chose QT because it is platform independent), that is the only interface between the two applications

If it is as you suspected the case to be false sharing.

And if you have two threads accessing a variable right next to each other in an array.
thats bad mkay.

use a local var to store first then update.

 

[AdrianHagh81]

They are class variables as well, so they do not go out of scope.

You know linux allocator have a set limit of 64k i think before they go hit up.