Man....Go is FAST!

[bjohnson777]

So, to answer your question what real world experience I have on the administrative side: none!

So why do you constantly criticize and contradict everything I say? I've given you plenty to look up on your own and verify. And they're not my standards. These are just the basics of the basics. You seem insistent on constantly leading people astray.

If you really think you're so good, then I challenge you to write a back tester that will take its instructions by command line or config file, load 1 million bars minimum, do something more meaningful than a MA crossover test, write a brief output report to a file (preferrably an HTML segment that can be inlined), and can run 1000 instances at the same time. Take the run time with the computer unloaded and then take the average run time with the computer fully loaded. Go ahead and use C# since you like it so much. It may be fine for individual workstation use, but we're talking multi-use servers in this thread.

 

[Zzzz1]

"So why do you constantly criticize and contradict everything I say?"
Because what you are talking about in this thread is irrelevant to the thread topic. a) OP never talked about wanting to build a massively distributed backtest engine. b) OP never talked about wanting to run 1000 (your number quoted) concurrent backtests. c) OP focused this thread on "GO" and not C++ or C# or anything else. You are OFF TOPIC. Hence it is you who hijacked this thread, not others.

"If you really think you're so good, then I challenge you to write a back tester that will take its instructions by command line or config file, load 1 million bars minimum"
I have done so, as indicated. Not from command line but from a UI frontend that connects via message bus to a server backend. Not have I loaded only 1 million bars but I constantly process 10s if not 100s of millions of tick based data. The capabilities of the data loader lies at around 10 million tick price data points (with millisecond time stamps) per second. However, in reality this is never achieved because...see next comment...

"do something more meaningful than a MA crossover test"
...exactly for this reason, most strategies are way more complex than some simple MA crossover. The strategies are simply too complex in most cases to consume even more than 2 or 3 million price data points per second. Hence, a data loader faster than that is overkill and simply unnecessary. Even if you run multiple backtests on several cores on a the server side you still are bound by the PCI throughput caps and internal memory capacity. To give you a concrete example: Even a really simple data struct takes up 24 bytes of space. For even moderately complex architectural considerations (post backtest processing, TCA (transaction cost analysis), performance computations, ...) you need to retain the entire data set in memory after the strategy iterated over the data points and before the entire backtest with all its statistical computations is completed. That means 10 million data points (which is pretty moderate because for tick based data in spot fx [even much more for the complete order book in cash equities] constitute only around 50 days worth of data. Such amount of data will take up around 230 megabyte of memory. Only a few dozen such data series will entirely clog up your memory unless you are investing in massive amounts of memory (>=128gb). Also, loading those data series concurrently forces you to invest in a Xeon multi core engine that spots at least 20 cores (via 1 or 2, or more CPUs). This is completely out of topic unless the specification was to build a distributed backtest engine over multiple servers, something OP was NEVER talking nor hinting at. Again, you are OFF TOPIC. This topic is NOT about a massively distributed profiling and testing framework.

"Go ahead and use C# since you like it so much. It may be fine for individual workstation use, but we're talking multi-use servers in this thread"
While C# clearly would not be the best choice of language for a massively distributed testing architecture, there is nothing that would limit the use of C# in theory. It's not that your C++ structs are magically of smaller size than as C# struct for the same quote or bar. C# can out of the box distribute workloads over an unlimited amount of threads via TPL. In fact you can from within C# even spin up whole Azure instances. The possibilities are endless. But again, I am not promoting using C# for such project. However, point here is, this thread is NOT about a distributed testing architecture where hundreds or more clients concurrently connect and submit backtest requests.

I have nothing against you personally, I simply think you misunderstand the thread topic and what OP is looking to achieve. You are probably right with many of your points if the goal was to create something like the mulitiple available web-based frontends that submit backtests to a server or server farm to process. But you are imho totally off-topic here. I have made my point and am out of here. Good luck in your endeavor.

So why do you constantly criticize and contradict everything I say? I've given you plenty to look up on your own and verify. And they're not my standards. These are just the basics of the basics. You seem insistent on constantly leading people astray.

If you really think you're so good, then I challenge you to write a back tester that will take its instructions by command line or config file, load 1 million bars minimum, do something more meaningful than a MA crossover test, write a brief output report to a file (preferrably an HTML segment that can be inlined), and can run 1000 instances at the same time. Take the run time with the computer unloaded and then take the average run time with the computer fully loaded. Go ahead and use C# since you like it so much. It may be fine for individual workstation use, but we're talking multi-use servers in this thread.

 

[bjohnson777]

Since you never really bothered to read what I wrote, I'll thank you for making some of my previous points and making a fool of yourself in red.

 

[pstrusi]

I have started to rewrite the guts of my back-testing website in Go and I am blown away by its speed! I will be creating Go packages for each piece of functionality and will be open sourcing them. The first package I am going to write is the marketdataloader package. It will handle CSV data for now but other data adapters could be added. The data loader will be able to load multiple time frames of data (having them linked together) as well as market events (i.e. OPEC cuts production, etc.) and return the data in such a way that you can easily create complex backtest conditions. Stay tuned!

fan27

Interesting post. So, after quite some time, what's your opinion on Golang for fast trading systems?

Thanks!!

 

[Simples]

Interesting post. So, after quite some time, what's your opinion on Golang for fast trading systems?
Thanks!!

If I may chime in. Of course it always depends on your needs. If you and your system is highly mathematical, you'd maybe want to optimize using GPUs, ASIC, whatever. But that takes extra effort and time, which could be better spent getting everything to work together instead. So if you're like me, and want to avoid premature optimization and not afraid of later refactoring / rewrite, if required, you could fancy "The Go Way". It promotes a simple and easy code style where you just write code as it is supposed to execute, and much of Golang is actually optimized to work highly efficiently out of the box.

There are some dark sides to Golang as well:

1. It doesn't produce all logging-context needed in stack traces and doesn't point you directly to faulty code through exception handling. Java succeeded because it provides full function trace, though no additional / useful contexts like arguments, return values or contract-info. Golang provides even less in this area, is not hopeless, but you may miss errors or need to hunt down the right scope of the root cause on a few occations.
   
2. Sometimes the default handling in Golang is directed at power-users, and you can spike your hand if you're not extra careful how datastructures are actually handled in the specification (which is quite short and simple), ie. types like arrays, slices and maps are handled in machine-efficient, but slightly different and unintuitive ways than other languages. These ways saves memory and CPU by default, naive use, but may sometimes provide unintended sideeffects and the rare WTF scenario.

That's it really. The rest is beautiful, lean and mean. Using it for trading, though on very slow timeframes like daily bars. You may compare Golang speed to Java and C#, and it could approach (probably not exceed) C++ sometime, has a very concurrent GC, especially in latest versions, and kept stable and mature so language, stdlibs and tooling don't change too much under your feet. In terms of runtime memory and compile times, it's faster and smaller than anything comparable, unless you go with C/C++ and especially like to inspect coredumps and fancy an illegal instruction or two. Yeah, Golang is made to be safe as well, but runs natively without VM and minimal dependencies. Can't complain.

Though Golang is quite easy to learn, it is really a language geared at the niche: systems languages, like C, but with GC, safety and lots of standard libraries and tooling (though not exhaustively so). Golang is the programming language you'd want to use to build: cmdtools, databases, proxies, (micro)services, and yes, Trading systems.. Though actual needs and preferences always come first.

 

[fan27]

Interesting post. So, after quite some time, what's your opinion on Golang for fast trading systems?

Thanks!!

@Simples lays it out very nicely above. I switched to .NET Core because I plan on selling my product and C#/.NET is more marketable than Go.