Simples...thanks for that excellent post! Fantastic!!
Man....Go is FAST!
- Thread starter fan27
- Start date
Simples...thanks for that excellent post! Fantastic!!
I second that. From my experience one can choose from among 10 different languages for a backtest engine/framework, and if written well each of the 10 language implementations will iterate over historical quotes at speed that is only capped by file IO limits. As usual, choose a language that you are skilled in and that is widely supported. In that I question Go can compete with Java or C# and a lot of existing parts in other languages would have to be "reinvented" in Go.
Thanks for your post. I wonder do you mind commenting on the issue that a lot of existing frameworks and abstracted parts in other languages with wider and longer user support would have to be reinvented or written in Go? For example, streaming is well handled in C# via tpl data flow and reactive extensions. Or linq. In Go you would be forced to write tons of boiler plate code to get similar features. Another example is simple json converter. I looked at the code at the golang.org website under json/stream and it looks very wordy and long-winded to me, something that comes out of the box or one can pick from 2 dozen libraries in Java or c#. Or check out protocol buffers. It requires the c++ installation. I wonder what the usefulness of a language is if it only opens up libraries when linking to other languages.
Last edited:
Not sure what I plan to ultimately do with my site. For now I am re-writing the functional components into Go packages which could be run from the command line with the expectation that I will eventually incorporate them into my site. The packages will be open sourced so anyone with a little bit of programming knowledge should be able to use them for back-testing.
I am working on a marketdata package now that will be able to load all sorts of data from CSV (OHLCV (multiple time frames), Events, Economic and Fundamental data, etc). It is written in such a way that other data sources can easily be added. The aim will to be able to load the data so that the data sets can be used in a single condition.
Do you have access to tick data? If so, pm a sample data file I can use for testing (does not have to be big) and I can add a function for loading tick data.
thanks
fan27
Checkout the packages available here: http://go-search.org/.
I already found a Go implementation of ta-lib which has all standard indicators so the code I am writing is code I would have to write regardless of the language I chose. But to your point, Go is a new language so there will not be as many packages to choose from at the moment. Plus, learning Go and having tight Go repositories on GitHub is a good career move for me at the moment
fan27
As for me, I'm a Go newbie and naturally am very excited about the language. I've really been productive in the language from day 0
, just as I've been in Ruby, and have gained alot of insight from Golang videos and resources on the net. Having said that, please take my enthusiasm with a pinch of salt. Golang might not be for you or for your next project, as other languages and solutions might be better suited: ie. choose the right tool for the job. Often the right tool is the one that makes you happy 
As for Go, I'm not so sure reinventing existing frameworks and abstracted parts from other languages automatically is the right path, at least that's a different goal than what the Golang-developers tries to facilitate with Go. Golang is already derived from a minimal but functional subset of multiple language concepts, academic research, rich ideas about language design and exceptionally experienced developers at Google. So would recommend to learn more about how stuff is solved in Go, and just go through the boiler-plate coding period at first. For streams, io.Reader, io.Writer and interfaces are new abstraction-designs in Go that blows most existing stream-abstractions out of the water.
Mature frameworks and features are deceptively powerful. True, you can quickly generate very advanced functionality, many layers and indirections, but it comes at multiple costs, ie. complexity, performance, memory utilization, GC and dependency-vulnerabilities.
Instead, the go way is more about taking away. As the adage goes "less is more". Thinking things through many times becomes easier when building from small, independent, atomic parts ("lego bricks"), rather than gluing existing stale structures over your concepts. It may pay off in the end to spend more time simplifying, only making what you truly need, in the way that you need it, thus minimalizing unnecessary costs and complexities. At the start, these benefits and costs seems to be invisible, but tend to accumulate over solution lifecycle. So, just as there isn't necessarily more value in more lines of code, there might not be more value in adding complexity and dependencies.
I agree it's unfortunate that sometimes Go libraries have external language dependencies, and there are some areas Go is not currently adapted for, like games-development. However, any improvements not breaking current Go 1 code may be accepted into the language and developers are encouraging experimentation and discovery.
thanks for sharing your thoughts, valuable insights and experience. I am rather skeptical of all those new languages. Out of 100 languages generally less than 10 make it to a more mature stage. One of the most successful ones in the past few years being Python, imho. It did exactly what you described: Solving simple problems without much bells and whistles in an easy and efficient manner. Obviously Python is completely different than Go even in its core architecture but I see also parallels in terms of target audience. But it is probably too early to tell...
anyway thanks for sharing your thoughts...would be great if you could name a few use-cases where you think Go particularly fits well...
anyway thanks for sharing your thoughts...would be great if you could name a few use-cases where you think Go particularly fits well...
As for me, I'm a Go newbie and naturally am very excited about the language. I've really been productive in the language from day 0
As for Go, I'm not so sure reinventing existing frameworks and abstracted parts from other languages automatically is the right path, at least that's a different goal than what the Golang-developers tries to facilitate with Go. Golang is already derived from a minimal but functional subset of multiple language concepts, academic research, rich ideas about language design and exceptionally experienced developers at Google. So would recommend to learn more about how stuff is solved in Go, and just go through the boiler-plate coding period at first. For streams, io.Reader, io.Writer and interfaces are new abstraction-designs in Go that blows most existing stream-abstractions out of the water.
Mature frameworks and features are deceptively powerful. True, you can quickly generate very advanced functionality, many layers and indirections, but it comes at multiple costs, ie. complexity, performance, memory utilization, GC and dependency-vulnerabilities.
Instead, the go way is more about taking away. As the adage goes "less is more". Thinking things through many times becomes easier when building from small, independent, atomic parts ("lego bricks"), rather than gluing existing stale structures over your concepts. It may pay off in the end to spend more time simplifying, only making what you truly need, in the way that you need it, thus minimalizing unnecessary costs and complexities. At the start, these benefits and costs seems to be invisible, but tend to accumulate over solution lifecycle. So, just as there isn't necessarily more value in more lines of code, there might not be more value in adding complexity and dependencies.
I agree it's unfortunate that sometimes Go libraries have external language dependencies, and there are some areas Go is not currently adapted for, like games-development. However, any improvements not breaking current Go 1 code may be accepted into the language and developers are encouraging experimentation and discovery.
It's easier to say what Go is not particularly made for, like native interactivity, anything requiring GUI/widgets and huge, mature frameworks. I can take a wild guess at this list though: IO/text processing / command-line tools / automation, heavy computations, concurrency, micro services / "cloud", anything "small is good", basically most in the systems programming-category (which is what Go is made for), except for OS (at least not out-of-the-box, naturally).
Recommendations from various companies linked to languages: https://github.com/golang/go/wiki/FromXToGo
Lots of leading-edge tools are being created with go, like Kubernetes, OpenShift, Docker, Prometheus and CoreOS: https://github.com/golang/go/wiki/GoUsers
If you like Python for its design-choices, you might enjoy Go's design choices even better. This is legacy from C/C++, Pascal-era refined, and seem to set some programming constructs to new standards as well as simplifying the act of coding and sharing the same codebase (though not in the murky interdependency-area yet):
If it appears I have some insight, it's because of people like Rob Pike paving the way and tying it all together in go, a compiler now also compiled from Go itself. Really tremendous work already done and ready to use. Some rough edges here and there, but often they are there by design in order to keep options open for future development and force developers to rethink complex architecture, designs and usage.
Most importantly: Go is fun! It doesn't feel tedious at all. The language constructs supports Don't Repeat Yourself (DRY)-principles and are themselves DRY, but acknowledges that per default benefits of keeping DRY should outweight the accumulated costs (dependencies and fault/compatibility-cascades).
Languages like Python and Ruby made it simple to make complex things. Go takes a different approach, making it easier to make simple yet effective designs. The latter is much harder to accomplish and takes time. Where Go doesn't fit, it may make sense to use something else. Though is worth learning simply because it advances the bar for all C/Pascal-derived languages.
Last edited:
@fan27, I'll give you a different point of view from the admin world and capacity planning. Since you're expanding your web site, execution efficiency and memory footprint become very relevant.
If you have a program (or program module) that takes 30megs RAM to execute and a server with 8g RAM, that means you can roughly serve a couple hundred instances of that program at once. If you have 1000 users trying to access your services at once, you'll need 5 servers. If you rewrite your program in something more efficient (usually C/C++) and now it only takes 1meg RAM to run, you can handle all 1000 users on a single server with room to spare.
Similar goes for execution time. If one language takes 10 seconds to run per core and the more efficient language takes 1 second per core, you can handle 10x the load given the same hardware.
Most people don't think about this because they're only programming for themselves or clients running their own workstations. In these instances, most people don't have to care.
As a semi-relevant example: Recently I've been forced to learn python to work with selenium (web browser automation). I'm traditionally a C/C++ and bash programmer for linux. I've got around 1500 lines of python code right now to make selenium semi-useful. Since I'm navigating web pages, I'm dealing with small amounts of data in terms of strings and objects... probably less than 100k max (I'm not including the browser because it's running in its own program space and being accessed through an API). I was amazed that python took a dozen seconds to process and multi-search through several dozen strings when similar code I've used in C++ could do the same job in less than 1/4 second. I've also seen the python shell jump to 60megs RAM usage for something that should be less than 1meg for C++. I wasn't able to load the benchmark links in a previous post, so I can't compare... but I'm guessing python has a 30-50x overhead compared to the equivalent C++ in my case. I'm not going to be able to run very many instances of this program at once. :-(
Tying this back to Go, a decently compiled language will crush any interpreted language hands down... including bytecode interpreted java. I would suggest learning C++ in the future since you're building a public server, but for now, pay attention to the high speed low level Go abilities and exploit them where you can. If you're seeing some slow downs or memory flare ups, try to narrow down where the problems are and possibly rewrite them yourself in more efficient and simpler code. I'm not saying you have to do this all right now, but just keep an open mind about fixing things for the future. I think you mentioned Go boosting your resume. Thinking ahead like this is an even bigger boost if you ever decide to get into more server programming jobs. Many server programmers flippantly give the answer of "buy more hardware"... and most employers with real world experience will respond with, "wrong answer". If you can start learning capacity planning early on, that will put you far ahead of many wannabe programmers.
If you have a program (or program module) that takes 30megs RAM to execute and a server with 8g RAM, that means you can roughly serve a couple hundred instances of that program at once. If you have 1000 users trying to access your services at once, you'll need 5 servers. If you rewrite your program in something more efficient (usually C/C++) and now it only takes 1meg RAM to run, you can handle all 1000 users on a single server with room to spare.
Similar goes for execution time. If one language takes 10 seconds to run per core and the more efficient language takes 1 second per core, you can handle 10x the load given the same hardware.
Most people don't think about this because they're only programming for themselves or clients running their own workstations. In these instances, most people don't have to care.
As a semi-relevant example: Recently I've been forced to learn python to work with selenium (web browser automation). I'm traditionally a C/C++ and bash programmer for linux. I've got around 1500 lines of python code right now to make selenium semi-useful. Since I'm navigating web pages, I'm dealing with small amounts of data in terms of strings and objects... probably less than 100k max (I'm not including the browser because it's running in its own program space and being accessed through an API). I was amazed that python took a dozen seconds to process and multi-search through several dozen strings when similar code I've used in C++ could do the same job in less than 1/4 second. I've also seen the python shell jump to 60megs RAM usage for something that should be less than 1meg for C++. I wasn't able to load the benchmark links in a previous post, so I can't compare... but I'm guessing python has a 30-50x overhead compared to the equivalent C++ in my case. I'm not going to be able to run very many instances of this program at once. :-(
Tying this back to Go, a decently compiled language will crush any interpreted language hands down... including bytecode interpreted java. I would suggest learning C++ in the future since you're building a public server, but for now, pay attention to the high speed low level Go abilities and exploit them where you can. If you're seeing some slow downs or memory flare ups, try to narrow down where the problems are and possibly rewrite them yourself in more efficient and simpler code. I'm not saying you have to do this all right now, but just keep an open mind about fixing things for the future. I think you mentioned Go boosting your resume. Thinking ahead like this is an even bigger boost if you ever decide to get into more server programming jobs. Many server programmers flippantly give the answer of "buy more hardware"... and most employers with real world experience will respond with, "wrong answer". If you can start learning capacity planning early on, that will put you far ahead of many wannabe programmers.