New trading platform collaboration C# .NET, anyone?

[tendimsol]

Hi All,

Re data read/write performance; here are my 2 cents.

I think this is a very relative topic, and only relative to this thread since Chris is making a true trading class library 'platform'. What really matters imo is 'where' you are going to host your database, and 'how often' you are going to read/write data to it. Based on these dynamics driven by trading style - you have a plethora of options open to you. From local databases/file structures to cloud blobs / tables or relational database. Each can be appropriate (if not optimal) based on these underlying dynamics, and are somewhat relative to the complexity/scale of the platform design. It becomes a sliding scale at the end of the day, but some general patterns emerge. Use memory not a database/file for high frequency trading. Ensure the client/processing end of your trading system is hosted in the same location (data center) as any stored data. Balance your tick data time frames against requirements and associated processing loads. (there is no sense bottlenecking your system CPU and I/O unnecessarily) How many instruments do you also intend to trade? What kind of reliability / SLA factors do you need? Analyse hosting costs / initial setup fees / ongoing costs relative to your IO frequency and size - and incorporate this into your technology selection and hosting location choices.

Then, you can worry less about different individual platform/service performance metrics, understanding that at the end of the day, if you can afford performance - you can buy it, and it is only getting faster each day; and focus on what matters 'most' (the trading strategies) which if profitable, will allow you to buy/account for any future desired/required data i/o performance requirements. This can be summarized also by concept of 'cost of entry', as for example to build a per tick high frequency automated trading platform is obviously going to require more investment/resources/complexity relative to a system which only stores and process hourly or daily data.

So whilst there is not perfect system I think, I would recommend to anyone building any form of automated trading is to ensure you are using cloud based services and not local user/system run-time applications/files/databases. I see many great 'little applications' people have built that will never scale appropriately at a code level (even if they are profitable or look pretty) because they were not built with a platform architecture mindset. Cloud services allow you to be A) more modular in your designs and B) provide optimal stability / performance / redundancy scale-ability. I see so many blogs of python script hack job trading 'code' that it brings me to tears that they (and the MT4/5 community EA people) don't understand the value of a system that can reboot, update, and have an internet outage gracefully.

 

[tendimsol]

Free Code for .NET C# and Trading Enthusiasts:

 

[fan27]

Hi CJ,

I have read your post about collaboration and c# and would like to arrange a time to skype if you are interested.

Firstly, a little about my background; I have over 20 years in IT and have significant solution architecture and coding experience having founded my own software development company 6 years ago which makes CRM solutions. I have recently completed a rather epic undertaking; the creation of a Azure based cloud service consisting of 4 worker roles and dashboard web app which is currently capable of trading 47 instruments on 2 brokers (kraken and oanda). The platform can actually be very easily described in the following way.

Take Profit Targets:
I use Long Short Term Memory Recurrent Neural Network PER INSTRUMENT PERSISTENT (state-full) models - (using Microsoft Cognitive Toolkit Framework) - outputting predictions time-shifted -8 - 16 & -24 hours, 24 hours a day, 365 days a year based on continuous updated data.
Stop Loss Targets: all stop losses are 2:1 profit / loss (half distance between actual and predicted)

Entry Conditions:
(precondition = margin < 25% and market open etc)
Signal 1 = Price + spread above or below prediction - ( if above buy if below sell. )
Signal 2 = Instrument is in top or bottom 61.8 or 38.2 % 3, 9, or 27 day range. (Dynamically calculated term period overlay Fibonacci Term Period Levels) this brings 2 factors to the table. A) if prediction was in bottom 38.2% - we would not want to sell unless the current price was in the 61.8% or above range. (range = price fluctuation over time). this gives you 'statistically' improved and controlled 'trade entry ranges'. hence adding an edge. You can use chart indicator 'fibonacci arc' or Fibonacci range retracement tool on most charting software to get an idea of what its doing. difference is its doing this calculation every 5 minutes and updating its term min max ranger period percentages.
Signal 3 = Custom Bollinger Band Trend Signal for 8 / 16 / 24 hour time periods picking up trend toward prediction. - Bollinger well known chart indicator. I just use 3 of them overlayed using special values I developed to create a trend indicator.
Signal 4 = The last 2 Rate of Change Values for 8/ 16 / 24 Hour timer periods must be positive/negative match - again regular indicator you can add to chart. it provides additional 'confirmation' of trend toward prediction.
Signal 5 = Ichimoko Cloud Trend Indicator. Red Above Blue = short blue above red = long match - regular (complex) trend indicator - again used to confirm market is moving toward prediction.

Trade Management Rules:
A) If Price has moved more than 38.2% toward predicted, update order with trailing stop = purchase price
B) If Price has moved more than 61.8% toward predicted price AND margin use is greater than 25%, then close the trade.

Volume Sizing:
Currently something I am working on. I have some advanced pip value / exchange rate function now in testing however currently used minimum fixed trade amounts per instrument multiplied by (if over 1) then percent difference between the current actual and predicted. (this gives 'a form' of weighting which I still think is a good idea to keep during implementation of the proper pip value calculated volumes.

Whilst obviously I am a little biased, having created this myself, I do believe it is one of the most state of the art 'personal proprietary' platform 'architectures' around, but it has take me a huge amount of time and effort and money to be honest, which I generally dont have available. I can say that it is trading positively now, and making good profit - however there is always room for improvement.

I totally understand where you are coming from with all the other platforms/libraries. One reason I have the system I do now, is because I has some 'temporary success' with MT4 EA's like trio dancer and Forex Benz many years ago. Of course I quickly learned their limitations and why they would blow accounts, but at the time didn't have the necessary coding skills or knowledge to embark on my own platform. What I did take away from that experience was the 'massive opportunity to earn' that existed - I just had to find a way to take full advantage of it in a 'reliable fashion'. To this end I have sourced (the best predictive neural network model/infrastructure and tweaked it), created all the necessary back end data store/retrieval services etc, and then embarked on applying traditional market analysis indicator rules and typical trade rule layers on top. I firmly believe in an iterative approach to development, and first recognizing underlying market principles/methods - and then finding a way to incorporate and encode these in a systematic way toward building a 'comprehensive automated trading model'. Also I'm sure you understand the limitations of working with web based platforms like ninjatrader or quantconnect etc even if the do offer some c# or f# no.

I am interested to learn about the current state of your platform and if you are interested in collaborating in a serious capacity, we can then look at merging relevant value add code base features toward this end. Whilst I have come a long way alone on this, I cant devote any of my own dev resource to this as he has another product to maintain - and so I need to make a C# trading friend so to speak. 2 brains are definitely better than one when it comes to this. The C# / platform is the easy part! For example it took me 2 weeks to fully understand different unit size calculation methods and resource snippets and only 60 minutes to cut the code. I am looking for someone who has a real passion and dedication to realizing life changing profits using all the knowledge and resources at their disposal, be that time/skill/money etc to hopefully work with me to further refine and maintain the current trajectory. Current goals include testing/implementation of pipvalue/risk/pipstep unit size ( TradeUnitSize = int.Parse(((Convert.ToDouble(accountSummary.balance) * MaximumStopLossAccountRiskPercent / 100) / (StopLossInPips * InstrumentPipValue)).ToString()); ) I want to add instrument leverage to this calculation. (somehow) + some additional dashboard refinements for live trading summary analysis + further debugging where less than 1% of applied trailing stops get rejected. Once volume is perfect for forex/commodities - review of crypto volumes (currently using 'crude' way to determine purchase volume) instead of instrument class like in oanda. Once get to this level, extend to as many instruments can afford infrastructure for (potentially 100 +) to further increase profits, whilst dropping any isolated non performing instruments along the way.

https://tds-trading-dashboardwebapp.azurewebsites.net - predictions
https://tds-trading-dashboardwebapp.azurewebsites.net/Trading_Signals/Index - signals

I also live in Australia which would make collaboration easier. my skype is owenrosswilliams and my email is owen dot ross dot williams at gsnail dot com

I really hope to hear from you if you feel this is something you would like to become involved in.

KR - Owen

Sounds like a great project! Curious as to how exactly you incorporate the Microsoft Cognitive Toolkit into your algo discovery.

 

[tendimsol]

Hi Fan,

thanks for you up late at night question

Yes I hope to build a private community here and take this project all the way to the bank so to speak. Below is some description of CNTK (available for c#). It is Microsoft's 'version' of Tensorflow and Google Deepmind etc, or the underlying components used (imo theorizing) behind such online services as Azure Machine Learning (web drag and drop etc/prebuilt webservices etc)

LSTM is in essence the 'algo' in itself. It is a 'type' of neural network architecture which is both very 'new' relatively speaking to the scene but has also been identified by the industry (as far as I can tell) as yielding the best results for time series analysis when it comes to machine learning 'models'. After all it is predicting the price quite accurately after tweaking. However in order to make money from a prediction - you need many more layers of logic applied to your order entries/exists. This is where my final 'strategy' incorporates using 'best of breed' variables from both the machine learning world and traditional statistical analysis / market indicators in order to make a decision.

If you are interested to learn about just how cool CNTK is I highly recommend checking out Bahrudin Hrnjica's blog @ https://bhrnjica.net/ He has a PhD and is a professor and has valuable information and code examples that can be experimented with / extended / learn from and thankfully he works in C# (most who use CNTK currently using ugly python(not suitable for enterprise platform architecture implementations)).

P.S. The logical extension to the platform design I have described is to extend the trade and account management layers (and potentially execution) to DQN model. This is much more complicated when it comes to trading as one must design 'agents' and an architecture to manage them, however I have learnt a lot on this topic recently and believe that I have the right agent definitions and variables to work with. I have some 'basic' code for DQN using CNTK in c# from a microsoft researcher but this project would probably need more funding and people to achieve such an outcome in reasonable timeframe. The hardest part is finding people to work with which for example I could hand that DQN code template to (them already understanding my LSTM) and have a rational conversation about it with - both understanding the terminology and underlying implementation requirements. Oh.. i forgot to mention. DQN would be like proper AI, not 'machine learning' which imo is just glorified iterative statistical analysis.

The Microsoft Cognitive Toolkit (https://cntk.ai) is a unified deep learning toolkit that describes neural networks as a series of computational steps via a directed graph. In this directed graph, leaf nodes represent input values or network parameters, while other nodes represent matrix operations upon their inputs. CNTK allows users to easily realize and combine popular model types such as feed-forward DNNs, convolutional nets (CNNs), and recurrent networks (RNNs/LSTMs). It implements stochastic gradient descent (SGD, error backpropagation) learning with automatic differentiation and parallelization across multiple GPUs and servers. CNTK has been available under an open-source license since April 2015. It is our hope that the community will take advantage of CNTK to share ideas more quickly through the exchange of open source working code.

 

[qlai]

 

[tendimsol]

Hello 'source player' lol.

 

[fan27]

Hi Fan,

thanks for you up late at night question

Yes I hope to build a private community here and take this project all the way to the bank so to speak. Below is some description of CNTK (available for c#). It is Microsoft's 'version' of Tensorflow and Google Deepmind etc, or the underlying components used (imo theorizing) behind such online services as Azure Machine Learning (web drag and drop etc/prebuilt webservices etc)

LSTM is in essence the 'algo' in itself. It is a 'type' of neural network architecture which is both very 'new' relatively speaking to the scene but has also been identified by the industry (as far as I can tell) as yielding the best results for time series analysis when it comes to machine learning 'models'. After all it is predicting the price quite accurately after tweaking. However in order to make money from a prediction - you need many more layers of logic applied to your order entries/exists. This is where my final 'strategy' incorporates using 'best of breed' variables from both the machine learning world and traditional statistical analysis / market indicators in order to make a decision.

If you are interested to learn about just how cool CNTK is I highly recommend checking out Bahrudin Hrnjica's blog @ https://bhrnjica.net/ He has a PhD and is a professor and has valuable information and code examples that can be experimented with / extended / learn from and thankfully he works in C# (most who use CNTK currently using ugly python(not suitable for enterprise platform architecture implementations)).

P.S. The logical extension to the platform design I have described is to extend the trade and account management layers (and potentially execution) to DQN model. This is much more complicated when it comes to trading as one must design 'agents' and an architecture to manage them, however I have learnt a lot on this topic recently and believe that I have the right agent definitions and variables to work with. I have some 'basic' code for DQN using CNTK in c# from a microsoft researcher but this project would probably need more funding and people to achieve such an outcome in reasonable timeframe. The hardest part is finding people to work with which for example I could hand that DQN code template to (them already understanding my LSTM) and have a rational conversation about it with - both understanding the terminology and underlying implementation requirements. Oh.. i forgot to mention. DQN would be like proper AI, not 'machine learning' which imo is just glorified iterative statistical analysis.

The Microsoft Cognitive Toolkit (https://cntk.ai) is a unified deep learning toolkit that describes neural networks as a series of computational steps via a directed graph. In this directed graph, leaf nodes represent input values or network parameters, while other nodes represent matrix operations upon their inputs. CNTK allows users to easily realize and combine popular model types such as feed-forward DNNs, convolutional nets (CNNs), and recurrent networks (RNNs/LSTMs). It implements stochastic gradient descent (SGD, error backpropagation) learning with automatic differentiation and parallelization across multiple GPUs and servers. CNTK has been available under an open-source license since April 2015. It is our hope that the community will take advantage of CNTK to share ideas more quickly through the exchange of open source working code.

I will take a look at the blog you referenced. The project you are working on does sound like an epic undertaking as you mentioned. The approach I am taking is to only build software for specific problems that are not adequately solved by existing solutions. IMO, there are solutions which do a pretty good job with charting, live trading and metrics. I am evaluating AlgoTerminal right now and it can solve those problems adequately. What is cumbersome and time consuming with most platforms is finding Algos that "work". That is the problem I am working on where my platform can generate code to run the algo in an existing ATS. I much rather spend a few hundred bucks a month on an existing ATS solution than build one from scratch. That being said, you may have specific requirements not met with existing platforms. If that is the case, I would be curious to know what those requirements are.

 

[tendimsol]

Hi again Fan,

In one word, I can answer by saying 'flexibility'. One can think of platforms like AlgoTerminal as a constructed layer within which you can work. Don't get me wrong, working with such platforms can be more than adequate but as you question - 'what more could I possibly need'. Indeed such platforms (by their inherent business model) are design to make it 'easier' for people to implement automated trading strategies (even in c#!). Well I can think of many, news feeds, sentiment analysis, AI/machine learning, even basic things like exchange rates. Due to their business model such services (unlike some brokers) simply cannot keep pace with the rapid rates of change/advancements 'required imo' in order to continue to trade profitable in a market where over 60% of the trades are already automated. In short if you have the option/flexibility to do 'the same thing' independently and not rely on a 3rd party, then in my experience in life - whilst perhaps more difficult - usually the better long term option.

Speaking as a professional software developer however, you can almost form a basic hierarchy of auto trade solutions out there;
1) fully 100% custom built
2) an custom assimilated and compiled class / reference libraries derived from multiple sources/repositories/examples
3) an open source or proprietary broker / strategy/analysis 'code base' and or associated API service
4) web based service 'scripts' or client side application (MT4/5) 'hosted' scripts.

Since custom built would require genius level coding skills and trading knowledge, or corporate resources, lets assume this is out of selection even though it will offer highest degree of flexibility.

Next is a collection of classes or analysis functions/methods which have been obtained as a result of 'research and development' - effectively tailored to the requirements of your design. This requires a lot more time and effort to assimilate this 'IP' into your 'working platform stack' - however is 2nd best for providing flexibility.

3rd place goes to defined libraries and or service machinations, as the way they work within functional boundaries - this will restrict your to ability to 'truly' evaluate 'algos'. I will concede that such platforms make things like backtesting much easier but honestly, you are much better off simply using a broker that has these services offered directly even if that is desired functionality. As you say there are many algos available but how to find a profitable one? My advice on this topic is there is no silver bullet, which is why you will find the 'best' quant traders out there doing linear regression against multiple algorithms running in parallel to pick best one to run at different times. Also with regard to backtesting - whilst a valuable tool, I think more focus should go into real live trading on practice account in conjunction to an 'agile' or iterative monitoring and development approach.

The least flexible is of course web service that will run a 'script' (pseudo code) for you, or tie you down to implementation 'within' a client side application which for automated trading is just a dumb idea for so many stability / functionality restrictions.

KR,

Owen

 

[fan27]

Hi again Fan,

In one word, I can answer by saying 'flexibility'. One can think of platforms like AlgoTerminal as a constructed layer within which you can work. Don't get me wrong, working with such platforms can be more than adequate but as you question - 'what more could I possibly need'. Indeed such platforms (by their inherent business model) are design to make it 'easier' for people to implement automated trading strategies (even in c#!). Well I can think of many, news feeds, sentiment analysis, AI/machine learning, even basic things like exchange rates. Due to their business model such services (unlike some brokers) simply cannot keep pace with the rapid rates of change/advancements 'required imo' in order to continue to trade profitable in a market where over 60% of the trades are already automated. In short if you have the option/flexibility to do 'the same thing' independently and not rely on a 3rd party, then in my experience in life - whilst perhaps more difficult - usually the better long term option.

Speaking as a professional software developer however, you can almost form a basic hierarchy of auto trade solutions out there;
1) fully 100% custom built
2) an custom assimilated and compiled class / reference libraries derived from multiple sources/repositories/examples
3) an open source or proprietary broker / strategy/analysis 'code base' and or associated API service
4) web based service 'scripts' or client side application (MT4/5) 'hosted' scripts.

Since custom built would require genius level coding skills and trading knowledge, or corporate resources, lets assume this is out of selection even though it will offer highest degree of flexibility.

Next is a collection of classes or analysis functions/methods which have been obtained as a result of 'research and development' - effectively tailored to the requirements of your design. This requires a lot more time and effort to assimilate this 'IP' into your 'working platform stack' - however is 2nd best for providing flexibility.

3rd place goes to defined libraries and or service machinations, as the way they work within functional boundaries - this will restrict your to ability to 'truly' evaluate 'algos'. I will concede that such platforms make things like backtesting much easier but honestly, you are much better off simply using a broker that has these services offered directly even if that is desired functionality. As you say there are many algos available but how to find a profitable one? My advice on this topic is there is no silver bullet, which is why you will find the 'best' quant traders out there doing linear regression against multiple algorithms running in parallel to pick best one to run at different times. Also with regard to backtesting - whilst a valuable tool, I think more focus should go into real live trading on practice account in conjunction to an 'agile' or iterative monitoring and development approach.

The least flexible is of course web service that will run a 'script' (pseudo code) for you, or tie you down to implementation 'within' a client side application which for automated trading is just a dumb idea for so many stability / functionality restrictions.

KR,

Owen

I prefer a hybrid approach of 2 and 3. The key is to find a solution in 3 (off the shelf) that allows for 2 (customization and extension). That being said, most of the off the shelf platforms have not met my needs. I have personally chatted with several people on this site who have fully functional solutions for algo trading that they mostly built from scratch. While impressive, I also know the level of effort put into building those platforms and that is time I do not have. Anyhow, not sure how active the OP is on the site but looks like you have some great coding skills and could likely recruit someone to your effort.

 

[tendimsol]

I prefer a hybrid approach of 2 and 3. The key is to find a solution in 3 (off the shelf) that allows for 2 (customization and extension).

Yeah I hope OP is active, but i'm guessing others may contact also. You know its a interesting thing the politics behind creating such a thing. In reality it is effectively a license to 'print money'. As such I am already cautious about who I work with / sell code to or collaborate with. For example I asked a friend if they wanted to invest in the platform for a small one off fee (to help me start funding 'actual serious go live account with sufficient capital to cover all the instruments being traded (after all I am just a broke entrepreneur trying to fund an actual unrelated product release)) I know his story in wanting to make money etc for his kids future etc etc, and since I know my platform is profitable - I proposed to provide all upgrades also to strategy files etc. I explained to him as best I could the system, and all he can ask is 'what is the % profit daily/weekly like in the emails we used to get'. I don't plan to reply because I now understand that that is 'too easy' an option. Nobody ever came to me offering guaranteed results, saying here is free money with no risk/effort required. I now reply by saying ok, so do the calculations yourself from my signals if you can be bothered. This is from someone who actually enjoys helping people. I am guessing this is the same dynamic many people on these types of forums have picked up on. Where are all the people making money and what tools/codebases are 'they' using and why they are so quiet. (well to anyone appropriately aligned reading this (my) exception to the rule and looking to become part of this exclusive hidden class of traders hopefully reply haha) All I can say is stick with it an align yourself to people with similar goals. Also from your approach you should probably take a step back with a whiteboard in terms of 'what matters' and then base your coding around this. 3 providing 2 does not exist by rule of nature I just described. We live in an interesting age. Personally with advent of quantum computing / laws banning them for personal use etc etc. I think this type of trading has a definite time limit, beyond this it is AI and systemic breakdown of traditional systems. Think of it like this , if someone releases to public the perfect algorithm and simple method of application, this will spread like wildfire. Very soon its strategy across many players will 'MAKE' the market. It will do this up to a point, much like the time to sell is when interest is at its maximum; however beyond this - the market will then quickly adapt, and said strategy will no longer function as it once did.

You may find it is actually less code than you think and if you know what you are looking for its just a search away. for example indicators. why be limited by one single api computing them for me or the range they have available. better to source from open source libraries and compile your own library of 100's to choose from. Also, how much % of their exposed features will actually be useful? One tip I can give, is just to stress the real-time monitoring 'in conjunction' with your applied strategy 'rules' or signals. before i created the 'signals' dashboard it was very difficult to perform any meaningful tweaking of indicators relative to different timeframes and get them in meaningful alignment. Whilst not 100% necessary - this visual type feedback is very helpful to determine what combination of indicators/settings/processing variables works best. such a dashboard is really small amount of work in reality. I guess i should clarify that my approach to 2, is to KISS, and only add things you actually need to help achieve positive returns reliably. Everything else, including many of the features / subscription services offered, and including things like charting etc is really not necessary but just a value add on top. Just don't get bogged down in platforms/libraries. Formulate or adopt/adapt a strategy of your own and then build the platform 'around it'. This will be way less work than you are imagining in terms of actual effort. Another tip would be don't rely on other peoples recommended settings. Always experiment with your own values in order to morph them to your needs. So understand that again with these online platforms, may of the functions or template recommendations are just that, recommendations. So whilst someone might recommend 70% training/20 evaluation and 10% dataset split as a industry recommendation (understand they are reciting what they were told) and whilst in 'general' these things are true - may not always be accurate to the model outcome you are trying to produce. so for example in above machine learning data if one were to use this 'recommendation' for any price movement outside known boundaries no matter how much past historical data you have - your adjustment is going to up to 30% lagged in time. Lets take recent oil over 80 scenario... it will fail epic-ally.