Starting automated trading, what are most common setups?

[volpunter]

One of the most honest but above all accurate posts in a long time on this site. Most would be well advised to invest their efforts in other areas for much higher returns financially and emotionally.

Maybe you've lost before the game has even started? If that is true, wouldn't you like to know as soon and as cheap as possible?



Invest your 4 hours/day in something more productive. Trading is not a growth industry.

You're unlikely to compete in that product with a short term automated strategy and certainly not with an infrastructure budget of $100/month. To give you some perspective, there is a guy here with clearing costs around a tenth of what you'll pay at IB, IOM seats, and who writes direct to the exchange. He said he had no automated strategies running on the S&P futures.

It seems you've made the common error of starting with what you know about (programming), instead of learning about the environment you want to enter. There probably isn't much point going into why this is a poor choice of contract / trade frequency and also data feed, as
1) the information is already available on a search
2) if you need help getting past the easy bits, you aren't going to crack the hard bits

Perhaps look at other ways into the industry if you have a keen interest...it attracts a lot of bright people who like solving problems...but also those who lack the honest self-evaluation necessary to recognise whether they have a realistic chance of competing effectively...like moths to a flame and it is sad to see when their efforts could be so much better employed elsewhere.


Then I cannot fathom why you've missed his excellent points about transaction costs and have opted for a high frequency / daytrading strategy. Will just re-state my point that if you cannot make the best of what is already available for free, the simple stuff, you haven't a prayer when you run up against something difficult.

I've recently learned that I have a compulsion to try to rescue. To save others from the adverse consequences of their decisions. I've acted this out on ET trying to discourage newbies who obviously weren't going to make it, as well as in my personal life. I've earned a little more awareness of this tendency but I've had a little bit of a slip up today.

Cheers.

 

[tradingcomputer]

I will have to try this automated trading things for a year, either success or fail, it is ok, I still have my day job. If success, I will quit my day job, if fail I will stop trading.

One of the most honest but above all accurate posts in a long time on this site. Most would be well advised to invest their efforts in other areas for much higher returns financially and emotionally.

 

[Johanmul]

They do exist. There were two guys at a British trading company keeping at least $7/round trip in ES and doing a few tens of thousands per day. So they could start up with $20k and an IOM lease.

They sure do. I used to work for a guy who averaged close to two handles on the ES for several years, he'd trade infrequently, but "in size".

tradingcomputer : don't get me wrong but you are setting yourself up for failure. If you think you know more than a trader who used to be a musician because you've taken a machine learning class, that arrogance will cost you money. When you start trading you'll be against traders who know more about markets than you do, so you'd better step out of your comfort zone and learn the basics first.

 

[tradingcomputer]

tradingcomputer : don't get me wrong but you are setting yourself up for failure. If you think you know more than a trader who used to be a musician because you've taken a machine learning class, that arrogance will cost you money. When you start trading you'll be against traders who know more about markets than you do, so you'd better step out of your comfort zone and learn the basics first.

Johanmul,
Thank you for your advice.

Here my honest self-assessment of my skill set:
Machine Learning: intermediate to expert
Algorithm: expert
Programming: intermediate to expert
Data Processing: expert
Trading/markets: basic, I believe I need to learn a lot in trading/markets, there is no way I can apply my other skill set effectively without knowing the trading/markets knowledge in depth.

Many years ago, someone who claim that he is a machine learning (ML) expert, told me that he does not to understand the domain, in this example fraud, to apply machine learning. He failed, as the features that he built --as inputs of ML algorithm-- are too generic, not using any domain knowledge.

In depth understanding the domain --in this use case-- trading/markets is paramount to the overall success. Machine learning, algorithm, programming, data processing are just supporting tools.

First, need to understand the trading/markets domain in depth, then formulate the problem to be solved (that is asking the right questions), finally applying the right tools correctly to solve the problem (the answers).

 

[zbestoch]

They do exist. There were two guys at a British trading company keeping at least $7/round trip in ES and doing a few tens of thousands per day.

They sure do. I used to work for a guy who averaged close to two handles on the ES for several years, he'd trade infrequently, but "in size".

Except to say that not everything which counts can be counted. Just because we cannot see a structure which would allow such high P&L over millions of contracts traded doesn't mean it isn't there. I'd venture to say that computer/maths guys are especially unlikely to discover it because of their approach. The information isn't in the data.

"not everything which counts can be counted" - How true. Edges differ in kind/class. Nice post G_o_B

 

[monoid]

@tradingcomputer:

I know I am little late to this thread. But I do want to share a few thoughts.

(1) Read @Ghost_of_Blotto's post again. Re-read it; then, re-read it again. If you still want to move forward continue;
(2) As a newbie, take @garachen's advice

"stick with low frequency - maybe a dozen or fewer trades a day"​

If this hold time (or longer hold time) fits your personality, move forward otherwise ... (go back to (1)?)

I have been running a fully automated system for the past 2 months. I had 'manually' traded the system for about a year before that. I used ML to "mine" my trade setups -- took me about 2 years, working full time, to formulate the research problem, implement, and validate the trade setups.

My current automated trading setup (home grown, self coded) consists of:

• a broker interface for Market data (FIX coded in Haskell);
• a broker market order interface (FIX coded in Haskell);
• A messaging bus (RabbitMQ)
• Trading Strategies (Haskell);
• Redis to hold detailed monitoring data from trading strategies.
• Web server to serve the day's trade signals generated by the Strategies, and other data held in Redis (Node.js with node-redis)

• Monitoring (Monitis -- server monitoring, RabbitMQ monitoring, network monitoring, broker interface app monitoring, and trading strategies app monitoring. Monitis has an ios app that is very handy).

My FCM provides me with the ability to look at my order book and make modifications if need be over a web interface (in case I get failure alerts from monitoring application and I know there are open orders working in market(s)). So, I didn't have to recreate it.

A couple of thoughts on how I approached my research, which I think might be of help to you. I looked at the problem as pattern detection in the context of a complex adaptive system. By framing the system as a CAS, I had the theoretical justification to look for patterns in the first place, and the knowledge that the goal is not to find similar instances of patterns but to extract a pattern class from similar pattern instances. It turned out, it was not easy to even 'encode' patterns. I finally settled on techniques from Bioinformatics to encode patterns, use probabilistic clustering algorithms to group 'similar patterns' (yes, not a supervised learning approach!; did consider MDPs/reinforcement learning but couldn't succeed in formulating a research statement). Going from here to finding a trading strategy was a process of iteration -- going back and forth between raw data and the similarity sequences within a cluster.

Then comes the easy part of testing the strategies in an out-of-sample data set to determine statistical validity. However, I want to emphasis that the statistical testing done is almost useless given the assumption of CAS. In a CAS, the system evolves. This means, the distribution of a class of pattern 'holding' will not be Gaussian -- it could be clustered, have power law distribution, or in the worst case, be total unstable. So, however good the out-of-sample results were, I had to take it with a grain of salt.

All the best with your research -- don't underestimate the effort required.

Regards,
Monoid.

 

[Gambit]

It looks very complicated (multiple languages is asking for trouble)

My setup is IB for execution and data, and my own python package for both implementation and back testing. I spend 10 bucks a month on data. More information on my blog qoppac.blogspot.co.uk

GAT

You've got the perfect set up for the home trader, inclusive of strategy and approach. You really should consider selling your platform. There are tons of algo traders using IB who would pay for a package like that.

 

[globalarbtrader]

You've got the perfect set up for the home trader, inclusive of strategy and approach. You really should consider selling your platform. There are tons of algo traders using IB who would pay for a package like that.

It sounds superficially appealing, and I've thought about it. But there is more work involved than you might think. The code would need to be rewritten, documented and fully tested to be a commercial product. There would be ongoing support and upgrades.

All in all it's something close to a full time job. If I wanted a full time job I would return to my former career (systematic hedge fund manager) - I'd make a lot more money than I would selling software.

So it doesn't make any sense for me at least.

GAT

 

[RobSwe]

Thank you for this post tradingcomputer and all commenters.
I learned and realized a lot from reading them.

The only contribution I can add is inspiration.
The fact you put four hours after the working day into making your goals a reality
is simply put impressive. It inspires me to up my own game.

I manage realistically on average two hours after a work day and I see myself
as dedicated.

With that determination combined with your skills any obstacle will literally
be a small bump in the road. You will make it happen, 101% sure of that.

That's all of value I can add, but if only it gives you energy I think I contributed
with some small thing

/Robert

 

[tradingcomputer]

(1) Read @Ghost_of_Blotto's post again. Re-read it; then, re-read it again. If you still want to move forward continue;
(2) As a newbie, take @garachen's advice

"stick with low frequency - maybe a dozen or fewer trades a day"​

If this hold time (or longer hold time) fits your personality, move forward otherwise ... (go back to (1)?)

I re-read @Ghost_of_Blotto's post, especially on "the information is not in the data"

I have been running a fully automated system for the past 2 months. I had 'manually' traded the system for about a year before that. I used ML to "mine" my trade setups -- took me about 2 years, working full time, to formulate the research problem, implement, and validate the trade setups.

My current automated trading setup (home grown, self coded) consists of:

• a broker interface for Market data (FIX coded in Haskell);
• a broker market order interface (FIX coded in Haskell);
• A messaging bus (RabbitMQ)
• Trading Strategies (Haskell);
• Redis to hold detailed monitoring data from trading strategies.
• Web server to serve the day's trade signals generated by the Strategies, and other data held in Redis (Node.js with node-redis)

• Monitoring (Monitis -- server monitoring, RabbitMQ monitoring, network monitoring, broker interface app monitoring, and trading strategies app monitoring. Monitis has an ios app that is very handy).

Thanks for sharing the setup. I am looking into RabbitMQ, Redis and Monitis. By the way how much does Monitis cost you per month for your setup?

My FCM provides me with the ability to look at my order book and make modifications if need be over a web interface (in case I get failure alerts from monitoring application and I know there are open orders working in market(s)). So, I didn't have to recreate it.

Would you let us know which company is your FCM?

A couple of thoughts on how I approached my research, which I think might be of help to you. I looked at the problem as pattern detection in the context of a complex adaptive system. By framing the system as a CAS, I had the theoretical justification to look for patterns in the first place, and the knowledge that the goal is not to find similar instances of patterns but to extract a pattern class from similar pattern instances. It turned out, it was not easy to even 'encode' patterns. I finally settled on techniques from Bioinformatics to encode patterns, use probabilistic clustering algorithms to group 'similar patterns' (yes, not a supervised learning approach!; did consider MDPs/reinforcement learning but couldn't succeed in formulating a research statement). Going from here to finding a trading strategy was a process of iteration -- going back and forth between raw data and the similarity sequences within a cluster.

Then comes the easy part of testing the strategies in an out-of-sample data set to determine statistical validity. However, I want to emphasis that the statistical testing done is almost useless given the assumption of CAS. In a CAS, the system evolves. This means, the distribution of a class of pattern 'holding' will not be Gaussian -- it could be clustered, have power law distribution, or in the worst case, be total unstable. So, however good the out-of-sample results were, I had to take it with a grain of salt.

By the way, what is your Sharpe ratio?

I am working on two approaches:
1. Deep Learning with millions of hyperparameters to optimize
2. Feature Constructions followed by multiple Rule Learning algorithms combined within Ensemble Learner setup.

All models will need to go through regularization process to avoid overfitting.

Both approaches produces models, the test will be out of sample test, followed by forward test of paper trading, then forward test on real money 1 emini contract - only a few trades per day, then increase then number of contract and frequency of trades.

All the best with your research -- don't underestimate the effort required.

Thanks, I don't underestimate, I understand this is very hard. Even for someone with a PhD in Machine Learning and 20 years industrial experience this problem is challenging.


Regards,
Trading.Computer