Glad to have helped. Good Luck!
My stab at an automated trading system
- Thread starter Bigchazza
- Start date
Here is is promised (brief, unfocussed) update - with a question at the end for those who make it that far.
As hinted at, my auto system has undergone many changes, as has my trading knowledge. I look at the system now as a kind of robot assistant that supports what I do by crunching data and giving me insights. I still run a strategy on it that gives me signals when it finds them, but it also gives me a run-down of the sectors and industries each morning and lets me run an adhoc (and very crude) fundamental analysis of an individual stock as required.
Although I call it a system it is more of a platform. And rather than simply a platform for running algos it is more akin to a kind of hand-stitched TWS, made out of bits of driftwood and old bottle-tops and smelling faintly of craft glue. I even have a front end for it so I can plug in trades and show me my positions at any given moment based on a GPB/USD rate I update every half hour. Why go to all this trouble when there is perfectly good software out there already that does this 100 times better than my clunky java contraption? Because it's my way of driving my learning home. I'm a data person, so everything I learn about the markets comes with the question "how can I look at that in data terms?" Also it's rewarding. Some people are good at DIY and build things with their hands that enrich their lives. This is the geek equivalent of that.
But I freely admit to being a geek who is behind the times. I my day job world I moved into data architecture years ago, so my hacking skills are a bit lame. And I was never a front-end hacker anyway - back end all the way for me! Which leads me to a rather embarrassing admission - this super dooper system I have been raving about? The front-end is a windows batch file (*blushes*) I've been meaning to work out how to use something like Slack for my interfacial needs, but there is always something more relevant to do.
HERE IS THE QUESTION
One thing I want to nail ASAP is performance measurement. Profit factor, gain to pain ratio... all that good and slightly scary stuff. But I am finding it hard.. possibly because of the way I have done my trade data model - essentially I have one top-level trade representing the position in an stock, then under it I have trade_transactions representing buys or sells. If I buy a stock one day and then sell it another day, there will be one trade record for that position with two trade_transaction records for the buy and the sell. And if I buy 100 shares in a stock one day, sell 5, add 50, sell another 25 and then sell the rest - that will be 5 trade_transactions.
Is this a normal way to structure the data? What I am finding is that I am not sure how to measure performance that way - each time you sell a partial position, that becomes a realised profit, with the rest of the position unrealised. So it becomes a bit ambiguous if I want to show the current state of my performance in that position, which has grown and shrunk over time. I may be overthinking this. But also I may be underthinking it. Who knows!
As hinted at, my auto system has undergone many changes, as has my trading knowledge. I look at the system now as a kind of robot assistant that supports what I do by crunching data and giving me insights. I still run a strategy on it that gives me signals when it finds them, but it also gives me a run-down of the sectors and industries each morning and lets me run an adhoc (and very crude) fundamental analysis of an individual stock as required.
Although I call it a system it is more of a platform. And rather than simply a platform for running algos it is more akin to a kind of hand-stitched TWS, made out of bits of driftwood and old bottle-tops and smelling faintly of craft glue. I even have a front end for it so I can plug in trades and show me my positions at any given moment based on a GPB/USD rate I update every half hour. Why go to all this trouble when there is perfectly good software out there already that does this 100 times better than my clunky java contraption? Because it's my way of driving my learning home. I'm a data person, so everything I learn about the markets comes with the question "how can I look at that in data terms?" Also it's rewarding. Some people are good at DIY and build things with their hands that enrich their lives. This is the geek equivalent of that.
But I freely admit to being a geek who is behind the times. I my day job world I moved into data architecture years ago, so my hacking skills are a bit lame. And I was never a front-end hacker anyway - back end all the way for me! Which leads me to a rather embarrassing admission - this super dooper system I have been raving about? The front-end is a windows batch file (*blushes*) I've been meaning to work out how to use something like Slack for my interfacial needs, but there is always something more relevant to do.
HERE IS THE QUESTION
One thing I want to nail ASAP is performance measurement. Profit factor, gain to pain ratio... all that good and slightly scary stuff. But I am finding it hard.. possibly because of the way I have done my trade data model - essentially I have one top-level trade representing the position in an stock, then under it I have trade_transactions representing buys or sells. If I buy a stock one day and then sell it another day, there will be one trade record for that position with two trade_transaction records for the buy and the sell. And if I buy 100 shares in a stock one day, sell 5, add 50, sell another 25 and then sell the rest - that will be 5 trade_transactions.
Is this a normal way to structure the data? What I am finding is that I am not sure how to measure performance that way - each time you sell a partial position, that becomes a realised profit, with the rest of the position unrealised. So it becomes a bit ambiguous if I want to show the current state of my performance in that position, which has grown and shrunk over time. I may be overthinking this. But also I may be underthinking it. Who knows!
It is good to see that your software developed in such a way that it serves you best. You don't need all kind of bells and whistles which you don't use. What remains important though is that you fully understand the way your software collects and analyses data. If that isn't the case you run the risk of ending up in a situation which can be classified as "garbage in, garbage out".
I can't answer your question about how to measure performance of a position which "breaths", grows and shrinks over time. Of course you can calculate the realized profit over time, and compare that to the unrealized profit. Or compare it to other positions, or your entire portfolio. You may have to consider what performance parameter(s) is/are most relevant to you.
I can't answer your question about how to measure performance of a position which "breaths", grows and shrinks over time. Of course you can calculate the realized profit over time, and compare that to the unrealized profit. Or compare it to other positions, or your entire portfolio. You may have to consider what performance parameter(s) is/are most relevant to you.
Life becomes much simpler if you think of your position in MTM terms and don't care about open or closed equity, or even individual trades. Just a single series of positions where you have or lost money in a given stock.
GAT
MTM = Mark to Market? TBH I thought I was doing that anyway in an unsophisticated way but I am probably being incredibly naive. Have you covered this in any of your blog posts?
given your transactions you can derive your position. you could persist eod and during day compute it incrementally
Just letting folks know that I am still here and still busy with this project - in the spaces between my life obligations at least. But things have changed a lot. The system as described piecemeal in the above posts is still running but is not something I react to. In other words the data refresh schedules are executed, signals are sent for the few strategies I run but I don't do anything with them.
I went through a period of trying to learn more about discretionary stock trading. After all, this is the thing we are automating, right? If "automation" didn't exist then we would only have discretionary trading. Therefore discretionary trading is what automation is seeking to distil... for one reason or another. My reason - the motivation for doing all this - is that I am an awful discretionary trader. I may have learned a lot more about technical and fundamental analysis but I still change my mind all of the time. The signal says sell but I don't want to. Make me!
The other thing I have been doing is biting the bullet and learning python. For new readers to this thread, this whole project is an attempt to learn my way into trading and automation through fairly standard data manipulation technology (Talend and MySQL) - i.e. having to learn how to manually do the signal scanning rather than relying on libraries that have that functionality. Sounds harder, but I'm actually being lazy because I have a background in these technologies. If you go the hand-rolled route you can end up with exactly the system you want, but you are going to have to leave a shitload of stuff out. Or work work work to get it all in. So when I looked into different ways to evaluate my backtests (e.g. Sharpe and Sortono ratios) I put my Talend hat on, sat at the keyboard and thought... fuck this. Life is too short!
Cue Andreas Clenow's book TRADING EVOLVED. It is good. You get an education on the asset classes and investment strategies and all the while he is stealthily tooling you up with the knowledge and tech setup (you have to provide the computer though) to do what you want to do. I tried to learn python more formal ways but I just couldn't get traction. Clenow's book has not made me an expert or even an intermediate level python quant, but it has given me the traction to go back the more formal routes and make some progress.
I don't think my Talend work is wasted. Whatever python gives me in terms of backtest capability, the goal is still to have a completely automated system running the strategies I want, and Talend could work for that. I just ran up against its limitations as a lens to learn trading through.
I went through a period of trying to learn more about discretionary stock trading. After all, this is the thing we are automating, right? If "automation" didn't exist then we would only have discretionary trading. Therefore discretionary trading is what automation is seeking to distil... for one reason or another. My reason - the motivation for doing all this - is that I am an awful discretionary trader. I may have learned a lot more about technical and fundamental analysis but I still change my mind all of the time. The signal says sell but I don't want to. Make me!
The other thing I have been doing is biting the bullet and learning python. For new readers to this thread, this whole project is an attempt to learn my way into trading and automation through fairly standard data manipulation technology (Talend and MySQL) - i.e. having to learn how to manually do the signal scanning rather than relying on libraries that have that functionality. Sounds harder, but I'm actually being lazy because I have a background in these technologies. If you go the hand-rolled route you can end up with exactly the system you want, but you are going to have to leave a shitload of stuff out. Or work work work to get it all in. So when I looked into different ways to evaluate my backtests (e.g. Sharpe and Sortono ratios) I put my Talend hat on, sat at the keyboard and thought... fuck this. Life is too short!
Cue Andreas Clenow's book TRADING EVOLVED. It is good. You get an education on the asset classes and investment strategies and all the while he is stealthily tooling you up with the knowledge and tech setup (you have to provide the computer though) to do what you want to do. I tried to learn python more formal ways but I just couldn't get traction. Clenow's book has not made me an expert or even an intermediate level python quant, but it has given me the traction to go back the more formal routes and make some progress.
I don't think my Talend work is wasted. Whatever python gives me in terms of backtest capability, the goal is still to have a completely automated system running the strategies I want, and Talend could work for that. I just ran up against its limitations as a lens to learn trading through.
Don't suppose anyone reading this has experience of the python zipline library (I think it came from Quantopian)? Got two examples of strange behaviour I can't seem to account for...
1. I'm working on daily closing prices. I put an order in for a stock and it splits it in two, always buying two shares the next day and the rest the FOLLOWING day. It doesn't matter if I order 3 or 1000 shares - 2 the next day, the remainder later. Always on this one stock on a particular date!
2. Might be the same thing, but if I close a short position for 100 shares (order_target for 0 shares) sometimes it fails to execute the next day, so my algo detects again that the stoploss has been hit and again sends in an order_target for 0 shares. The following day the two both execute and I end up long 100 shares. WTF?
I am sure I am just a moron, but there isn't much out the on this.
1. I'm working on daily closing prices. I put an order in for a stock and it splits it in two, always buying two shares the next day and the rest the FOLLOWING day. It doesn't matter if I order 3 or 1000 shares - 2 the next day, the remainder later. Always on this one stock on a particular date!
2. Might be the same thing, but if I close a short position for 100 shares (order_target for 0 shares) sometimes it fails to execute the next day, so my algo detects again that the stoploss has been hit and again sends in an order_target for 0 shares. The following day the two both execute and I end up long 100 shares. WTF?
I am sure I am just a moron, but there isn't much out the on this.