I thought it was just me. I have trouble grasping the concept because of the language. I've followed this thread a long time but don't understand most of it.
Trading with automation (with IB)
- Thread starter fullautotrading
- Start date
I thought it was just me. I have trouble grasping the concept because of the language. I've followed this thread a long time but don't understand most of it.
Today we had some recovery, after the sharp DD of last week.
We are not safe yet with our investment on crude oil, as there is still a large exposure, however, the situation looks better now, with a nice PNL and a large potential profit (but also significant risk). So, no worries, I can still blow up
Actually, as counter-intuitive it may superficially seem, it's just in the periods of sharp DD that this approach reaches its maximum relative efficiency. The reason is quite obvious, and essentially is that the large exposure on one side allows the application to place larger trades on the opposite side (since they are "covered"), the so called "protective players", and this provides a huge and unseen scalping action. The continuous hammering of large protective players (promptly closed on reverse) has the effect to hugely grow our potential and provide further resources to hold on in case of further unfavorable moves.
Clearly, one needs the right capital (and understanding) to withstand the DD, or else the game can be over if the DD is so extensive that the application can no more continue it's scalping/hedging game.
Further, once a great DD has been experienced and there is some recovery, with consequent realized gains, it becomes less and less likely that the mkt will be able to throw us out.
During the DD, I even stopped looking at it, as I could only make the situation worse and the engine is now full equipped to deal with "extreme" situations, as we have faced them over and over and perfected all the necessary mechanisms, which are frankly quite counter-intuitive or at least, in strong conflict with "fear-driven" instincts.
I just did some player "splits" (I think we talked about those in a previous thread, when we implemented this useful device), to get rid of some excessively large protective players and make easier the recovery of those "stops".
I am currently also working on a few code refinements in the area of the game rules... and investing some time in the gym
We are not safe yet with our investment on crude oil, as there is still a large exposure, however, the situation looks better now, with a nice PNL and a large potential profit (but also significant risk). So, no worries, I can still blow up
Actually, as counter-intuitive it may superficially seem, it's just in the periods of sharp DD that this approach reaches its maximum relative efficiency. The reason is quite obvious, and essentially is that the large exposure on one side allows the application to place larger trades on the opposite side (since they are "covered"), the so called "protective players", and this provides a huge and unseen scalping action. The continuous hammering of large protective players (promptly closed on reverse) has the effect to hugely grow our potential and provide further resources to hold on in case of further unfavorable moves.
Clearly, one needs the right capital (and understanding) to withstand the DD, or else the game can be over if the DD is so extensive that the application can no more continue it's scalping/hedging game.
Further, once a great DD has been experienced and there is some recovery, with consequent realized gains, it becomes less and less likely that the mkt will be able to throw us out.
During the DD, I even stopped looking at it, as I could only make the situation worse and the engine is now full equipped to deal with "extreme" situations, as we have faced them over and over and perfected all the necessary mechanisms, which are frankly quite counter-intuitive or at least, in strong conflict with "fear-driven" instincts.
I just did some player "splits" (I think we talked about those in a previous thread, when we implemented this useful device), to get rid of some excessively large protective players and make easier the recovery of those "stops".
I am currently also working on a few code refinements in the area of the game rules... and investing some time in the gym
Last edited:
For the colleagues who have been asking for long term updates, I am posting the state of the demo account after a total of about 421 days, 10,449 fills, for a total of about 80K in commissions. Complete trade report is also attached.
(I have no idea why IB does not allow to select an earlier start date for the report (2015, Sept 8), looks like some limitation or bug. Anyway, I posted the various previous complete reports too within the thread. We started with NetLiquidation 1,525,169.97 USD )
BuyingPower 12,385,730.06 USD [Min: 0.00, Max: 13,989,619.42] (6.98 x 1,775,228.44)
FullAvailableFunds 1,775,228.44 USD [Min: -481,222.61, Max: 2,098,442.91]
FullExcessLiquidity 1,863,741.80 USD [Min: -202,245.09, Max: 2,098,442.91]
FullInitMarginReq 511,433.68 USD [Min: 0.00, Max: 1,656,309.60]
FullMaintMarginReq 422,920.32 USD [Min: 0.00, Max: 1,319,107.31]
NetLiquidation 2,286,662.12 USD [Min: 301,505.35, Max: 2,292,639.14]
As we know, this run had DD which imho are not viable for most beginner/casual traders/investors (they would quit way before seeing any positive result). Requiring a WB type capital and attitude. The aggressive use of the futures has also much worsened the situation in this particular test.
However, the particular algorithmic architecture has shown it can practically bear and recover any DD, mostly because of the exploitation of the liquidation orders (issued by the broker), which are actively incorporated (virtual orders + player splitting) within the trading games. In practice, it means the the larger is the DD, the more efficient becomes the application, and in practice if an extreme DD happens, it sets up for a tug of war and enormous increase of efficiency which normally, given enough time, turns out favorable.
Here is a look at one of the CL layer which is still in DD, with a look at all the player splitting:
As it is evident, all the broker liquidation activity has been transformed into a front of unfavorable trades (see the buys on top and the sells on the bottom) which the application slowly but inexorably will "undo" one by one (eventually turning each stop order into a favorable trade). [Clearly, this works most with these type of instruments.]
Frankly, not an approach for "small" capital or "casual" traders. More viable for "relatively" smaller capital, appears the use of ETF with a strong structural drift.
For large capital (destined only to the purpose of trading), it looks fine.
I am looking forward to starting a new test, but I would like to keep this one alive, if possible.
Please, let me know if anyone has a paper trading account available which I can use for another public test here on ET. If no one has it, I will just reset this one...
I would like to make another test run focusing mostly on ETFs, to reduce the max DD.
Last edited:
Nov 9, it's a new president for USA, and the application has coincidentally "celebrated" touching the milestone of 1M profit high watermark.
It has taken a bit more than 1 year to add up a 60% to the initial 1.5M. We have recovered most of the layers which were in DD, mostly crude oil, with exception of those (ES, NG), which were suspended during peak DD to recover some funds, and need some time to "undo" all stop orders memorized on these layers.
It has taken a bit more than 1 year to add up a 60% to the initial 1.5M. We have recovered most of the layers which were in DD, mostly crude oil, with exception of those (ES, NG), which were suspended during peak DD to recover some funds, and need some time to "undo" all stop orders memorized on these layers.
Here is another "long term" update, since we broke today a new high watermark, practically finally reaching a 100% profit. It has taken 490 days, starting with about 1.5M, and with about 90K in commissions (for 11,627 fills). While we have been trading 106 layers and 58 distinct instruments (including ETFs, Futures and FOPs), there are still a few layers which are in a big "load phase" (CL and NG), so there still potential for good gains (as well as new DD of course).
The newest layer (a SI layer) added (only 50 days old) has been scalping pretty nicely in complete automation, making 16K. Here is a summary view of the orders (buy=blue, sell=red):
In practice, we have been systematically "undoing" most of the stop orders on almost all layers, thus turning the previous DD, into profits.
As noted, this process clearly takes time and capital.
The newest layer (a SI layer) added (only 50 days old) has been scalping pretty nicely in complete automation, making 16K. Here is a summary view of the orders (buy=blue, sell=red):
In practice, we have been systematically "undoing" most of the stop orders on almost all layers, thus turning the previous DD, into profits.
As noted, this process clearly takes time and capital.
whats the most amount of 'players' active any time, and the number of 'games' a player is constrained to?
If using only one IB account and players trading the same instrument have offsetting orders, the account would treat it as no position..
If using only one IB account and players trading the same instrument have offsetting orders, the account would treat it as no position..
The number of players (on both sides buy/sell) is (initially) the result of the "game rules" or game settings, which are set up by the user. The game settings are in practice the "center of control" of the "trading game". There is a good number of those, which I have been evolving in time, to provide a fairly good level of control on the trading game being played. These look like the screenshot below:
For instance, in the case of the previously shown SI layer, the overall position has been within -4 (so 4 contracts short) and +1 (one long).
The game settings are in practice the set of rules specified by the user which says how the automatically generated buy/sell players have to come into existence, behave (for instance, enforcing reciprocal constraints) and close.
Note that, in addition, to the players generated by the trading engine we can also have further players which can be generated either by:
1) the user: for instance manual injection of players (for either "overloading", or additional "stopping", or whatever)
2) the broker: for instance liquidations, due to lack of funds, forced rollovers, or any other cause
in any case, once generated, the players will abide the user rules. (In order to for a player not to be subjected to the game rules, there is also the additional possibility for the user to transform it into a so called "manual position", which will be let alone by the algorithmic engine.)
Another useful feature is that the players can be "split" (there is a specific facility for that), so that it is possible to spread out those larger orders due to liquidations (stops) or user injection (panic stop or whatever) into several smaller players, and let the application recover them automatically.
(There would be a lot of things to say about the game rules, as I have been evolving them in years, but for now I think that this concise description can suffice. Most of the details can only be appreciated when actually playing with it, as they all were shaped after practical needs.)
That is right, you got it. The concept of (virtual) buy/sell players can be interpreted as a level of abstraction where you have simultaneously long/short position, but it is done on 1 account instead of 2 distinct accounts (clearly the result in principle is mathematically the same , but running on 2 accounts, apart that would not be allowed, would also be a technical nightmare and difficult to control and organize in a "centralized" way, as we can instead easily do here.)
(As an analogy, If you are Photoshop user, you know well that an image can be the result of the of different layers. Here we have something similar, with the obvious difference that we are "overlaying" trading games. And, in addition, a trading game is itself a "superposition" of buy/sell players. Clearly, the final user will only the the final "flatten" image resulting by the layer merge and to him it does not matter how the image was created), but nevertheless we maintain all the separate information which allow us to easily work on/edit the image.
For instance, on that SI layer, currently, we have 2 buys and 5 sells, resulting in a "position" of -3. So, the overall "real" position of SI would be -3 if we had only this layer only active for SI (but, in this case, we also have other two SI layers currently active).
Last edited:
I would test out your players/game/rules only on notable days/dates going forward.
For testing, have included in the application a simulation facility which can generate years and years of millisec tickdata, so that one can already envision/explore what the trading engine will do with real tickdata and different setups, as well as explore "extreme" situations. The scenarios created by the simulation are based on mixtures of various stochastic processes, and can be quite extreme (possibly much more that real instruments do).
In practice, one would adjust the trading games to the specific instrument, depending on its typical behavior. A most notable discrimination is to be made, imho, between instruments which exhibit "structural drifts" (such us many mkt inverse ETFs) and instruments which move with fluctuations in wide price ranges (such as various commodity FUTs).
Many other game adjustments are not even crucial, as the "long term" results are essentially due more to the "stop recovery" mechanisms than any game details one can work out. (However, they can be useful to "align" the game to the user psychology. )
It is also important to study the instrument "microstructure", as it turns out that in practice some instruments are systematically better at providing scalping based profits. (In practice, it turns out that some instruments are just impractical, for lack of "right" kind of volatility.)
What price would you put on your platform without the proprietary rules you use for the players. I would like to check out your platform using my own rules.