Do you remember or have the stats for the other adaptive execution options compared to the Normal setting? Was it a significant difference between them?
Fully automated futures trading
- Thread starter globalarbtrader
- Start date
The largest test I did (over 3,000 trades) yielded, in terms of spread paid:
Patient 56.1%
Normal 60.2%
Urgent 88.3%
Another factor is the time to execution. Depends how your system works, but I don't want my system waiting around too long to get filled.
Median/average fill times:
Patient 73.4 / 553 s
Normal 9.8 / 38.8 s
Urgent 0 / 0.27 s
The problem I had with patient is that some trades literally took hours to execute (usually STIRs).
Normal works best for me.
Patient 56.1%
Normal 60.2%
Urgent 88.3%
Another factor is the time to execution. Depends how your system works, but I don't want my system waiting around too long to get filled.
Median/average fill times:
Patient 73.4 / 553 s
Normal 9.8 / 38.8 s
Urgent 0 / 0.27 s
The problem I had with patient is that some trades literally took hours to execute (usually STIRs).
Normal works best for me.
1. isn't it only for stocks? anyway, I never bothered with that (maybe unwisely?)
2. I think pysystemtrade's roll calendar is a good start, but then in my system for "hard" and "soft" roll periods I refined overlapping periods with maximum volumes for previous and next contracts for each instrument, which I then "hardcoded" for the next 100 years (a lot of painful manual work multiplied by 100x, or more if you include all tradable universe of IB futures..), That's mostly because I have automatic rolls and don't want my system placing orders in an empty market while I might be sleeping. And I don't do any spread orders, too much hassle.. But not everyone is doing rolls automatically..
3. For instruments without market data subscription I use one of the IB's fancy order types, Snap-to-mid or something like that (simple limit orders for instruments with data subscription)..
4.Doing it in one account might help using capital more efficiently? Might be a bit messy, though? not sure if it matters much.. How is your stock mean reversion experience btw, did you trade pairs or single stocks? I traded ETF\stock pairs a long time ago - basically a complete failure
the system made 3-4% at first and then kept loosing slowly till I shut it down The capital usage was way too high compared to futures, and the companies kept merging, paying dividends and doing stock splits, which I was often missing from IB data - a total headache..5. no idea, never used it. I doubt that the effort is worth the benefits..
Btw, this reminded me to check my costs (which I haven't done for many months), so I might as well post the results here:
So looks like over 4 years I saved about 3,000$ in execution costs comparing to paying half of the spread every trade (half of the current spread + commissions vs FillPrice-MidPrice+commissions). 2,750 trades and 5,824$ in commissions paid in total over this time..
Last edited:
I am running my automated futures trading system live for over five years. I have had a few occasions where it encountered a circuit breaker. That was almost entirely limited to agriculture futures such as LE and HE, or grains. My system trades near the end of the trading day. If a circuit breaker gets activated during the day it mostly remains active for the remainder of that day. My software does not recognize that, so when it submits an order it gets not filled. I decided to ignore this, let the system try the following day. I'm too lazy to try and develop fancy code to recognize this situation and respond differently.
Thank you so much for all your replies! I really appreciate the know how as this safes me from running some potentially costly experiments.
To reiterate the questions:
1. I will go down the route of HobbyTrading, not caring at all. In my stocks backtests I detect trading halts by identifying unusual volume. This is necessary in stocks land because I have much more data there and actually do a light form of parameter search. Such a method needs are very precise backtest. Not needed for futures trading.
2. As Kernfusion mentioned my stitching method needs a little bit more work. As this does not affect the models strongly, I will postpone the implementation - something I can work on when paper trading is up and running. This test will take months anyway, because I want to have autorolled a few contracts.
3. Newbunch's analysis is pure gold! I will definitely stick to normal going forward. The adavantage of "patient" is not big enough even I practically do not trade STIRs. They are excluded nearly all of the time because of the minimum volatility requirement. I only use 3% minimum volatility for the "mixed" volatility and only CADStir sometimes is above that.
4. I will stick to one account for margin reasons, even if the PnL attribution could be messy. Regarding my experience with mean reversion stocks: I started automated pairs trading in 2009 and hanged on to that till about 2014 - but was roughly flat pnl wise. I used intraday data to calibrate the models back then. As a freshly minted PhD in AI back then, I was way too confident in complicated models. I was just too young and dumb back then. A few years ago I started mean reversion on single stocks without any complicated stuff in the background and it works much better but is less reliable since interest rates started to go up.
5. As I have guessed, spread trades are something to go for when manually rolling. I am not quite sure yet if Adaptive algo is the way to go for rolling because you want to make sure that both legs execute at the same time. If so, market orders it is. I hope IB's paper trading allows for adaptive algo. I did have some order types in the past that I could not test in paper trading. A real bummer!
To reiterate the questions:
1. I will go down the route of HobbyTrading, not caring at all. In my stocks backtests I detect trading halts by identifying unusual volume. This is necessary in stocks land because I have much more data there and actually do a light form of parameter search. Such a method needs are very precise backtest. Not needed for futures trading.
2. As Kernfusion mentioned my stitching method needs a little bit more work. As this does not affect the models strongly, I will postpone the implementation - something I can work on when paper trading is up and running. This test will take months anyway, because I want to have autorolled a few contracts.
3. Newbunch's analysis is pure gold! I will definitely stick to normal going forward. The adavantage of "patient" is not big enough even I practically do not trade STIRs. They are excluded nearly all of the time because of the minimum volatility requirement. I only use 3% minimum volatility for the "mixed" volatility and only CADStir sometimes is above that.
4. I will stick to one account for margin reasons, even if the PnL attribution could be messy. Regarding my experience with mean reversion stocks: I started automated pairs trading in 2009 and hanged on to that till about 2014 - but was roughly flat pnl wise. I used intraday data to calibrate the models back then. As a freshly minted PhD in AI back then, I was way too confident in complicated models. I was just too young and dumb back then. A few years ago I started mean reversion on single stocks without any complicated stuff in the background and it works much better but is less reliable since interest rates started to go up.
5. As I have guessed, spread trades are something to go for when manually rolling. I am not quite sure yet if Adaptive algo is the way to go for rolling because you want to make sure that both legs execute at the same time. If so, market orders it is. I hope IB's paper trading allows for adaptive algo. I did have some order types in the past that I could not test in paper trading. A real bummer!
I forgot to reply to your question #4. I use two separate sub-accounts, for the specific reason you mention. Having a separate sub-account for futures trading makes it easier to calculate PnL, margin usage, volatility and so on. I am aware that it causes inefficiency in my capital usage as I must keep capital in this futures sub-account which I can't invest in e.g. stocks or ETFs. I decided (long ago) that that would be fine with me. At least IB is paying me some credit interest on it...
The benefit of this separate sub-account is that it "isolates" multi currency issues. My other sub-accounts are USD only. The futures sub-account is the only one which has USD, EUR, JPY, KRW and so on in it.
I think with handcrafting the top level weight allocation shouldn't really be 1\n, exactly because some groups have too few instruments in them (or we should try to come up with some even higher-level groups e.g. 'financials' for all stocks, bonds and volatility).. And on the other hand, the groups with many diversifying instruments should probably receive higher allocation.. E.g. My agriculture top-level group has 19 instruments (and it contains weird\diversifying things like Coffee, Milk and Orange Juice) and my InterestRate group only 6, so it sort of makes sense to give more weight to agriculture.. This logic makes things kind of arbitrary I think, because we could end up forcing the group weight that we (almost subjectively) like.. Here's my current top-level weights, and I can't really explain how I came up with them, they started as the result of pysystemtrade optimization, but I then adjusted them manually..
For pysystemtrade users, I've created a script that scans the config.yaml file for instrument weights and moreinstrumentinfo.csv to gather asset class, subclass, subsubclass, and country information, allowing for a quick overview of weights by category. I've also added an option to rebalance weights from one main category to another. For example, you can move 5% from equities to volatility while maintaining the same proportion for each instrument within the respective category.
I don’t know if it will be useful for anyone out there, but here is the script. I haven't add the option to update the new weight after a rebalance but that could be done manually.
Code:
import yaml
import csv
def parse_input(input_data):
""" Parse input string into structured dictionary. """
portfolio = {}
for line in input_data:
parts = line.split(';')
instrument, weight, categories = parts[0], float(parts[1]), parts[2:]
# Navigate and create dictionary structure
current_level = portfolio
for category in categories:
if category not in current_level:
current_level[category] = {}
current_level = current_level[category]
current_level[instrument] = weight
return portfolio
def sum_weights(data):
""" Recursively sum weights to calculate total weights for categories. """
if isinstance(data, dict):
return sum(sum_weights(value) for value in data.values())
return data
def display_weights(data, level=0):
global current_category
if isinstance(data, dict):
total_weight = sum_weights(data)
formatted_weight = f"{total_weight * 100:.2f}%" # Convert to percentage and format
print(' ' * (level * 4) + f'{current_category}: {formatted_weight}')
for key, value in data.items():
current_category = key # Update the current category
display_weights(value, level + 1)
else:
formatted_weight = f"{data * 100:.2f}%" # Convert to percentage and format
print(' ' * (level * 4) + f'{current_category} (Instrument): {formatted_weight}')
def adjust_weights(data, category_total_weight, percent_change, add=True):
""" Adjust weights for a specific category by a percentage based on proportional contributions. """
adjustment_ratio = percent_change / 100 # Convert percentage to decimal
def adjust_recursive(sub_data, category_total_weight):
""" Recursively adjust weights in a nested dictionary. """
if isinstance(sub_data, dict):
for key, value in sub_data.items():
if isinstance(value, dict):
adjust_recursive(value, category_total_weight)
else:
# Calculate the proportion of the total category weight that this instrument represents
proportion_of_category = value / category_total_weight
# Calculate the absolute amount to adjust this instrument by
adjustment_amount = proportion_of_category * adjustment_ratio
if add:
sub_data[key] = value + adjustment_amount
else:
sub_data[key] = value - adjustment_amount
# Initiate recursive adjustment
adjust_recursive(data, category_total_weight)
def rebalance_portfolio(portfolio, source_category, target_category, percent_change):
if source_category in portfolio and target_category in portfolio:
source_category_total_weight = sum_weights(portfolio[source_category])
target_category_total_weight = sum_weights(portfolio[target_category])
# Deduct weight proportionally from the source category
adjust_weights(portfolio[source_category], source_category_total_weight, percent_change, add=False)
# Add weight proportionally to the target category
adjust_weights(portfolio[target_category], target_category_total_weight, percent_change, add=True)
print(f"\nUpdated portfolio weights after transferring {percent_change}% from {source_category} to {target_category}:")
display_weights(portfolio)
else:
print("One or both categories not found in the portfolio.")
def main_menu(portfolio):
while True:
print("\nMenu:")
print("1 - Display All Weights")
print("2 - Rebalance Portfolio Weights")
print("3 - Exit")
choice = input("Enter your choice: ")
if choice == '1':
current_category = "Portfolio"
display_weights(portfolio)
elif choice == '2':
source_category = input("Enter the source category to deduct weight from: ")
target_category = input("Enter the target category to add weight to: ")
percent_change = float(input("Enter the percentage to move (e.g., 5 for 5%): "))
rebalance_portfolio(portfolio, source_category, target_category, percent_change)
elif choice == '3':
print("Exiting program.")
break
else:
print("Invalid choice, please try again.")
def load_yaml_data(filepath):
""" Load data from a YAML file. """
with open(filepath, 'r') as file:
return yaml.safe_load(file)
def load_csv_data(filepath):
""" Load data from a CSV file. """
data = {}
with open(filepath, newline='') as csvfile:
reader = csv.DictReader(csvfile)
for row in reader:
data[row['Instrument']] = {
'AssetClass': row['AssetClass'],
'SubClass': row['SubClass'],
'SubSubClass': row['SubSubClass'],
'Country': row['Country']
}
return data
def build_input_data(weights_config, instruments_info):
""" Build input data combining weights and asset class info. """
input_data = []
for instrument, weight in weights_config.get('instrument_weights', {}).items():
if instrument in instruments_info:
info = instruments_info[instrument]
path = [instrument, str(weight), info['AssetClass'], info['SubClass']]
if info['SubSubClass']:
path.append(info['SubSubClass'])
if info['Country']:
path.append(info['Country'])
input_data.append(";".join(path))
return input_data
# Load the data from files
config_data = load_yaml_data('PATH TO YOUR config.yaml WITH INSTRUMENT WEIGHT')
instrument_info = load_csv_data('PATH TO moreinstrumentinfo.csv NORMALLY UNDER data.futures.csvconfig')
# Build the input data
input_data = build_input_data(config_data, instrument_info)
print(input_data)
# Parsing the input
portfolio = parse_input(input_data)
# Display weights
current_category = "Portfolio"
# Start menu
main_menu(portfolio)I'm not bothering with simultaneous leg-execution, the only thing I enforce when rolling is that I first close the old leg before placing order on the next one to not allow my exposure to increase (if it temporary decreases no big deal).
Another potential argument for separate accounts might be if you want to treat them differently for tax purposes somehow, but this is doubtful.. And another downside is that you might not be able to use the same IB data subscriptions and would have to pay twice, but not sure about that..
I do it slightly different. From a few weeks before the expiry of a contract I allow the system to gradually convert from old to new. When the position size is to be expanded it uses the new contract, while if the position size needs to be reduced it closes a portion of the old contract. Until a "deadline date" is reached. If I still have a position in the old contract remaining on that date will it be forcefully closed. Once it is closed will it be replaced by a same position size in the new contract.
This is not correct. You subscribe to data on the IB username&password level. This is independent of how many account numbers (e.g. Uxxx, Dxxx) you have.
This is similar to what I do; pst has varous roll modes to support this:
- passive as described above
- actively closing the position in the current contract (DO will then decide if it wants to reopen in the new contract; if it doesn't this is a nice way of saving money)
- actively rolling to new contract eithier as two outright trades or a spread trade.
It's not automated entirely, but I run a script that changes to the best mode depending on various rules about relative volume and time to expiry; and then rolls once there is no position in the current contract.
There is a discussion in this in one of the final chapters of AFTS which everyone on this thread should already own 3 copies of, one to read, one to put on the shelf and admire, and the other as a spare in case the one you are reading gets nicked.
Rob