My pleasure. You're welcome.
Developing "Spartan"
- Thread starter nooby_mcnoob
- Start date
Volume
According to support, it turns out the fact that I can subscribe to all these is a bug on their end. I'm literally not supposed to be able to do so no matter how much money I throw at them until you get to institutional levels (according to them anyway).
Oops.
According to support, it turns out the fact that I can subscribe to all these is a bug on their end. I'm literally not supposed to be able to do so no matter how much money I throw at them until you get to institutional levels (according to them anyway).
Oops.
Volume
I let the quote storage process run overnight, looks like it added about 1.2 million rows in ~810 hours and while quotes are being stored, I'm running another process to update another table estimating tick volume based on historical data. SQLite keeps on ticking!
The next step in the volume saga is to periodically sweep the quotes table and convert it to a volume estimate.
I let the quote storage process run overnight, looks like it added about 1.2 million rows in ~
The next step in the volume saga is to periodically sweep the quotes table and convert it to a volume estimate.
Last edited:
Buy vs build
The way I like to write code is to make it as painless as possible today, while pushing off potential pain into the future. The simple reason for this is that I cannot predict the future on most new software projects with any reasonable amount of certainty. However, I made a mistake early on in this project where a core data structure had an additional level of indirection that proved to be unnecessary (edit: and actually was always unnecessary). This became evident when I was recording quotes and I had the choice of either continuing with the pain, or removing it altogether.
I decided to remove it. While in the end, the solution to removing it was about 1500 lines of changes, the core change was ~ 10 lines. Figuring out those 10 lines took all afternoon, where in the meantime I had paused the quote collection. D'oh. It's back on now.
In the end, the code is much cleaner, and the language of solving problems is much more efficient without the additional indirection.
30 million dollars
I did the calculation with support, and in order for me to be able to have the appropriate # of reqTickByTick, I would need to have 3000 data lines which (I believe) roughly translates to 30 million dollars in equity. Sadly, I am nowhere close to that amount. I would need to marry and then divorce @sle to get that amount. Alternatively, I can generate $20K in commissions. Also not there yet.
Progress
I had expected to get backtest/forward test persistence worked out by this time but unfortunately, dealing with the volume issue took longer than I expected. I did manage to finish half of it, where the account balance is stored/updated but I still need to save trades. In the future, I would remove the account balance and create the concept of "transactions" where the first "transaction" would be a deposit. But that isn't helpful yet, so I'll just leave it as is.
Another issue I have at the moment with back testing is even if I save the trades/account balance, I lose the current backtest state. This means I have to do the entire backtest at one sitting. Not a problem yet, but likely will be at some point in the future.
Main focus now is nailing the backtest/forward test trade persistence and reporting once I implement sweeping the quotes table to estimate volume.
The way I like to write code is to make it as painless as possible today, while pushing off potential pain into the future. The simple reason for this is that I cannot predict the future on most new software projects with any reasonable amount of certainty. However, I made a mistake early on in this project where a core data structure had an additional level of indirection that proved to be unnecessary (edit: and actually was always unnecessary). This became evident when I was recording quotes and I had the choice of either continuing with the pain, or removing it altogether.
I decided to remove it. While in the end, the solution to removing it was about 1500 lines of changes, the core change was ~ 10 lines. Figuring out those 10 lines took all afternoon, where in the meantime I had paused the quote collection. D'oh. It's back on now.
In the end, the code is much cleaner, and the language of solving problems is much more efficient without the additional indirection.
30 million dollars
I did the calculation with support, and in order for me to be able to have the appropriate # of reqTickByTick, I would need to have 3000 data lines which (I believe) roughly translates to 30 million dollars in equity. Sadly, I am nowhere close to that amount. I would need to marry and then divorce @sle to get that amount. Alternatively, I can generate $20K in commissions. Also not there yet.
Progress
I had expected to get backtest/forward test persistence worked out by this time but unfortunately, dealing with the volume issue took longer than I expected. I did manage to finish half of it, where the account balance is stored/updated but I still need to save trades. In the future, I would remove the account balance and create the concept of "transactions" where the first "transaction" would be a deposit. But that isn't helpful yet, so I'll just leave it as is.
Another issue I have at the moment with back testing is even if I save the trades/account balance, I lose the current backtest state. This means I have to do the entire backtest at one sitting. Not a problem yet, but likely will be at some point in the future.
Main focus now is nailing the backtest/forward test trade persistence and reporting once I implement sweeping the quotes table to estimate volume.
Last edited:
Ah, the joy of refactoring existing software. It always appears to be a huge hassle, but once you've bitten the bullet and made the changes you're happy that you did it.
The worst part is that it was entirely unnecessary even at the start. I just thought it was a nice concept to have. You have to be ruthless to avoid garbage in your codebase.
Avoiding trading around news events
One of the things that's been tripping me up is trading around news events. While this is fine in forward testing, as I can regularly check upcoming news events it's a little more difficult in backtesting. To that effect, I've been manually checking instances in which trades don't go the way I expected and a bunch of them are around news events. For example:
forexfactory.com has a decent economic calendar going back to 2007 which is more than I would need as I am mostly back testing against random portions of the last 6 years of data.
At the moment, I don't want to automate the decision to trade or not trade based on news events but I will use this economic calendar and mark the important events on the charts so I can know to avoid them.
Trade persistence
This has been implemented.
One of the things that's been tripping me up is trading around news events. While this is fine in forward testing, as I can regularly check upcoming news events it's a little more difficult in backtesting. To that effect, I've been manually checking instances in which trades don't go the way I expected and a bunch of them are around news events. For example:
forexfactory.com has a decent economic calendar going back to 2007 which is more than I would need as I am mostly back testing against random portions of the last 6 years of data.
At the moment, I don't want to automate the decision to trade or not trade based on news events but I will use this economic calendar and mark the important events on the charts so I can know to avoid them.
Trade persistence
This has been implemented.
Last edited:
News events
Had some time to myself today so I managed to import and display events relevant to the currency pair I'm trading. There seem to be... quite a few. I expect experience will teach me how to prioritize, but for now, I'm going to avoid any GDP/inflation/Fed type of events.
It's starting to look really messy, but I expect I will clean it up at some point once I get more familiar with the workflow around the news. The triangles near the bottom of the chart show potentially high impact (red) news events all the way down to gray (no impact/holiday). At the least, I expect I will have a list of things happening in the "next 10 days" in addition to a graphical view.
This screenshot also points out one of the things I'm working with:
1. No date/time information
2. No price information
3. No currency pair information
Had some time to myself today so I managed to import and display events relevant to the currency pair I'm trading. There seem to be... quite a few. I expect experience will teach me how to prioritize, but for now, I'm going to avoid any GDP/inflation/Fed type of events.
It's starting to look really messy, but I expect I will clean it up at some point once I get more familiar with the workflow around the news. The triangles near the bottom of the chart show potentially high impact (red) news events all the way down to gray (no impact/holiday). At the least, I expect I will have a list of things happening in the "next 10 days" in addition to a graphical view.
This screenshot also points out one of the things I'm working with:
1. No date/time information
2. No price information
3. No currency pair information
What do you mean by this?
Saving trades to disk (so I can run reports on the back tests)