Standard trend following stuff with a little extra sauce
Developing "Spartan"
- Thread starter nooby_mcnoob
- Start date
Volume
Unsurprisingly, computing volume based on ticks is taking forever. Between last night and tonight, I have managed to replicate volume for 2 months of one currency. At this rate, I should have 6 months in another day or so, which means it would take about 1 month to download 6 month tick data for all currencies that I intend to trade.
So perhaps it is time to look for a dataset. Ah there is one from Quandl:
!@#$
Ah well.
Unsurprisingly, computing volume based on ticks is taking forever. Between last night and tonight, I have managed to replicate volume for 2 months of one currency. At this rate, I should have 6 months in another day or so, which means it would take about 1 month to download 6 month tick data for all currencies that I intend to trade.
So perhaps it is time to look for a dataset. Ah there is one from Quandl:
!@#$
Ah well.
Out of curiosity: how is volume calculated for spot forex?
Take for example the EURUSD pair: is it the total dollar value of all euro buying and selling during a specified period of time? Something like xxx million USD during the last minute?
Take for example the EURUSD pair: is it the total dollar value of all euro buying and selling during a specified period of time? Something like xxx million USD during the last minute?
My understaning is that they track along two dimensions: # of trades and dollar volume.
Volume
I've decided that I can't wait a month for this tick data to be pulled down so I've decided to randomly sample ticks for 1 hour in each quadrant of the day, for each currency. By repeatedly running this process, eventually the holes will be filled in but it allows me to start working on the rest of things having at least _some_ idea of what's going on.
The ugly-ass code for this:
(Edit: added this chart using current volume vs price data for AUDUSD) Looks like volume spikes with big moves which is what I'm looking for with respect to validation. This does not use the randomly sampled data, however.
I've decided that I can't wait a month for this tick data to be pulled down so I've decided to randomly sample ticks for 1 hour in each quadrant of the day, for each currency. By repeatedly running this process, eventually the holes will be filled in but it allows me to start working on the rest of things having at least _some_ idea of what's going on.
The ugly-ass code for this:
Code:
hoursToSample = []
for i in range(int(24/6)):
while True:
which = random.randrange(0,4,1)
hour = 6*i+which
if hour < 24 and hour not in hoursToSample:
break
hoursToSample.append(hour)
bar:model.Bar
self._logger.info("Sampling %s for %s",hoursToSample,instrument)
for bar in map(model.Bar,barsWithoutVolume):
if bar.baropen_datetime.time().hour not in hoursToSample: continue
bar.volume = ib.getTickVolumeForBar(instrument,bar,barSize)(Edit: added this chart using current volume vs price data for AUDUSD) Looks like volume spikes with big moves which is what I'm looking for with respect to validation. This does not use the randomly sampled data, however.
Last edited:
Writing "apps"
One of the patterns I've developed organically is for every little task I want to do ingesting data or whatever, I write a little "app". The pattern looks like this:
I then run this using the command:
I expect that once these grow a little more organically, they will be refactored into interactive tasks with parameters/etc. I've found this incredibly productive.
Volume
I wrote an "app" to download CSVs from a new datafeed I found (histdata.com) and implemented my custom volume studies on top of it. So far, the results are encouraging.
Hopefully the historical download will wrap up by the end of this week. Once that is done, I will need to catch up on ticks through IB as the above data source is only updated periodically, and not daily.
One of the patterns I've developed organically is for every little task I want to do ingesting data or whatever, I write a little "app". The pattern looks like this:
Code:
class MyLittleJob(SpartanApp):
def run(self):
# model/database initialized, any IB connections created using centralized configuration
app = MyLittleJob()
app.run()I then run this using the command:
Code:
pipenv run python -m spartan.cli.mylittlejobI expect that once these grow a little more organically, they will be refactored into interactive tasks with parameters/etc. I've found this incredibly productive.
Volume
I wrote an "app" to download CSVs from a new datafeed I found (histdata.com) and implemented my custom volume studies on top of it. So far, the results are encouraging.
Hopefully the historical download will wrap up by the end of this week. Once that is done, I will need to catch up on ticks through IB as the above data source is only updated periodically, and not daily.
Code:
$ du -hcs __histdata_cache/
9.0G __histdata_cache/
$ _Volume
I've formulated two equations to run on top of tick volume that estimate money flow. It's nothing as ridiculous as the actual money flow indicator but as far as I can tell, tick volume is actually a great source of information. Now that the volume story is defined, it's time to get cracking.
Progress
According to the list I made initially, where my optimistic estimate was 4-6 weeks, and my realistic estimate was 8-12 weeks, I'm about halfway through and it is ~3 weeks in. This week should see some progress as well.
* Data storage/download (done)
* Rough-in charting (done)
* Rough-in custom indicators (done)
* Basic backtesting console (done)
* Connect backtester with charting (done)
* Backtest report
* Do full backtest (done - with caveats)
* Do forward test (doing)
* Connect executions to IB
* Switch to paper account
* Switch to live trading
The problem I currently have is that my backtests are still in memory, I haven't persisted them to the database. It isn't going to be too big of a problem as the model there has been pretty consistent but just something I have to get to. Then I will need to write some reporting on top of it.
I will post those reports here. I think that's about 2 weeks away.
Game theory
When I posted my plan to use tick volume, people came in and said "BRUH DAT AIN'T REAL VOLUME."
True. But let's assume that we have a bunch of parties around the world whose sole job it is to jerk around the bid/ask spread. If that were true, we'd have very low forex market volume. But the global forex market volume is estimated to be 1-2 trillion/month so that jerking around isn't very effective.
I have this thought out much better in my head but it's getting late now so maybe I'll talk about the game theory part of this another day.
I've formulated two equations to run on top of tick volume that estimate money flow. It's nothing as ridiculous as the actual money flow indicator but as far as I can tell, tick volume is actually a great source of information. Now that the volume story is defined, it's time to get cracking.
Progress
According to the list I made initially, where my optimistic estimate was 4-6 weeks, and my realistic estimate was 8-12 weeks, I'm about halfway through and it is ~3 weeks in. This week should see some progress as well.
* Data storage/download (done)
* Rough-in charting (done)
* Rough-in custom indicators (done)
* Basic backtesting console (done)
* Connect backtester with charting (done)
* Backtest report
* Do full backtest (done - with caveats)
* Do forward test (doing)
* Connect executions to IB
* Switch to paper account
* Switch to live trading
The problem I currently have is that my backtests are still in memory, I haven't persisted them to the database. It isn't going to be too big of a problem as the model there has been pretty consistent but just something I have to get to. Then I will need to write some reporting on top of it.
I will post those reports here. I think that's about 2 weeks away.
Game theory
When I posted my plan to use tick volume, people came in and said "BRUH DAT AIN'T REAL VOLUME."
True. But let's assume that we have a bunch of parties around the world whose sole job it is to jerk around the bid/ask spread. If that were true, we'd have very low forex market volume. But the global forex market volume is estimated to be 1-2 trillion/month so that jerking around isn't very effective.
I have this thought out much better in my head but it's getting late now so maybe I'll talk about the game theory part of this another day.
Last edited:
Be careful in combining data from two different sources. They might not be aligned fully, resulting in strange results. Prices can be different, and so can trade volume.
You're right, this is going to be a tough problem to solve. I think my formulas handle this correctly I'll know more once the data is in. And also, once caught up, I'll be using IB tick volume. This is mainly to let me back test.