Why confidential? How are you gonna attract investors if you don't disclose it? Or do you only do proprietary trading? Don't get me wrong - I'm sure your setup is super duper great, but there's way too much hypothetical talks here on ET; coupling strategy/system/algo talks with some sort of output helps filtering the noise. Just my 2c.
Automated Trading for a Living
- Thread starter Allistah
- Start date
Why confidential? How are you gonna attract investors if you don't disclose it? Or do you only do proprietary trading? Don't get me wrong - I'm sure your setup is super duper great, but there's way too much hypothetical talks here on ET; coupling strategy/system/algo talks with some sort of output helps filtering the noise. Just my 2c.
I have zero interest to attract investors. Simply zero. I have interest to talk to similar minded people, but I simply have ZERO interest to get any investor into the game. None. Zilch. Nada. Nothing.
I only do internal own trading. Me and 2 people that I employ for this, and a possible third one (open position, still looking for the right candidate locally here).
I started once with NinjaTrader, too - and we spent literally months trying to get a strategy tested. From bad data to system just crashing. After one week with zero finished optimizations - I was pissed and together with another guy (also here) we have now our own platform. And this thread is only about the technical part. Where in the past - same hardware - we did not et more than 2-3 attempts per weekend (who wants to watch half a dozen machines when and where the test is finished), we are now more than half in the 4th of the 44 optimizations we are doing these days (today is holiday, plus weekend).
Plus, this thread here is not about anything that regarding strategies, but about the technical aspect. As such, any information about specific strategies would also be off topic. Have some decency and respect to the OP. Open your own thread.
This thread is about - well - the technical aspect, and seriously, pretty much every piece of retail software is a burden. I have not seen a single one that is decent.
Anyhow, here is a simple image from the grid control - this is really not a lot to see:
What you see there is the currently ongoing backtest. 3196 tasks are completed, 53 are shown as running (of 54 active cores, so one must just have finished and the node not picked up another one) and 1336 are still scheduled. 0 Failed. On the right are some buttons - most off - to pause the job, cancel it etc.
This stuff really is not exciting. It also is not really trading related - when we reworked the grid recently we made sure there is NO trading reference in there, I can use this subsystem to do anything else. It takes an exe file and a set of commands and executes it. Our trading framework has a shell interface and using that you start a backtest or optimization and it copies the file to a job specific folder and generates the task entries. This way I can also use it for example to extract data without rolling out new agents.
The other areas are not really something to show around - we work on a good easy analysis front to add to our selection of SQL Statements as well as to all the - excel stuff that goes to the database that the optimizations generate. This is ongoing work - "as good as it needs to be" with a progframmer searched to spend time making it round. Again, this is internal - and not something we want to share or looking for investors for.
It is something, though, that is a LOT more valuable to us than Ninja can ever be. The executions are correct (using full market replay - ninja can not do that in backtest). There is no continuous contract - when contracts switch, we generate 2 jobs for the week, one for the old, one for the new.
The cash trading side sadly is a lot less impressive - it is still a command line program with a shell to start the strategies. It is scheduled for rework next.... turning to a proper server and getting a small web front end so one can see what is going on, strategies that are running etc.
Anyone thinking this picture looks "good" - that is pretty much stock bootstrap. Nice toolkit for web applications. The URL is also localhost because that is the version I just now am checking into our version control and then rolling out to our server.... I added some functionality.
Net Thanks for the screenshot and lucid explanation. Your framework is very good. One question on data storage and retrieval though. Almost everyone uses market data specific databases like KDB+ or flat binary files. With otherwise such good infrastructure why did you decide to store everything in sqlserver instead of say flat files (even if KDB+ is too expensive)?
I never said I did that. We use binary tick files for the backtest sources. As those are never searched, SQL Server would be idiotic as a choice. RIght now it is a file per symbol per day.
The SQL Server is for the results, which are analyzed. Right now we have 1.53 billion trades stored there - 899 million using an old schema, to be deleted, 629 million using a new one. That are 838 million executions (we also store cancelled trades - we want to see when a strategy wanted to trade, even if a position was not entered). That is the result of (we pu the new grid software in place some weeks ago - this weekend is the first big run) 582 jobs and 1.5 million tasks (out of which at the moment 242000 are still in the queue - as I said, we now have the big launch weekend for a new version of the grid, which is a LOT faster than the last one).
It makes NO sense to store raw event data in SQL Server, so we do not
Our current archive of 29 complete months, for 22 markets CME, futures only (not options) as 343gb binary compressed data in ,364730 files. Most are small and not used (we export all months). We actively test 15 markets for every strat in 3 phases of 5 markets - that is a LOT of trials to see what works where. Trying not to throw processing power at the problem, but when you check 15 markets it is 15 times the work of one market.End of next months we get another 35 months of data (3 years - 1 month - we right now start in a december).
The grid adds about 50 million trades per day now under full load, goal is to reach more than 250 millions (i.e. 5x the processing power). Not decided yet what hardware I will get here. I possibly add 6 workstations and put 6 core + hyperthreading Intel hardware in them - that would add a nice 72 concurrent tasks
Ok, this is quite showing off - but seriously, you will have to go in a similar direction. Retail software does not work, from our experience, especialyl Ninja with the stupid idiocies of giving random results´, but nothing that does not allow you to distribute tests.
For example, you optimize a strat in code - in code, only.
How can you check things have not changed? All executions should be similar. Hard to do in Ninja or MC - easy for us, we just look up the last "valid" backtest, run another one, a sql statement compares trades and executions. Should be similar... or we investigate.
Quote from NetTecture:
I never said I did that. We use binary tick files for the backtest sources. As those are never searched, SQL Server would be idiotic as a choice. RIght now it is a file per symbol per day.
The SQL Server is for the results, which are analyzed. Right now we have 1.53 billion trades stored there - 899 million using an old schema, to be deleted, 629 million using a new one. That are 838 million executions (we also store cancelled trades - we want to see when a strategy wanted to trade, even if a position was not entered). That is the result of (we pu the new grid software in place some weeks ago - this weekend is the first big run) 582 jobs and 1.5 million tasks (out of which at the moment 242000 are still in the queue - as I said, we now have the big launch weekend for a new version of the grid, which is a LOT faster than the last one).
It makes NO sense to store raw event data in SQL Server, so we do not
End of next months we get another 35 months of data (3 years - 1 month - we right now start in a december).
The grid adds about 50 million trades per day now under full load, goal is to reach more than 250 millions (i.e. 5x the processing power). Not decided yet what hardware I will get here. I possibly add 6 workstations and put 6 core + hyperthreading Intel hardware in them - that would add a nice 72 concurrent tasks
Ok, this is quite showing off - but seriously, you will have to go in a similar direction. Retail software does not work, from our experience, especialyl Ninja with the stupid idiocies of giving random results´, but nothing that does not allow you to distribute tests.
For example, you optimize a strat in code - in code, only.
How can you check things have not changed? All executions should be similar. Hard to do in Ninja or MC - easy for us, we just look up the last "valid" backtest, run another one, a sql statement compares trades and executions. Should be similar... or we investigate.
Ever considered using Nvidia Tesla for computations?
Yes, but that is not worth the effort. We run all code in C# and I really want the backtests to run exactly the same code base. And I would not trade on a tesla.
It is crucial for strategy quality - code wise - that you backtest on an event stream. Allows a lot more analysis and makes sure the code does not need changes to handling the real data. Big problem with Ninja.
No problem at all.
See, we are:
* Storing them on a RAID system that is refreshing every month.
* On two locations
* In a checksum secured file system
If tings get bad we get a read error and regenerate them.Question is: There are code libraries available for C/C++ for Tesla; shouldn't be too tricky to convert...?
Ah, yes, but why invest in this area when I am happy with the rest? This is a focus question - maybe one day, but not now. No interest. You have to focus - and we are happy now, plus I can do things right now that I can not do on a tesla. We are not there yet - i like having the machien around in case I need them for other things.