Back-testing software, my progression to freedom

lolatency · Apr 12, 2009

I'm trying to break-free from the man and have been working with strategies in R. We have similar testing software at work, but I wanted a clean break from stuff at work. I'm scared to test my ideas at work in the event that they want to own my ideas.

I've got two options:

1) Use a retail software package to backtest, and/or
2) Write my own framework

I have no interest in rewriting my own framework. TradeStation's platform is too expensive as a standalone and their execution is too slow.

What I did this weekend was try to automate my strategy in E-Signal. I used their EFS package, but the interface for backtesting seems really broken and doesn't generate those equity graphs over time. When I used this software about 5 years ago, it seemed to work then. Now, it doesn't seem like it works.

What are you guys using? I don't care about speed -- I just want back-testing results quick. When I go to market, I intend to write my own execution framework and trade managers.

Fractals 'R Us · Apr 12, 2009

When you are sort of in the development stage, when you want to be looking at charts, testing, getting new ideas and incorporating them, maybe Ninjatrader is the best [that I've tried]. When you are pretty firmed up regarding your ideas but want to test in some big ways, lots of variables with lots of settings, Openquant is going to make you real happy, they have simulated annealing, it can test a very big parameter space in [relatively] little time. Both program in C# which is great, you can pull in all the Microsoft libraries and incorporate their features in your script. Ninjatrader's support forum is especially good, pros from Ninjatrader attend to all questions, it's not a kindergarten iow... Openquant's forum is fair, they definitely don't cater to retail traders that need programming handholding, and their documentation is... well... it leaves a bit to the imagination sometimes, it's useful, it's just not at the tutorial level with examples of everything as Ninjatrader's is... I use both, I can do a little live trading and informal testing with Ninjatrader [love that Chart Trader feature] and do some heavy duty testing with Openquant...

xiaohu · Apr 12, 2009

Quote from Fractals 'R Us:

When you are sort of in the development stage, when you want to be looking at charts, testing, getting new ideas and incorporating them, maybe Ninjatrader is the best [that I've tried]. When you are pretty firmed up regarding your ideas but want to test in some big ways, lots of variables with lots of settings, Openquant is going to make you real happy, they have simulated annealing, it can test a very big parameter space in [relatively] little time. Both program in C# which is great, you can pull in all the Microsoft libraries and incorporate their features in your script. Ninjatrader's support forum is especially good, pros from Ninjatrader attend to all questions, it's not a kindergarten iow... Openquant's forum is fair, they definitely don't cater to retail traders that need programming handholding, and their documentation is... well... it leaves a bit to the imagination sometimes, it's useful, it's just not at the tutorial level with examples of everything as Ninjatrader's is... I use both, I can do a little live trading and informal testing with Ninjatrader [love that Chart Trader feature] and do some heavy duty testing with Openquant...

hi,
Do you automate your trading ?
If so, which do you use Ninjatrader or OpenQuant for live trading ?

thanks for you input.

Trader922 · Apr 12, 2009

Quote from lolatency:

I'm trying to break-free from the man and have been working with strategies in R. We have similar testing software at work, but I wanted a clean break from stuff at work. I'm scared to test my ideas at work in the event that they want to own my ideas.

I've got two options:

1) Use a retail software package to backtest, and/or
2) Write my own framework

I have no interest in rewriting my own framework. TradeStation's platform is too expensive as a standalone and their execution is too slow.

What I did this weekend was try to automate my strategy in E-Signal. I used their EFS package, but the interface for backtesting seems really broken and doesn't generate those equity graphs over time. When I used this software about 5 years ago, it seemed to work then. Now, it doesn't seem like it works.

What are you guys using? I don't care about speed -- I just want back-testing results quick. When I go to market, I intend to write my own execution framework and trade managers.

I use OpenQuant for some things, and Matlab for others. I find Matlab to be much better for testing an idea on many instruments (i.e. all SP500 stocks). I am going to start playing around with some other open source packages to see how well they do things.

I think the strong point about OpenQuant is how quickly one can take a strategy from backtesting to production, but if that isn't your goal, then it may not be a good product for you. I think some of the hangups with the product such as out of memory issues when running a strategy on many instruments will become major headaches for you.

Why not get a strategy going that can be run by running something once or twice a day and generating a flat file with MOC orders that can be uploaded to the broker???? There are a lot of profitable strategies that fit into that arena.

Also, you may want to look into SciLab, that may do what you need.

Fractals 'R Us · Apr 12, 2009

I do test and trade just one insturment at a time, true, I did not take that into consideration.. I automate with Openquant but lately I've been live trading, that's the only time I get new ideas.

Trader922 · Apr 12, 2009

Quote from Fractals 'R Us:

I do test and trade just one insturment at a time, true, I did not take that into consideration.. I automate with Openquant but lately I've been live trading, that's the only time I get new ideas.

Many of the ways that OpenQuant handles errors are pretty frustrating, i.e. its not that difficult to corrupt the instrument database when you attempt to download data for many symbols, its not very difficult to get some nasty out of memory errors while backtesting, etc.

I don't know if you would run into these same issues with a large amount of data for a single symbol.

More and more I like the idea of adapting a general purpose science or statistics software package to fit my needs as a trader, rather than trying to find a retail-level trading software package to do it all.

Try backtesting an idea like 'Buy a stock MOC that is down 5% from the prior day's close & close the trade on the following days close using a MOC Order' in SmartQuant and Matlab. Run this test for all SP 500 stocks (or more) for several years and you'll see the difference I am referring to.

Fractals 'R Us · Apr 12, 2009

Maybe the available software has shaped my strategies... they are mind numbingly simple and run on one instrument. I don't have any problems at all with software that way really.

I ran strategies a few years ago with Tradestation 2000 and Easy Language and interfaced to IB with a third party dll... the programming for more than one instrument is way more complicated.. you have to involve a lot of calls to IB's API regarding positions and account buying power and such, what goes on with one instrument affects the account and affects the other trades, and IB has a couple of quirks in that area that threw me some headaches... the thing is that TS2000's data server could handle whatever you threw at it, some had DTN Satellite and collected ticks on 30,000 instruments !!

Openquant is a gateway product, it serves to show what they can do and it seems like they are not putting a lot of resources into the product, they want to steer business to Smartquant products with it, they have said as much. I'm self taught in C# so as a really weak programmer I have fewer choices of software to work with, Openquant is fine for what I'm doing... besides I can run it behind a very tight firewall, Ninjatrader has to phone home before it works and that is against my security philosophy.

Tickzoom is said to be extremely fast, probably were I going to complicate things I would build on that..

lolatency · Apr 13, 2009

Quote from Fractals 'R Us:

I do test and trade just one insturment at a time, true, I did not take that into consideration.. I automate with Openquant but lately I've been live trading, that's the only time I get new ideas.

Well, how abou a ninjatrader hint then. How do I reference the farthest bar in time back possible in the code, and how do I make sure that it was the first 1minute from 6 months ago?

heech · Apr 13, 2009

Quote from lolatency:

Well, how abou a ninjatrader hint then. How do I reference the farthest bar in time back possible in the code, and how do I make sure that it was the first 1minute from 6 months ago?

Farthest bar back in time (loaded in the current backtest/chart):

Close[CurrentBar]

How do you make sure it was the first 1 minute from 6 months ago? I don't know I understand the question, but you can always use Time[CurrentBar] to get the DateTime timestamp attached to that specific bar and do whatever you need to do with it.

That said, NinjaTrader can be very frustrating in the way it deals with missing historical data on backtesting. There's some strange meta-state it can get in, where there's days/hours of missing data in the middle of a block... but NT doesn't know to request it from your data feeds, even if connected.

lolatency · Apr 13, 2009

Quote from heech:

Farthest bar back in time (loaded in the current backtest/chart):

Close[CurrentBar]

How do you make sure it was the first 1 minute from 6 months ago? I don't know I understand the question, but you can always use Time[CurrentBar] to get the DateTime timestamp attached to that specific bar and do whatever you need to do with it.

That said, NinjaTrader can be very frustrating in the way it deals with missing historical data on backtesting. There's some strange meta-state it can get in, where there's days/hours of missing data in the middle of a block... but NT doesn't know to request it from your data feeds, even if connected.

This is what frustrates me so much about the backtest software out there.

If I design something using python, for example, I am wondering what features I should lift from these backtesters.