The 'Ol Backtesting Dilemma

Quote from tickzoom:

This is so *almost* true. I mean that for the longest time I quit back testing and just did market replay, would see a new condition and add a rule.

But the biggest problem with back testing was that the data doesn't close enough simulate a real live trading situation.

For example, I work in USD/JPY and the most realistic data I can get for 5 years has only non-duplicate ticks, and only the bid prices.

Only bid prices. I started collected my own data with bid/ask and 5 levels of the DOM using tickZOOM.

When replaying that data and graphing the bid/ask spread, you see that it has MEANING.

Also, the total depth of market. It's shocking what those things can tell you.

I have come to the conclusion that the reason that people say you need "screen time" (which I have quite a bit) is watching the DOM, Time and Sales, and charts with volume, of course.

With all that information on the screen you can see when the market goes "crazy" or slows down. It kind of breathes.

Well, I committed to building TRUE market simulation much like a flight simulator with full market data, bid/ask, last trade, Time & Sales, and DOM all in tick data.

So you can replay or test historically with a true to live market simulation.

Hey it totally changes the meaning of testing strategies.

I have rules about the the DOM and bid/ask spread change. And that's significant for controlling risk.

Wait, I shouldn't be giving these secrets out in public.

So my theory is that it isn't historical testing per se that is a problem, it's the quality of historical testing that matters.

I'll attempt to prove it with this point.

Successful traders all say you need "screen time" versus historical testing to succeed.

But what if you could go back in time and sit in front of the same screens and charts as that trader did. Then you would learn the same as she did, right?

Well, you can do that with historical data as long as you have COMPLETE historical data.

You can then use historical data to train yourself or your strategies just like the aviation industry uses flight simulators for ground school and the air force uses them for even advanced training.

Sincerely,
Wayne
More...

good first post....
 
Quote from pdmoney:
I got started using Amritrade's Strategy Desk. I have read that their backtesting leaves much to be desired and I can see why. After creating a few strategies and testing them they work almost nothing like they do in real life as they do in the backtests. My question is, is there ANY software that somewhat comes close to getting the same results LIVE as they do in backtests? Not looking for the Magic Bullet, just looking for what works best for a novice just getting started in Automated/Strategy trading. Any help would be greatly appreciated.
More...

what are you backtesting?

stocks? futures? forex?

what time resolution do you use?

daily bars? minute bars?

how's the order executed?

end of bar only?
intrabar order generation?



these issues make a difference!
 
Quote from tickzoom:

Thanks, technically it's not my first or second though. I registered another account since I'm the author of TickZOOM. That's the quote collector, execution server, and historical testing platform I built around the TickZOOM engine which can handle 10 million ticks in 40 seconds. Anyway, it will be free but not released for a couple weeks.

I decided to get the user name tickzoom which is at tickzoom.org

Wayne
More...

ahh cool ill take a look at it im interested.
 
what are you backtesting?

- Stocks Only

what time resolution do you use?

-I have tried 3 minute, 5 minute and 60 minute time frame charts. Did not have much luck with reults of other time frames.


how's the order executed? Not sure what you mean...I use Ameritrade's Strategy Desk.. I also just started trying out NinjaTrader with limited success.

end of bar only? I tried calculations on Open, AND close bars...have not tried any intrabar tests..

intrabar order generation?

these issues make a difference!

I am very curious to see what TickZoom will be all about when its released. I havent seen anyone's thoughts on NinjaTrader Vs. Tradestation..any comments there?
 
Quote from pdmoney:

what are you backtesting?

- Stocks Only

what time resolution do you use?

-I have tried 3 minute, 5 minute and 60 minute time frame charts. Did not have much luck with reults of other time frames.


how's the order executed? Not sure what you mean...I use Ameritrade's Strategy Desk.. I also just started trying out NinjaTrader with limited success.

end of bar only? I tried calculations on Open, AND close bars...have not tried any intrabar tests..

intrabar order generation?

these issues make a difference!

I am very curious to see what TickZoom will be all about when its released. I havent seen anyone's thoughts on NinjaTrader Vs. Tradestation..any comments there?
More...

PDmoney look up my post on the refined macd-r2 paper research i think that it may help you if your trying to go the ol indicator backtesting route....
 
Thanks Nuke..interesting thread. However, I only have access to Ninjatrader at the moment. Also, does the idea apply to Stocks too? I only trade stocks.
 
I am very curious to see what TickZoom will be all about when its released. I havent seen anyone's thoughts on NinjaTrader Vs. Tradestation..any comments there?
More...

It's all relative. It seem every tools shines at a certain functionality which it focuses on.

TradeStation seems to be the major player at trading equities where you have perhaps thousands of instruments in radar station.

They originally invented EasyLanguage which really does make certain things much easier to write in trading rules but it limits in other ways.

NinjaTrader shines at discretionary trading where you can run historical test ideas and semi-automate your work. It uses straight C# for trading rules which is a little painful when you're used to EasyLanguage.

None of those are worth much when you want to test based on raw tick data. They need bar data or memory and performance goes to pot.

It really depends on what you want to trade and how you want to do it.

TickZOOM really shines for testing ideas on a TRUE market simulation with full, detailed tick data and then seamlessly automating your trading rules to run hands free.

It also uses C# but incorporates all the stuff (and more) that makes EasyLanguage easy for writing trading rules.

But TickZOOM only supports an extremely simplistic discretionary trading GUI.

In theory, all these tools could be improved or modified to handle everybody's needs. But reality shows it's very difficult to excel at everything.

Sincerely,
Wayne
 
Quote from pdmoney:

Thanks Nuke..interesting thread. However, I only have access to Ninjatrader at the moment. Also, does the idea apply to Stocks too? I only trade stocks.
More...

the paper is in the thread somewhere i believe but yeah it was originally meant for stocks i believe i modified the hell out of it to make it work in FX

i have the MT4 code if you would like it as i dont know anything about ninjatrader

note its accurate as hell but accuracy isnt everything as i trade FX i can say its profitable in FX with my modified version but i cant say ive tested this enough in stocks to say that is profitable but backtesting wise its in the 90% success rate and has huge profits with minor losses.

origin paper i attached... read through it its very interesting its a good up yours to random walk
 

Attachments

If you are trading with indicators and oscillators at Ameritrade go out to the TASC magazine and look for the past year or so. TD Ameritrade has posted quite a few indicators in the past year written by some well know traders. If you back test it paper trade it and watch for changing market conditions.

You will find that many traders with programming back grounds, like my self, want software that is much more sophisticated than TD Ameritrade strategy desk (see attached). This allows for more extensive price bar testing in concert with indicators and oscillators. I discretionary trade and automated trade stocks. I prefer discretionary trading with multiple indicators with confirmation. Automated trade with indicators is complex topic especially back testing. For this I use Tradestation.
 

Attachments

You must log in or register to reply here.
Back
Top