Quote from jonnyman:
I've been reading these forums for over a year now and I've recently begun development of an automated trading system. It's going well, but I was just curious to those who have systems running:
What was the accuracy of the backtesting results vs the realtime numbers?
Was there any reliable standard that existed? i.e. if your backtesting makes $x then real time it made 10% of $x
Or were they completely off.
Just curious to see the difference, it'd be helpful if some people experienced in this could share their thoughts.
I'm not looking for specific numbers or strategies, just generalizations (unless you want to be specific).
Thanks - I've had fun perusing these forums for a long time.
This may not be too helpful but it does give you some values.
Over time I have observed a range of backtesters who have had a common focus. Backtesters come in a lot of sizes it turns out.
There are two modes over the range. One mode does have a greater # of contributors it turns out.
One group misses by a factor of 2 to 4 and the other group is usually off by an order of magnitude all are to one side of the real results.
The opposite side to which you are biased in your thinking. One other poster here who both trades and backtests has the same bias as above which is the oppositie of yours.
You can see that there are two streams of results and one stream has two modes that are isolated from oneanother.
If an electrical engineer were to look at back testing in the same kind of context as considering the use of energy to produce fidelity in music, then he would mostly reject any results from backtesting trading models or approaches.
Kelly is an example from the hallowed halls of BTL and who did not have an interest in music. He made part of the trip across the bridge from real trading to assessing the models.
What are the causal factors of how backtesting misses o both sides of reality and how an order of magnitude difference occurs when simultaneous efforts are made by differing classes of backtesters?
It is actually analagous to the differences between markets and the commercial or custom software sold to operate in markets. There is a convergence of skills sets for programming and trading, rarely. In this thread, so far, you see but one person so far.
So why do the great traders not use computer sciences, commonly. Why is computer science not a necessary componet of trading well and with expertise.
Who among those with high money velocity wealth building need things like sharpe ratios and even Kelly stuff?
When the applications of magnetron and klystrons were made at first, they had short lives. Analysis of the variables lengthened their lives by factors between 10 and 100. The new series of related patents (contributed to BTL) were specifically related to, of all things, power supplies.
So why don't most people in the profession of improving trading results have a set of tools for doing trading analysis? Is there any reason to believe analyzing the markets is a possible pragmatic method for determining the efficacy and effectiveness of a trader (even though he is using a black box)?
Is there anything more off the mark as in looking at a given set of market data from varying sets of assortments of rules?
When I look at any profferred set of analysis tools and their results, the first thing I examine is the volume component and specifically how it is tied to the price component and then how sophisticated the multivarient analysis proceeds from there.
I find that generally the volume component of backtesting is weak. Secondly, the interrelationship of volume and price is often poorly articulated. Finally, the relationship of the near past and near future is not weighted in a manner that varies in correspondence to any frequency of occurrance in these time bound pre and post relationships.
