Automation Torture Test

N

Norm

It seems to me that it would be beneficial to have a program that checks strategies to define:

1. The data that would have the worst impact upon profitability and the impact of that data. That is, it would determine what the worst thing is that the market could do to that strategy.

2. The data that would have the best impact and the impact of that data.

3. The impact of the most likely data that the strategy will encounter.

This program could use artificial data streams or could analyze the code without using any data. Real data would not work since real data is not likely to have the worst and best scenarios.

Has anyone seen anything like this?

Norm
 
Quote from Norm:

It seems to me that it would be beneficial to have a program that checks strategies to define:

1. The data that would have the worst impact upon profitability and the impact of that data. That is, it would determine what the worst thing is that the market could do to that strategy.

2. The data that would have the best impact and the impact of that data.

3. The impact of the most likely data that the strategy will encounter.

This program could use artificial data streams or could analyze the code without using any data. Real data would not work since real data is not likely to have the worst and best scenarios.

Has anyone seen anything like this?

Norm
More...

It's called sensitivity analysis. Check google.
 
I am familiar with sensitivity analisys as it applies financial projections. I suspect the concept applies to automated strategies.

However, with financial projections, you merely change some assumptions, typically one at a time, and see how the bottom line changes. This is generally a straight forward process.

With a trading strategy, the point is to find the data streams that will be best and worst for the strategy. This is more difficult since there is not necessary a readily identifiable best data steam and a worst data stream.

For example, I am developing a mean reversion strategy. For many strategies, a downward plunge in the market would be the worst scenario. If this happened to my strategy, it would not likely be catastrophic. My program may enter and exit unprofitably only once. Or, it may not enter at all. No big deal.

Actually, last Friday this strategy was operating (live data, fake money) on a stock that took a 5% dive. The strategy, much to my pleasant surprise, only lost a fraction if what I would have assumed that it would lose. Careful analysis of the program revealed why this is the case, but not what the worst possible data stream would be.

However, if the market were to repeatedly trick my program into making entries (by repeatedly meeting my specific entry criteria) and then cause a stop loss to be executed (by repeatedly meeting my specific stop loss criteria), over and over many times in a row, that would be catastrophic. This may take a very unusual, very specific data stream.

So, the worst case is not necessarily obvious. My question is, does a program exist for determining such things?

Norm
 
Quote from Norm:

I am familiar with sensitivity analisys as it applies financial projections. I suspect the concept applies to automated strategies.

However, with financial projections, you merely change some assumptions, typically one at a time, and see how the bottom line changes. This is generally a straight forward process.

With a trading strategy, the point is to find the data streams that will be best and worst for the strategy. This is more difficult since there is not necessary a readily identifiable best data steam and a worst data stream.

For example, I am developing a mean reversion strategy. For many strategies, a downward plunge in the market would be the worst scenario. If this happened to my strategy, it would not likely be catastrophic. My program may enter and exit unprofitably only once. Or, it may not enter at all. No big deal.

Actually, last Friday this strategy was operating (live data, fake money) on a stock that took a 5% dive. The strategy, much to my pleasant surprise, only lost a fraction if what I would have assumed that it would lose. Careful analysis of the program revealed why this is the case, but not what the worst possible data stream would be.

However, if the market were to repeatedly trick my program into making entries (by repeatedly meeting my specific entry criteria) and then cause a stop loss to be executed (by repeatedly meeting my specific stop loss criteria), over and over many times in a row, that would be catastrophic. This may take a very unusual, very specific data stream.

So, the worst case is not necessarily obvious. My question is, does a program exist for determining such things?

Norm
More...
It's not the big one that's going to ruin you. Not implausible.
It is the chop. Not such an "unusual datastream" as you think it is. Very common!
 
there are a number of programs that do this but aimed more at institutions. RiskMetrics and algorithmics provide you with historical scenario analysis that will provide you with p%l returns on your portfolio if you had held your current positions over historic moves such as the number bond rallies, gulf war, ltcm etc..
Ultimately, this can all be done yourself but gathering the data for this is not simple
 
You must log in or register to reply here.
Back
Top