Methods to test robustness of your algo trading strategy.

D

Dicer

Methods are shown below:

1. Resampling

2. Monte Carlo

3. Regularization--Number of Variables (no. optimization steps<= no. datapoints)
eg. LASSO

4. Regularization--Structure (early stopping, drop-out)

5. FWER/Deflated Sharpe Ratio (I don't use them because no. of trials is not the only factor affecting overfitting probability)

6. Explainable (Examine whether the result is reasonable.)


What is your method to test the robustness of your strategy?
 

Attachments

  • 93065734_115177743480382_6425433609263382528_o.jpg
    93065734_115177743480382_6425433609263382528_o.jpg
    47 KB · Views: 98
Hi Dicer, I am sure you will get a lot of comments from traders including dismissing my comments, but I would include things that are trading independent and then customize them for trading. Also those things you are listing are more like metrics or metric gathering methods. Sort of like telemetry. They are not really robustness tests per se, imo.

Robustness has to do with a systems ability to handle situations outside the common designed scenario. In any case here is what I did for my system.

1) Boundary conditions. e.g. Running the system outside its bounds WRT to variable assumption and variable qualitative types. Vague I know, but without details...
2) Limits - Load testing. e.g. How far can the system run under load? e.g. what happens when your size gets beyond the liquidity of the market? What happens when the system gets into a disorderly market? Each exchange has measures in place to handle the crazy markets. What is your system going to do then with e.g. a lock limit, partial halting, or velocity lock?
3) MTTF: Mean time to failure: How long can it run before it starts to fail.
4) Error conditions: e.g. what happens when you get a disconnect mid trade?
5) and the Biggie: All of the above is various combinations at the same time or various sequences.

These are all just examples of categories with small examples within each. Hope that helps because maybe you really wanted optimization and meta optimization comments.
 
the question for me is always how does it enter trades

is it running with the price action trying to get a ride or is it taking some heat trying to enter on a pullback etc.

if you can't trade size and it perform the same as a one lot, it will blow up.
 
I do not have a method as I do not have an automated strategy (yet?), but I can comment on your post, at least the thread stays on top of the active list :)

You (and the attached figure) seem to take some methods from machine learning. My question is: To what extent is trading analogous to more traditional machine learning problems, as the market is an adaptive system?

If we take the analogy anyway, then the signal to noise ratio tends to be extremely small, requiring a lot of data, which is only available as high frequency, and even then, probably (I hear through the grapevine that high frequency trading firms don't go beyond simple linear regression), speed requirements mean complicated models are not used.

So in general, from your list, without experience to back it up, I would say #6 is the most important. And of course the other comments above intuitively hit the nail on the head.
 
Last edited:
Well, IMHO Kevin Davey is pretty much an expert on this subject and he has his own methodology....which he sells, natch. https://kjtradingsystems.com/index.html
HOWEVER, the landscape has really changed in this regard, especially with respect to trading the E-Minis: The new micro contracts provide a cheap way to test a strategy with small downside risk. Now, that being said, one must acknowledge that micros are only advantageous for this purpose as the commissions and fees for trading micros is pretty much outrageous relative to the full contracts.
All brokers benefit greatly from the micro contracts, as does the CME Group. They are great for short-term testing, and that's about it.
 
syswizard said:
HOWEVER, the landscape has really changed in this regard, especially with respect to trading the E-Minis: The new micro contracts provide a cheap way to test a strategy with small downside risk.
More...

the exact reason i am trading the micro's for system testing, if something goes wrong it's cheap and nothing beats real money testing to know what is working or not.
 
MarkBrown said:
the exact reason i am trading the micro's for system testing, if something goes wrong it's cheap and nothing beats real money testing to know what is working or not.
More...
That's exactly right Mark, and it now appears the Micros have made Emini SIM trading accounts moot. However, don't tell that to the prop firms....they'll get upset...LOL !
 
This is the method that Kevin Davey uses:



Here there are 2 videos in which it explains briefly how to test the Trading Systems and the errors typical to avoid in the tests:



There are excellent books to explore the valuation of Trading Systems:

- The Evaluation and Optimization of Trading Strategies (Robert Pardo)

- Trading Systems 2nd edition: A new approach to system development and portfolio optimisation (E. Tomasini-U. Jaekle)

- Trading System that Work (Thomas Stridsman)

- The Ultimate Trading Guide (G. Pruitt, J. Hill).

----------------------------------------------------

The most important test, however, always remains the so-called: paper trading, to be carried out for some time.
 
You must log in or register to reply here.
Back
Top