Out of sample testing

Q.E.D. said:
There is nothing in this universe that is "unique non optimised entry filters." But we are entering the field of epistemology.
More...

Actually, we've seemed to have entered The Twilight Zone. You, with your original response to the OP; and me, by responding with this post.

I never answered the question: "Are there unique non-optimized entry filters;" as this wasn't what the OP asked.

The OP's hypothetical or thought experiment supposed and defined such filters. 'Hypotheticals' and thought experiments are common in Mathematics and Science and etc.

You seem to be unfamiliar to 'hypotheticals', and/or thought experiments, please consider the following Google searches:

https://www.google.com/search?num=1......0i71j35i39j0i131j0i67j0i20i263.VodkI3OcIeE

https://www.google.com/search?num=1..........0i71j35i39j0i131j0i67j0i10.UCRXzIbOeow
 
Dhalsim said:
When is it okay to use out of sample data again?

If I have a basic non-optimised system with just entries tested and it works on the in sample and then I test on out of sample which it passes. If I add a secondary entry rule or filter which improves performance further, can I be justified in using the same out of sample data? This is usually a situation when I think of something that can improve strategies much later after completing the original system.

If we then add stops (which is going to have a big effect on performance) can we then use the out of sample data again?

Same for targets, can we use out of sample data again for this?
More...

Dhalsim,
I am sorry to intrude on this thread. Actually I want to bring up a non-related issue.
It had been a while since I came to this forum, and today, I came across your postings and I was very impressed.
I see that we do several things in a similar way.
(On a side note, I was really disgusted with how rude and disrespectful are some people in these threads. I don't think that a serious trader would write like this. So I apologize for them).
Here is what we do similarly:
(a) I have been trading up to 150 ES contracts, intraday.
(b) No, I am not a millionaire and I don't trade other people's money. (Think of a "scalper" who only gets in and stays in for less than 5 min, only once a day at about 9:40am ET).
(c) My stop-loss policy is the most austere, a few ticks the most. So the losses are very tight. And accordingly my profit per contract is only 4 or 5 index-points at most.
(d) To be able to make money within these boundary conditions, I must trade a large number of contracts at a low margin. (This is why I chose AMP Futures).
(e) I trade 100% automated strategies. I use TradeStation EasyLanguage, linked to AMP Futures via BlueWater Platform.

Well, Dhalsim, by reading your postings, I see that I have most of the same challenges that you do. I would have a note or a comment for most of your questions.

I would like to start with the issue of slippage.
As a background, I believe that the "trick" of my strategy is how I determine the Stop-Loss and Profit-Limit. They are not a fixed value, and they are not "trailing". I have lots of "if then else" conditions which dynamically changes my stops and limits at every tick.
For this reason I cannot send them to the broker's server. When they are triggered I have to send them as a "Market Order".
As you can expect, slippage is an serious issue for me: My theoretical profit is, let's say, 6 index points. Trading ES, from these 6 points I lose in average 1.5 points for slippage.
Very well, I am used to that and I still make some money.
However, Dhalsim, things went sour last week.
For more than 6 months, I have been working adapting the same strategy for FDAX, and last week I went live.
With FDAX, my average profit is 12 index-points. "AND OUT OF THESE 12, AT LEAST 9 ARE BEING EATEN BY SLIPPAGE".
Besides, since I send my stops as Market Orders, I would expect that one in a while I would see some "positive slippage", however this never happens????!!!!!
Dhalsim, would you have a guess as to where to problem may be? EasyLanguage to Bluewater? My internet? Bluewater to AMP Futures? AMP Futures to the Frankfurt Boerse?

How did you resolve the slippage issue that you posted about 2 years back?

Thank you for reading me and I look forward to hearing from you.
 
Dhalsim said:
When is it okay to use out of sample data again?

If I have a basic non-optimised system with just entries tested and it works on the in sample and then I test on out of sample which it passes. If I add a secondary entry rule or filter which improves performance further, can I be justified in using the same out of sample data? This is usually a situation when I think of something that can improve strategies much later after completing the original system.

If we then add stops (which is going to have a big effect on performance) can we then use the out of sample data again?

Same for targets, can we use out of sample data again for this?
More...

Dhalsim,
This is what I would do:
(a) I would run the new version on the same Out-of-Sample data chunk.
(b) By the results that you get, you may decide to change some stuff on the new rules and filters that you are introducing. And many times this turns into a loop.
(c) However, when (and if) you are happy with the results, you must find another chuck of Out-Of-Sample data for a final confirmation.

One time I ran into this same situation, with S&P Emini, however the only Out-of-Sample data that I had left were too old and I would not trust the results enough "neither good nor bad". Because many things might have changed in the last 8 years.

Therefore I chose the narrow (and painful way).

I thought like this: Based on the rationale of my strategy, "if it works for S&P, it should also work for Nasdaq". Next I adapted that strategy for Nasdaq, buy I only used the Nasdaq data 5 years or older to set up the parameters, etc. After that I had 5 years of brand new Nasdaq data to make a Walk-Forward Out-of-Sample verification.
It took me a lot of time.

I understand that this approach may not be applicable for most cases, because it requires "a lot of work"!!!!

In my case it was worth it because I really trusted the potential of that strategy and once verified, my plan was to use it for may years ahead.
 
You must log in or register to reply here.
Back
Top