Now... a bit more in detail.
There are a few pre-requisites to run this test. First requirement is for the developer to isolate a specific condition that the system would be profitable under. If the system is simple, the required condition of the test data would be simple. So using the example:
I have a trend-following system. I'm going to use a simple Turtle entry/exit which would be:
Entry - 20-bar breakout.
Exit - 10-bar breakout.
In this case, the condition required for the model to be profitable is quite simple. You want to code up a time-series data generator that frequently breaks out after 20 bars and you want the price to be above the entry price after a reversal 10-day breakout. You generate the data and you have a 100% profitable dataset for your trend-following system.
But then you would also want to add a few more things to the dataset to make it more concrete. For the trend-following system to be profitable, you want the balance between the entry and the exit to reflect key statistical measures (the tough part of this is it's very model centered. Each model has their own set of "fitness" that matches). Keeping things simple, for the system to be profitable, you want the ratios between the % profitability (or the frequency of an upside move) and the risk/reward to be in balance. You can (should) generate both kinds of datasets for future tests.
If the system is simple, the "data filter logic" is relatively simple. But more complicated the system, more coding you have to do to make the dataset. This process is not only for running slip-forwards, it actually becomes an important part of the management models when you are running it live.
More you know about the system, the better... Plus, you're still dealing with the required condition. This has to be taken further into exposing the tendency it requires for the model to work. If you can expose the models to the tendency level, then you've got more control and clarity over how you should be trading the model. (Most models can't be exposed to the core tendency due to most systems working under multiple tendencies but you can get close and the process of drilling down is what's important)
A man has to sleep so I'm fuckin' out for the night. Will continue when I feel like writing again.