Quote from Jerry030:
Criteria:
Random Number = Excel RAND() assigned to each bar and recalculated for each run
Random selection for trade: Rand() > .30 AND < .3205
which gives about 200 trades, of which about 50 will have a fill based on the Stop Limit order cited in the Rules
For Predictive: value of NN model predicts higher prices tomorrow, using same Stop Limit order
Time Window 11,000 bars, all available data
I think I follow the algorithm a bit. Without trying to ascertain the specifics, you are looking at a predefined decision rule (although they are both slightly different, why not find a rule and apply it identically to both cases, i.e. rather than chose between .3 to .32 for entry criteria, why not use close +/-high/low relative closing criteria from real data nn to random data set: apples to apples) and comparing it to a random set.
Firstly, as you are using 11,000 samples, it seems fair to use rand() function as the uniform dist permutation characteristics are a close approximation of gaussian (binomial approaches gaussian over long runs).
However, again to compare apples to apples (as each gain of the random run may not be comparable to each gain of the real data run, although maybe ratios are ok), it seems better to model with GBM as I mentioned earlier, and RUN A VERY LARGE monte carlo of the PL runs (not just 6 as in your example). Look at the range of PL over the simulations. By virtue of randomness and the model boundaries (unless you have large jumps in the market data you used), by its very definition, the real results you ran should be contained within the data set.
If it is not, the GBM data is not sufficient to model the random data behavior of the market you are testing. As maestro mentioned, you could get into more detailed models such as stable distributions or jump diffusions to get more accuracy. If you find that your real market results are on the higher end of the PL distribution from the monte carlo analysis then that shows your system is better than the average random results, but by it's very nature, its performance must be a subset of the monte carlo results or the random data is not being modeled properly.
If you tested this across all markets and all data, although it would be inconclusive (since it is not a closed system), you could argue that your system performed better than average over random data.
If you found that the market data was closely modeled by gaussian dist, you could then apply some sort of confidence interval to test your hypothesis (i.e. if your results were greater than 95% of monte carlo, you could say your results were statistically better than a system applied to random data).
However, as (I think) mastro argues, it does not guarantee equal likelyhood of future performance. Your next run of 11,000 data pts. may perform at the lower end of the monte carlo curve.
That being said, I would argue that your approach (if it is in the upper percentile of PL results), is much better than impulsively visually trading off TA signals.