I have spent a few years working on systems exactly like what you describe, except that my indicators, networks and rules operate per-tick and incorporate T&S and depth information for added flexibility over bar-based or pure-price methods. However, this comes with a performance penalty, especially for searching indicator parameter spaces. Core routines have to be coded in C.
This isn't easy by far. Indeed modern PCs have more than enough CPU power to discover and test highly sophisticated quant systems. Just don't let this fact fool you into jumping into more complexity than you can understand (intuitively) or search (computationally). You will probably find profitable, sophisticated market inefficiencies. Yet because of the sophistication, you may get locked into something that is a computational nightmare, or too dark of a black box that you can't understand, generalize and develop as easily.
I made this mistake early on. I pursued systems that eventually could vector-forecast a block of 5000 most-liquid US stocks out to 4 weeks into the future. Although this sounds amazing, the drawdowns were not low enough for me to trade personally, and the computational and data infrastructure, data scrubbing burden was too time consuming and expensive to continue development, at least working alone. I now work on much leaner, simpler and easier to understand single-market systems that trade ES, NQ and 6E. I still seek cleaner data for the multi-stock forecasting systems.
Indeed rolling train/test periods are essential for final validation of a system. However it is often helpful to explore low-level indicators or models without rolling periods. This saves computation time and avoids getting biased by the particular train/test period lengths.
One major challenge with adaptive methods is the problem of distinguishing between truly causal and seemingly causal relationships due to insufficient data, too many degrees of freedom, or more generally, lack of statistical significance.
The best solution I've found for insufficient statistical significance is this... Avoid big networks or iteratively-trained neural networks unless you can demonstrate profitability on fewer degrees of freedom, and have a good handle on convergence. It is better to linearly balance individually-profitable elements than subtract or multiply two elements that are unprofitable on their own where the difference "happens" to be profitable. It's better to start with smaller networks, single or multiple regression over a few inputs, or simple correlations. See what the simple analysis says before adding complexity.
Good luck!
