Quote from illiquid:
You are already ahead of the pack in this respect -- that is, you are anticipating the reasons why long-term success in trading is so elusive. Can you already see the problems with backtesting itself as so-called "proof" of an edge?
You seem to want to pit a longer-term trading time frame based on research vs a shorter, more immediate methodology based on direct observation. Whether these two are diametrically opposed at heart is questionable, but I think the instinct you have is correct. Backtesting, unfortunately, can only be done with archived data, looked at in isolation, whereas those trading primarily off price action get nearly immediate reinforcement that includes all other conditions (technical, fundamental, news flow) in real-time.
I guess what I'm saying is this: if you can trade directly off price action itself, you will be far less prone to falling into that trap of random luck masquerading as long-term edge that you fear. The nature of PA itself is never stable for long, as different market conditions even day to day will change the way prices move. But if you can master this mutability and come to terms with uncertainty, you will always necessarily be on your toes, always a bit unsure of yourself but also wary of never becoming complacent. That is how you keep yourself from being fooled by randomness, so to speak.
For me, the glaring difference between using backtesting vs price action is not the technique in itself, but the attitude of the trader in relationship to the market. Does a trader need certainty in order to prosper long term, or can he succeed by just always staying ahead of an ever-shifting playing field?