I put a new strategy through four stages (roughly speaking, I'm simplifying a bit):
1) Historical Data
2) Hold Out Data
3) Small Size Live
4) Full Live
First I test the idea or strategy on historical data. If it doesn't pass this test, then it definitely doesn't go any further; if it does pass this test, that's no guarantee of success.
After that (not always after, but good enough), I chop my historical data into pieces. If I have any parameters that I optimize at all (sometimes these parameters are hidden and don't look like parameters - be careful!), I 'optimize' (this can be defined many different ways - I don't just mean maximize profit) in certain time periods and test with those same parameter values on other time periods.
Some people will do this with rolling walk forward windows. I might do that sometimes, but I have other techniques that are a little more sophisticated and robust. In short, I sometimes chop the time periods in ways that reflect other covariates that I could use or factors related to other existing strategies. Ideally, I want my new strategy to play nicely with my existing portfolio, not just do well on its own. (
@globalarbtrader linked to Shannons' demon earlier today, which would point at what I'm doing as a jumping off point.)
In SOME cases, I actually put a strategy to the side for several months or years to let new data accumulate. It's rare that I'll do this, but occasionally I decide to do so.
If it passes all of that testing, then I go live in small size, just in case there's something I'm missing. This can be especially helpful with unexpected fills, variations in bid/ask spread, finding ways to improve total commissions, etc. Some people might do this with a paper account, but I find a small size keeps me more honest and it's okay with me if I blow it up. I'm talking usually <0.1% of assets (notional), sometimes a little more.
Once that's all okay, I scale up until I'm at the "bet size" (to use terminology from Kelly / Vince) I want to run that strategy at. Choosing that fraction (asset allocation) is another conversation entirely (one that is important, but seems not to be discussed often on ET.)
To touch on a few questions directly:
If you are trading a new strategy, can you have confidence in it without trading it hundreds of times? Is it possible to know if it is worth your time before then?
Yes, with careful testing, but I never let down my guard.
but it takes hundreds of individual results to account for randomness.
Depending on your strategy, this can certainly be true. So, backtesting must have at least this many trades. I recommend starting with some basic statistics to at least be able to calculate if the average return is statistically significant or not. If you don't even have enough data for that, then you're in trouble IMO.
Also, depending on the nature of your strategy, it could be vulnerable to different types of macroeconomic environments or other types of environment. I would like to have data that goes through as many of these as possible. In my long-term pseudo-investing model, I use data that goes back to the 1800s. Ray Dalio famously uses hundreds if not thousands of years of data. (Caution here to those who think the USD will be the global reserve currency forever.)
this implies it could take a beginner several years just to know whether or not a handful of different strategies produce results.
Personally I wouldn't wait this long, although you could. You definitely need several years of data in your backtesting as a minimum (unless you are doing some specialized HFT thing, which you and I are most assuredly not).
Even with the aid of backtesting, it seems there is a thin line between self-belief and delusion that traders must balance with to continue trading whatever approach they believe will be successful.
Indeed. Madness of crowds and delusions of traders. Psychology. I find statistics help keep me grounded in my psychology. I can trust statistics even when things are going poorly better than I can trust the intuition I had two months ago.
