Testing ideas is of no real value unless it is live trading with low risk.
----
You can not fully automate and expect consistent profits, as there are far too many variables to programme...
"There can be as much value in the blink of an eye, as in months of rational analysis"
I agree with the above segments of what you have said.
My 8 step process is:
------------------------
1 Look at charts/place trades
2 Come up with an idea/recognize a pattern
3 Codify the pattern
4 Run a crude test to see if it does better than random
5 Test different types of entry/exit
6 Test context (is an inside day, is above prior day VWAP etc)
7 Backtest
Backtest notes:
I penalize the test fills:
a. Excessive slippage for market/stop orders in backtest
b. Only count limits as filled if the limit is traded through in backtest
Both of these trash most results.
I don't optimize but I will check parameter sensitivity after the initial tests to see if it is a curve fit.If something stands out I try to figure out why.
8. Run it with small size to see what happens.
--------------------
When I'm actively doing this, I find that I have better market feel because I am anticipating what will happen with the systems. I trade along side or even against them in ways that are too challenging to articulate clearly let alone code.
The last time I was doing this actively was in 2014 and the results of the
discretionary trading Idid along side were good.