Quote from hft:
My definition of backtesting might not match yours. I define it strictly as simulating how a strategy performs through historical data. It's really hard to do this well with a high-frequency market-making strategy. Instead of that, I rely other forms of analysis (and admittedly 'educated guesses').
- Run blind: Simply rest small size and see if it makes money
- Trial and error: Tweak some settings and run it concurrently with a control strategy to see if it's more effective.
- Post-trade analytics: Analyze the trades you got into in order to improve the strategy.
- Historical data analytics: Analyze historical market data, but not necessarily simulated fills as in a typical backtest.
Some of that is quite vague, which is both to not reveal secrets as well as indicative of the ad-hoc nature of the process anyway.