I'm running into a sort of similar issue as well.
I have more data including bid + ask volume + whether the trades happen on the bid or ask (not tick by tick but at least on 5 second intervals).
I have a short term trading system that so far forward-tests / real-time sim-tests profitably on the ES-- if there are no transaction costs. But paying 2 ticks round-trip for the bid/ask spread, slippage, and commissions makes the strategy not worthwhile. I've tried to adapt the strategy to high-time frames and using SPY + ER2 but the edge disappears.
The good thing is I have no filters on the strategy, it is very basic, so I think there is room for improving performance , though doing so would be as difficult as coming up with the original "edge/character". I basically need to stack edges that are compatible so to speak to make it worth trading if transaction costs are what I think they are. And/or I need to look extensively at the effect of limit orders instead of market, etc, to see if the missed fills and worse performance on profit targets is a larger or smaller hit than reducing transaction costs. I'm using a custom Java program for research so I will need to figure out how the priority queues work to test limit orders.. not quite sure how to simulate them at this point.
-Taric
