Sure, I'll tell you my bug.
The strategy of mine posts multiple passive orders to get out. Usually, when one gets hit, I cancel the others and move along. Occasionally, someone sweeps the market, and I'll get overfilled. Normally, when I go into an overfill situation, the strategy turns off quoting, switching into a remove liquidity mode, and just waits for the next best time to remove shares. Then it stays in this situation until I look at the situation and flip a switch back on.
What happened was the broker server delayed fill messages for an extremely long period of time because the market was moving super fast, so the passive quotation engine was quoting in a fast market while the signal to flip into "escape mode" didn't arrive soon enough. During most markets, even somewhat fast markets, the fill notifications happen within (at most) a few milliseconds. For whatever reason, yesterday, they started taking seconds.
I was receiving fill notifications while the NBBO had moved a full 8-10 cents away from the fill-price. When I got into this situation, the only protection I had left was "max outstanding orders", which I had set to around 20. The problem was, even with 20 outstanding orders, that was still a lot of shares, and on those outstanding orders, even those fills were delayed.
The bug-fix is just to enforce an even harder cap on "potential exposure" on the code that does the passive quotation. It seems so obvious now, but I didn't have it in there because I never saw this situation come up before.
