Since you believe this may be a software issue, have you tried to recreate this scenario using a papertrading account?
send the basket order and purposely go over margin, wait for the rejected order, unplug the internet until tws shows disconnected then reconnect, see what happens. Maybe you...