Your box sucks. Or maybe you're misreading a decimal point, my not-so-great workstation pings localhost at 0.05 millis :). I see your point if you're at 0.5 millis, but then again, you're already sub-millisecond!
We don't use customized linux, just a standard distro. There are a few methods...