I've currently got a 0.05 sec delay from Chicago.
Try running a tracert to various servers and see if you see a common point of congestion (maybe within your ISP's net or maybe some common regional router). Traceroute traffic is prioritized lower than other traffic, so it's possible to get a false impression sometimes. But you might be able to see something common when comparing the results across multiple server locations.