This chart demonstrates the underlying issue here:
http://www.charthub.com/images/2007/12/16/Weekly_Volume.png
The lower pane shows a 20 period smoothing of weekly volume on the ES....which has doubled since the beginning of this year to almost 10 million contracts...almost 2 million contracts a day. This has doubled just since the beginning of this year. This is a pretty consistent trend with many instruments. So the number of trades/ticks that are getting piped through are twice what were last year. The servers, data feed clients, software programs, bandwidth...all are required to do twice the work...all else being equal.
But this is only part of the story...the data feed (at least eSignal) is also sending all bid/ask/size changes....that occur between actual trades....and in some situations, the market depth information at prices outside the bid/ask. This data alone can be more than the actual trades themselves. Much more in some cases.
The fact is, many users are getting sent alot more information than they need. Some are getting sent market depth but never look at it. Most are getting sent the bid/ask/size changes but don't care about it. Some are getting every tick, but really only care about every price change.
The problem is, there are some that use tickbar charts...and need EVERY tick...and would complain if they started getting fewer bars than a competing package was showing. There are some users that need per bar and per price delta values, so they need the bid/ask price along with each trade. So the demands of some software and some users require the data vendors to send an excessive amount of information, which results in data lags in some situations.
It's a tough situation. Who knows what the volume will be tomorrow, or a year from now....but I think we should assume it will continue to grow. Processing power and bandwidth will also continue to grow, but not likely at a rate that will keep pace with volume growth. It seems inevitable that the information is going to have to be sent in a more efficient manner. Some data feeds already do this by combining ticks (that may occur at the same price and time)....IB uses what they call "snapshot" quotes which tends to limit the pace of incoming quotes....but which many users complain about for reasons mentioned above (fewer tickbars, false marketdelta values, etc). I think we'll see various forms or combinations of tick compression, combining ticks, and/or paced quotes, as we go forward.