I should have added one further point to my note.
In one respect, my figures do support your position. The upstream traffic,
which was required a larger cache, was also a significantly slower stream,
both in packets and in bytes. That provides a lot more headroom for key
setup. And while that is based on today's traffic mix, I think that in many
respects that is unlikely to change -- central sites will always be bigger,
with more data, and thus the dominant data flow will be towards the remote
sites. (There may also be balanced streams, such as voice or video, but I
don't expect data transfers to go away.)
--Steve Bellovin