On Sat, Feb 4, 2012 at 10:06 AM, Jeff Janes <jeff.ja...@gmail.com> wrote: > > The worst thing about the current memory usage is probably that big > machines can't qsort more than 16,777,216 tuples no matter how much > memory they have, because memtuples has to live entirely within a > single allocation, which is currently limited to 1GB. It is probably > too late to fix this problem for 9.2. I wish I had started there > rather than here. > > This 16,777,216 tuple limitation will get even more unfortunate if the > specializations that speed up qsort but not external sort get > accepted. >
I think it's a fair ask to extend our palloc limitation of 1GB to 64bit space. I see there are a lot of applications that want more memory by one palloc call in user-defined functions, aggregates, etc. As you may notice, however, the area in postgres to accomplish it needs to be investigated deeply. I don't know where it's safe to allow it and where not. varlena types is unsafe, but it might be possible to extend varlena header to 64 bit in major release somehow. > Attached is a completely uncommitable proof of concept/work in > progress patch to get around the limitation. It shows a 2 fold > improvement when indexing an integer column on a 50,000,000 row > randomly ordered table. In any case, we do need bird-eye sketch to attack it but I guess it's worth and at some future point we definitely must do, though I don't know if it's the next release or third next release from now. Thanks, -- Hitoshi Harada -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
