On Wed, Feb 21, 2024 at 7:34 PM Andrei Lepikhov <a.lepik...@postgrespro.ru> wrote: > I see in [1] that the reporter mentioned a delay between the error > message in parallel HashJoin and the return control back from PSQL. Your > patch might reduce this delay. > Also, I have the same complaint from users who processed gigabytes of > data in parallel HashJoin. Presumably, they also stuck into the unlink > of tons of temporary files. So, are you going to do something with this > code?
Yeah, right. I will aim to get this into the tree next week. First, there are a couple of minor issues to resolve around freeing that Heikki mentioned. Then there is the question of whether we think this might be a candidate for back-patching, given the complaints you mention. Opinions? I would add that the problems you reach when you get to very large number of partitions are hard (see several very long threads about extreme skew for one version of the problem, but even with zero/normal skewness and perfect estimation of the number of partitions, if you ask a computer to partition 42TB of data into partitions that fit in a work_mem suitable for a Commodore 64, it's gonna hurt on several levels) and this would only slightly improve one symptom. One idea that might improve just the directory entry and file descriptor aspect, would be to scatter the partitions into (say) 1MB chunks within the file, and hope that the file system supports holes (a bit like logtape.c's multiplexing but I wouldn't do it quite like that).
