Re: SELECT creates millions of temp files in a single directory

2022-04-23 Thread Thomas Munro
On Sun, Apr 24, 2022 at 8:00 AM Peter wrote: > More than a million files in a single directory, this is > inacceptable. You didn't show EXPLAIN (ANALYZE) but if [Parallel] Hash is making insane numbers of temporary files then something is not working as intended... and I can take a guess at what

Re: SELECT creates millions of temp files in a single directory

2022-04-23 Thread Adrian Klaver
On 4/23/22 15:40, Peter wrote: On Sat, Apr 23, 2022 at 05:13:41PM -0400, Tom Lane wrote: And btw, I probably lost-in-translation the relevant info about the running version: Name : postgresql12-server Version: 12.10 Installed on : Mon Apr 4 04:13:18 2022 CEST Origin

Re: SELECT creates millions of temp files in a single directory

2022-04-23 Thread Peter
On Sat, Apr 23, 2022 at 05:13:41PM -0400, Tom Lane wrote: ! "David G. Johnston" writes: ! > I'll add that given the nature of the problem that changing temp_file_limit ! > away from its default of unlimited may be useful. ! > https://www.postgresql.org/docs/current/runtime-config-resource.html#RU

Re: SELECT creates millions of temp files in a single directory

2022-04-23 Thread Adrian Klaver
On 4/23/22 14:58, Peter wrote: On Sat, Apr 23, 2022 at 02:11:00PM -0700, Adrian Klaver wrote: ! On 4/23/22 12:50, Peter wrote: ! ! ! > People seem to have been brainwashed by Web-Services and OLTP, ! > and now think the working set must always fit in memory. But this ! > is only one p

Re: SELECT creates millions of temp files in a single directory

2022-04-23 Thread Peter
On Sat, Apr 23, 2022 at 02:11:00PM -0700, Adrian Klaver wrote: ! On 4/23/22 12:50, Peter wrote: ! ! ! > People seem to have been brainwashed by Web-Services and OLTP, ! > and now think the working set must always fit in memory. But this ! > is only one possible usecase, it is not the

Re: SELECT creates millions of temp files in a single directory

2022-04-23 Thread Tom Lane
"David G. Johnston" writes: > I'll add that given the nature of the problem that changing temp_file_limit > away from its default of unlimited may be useful. > https://www.postgresql.org/docs/current/runtime-config-resource.html#RUNTIME-CONFIG-RESOURCE-DISK Maybe ... that limits the total space i

Re: SELECT creates millions of temp files in a single directory

2022-04-23 Thread Adrian Klaver
On 4/23/22 12:50, Peter wrote: People seem to have been brainwashed by Web-Services and OLTP, and now think the working set must always fit in memory. But this is only one possible usecase, it is not the exclusive only one. This is no-win situation as most of the complaints in rec

Re: SELECT creates millions of temp files in a single directory

2022-04-23 Thread David G. Johnston
On Sat, Apr 23, 2022 at 1:58 PM Tom Lane wrote: > "David G. Johnston" writes: > > v12 what? > > That ... > > > It would help if you can provide a self-contained demonstration > > that others can then verify and debug (or explain). > > ... and that. As this message stands, it's undocumented whin

Re: SELECT creates millions of temp files in a single directory

2022-04-23 Thread Tom Lane
"David G. Johnston" writes: > v12 what? That ... > It would help if you can provide a self-contained demonstration > that others can then verify and debug (or explain). ... and that. As this message stands, it's undocumented whining. Please see https://wiki.postgresql.org/wiki/Slow_Query_Ques

Re: SELECT creates millions of temp files in a single directory

2022-04-23 Thread David G. Johnston
On Sat, Apr 23, 2022 at 1:00 PM Peter wrote: > In modern versions of postgres a simple SELECT writes a couple > of millions of individual temp files into a single directory under > pgsql_tmp. > I know of no filesystem that would take such lightly, and even > ZFS gets some problems with such extre

SELECT creates millions of temp files in a single directory

2022-04-23 Thread Peter
In modern versions of postgres a simple SELECT writes a couple of millions of individual temp files into a single directory under pgsql_tmp. I know of no filesystem that would take such lightly, and even ZFS gets some problems with such extremely long directories. What is the rationale in this beh