Re: PANIC: could not flush dirty data: Cannot allocate memory

2022-12-05 Thread klaus . mailinglists
Some more updates Did this start after upgrading to 22.04? Or after a certain kernel upgrade? For sure it only started with Ubuntu 22.04. We did not had and still not have any issues on servers with Ubuntu 20.04 and 18.04. It also happens with Ubuntu 22.10 (Kernel 5.19.0-23-generic). W

Re: PANIC: could not flush dirty data: Cannot allocate memory

2022-11-29 Thread klaus . mailinglists
Hello all! Thanks for the many hints to look for. We did some tuning and further debugging and here are the outcomes, answering all questions in a single email. In the meantime, you could experiment with setting checkpoint_flush_after to 0 We did this: # SHOW checkpoint_flush_after; chec

Re: PANIC: could not flush dirty data: Cannot allocate memory

2022-11-16 Thread Andres Freund
Hi, On 2022-11-16 09:16:56 -0800, Andres Freund wrote: > On 2022-11-15 13:23:56 +0100, klaus.mailingli...@pernau.at wrote: > > Filesystem is ext4. VM technology is mixed: VMware, KVM and XEN PV. Kernel > > is 5.15.0-52-generic. > > > > We have not seen this with Ubutnu 18.04 and 20.04 (although w

Re: PANIC: could not flush dirty data: Cannot allocate memory

2022-11-16 Thread Andres Freund
Hi, On 2022-11-15 13:23:56 +0100, klaus.mailingli...@pernau.at wrote: > Filesystem is ext4. VM technology is mixed: VMware, KVM and XEN PV. Kernel > is 5.15.0-52-generic. > > We have not seen this with Ubutnu 18.04 and 20.04 (although we might not > have noticed it). Did this start after upgradi

Re: PANIC: could not flush dirty data: Cannot allocate memory

2022-11-15 Thread Christoph Moench-Tegeder
## klaus.mailingli...@pernau.at (klaus.mailingli...@pernau.at): > AFAIU the problem is not related to the memory settings in > postgresql.conf. It is the kernel that > for whatever reasons report ENOMEM. Correct? Correct, there's a ENOMEM from the kernel when writing out data. > Filesystem is e

Re: PANIC: could not flush dirty data: Cannot allocate memory

2022-11-15 Thread Thomas Munro
On Wed, Nov 16, 2022 at 1:24 AM wrote: > Filesystem is ext4. VM technology is mixed: VMware, KVM and XEN PV. > Kernel is 5.15.0-52-generic. > > We have not seen this with Ubutnu 18.04 and 20.04 (although we might not > have noticed it). > > I guess upgrading to postgresql 13/14/15 does not help as

Re: PANIC: could not flush dirty data: Cannot allocate memory

2022-11-15 Thread klaus . mailinglists
...@pernau.at): On several servers we see the error message: PANIC: could not flush dirty data: Cannot allocate memory As far as I can see, that "could not flush dirty data" happens total three times in the code - there are other places where postgresql could PANIC on fsync()-and-stu

Re: PANIC: could not flush dirty data: Cannot allocate memory

2022-11-14 Thread Tom Lane
Thomas Munro writes: > It has been argued before that we might have been over-zealous > applying the PANIC promotion logic to sync_file_range(). It's used to > start asynchronous writeback to make the later fsync() call fast, so > it's "only a hint", but I have no idea if it could report a writeb

Re: PANIC: could not flush dirty data: Cannot allocate memory

2022-11-14 Thread Thomas Munro
On Tue, Nov 15, 2022 at 10:54 AM Christoph Moench-Tegeder wrote: > ## klaus.mailingli...@pernau.at (klaus.mailingli...@pernau.at): > > On several servers we see the error message: PANIC: could not flush > > dirty data: Cannot allocate memory > Of these three places, there&#x

Re: PANIC: could not flush dirty data: Cannot allocate memory

2022-11-14 Thread Christoph Moench-Tegeder
## klaus.mailingli...@pernau.at (klaus.mailingli...@pernau.at): > On several servers we see the error message: PANIC: could not flush > dirty data: Cannot allocate memory As far as I can see, that "could not flush dirty data" happens total three times in the code - there are ot

Re: PANIC: could not flush dirty data: Cannot allocate memory

2022-11-14 Thread Tom Lane
klaus.mailingli...@pernau.at writes: > On several servers we see the error message: PANIC: could not flush > dirty data: Cannot allocate memory What that's telling you is that fsync (or some equivalent OS call) returned ENOMEM, which would seem to be a kernel-level deficiency. Perhap

PANIC: could not flush dirty data: Cannot allocate memory

2022-11-14 Thread klaus . mailinglists
the error message: PANIC: could not flush dirty data: Cannot allocate memory Unfortunately I do not find any reference to this kind of error. Can you please describe what happens here in detail? Is it related to server memory? Or our memory settings? I am not so surprised that it happens with the