Re: meson vs. llvm bitcode files

2025-04-29 Thread Nazir Bilal Yavuz
Hi, On Thu, 13 Mar 2025 at 13:11, Nazir Bilal Yavuz wrote: > One thing I noticed is that gen_srcfiles['srcfiles'] seems wrong. > gen_sources is a better name compared to gen_srcfiles. So, I changed > it to gen_sources in v4. Rebase is needed due to b1720fe63f, v5 is att

Re: Add pg_buffercache_mark_dirty[_all] functions to the pg_buffercache

2025-04-28 Thread Nazir Bilal Yavuz
> Would love to hear if this makes sense or or am I overlooking something here. > Thanks for any feedback! I think what you said makes sense and is correct if we only want to simulate a buffer’s dirty state for testing/debugging, but if we want to replicate usual steps to marking buffers as dir

Re: New committer: Jacob Champion

2025-04-14 Thread Nazir Bilal Yavuz
! Congratulations Jacob! -- Regards, Nazir Bilal Yavuz Microsoft

Add pg_buffercache_mark_dirty[_all] functions to the pg_buffercache

2025-04-11 Thread Nazir Bilal Yavuz
already reviewed the patches are CCed. [1] postgr.es/m/CAN55FZ0h_YoSqqutxV6DES1RW8ig6wcA8CR9rJk358YRMxZFmw%40mail.gmail.com -- Regards, Nazir Bilal Yavuz Microsoft From 06f12f6174c0b6e1c5beeb4c1a8f4b33b89cc158 Mon Sep 17 00:00:00 2001 From: Nazir Bilal Yavuz Date: Fri, 4 Apr 2025 13:39:49 +0300

Re: Add pg_buffercache_evict_all() and pg_buffercache_mark_dirty[_all]() functions

2025-04-11 Thread Nazir Bilal Yavuz
hat. I am okay to replace these loops with [1] to make it standart everywhere: [1] for (int buf = 0; buf < NBuffers; buf++) { BufferDesc *desc = GetBufferDescriptor(buf); -- Regards, Nazir Bilal Yavuz Microsoft

Re: Add pg_buffercache_evict_all() and pg_buffercache_mark_dirty[_all]() functions

2025-04-11 Thread Nazir Bilal Yavuz
x27;d be happy to > work > on it). I think it looks useful, but I’m not too familiar with NUMA, so I’m not sure I can say much with confidence. Maybe someone else can chime in with a more solid take? -- Regards, Nazir Bilal Yavuz Microsoft

Re: Adding NetBSD and OpenBSD to Postgres CI

2025-04-08 Thread Nazir Bilal Yavuz
GET to the NetBSD and OpenBSD CI tasks. [1] https://cirrus-ci.com/task/5126123371102208?logs=test_world#L144 -- Regards, Nazir Bilal Yavuz Microsoft From 0a265615e7bd9d6da2cbf1d7edfbe68a8bdf39f1 Mon Sep 17 00:00:00 2001 From: Nazir Bilal Yavuz Date: Tue, 8 Apr 2025 14:21:42 +0300 Subject: [PATC

Re: Add pg_buffercache_evict_all() and pg_buffercache_mark_dirty[_all]() functions

2025-04-07 Thread Nazir Bilal Yavuz
Hi, On Tue, 8 Apr 2025 at 02:29, Andres Freund wrote: > On 2025-04-07 19:37:50 +0300, Nazir Bilal Yavuz wrote: > > > > > > + relation_close(rel, AccessExclusiveLock); > > > > > > > > > > Hm. Why are we dropping the lock here early? It's

Re: Add pg_buffercache_evict_all() and pg_buffercache_mark_dirty[_all]() functions

2025-04-07 Thread Nazir Bilal Yavuz
Hi, On Mon, 7 Apr 2025 at 16:38, Andres Freund wrote: > > Hi, > > On 2025-04-04 18:36:57 +0300, Nazir Bilal Yavuz wrote: > > v6 is attached. Additional changes prior to v5: > > > > * Tests are added. > > I don't think the mark-dirty stuff is quite ready

Re: ci: Allow running mingw tests by default via environment variable

2025-04-07 Thread Nazir Bilal Yavuz
rebasing them, I realized that we use 'REPO_CI_AUTOMATIC_TRIGGER_TASKS' for all branches. So, we don't have per branch 'REPO_CI_AUTOMATIC_TRIGGER_TASKS' configuration option. Is that a problem? -- Regards, Nazir Bilal Yavuz Microsoft From c872f537375416289d3fdc302716403266899c1f Mon Sep

Re: Add pg_buffercache_evict_all() and pg_buffercache_mark_dirty[_all]() functions

2025-04-05 Thread Nazir Bilal Yavuz
need to do this. I see EvictUnpinnedBuffer() as a central place that checks all conditions. If we add this pre-check then we should not do the same check in the EvictUnpinnedBuffer(), creating this logic would add unnecessary complexity to EvictUnpinnedBuffer(). I addressed these reviews in v3. I will share the patches in my next reply. -- Regards, Nazir Bilal Yavuz Microsoft

Re: Using read stream in autoprewarm

2025-04-05 Thread Nazir Bilal Yavuz
Hi, Thank you for looking into this! On Sat, 29 Mar 2025 at 23:10, Melanie Plageman wrote: > > On Fri, Nov 29, 2024 at 6:20 AM Nazir Bilal Yavuz wrote: > > > > v4 is attached. > > I've started looking at this version. What I don't like about this > structu

Re: Using read stream in autoprewarm

2025-04-04 Thread Nazir Bilal Yavuz
hanges, we do not check if it is changed from the database oid = 0. Handling this might require some structural changes. > What do you think? I think what you said is right but the current version of the patch looks more complicated to me. I may be biased because I do not like do-while loops and goto statements. -- Regards, Nazir Bilal Yavuz Microsoft

Re: Add pg_buffercache_evict_all() and pg_buffercache_mark_dirty[_all]() functions

2025-04-04 Thread Nazir Bilal Yavuz
Hi, Thanks for the review! On Wed, 2 Apr 2025 at 20:06, Andres Freund wrote: > > Hi, > > On 2025-03-31 19:49:25 +0300, Nazir Bilal Yavuz wrote: > > > After this discussion, I think it would be helpful if one of the more > > > experienced > > > hackers c

Re: Using read stream in autoprewarm

2025-04-04 Thread Nazir Bilal Yavuz
Hi, On Fri, 4 Apr 2025 at 10:59, Daniel Gustafsson wrote: > > This patch is already marked Ready for Committer and I concur with that. Same on my end, v14 LGTM. -- Regards, Nazir Bilal Yavuz Microsoft

Re: Using read stream in autoprewarm

2025-04-02 Thread Nazir Bilal Yavuz
Hi, On Wed, 2 Apr 2025 at 01:36, Melanie Plageman wrote: > > On Tue, Apr 1, 2025 at 8:50 AM Nazir Bilal Yavuz wrote: > > > > I am attaching v8, which is an updated version of the v7. I tried to > > get rid of these local variables and refactored code to make logic

Re: Using read stream in autoprewarm

2025-04-02 Thread Nazir Bilal Yavuz
Hi, On Wed, 2 Apr 2025 at 18:54, Melanie Plageman wrote: > > On Wed, Apr 2, 2025 at 6:26 AM Nazir Bilal Yavuz wrote: > > > > On Wed, 2 Apr 2025 at 01:36, Melanie Plageman > > wrote: > > > > > > I know you mentioned off-list that you don't

Re: Using read stream in autoprewarm

2025-04-01 Thread Nazir Bilal Yavuz
tforward instead of going back and forth. 0001 and 0002 are v8. 0003 is another refactoring attempt to make code more straightforward. I did not squash 0003 to previous patches as you might not like it. -- Regards, Nazir Bilal Yavuz Microsoft v8-0001-Optimize-autoprewarm-with-read-streams.patch

Re: Using read stream in autoprewarm

2025-03-31 Thread Nazir Bilal Yavuz
Hi, Thank you for looking into this! On Mon, 31 Mar 2025 at 17:42, Melanie Plageman wrote: > > On Sun, Mar 30, 2025 at 10:01 AM Nazir Bilal Yavuz wrote: > > > > > Some review feedback on your v4: I don't think we need the > > > rs_have_free_buffer

Re: Using read stream in autoprewarm

2025-03-31 Thread Nazir Bilal Yavuz
Hi, On Mon, 31 Mar 2025 at 21:15, Melanie Plageman wrote: > > On Mon, Mar 31, 2025 at 12:27 PM Nazir Bilal Yavuz wrote: > > > > I worked on an alternative approach, I refactored code a bit. It does > > not traverse the list two times and I think the code is more suitable

Re: Add pg_buffercache_evict_all() and pg_buffercache_mark_dirty[_all]() functions

2025-03-31 Thread Nazir Bilal Yavuz
y for sure about that. And in that > context, I'd > suggest referring to that awesome statement in the documentation: "If in > doubt, > functions should be labeled as UNSAFE, which is the default." You may be right. I thought they are parallel safe as we acquire the buffer he

Re: Add pg_buffercache_evict_all() and pg_buffercache_mark_dirty[_all]() functions

2025-03-28 Thread Nazir Bilal Yavuz
fercache_mark_dirty(): > Maybe edit the comment to: "Try to mark a shared buffer as dirty."? Done. > Maybe edit the elog text to: "bad shared buffer ID" - just to clarify > the case when > provided buffer number is negative (local buffer number). Although I think

Re: Draft for basic NUMA observability

2025-03-27 Thread Nazir Bilal Yavuz
Hi, Thank you for working on this! On Wed, 19 Mar 2025 at 12:06, Jakub Wartak wrote: > > On Tue, Mar 18, 2025 at 3:29 PM Bertrand Drouvot > wrote: > > Hi! v15 attached, rebased, CI-tested, all fixed incorporated. This needs to be rebased after 8eadd5c73c. -- Regards, Na

Re: read stream on amcheck

2025-03-27 Thread Nazir Bilal Yavuz
_NONE case and &rsdata for the rest. Example patch is attached, heapamcheck_rs_perblock_data is added to typedefs.list too. -- Regards, Nazir Bilal Yavuz Microsoft From 642a01c6298742879cefac1197d466b6fec5df7f Mon Sep 17 00:00:00 2001 From: Matheus Alcantara Date: Fri,

Re: CREATE DATABASE with filesystem cloning

2025-03-25 Thread Nazir Bilal Yavuz
Hi, Rebased version of the patch is attached. -- Regards, Nazir Bilal Yavuz Microsoft From d2eb46fb18258931f65e2b01ac6b406255f3c575 Mon Sep 17 00:00:00 2001 From: Nazir Bilal Yavuz Date: Thu, 8 Aug 2024 15:01:48 +0300 Subject: [PATCH v10] Introduce file_copy_method GUC This GUC can be set to

Re: Add pg_buffercache_evict_all() and pg_buffercache_mark_dirty[_all]() functions

2025-03-24 Thread Nazir Bilal Yavuz
ow that flush has happened, we just do not know if the buffer is flushed by us or someone else. [1] https://postgr.es/m/flat/76a550315baef9d7424b70144f1c6a2d%40postgrespro.ru -- Regards, Nazir Bilal Yavuz Microsoft

Re: Add pg_buffercache_evict_all() and pg_buffercache_mark_dirty[_all]() functions

2025-03-24 Thread Nazir Bilal Yavuz
d it does not guarantee that it has > been > written to disk. I think flushed means 'passing the buffer contents to the kernel' in the Postgres context (as it is explained in the FlushBuffer()). We know that flush has happened, we just do not know if the buffer is flushed by us o

Re: Add pg_buffercache_evict_all() and pg_buffercache_mark_dirty[_all]() functions

2025-03-24 Thread Nazir Bilal Yavuz
Hi, On Fri, 21 Mar 2025 at 01:25, Joseph Koshakow wrote: > > Hi I am working with Aidar to give a review and I am also a beginner > reviewer. Thank you so much for the review! > > From 813e5ec0da4c65970b4b1ce2ec2918e4652da9ab Mon Sep 17 00:00:00 2001 > > From: Nazir Bilal

Re: Improve monitoring of shared memory allocations

2025-03-21 Thread Nazir Bilal Yavuz
gh we assume both nbuckets and num_partition initialized as the same type, (*nbuckets) <<= 1 will cause an infinite loop if num_partition is bigger than MAX_TYPE / 2. So I think that the solution is to confirm that num_partition < MAX_NBUCKETS_TYPE / 2, what do you think? -- Regards, Nazir Bilal Yavuz Microsoft

Re: Show WAL write and fsync stats in pg_stat_io

2025-03-18 Thread Nazir Bilal Yavuz
hed a trivial patch. LGTM. -- Regards, Nazir Bilal Yavuz Microsoft

Re: [PoC] Federated Authn/z with OAUTHBEARER

2025-03-17 Thread Nazir Bilal Yavuz
on to server on socket "/tmp/xZKYtq40nL/.s.PGSQL.21400" failed: failed to obtain access token: Unsupported protocol (libcurl: Received HTTP/0.9 when not allowed) [11:09:56.548] # ' [11:09:56.548] # matches '(?^:connection to database failed)' [11:09:56.548] # Looks like you failed 2 tests of 121. [11:09:56.548] [11:09:56.548] (test program exited with status code 2) [1] https://cirrus-ci.com/task/4621590844932096 -- Regards, Nazir Bilal Yavuz Microsoft

Re: Add pg_buffercache_evict_all() and pg_buffercache_mark_dirty[_all]() functions

2025-03-14 Thread Nazir Bilal Yavuz
pg_buffercache_evict_all() functions do not necessarily mean these buffers are flushed by these functions. Remaining questions from the v1: On Wed, 12 Mar 2025 at 19:15, Nazir Bilal Yavuz wrote: > On Mon, 17 Feb 2025 at 19:59, Andres Freund wrote: > > On 2024-12-25 15:57:34 +0300, Nazir Bilal Ya

Re: per backend WAL statistics

2025-03-13 Thread Nazir Bilal Yavuz
Hi, On Mon, 10 Mar 2025 at 17:43, Bertrand Drouvot wrote: > On Mon, Mar 10, 2025 at 03:08:49PM +0300, Nazir Bilal Yavuz wrote: > > > I think that every time we flush IO or WAL stats, we want(?) to flush > > backend stats as well, > > Yeah, I think that's happening a

Re: meson vs. llvm bitcode files

2025-03-13 Thread Nazir Bilal Yavuz
well. One thing I noticed is that gen_srcfiles['srcfiles'] seems wrong. gen_sources is a better name compared to gen_srcfiles. So, I changed it to gen_sources in v4. -- Regards, Nazir Bilal Yavuz Microsoft From 578fbb407ce89293c8c955179e61b10aadc1709f Mon Sep 17 00:00:00 2001 From: Andr

Re: Add pg_buffercache_evict_all() and pg_buffercache_mark_dirty[_all]() functions

2025-03-12 Thread Nazir Bilal Yavuz
Hi, On Mon, 17 Feb 2025 at 19:59, Andres Freund wrote: > > Hi, Thanks for the review! And sorry for the late reply. > On 2024-12-25 15:57:34 +0300, Nazir Bilal Yavuz wrote: > > So, this patchset extends pg_buffercache with 3 functions: > > > > 1- pg_buffercach

Re: meson vs. llvm bitcode files

2025-03-12 Thread Nazir Bilal Yavuz
should get all or some of the generated backend sources > compiled to bitcode, similar to contrib modules. I think we can do this. I added other backend sources like you did in the PR but attached it as another patch (0007) because I wanted to hear other people's opinions on that first.

Re: per backend WAL statistics

2025-03-10 Thread Nazir Bilal Yavuz
code and did not find any example of that but I just wanted to be sure. If there is a case like that, then pgstat_backend_wal_have_pending() needs to check wal_bytes instead of wal_records. -- Regards, Nazir Bilal Yavuz Microsoft

Re: meson vs. llvm bitcode files

2025-03-07 Thread Nazir Bilal Yavuz
Hi, On Thu, 5 Sept 2024 at 12:24, Nazir Bilal Yavuz wrote: > > I found that Andres shared a patch > (v17-0021-meson-Add-LLVM-bitcode-emission.patch) a while ago [1]. Andres and I continued to work on that. I think the patches are in sharable state now and I wanted to hear opinio

Upgrade FreeBSD CI images to 14.2

2025-03-05 Thread Nazir Bilal Yavuz
upstream. Any kind of feedback would be appreciated. [1] https://github.com/anarazel/pg-vm-images/commit/3f2d728299808e76ce23b9b12ccfec944c86e318 [2] https://github.com/anarazel/pg-vm-images/commit/f1ea53c94ab864feb7c23d609bf85f0a1d568c9d -- Regards, Nazir Bilal Yavuz Microsoft From

Re: ci: Allow running mingw tests by default via environment variable

2025-03-04 Thread Nazir Bilal Yavuz
Hi, On Tue, 4 Mar 2025 at 03:28, Thomas Munro wrote: > > On Fri, Apr 26, 2024 at 12:02 AM Nazir Bilal Yavuz wrote: > > On Sat, 13 Apr 2024 at 05:12, Andres Freund wrote: > > > I propose that we instead run the task automatically if > > > $REPO_MINGW_TRIGGER_B

Re: Adding NetBSD and OpenBSD to Postgres CI

2025-02-28 Thread Nazir Bilal Yavuz
used 5G on /dev/sd0j +#swapon /dev/sd0j # fails with ENXIO... what am I missing? I think I found the problem, sd0j's fstype is not a swap. It worked like that: $ disklabel -E sd0 $ umount /usr/obj $ disklabel -E sd0 # prompts are: m -> j -> \n -> \n -> swap -> w -> q $ di

Re: read stream on amcheck

2025-02-17 Thread Nazir Bilal Yavuz
, LGTM. -- Regards, Nazir Bilal Yavuz Microsoft

Re: per backend WAL statistics

2025-02-16 Thread Nazir Bilal Yavuz
'll add it in the new patch serie for this thread (that simplifies the new > pg_stat_wal_build_tuple() among other things) unless Nazir beat me to it. > > [2]: https://www.postgresql.org/message-id/Z6L3ZNGCljZZouvN%40paquier.xyz I am working on some other stuff right now. Please feel

Re: Adding NetBSD and OpenBSD to Postgres CI

2025-02-13 Thread Nazir Bilal Yavuz
lse. Sure, I will work on this. Thank you for mentioning it. -- Regards, Nazir Bilal Yavuz Microsoft

Re: Show WAL write and fsync stats in pg_stat_io

2025-02-06 Thread Nazir Bilal Yavuz
g is similar to that. On Thu, 6 Feb 2025 at 10:32, Bertrand Drouvot wrote: > > s/and the primary WAL some writes/and the primary some WAL writes/ I am not a native speaker but 'primary writes some WAL with its checkpointer' sounds better to me. -- Regards, Nazir Bilal Yavuz Microsoft

Re: Show WAL write and fsync stats in pg_stat_io

2025-02-06 Thread Nazir Bilal Yavuz
pgstat_tracks_io_op() and pgstat_tracks_io_object(), do you mean converting if statements to switch statements or something else? -- Regards, Nazir Bilal Yavuz Microsoft

Re: Show WAL write and fsync stats in pg_stat_io

2025-02-05 Thread Nazir Bilal Yavuz
y -> ReadCheckpointRecord -> ReadRecord -> XLogPrefetcherReadRecord -> lrq_complete_lsn -> lrq_prefetch -> lrq->next = XLogPrefetcherNextBlock -> XLogReadAhead -> XLogDecodeNextRecord -> ReadPageInternal -> state->routine.page_read = XLogPageRead() ''' Is there a chance that the function chain above does not get triggered while running the stats.sql test? -- Regards, Nazir Bilal Yavuz Microsoft

Re: Show WAL write and fsync stats in pg_stat_io

2025-02-04 Thread Nazir Bilal Yavuz
Hi, On Tue, 4 Feb 2025 at 17:29, Nazir Bilal Yavuz wrote: > > Thanks! My benchmark results are the same [1]. I could not see the > noticable difference between master and patched version. I run the > benchmark a couple of times, sometimes the patched version, sometimes the > ma

Re: Show WAL write and fsync stats in pg_stat_io

2025-02-04 Thread Nazir Bilal Yavuz
Hi, On Tue, 4 Feb 2025 at 10:55, Michael Paquier wrote: > > On Mon, Feb 03, 2025 at 02:34:29PM +0300, Nazir Bilal Yavuz wrote: > > On Mon, 3 Feb 2025 at 11:50, Bertrand Drouvot < bertranddrouvot...@gmail.com> wrote: > >> === 1 > >> > >>

Re: Show WAL write and fsync stats in pg_stat_io

2025-02-03 Thread Nazir Bilal Yavuz
Hi, On Mon, 3 Feb 2025 at 11:50, Bertrand Drouvot wrote: > > Hi, > > On Mon, Feb 03, 2025 at 01:07:26PM +0900, Michael Paquier wrote: > > On Fri, Jan 31, 2025 at 11:29:31AM +0300, Nazir Bilal Yavuz wrote: > > > On Wed, 29 Jan 2025 at 18:16, Bertrand Drouvot > >

Re: Show WAL write and fsync stats in pg_stat_io

2025-01-31 Thread Nazir Bilal Yavuz
Hi, On Wed, 29 Jan 2025 at 18:16, Bertrand Drouvot wrote: > > Hi, > > On Wed, Jan 29, 2025 at 02:57:21PM +0300, Nazir Bilal Yavuz wrote: > > I agree with you but it was discussed before in this thread [2]. It > > was decided to use both track_wal_io_timing and track_io_t

Re: Windows CFBot is broken because ecpg dec_test.c error

2025-01-30 Thread Nazir Bilal Yavuz
Hi, On Wed, 29 Jan 2025 at 19:50, Andres Freund wrote: > > On 2025-01-29 18:24:45 +0300, Nazir Bilal Yavuz wrote: > > > The cause is that meson fixed a bug [1] in v.1.7.0. Before meson > > v1.7.0; although --no-rebuild is used while running tests, meson was > > bu

Re: BitmapHeapScan streaming read user and prelim refactoring

2025-01-30 Thread Nazir Bilal Yavuz
> > > > I could use review feedback on all of them. 0001 should be ready to > > push once I get some review. > > Bilal mentioned offlist that 0001 had some indentation issues in one > of the comments. v27 attached. I have no other comments, 0001 LGTM. -- Regards, Nazir Bilal Yavuz Microsoft

Re: Windows CFBot is broken because ecpg dec_test.c error

2025-01-29 Thread Nazir Bilal Yavuz
able for the CI. CI run after this change applied: https://cirrus-ci.com/build/6264369203380224 [1] https://mesonbuild.com/Release-notes-for-1-7-0.html#test-targets-no-longer-built-by-default -- Regards, Nazir Bilal Yavuz Microsoft

Re: Show WAL write and fsync stats in pg_stat_io

2025-01-29 Thread Nazir Bilal Yavuz
Hi, On Tue, 28 Jan 2025 at 07:23, Michael Paquier wrote: > > On Mon, Jan 27, 2025 at 05:13:39PM +0300, Nazir Bilal Yavuz wrote: > > On Mon, 27 Jan 2025 at 16:59, Nazir Bilal Yavuz wrote: > >> > >> On Mon, 27 Jan 2025 at 03:52, Michael Paquier wrote: > >

Re: Show WAL write and fsync stats in pg_stat_io

2025-01-29 Thread Nazir Bilal Yavuz
Hi, On Mon, 27 Jan 2025 at 10:45, Bertrand Drouvot wrote: > > Hi, > > On Fri, Jan 24, 2025 at 06:29:46PM +0300, Nazir Bilal Yavuz wrote: > > Hi, > > > > Thanks for looking into this! > > > > On Fri, 24 Jan 2025 at 17:20, Bertrand Drouvot > > wrote

Re: Show WAL write and fsync stats in pg_stat_io

2025-01-27 Thread Nazir Bilal Yavuz
Hi, On Mon, 27 Jan 2025 at 16:59, Nazir Bilal Yavuz wrote: > > On Mon, 27 Jan 2025 at 03:52, Michael Paquier wrote: > > > > In 0002, you are removing PendingWalStats.wal_write_time, which does > > not seem OK to me because we should still aggregate this data for >

Re: Show WAL write and fsync stats in pg_stat_io

2025-01-27 Thread Nazir Bilal Yavuz
Hi, On Mon, 27 Jan 2025 at 03:52, Michael Paquier wrote: > > On Fri, Jan 24, 2025 at 06:29:46PM +0300, Nazir Bilal Yavuz wrote: > > By saying that, this does not solve the problem; there is still a > > difference although you omit WAL initialization stats from the > > pg_

Re: Show WAL write and fsync stats in pg_stat_io

2025-01-24 Thread Nazir Bilal Yavuz
by stats_reset;'. By saying that, this does not solve the problem; there is still a difference although you omit WAL initialization stats from the pg_stat_io. -- Regards, Nazir Bilal Yavuz Microsoft

Re: Show WAL write and fsync stats in pg_stat_io

2025-01-24 Thread Nazir Bilal Yavuz
Hi, On Wed, 22 Jan 2025 at 03:14, Michael Paquier wrote: > > On Thu, Jan 16, 2025 at 11:40:51AM +0300, Nazir Bilal Yavuz wrote: > > I encountered another problem while rebasing the patch. The problem is > > basically we do not expect any pending stats while restoring the stats

Re: Make pg_stat_io view count IOs as bytes instead of blocks

2025-01-16 Thread Nazir Bilal Yavuz
//godbolt.org/z/cGosfzGne (we > can see the warning by using the current define and that the warning is gone > with the new define). Thanks all! I checked clang 4 as well on the link you sent and it also fixes the warning there. -- Regards, Nazir Bilal Yavuz Microsoft

Re: Show WAL write and fsync stats in pg_stat_io

2025-01-16 Thread Nazir Bilal Yavuz
Hi, On Fri, 19 Apr 2024 at 11:01, Nazir Bilal Yavuz wrote: > > If I am not missing any new changes, the only problem is reading > variable bytes now. We have discussed a couple of solutions: With the recent commit [1], pg_stat_io tracks IOs as bytes instead of blocks. This solves the

Re: per backend I/O statistics

2025-01-15 Thread Nazir Bilal Yavuz
Hi, On Wed, 15 Jan 2025 at 12:27, Michael Paquier wrote: > > On Wed, Jan 15, 2025 at 08:30:20AM +, Bertrand Drouvot wrote: > > On Wed, Jan 15, 2025 at 11:03:54AM +0300, Nazir Bilal Yavuz wrote: > >> With this commit it may not be possible to count IOs in the critical &

Re: per backend I/O statistics

2025-01-15 Thread Nazir Bilal Yavuz
e problem happens only if the local PgStat_BackendPending entry is being created for the first time for this backend in the critical section. -- Regards, Nazir Bilal Yavuz Microsoft

Re: Make pg_stat_io view count IOs as bytes instead of blocks

2025-01-14 Thread Nazir Bilal Yavuz
Hi, On Tue, 14 Jan 2025 at 06:18, Michael Paquier wrote: > > On Fri, Jan 10, 2025 at 08:23:46AM +, Bertrand Drouvot wrote: > > On Fri, Jan 10, 2025 at 10:15:52AM +0300, Nazir Bilal Yavuz wrote: > >> But I agree that having a macro has more benefits, > >> also th

Re: pgbench error: (setshell) of script 0; execution of meta-command failed

2025-01-10 Thread Nazir Bilal Yavuz
Hi, On Fri, 10 Jan 2025 at 17:58, Tom Lane wrote: > > Nazir Bilal Yavuz writes: > > I ran it ~500 times on HEAD but the issue did not occur on my machine. > > What platform are you testing on? My local machine is: Linux 6.12.8-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.8-1 (

Re: pgbench error: (setshell) of script 0; execution of meta-command failed

2025-01-10 Thread Nazir Bilal Yavuz
you want reproduce it quicker, > you can change the '-T5' to '-T1' in the pgbench command and run many times. I ran it ~500 times on HEAD but the issue did not occur on my machine. Is 500 times not enough? -- Regards, Nazir Bilal Yavuz Microsoft

Re: Make pg_stat_io view count IOs as bytes instead of blocks

2025-01-09 Thread Nazir Bilal Yavuz
one extend > batch, worth (extend_by * BLCKSZ). I agree with you. > > Two times my fault today that the main patch does not apply anymore > (three times at least in total), so I have rebased it myself, and did > an extra review while on it, switching the code to use a macro. That > seems OK here. Please let me know if you have more comments. No worries, thank you for all of these! -- Regards, Nazir Bilal Yavuz Microsoft

Re: Reorder shutdown sequence, to flush pgstats later

2025-01-09 Thread Nazir Bilal Yavuz
n it didn't, recovery will - * occur on next postmaster start.) + * told to shut down. We know it wrote a shutdown checkpoint, + * otherwise we'd still be in PM_SHUTDOWN. s/PM_SHUTDOWN/PM_WAIT_XLOG_SHUTDOWN/ I have these comments for now. I need to study 0006 and 0007 more. -- Regards, Nazir Bilal Yavuz Microsoft

Re: Make pg_stat_io view count IOs as bytes instead of blocks

2025-01-09 Thread Nazir Bilal Yavuz
Hi, On Thu, 9 Jan 2025 at 11:11, Michael Paquier wrote: > > On Thu, Jan 09, 2025 at 10:15:20AM +0300, Nazir Bilal Yavuz wrote: > > I am a bit confused, are you suggesting these two alternatives: > > 1- Making pgstat_count_io_op_n() static and continuing to use > > pgsta

Re: Make pg_stat_io view count IOs as bytes instead of blocks

2025-01-09 Thread Nazir Bilal Yavuz
Hi, On Thu, 9 Jan 2025 at 10:15, Nazir Bilal Yavuz wrote: > > On Thu, 9 Jan 2025 at 05:59, Michael Paquier wrote: > > > > > > +static inline bool > > +is_ioop_tracked_in_bytes(IOOp io_op) > > +{ > > +Assert((unsigned int) io_op < IOOP_NUM

Re: Make pg_stat_io view count IOs as bytes instead of blocks

2025-01-08 Thread Nazir Bilal Yavuz
Hi, Thanks for the review! On Thu, 9 Jan 2025 at 05:59, Michael Paquier wrote: > > On Thu, Dec 26, 2024 at 02:41:26PM +0300, Nazir Bilal Yavuz wrote: > > Thanks! v4 is attached. I quickly tested the pg_stat_get_backend_io() > > function and it seems it is working. > &

Re: Adding NetBSD and OpenBSD to Postgres CI

2025-01-08 Thread Nazir Bilal Yavuz
Hi, Thanks for the review! On Tue, 17 Dec 2024 at 19:21, Andres Freund wrote: > > Hi, > > On 2024-11-12 11:38:11 +0300, Nazir Bilal Yavuz wrote: > > On Fri, 1 Nov 2024 at 21:44, Andres Freund wrote: > > > > +CCACHE_DIR: /tmp/ccache_dir > > >

Re: add support for the old naming libs convention on windows (ssleay32.lib and libeay32.lib)

2024-12-30 Thread Nazir Bilal Yavuz
crypto_lib.found() and is_windows > + crypto_lib = cc.find_library('libeay32', > +dirs: test_lib_d, > +required: openssl_required) > +endif > + > if ssl_lib.found() and crypto_lib.found() >ssl_int = [ssl_lib, crypto_lib] >ssl = declare_dependency(dependencies: ssl_int, include_directories: > postgres_inc) Other than these LGTM. -- Regards, Nazir Bilal Yavuz Microsoft

Re: Make pg_stat_io view count IOs as bytes instead of blocks

2024-12-26 Thread Nazir Bilal Yavuz
Hi, On Tue, 24 Dec 2024 at 09:13, Michael Paquier wrote: > > On Fri, Dec 06, 2024 at 12:41:55PM +0300, Nazir Bilal Yavuz wrote: > > Thanks! I think 'track' is a better word in this context. I used > > 'tracked in ...', as it sounded more correct to me (I ho

Add pg_buffercache_evict_all() and pg_buffercache_mark_dirty[_all]() functions

2024-12-25 Thread Nazir Bilal Yavuz
buffers marked as dirty. I tested these functions with 16GB shared buffers. pg_buffercache_evict() -> 790ms pg_buffercache_evict_all() -> 270ms pg_buffercache_mark_dirty() -> 550ms pg_buffercache_mark_dirty_all() -> 70ms Any feedback would be appreciated. -- Regards, Nazir Bilal Yav

Re: Removing the pgstat_flush_io() call from the walwriter

2024-12-19 Thread Nazir Bilal Yavuz
size of the WAL read IO may vary and it is not possible to show this on the pg_stat_io view. Proposed a fix [2] for that (by counting IOs as bytes instead of blocks in the pg_stat_io) which you already reviewed :). [1] https://commitfest.postgresql.org/51/4950/ [2] https://commitfest.postgresql.org/51/5256/ -- Regards, Nazir Bilal Yavuz Microsoft

Re: Eagerly scan all-visible pages to amortize aggressive vacuum

2024-12-17 Thread Nazir Bilal Yavuz
eeded in both places to explain why vacrel->eager_pages.[remaining_fails | remaining_successes] is set to vacrel->rel_pages + 1 and why vacrel->eagerness should be VAC_EAGER when was_eager_scanned is true and vacrel->eager_pages.remaining_successes is 0. -- Regards, Nazir Bilal Yavuz Microsoft

Re: Count and log pages set all-frozen by vacuum

2024-12-16 Thread Nazir Bilal Yavuz
TYMAP_ALL_FROZEN) == 0) +{ +vacrel->vm_new_visible_pages++; +if (presult.all_frozen) +vacrel->vm_new_visible_frozen_pages++; +} I think there is no need to check VISIBILITYMAP_ALL_FROZEN in these if conditions. If the page is not all-visible; it can not be all-frozen, right? -- Regards, Nazir Bilal Yavuz Microsoft

Re: Why we need to check for local buffers in BufferIsExclusiveLocked and BufferIsDirty?

2024-12-08 Thread Nazir Bilal Yavuz
Hi Srinath, On Sat, 7 Dec 2024 at 11:17, Srinath Reddy Sadipiralla wrote: > > On Fri, 06 Dec 2024 16:40:51 +0530 Nazir Bilal Yavuz > > wrote --- > > LGTM. > > sorry i didn't get,what you meant to say is the assert failure which i said > is correct and doe

Re: Why we need to check for local buffers in BufferIsExclusiveLocked and BufferIsDirty?

2024-12-06 Thread Nazir Bilal Yavuz
yours. > > > - Andres > > > hmm got it,if thats the case, for local buffers lockbuffer will skip > acquiring content lock, so assert will fail in BufferIsDirty. LGTM. Adding Daniil to CC as he too started a similar thread [1]. [1] postgr.es/m/CAJDiXggGznOttwREfyZRE4f7oLRz1%3DjTA4xA7u-t6_8CX7j%3D0g%40mail.gmail.com -- Regards, Nazir Bilal Yavuz Microsoft

Re: Make pg_stat_io view count IOs as bytes instead of blocks

2024-12-06 Thread Nazir Bilal Yavuz
Hi, On Thu, 5 Dec 2024 at 12:13, Bertrand Drouvot wrote: > > Hi, > > On Wed, Dec 04, 2024 at 02:49:11PM +0300, Nazir Bilal Yavuz wrote: > > On Thu, 28 Nov 2024 at 16:39, Bertrand Drouvot > > wrote: > > > > > You are right, no need to have this check; it c

Re: Make pg_stat_io view count IOs as bytes instead of blocks

2024-12-04 Thread Nazir Bilal Yavuz
Hi, Thanks for looking into this! On Thu, 28 Nov 2024 at 16:39, Bertrand Drouvot wrote: > > Hi, > > On Wed, Nov 27, 2024 at 11:08:01AM -0500, Melanie Plageman wrote: > > On Wed, Sep 11, 2024 at 7:19 AM Nazir Bilal Yavuz > > wrote: > > > > > > Currentl

Re: meson missing test dependencies

2024-12-03 Thread Nazir Bilal Yavuz
Hi, On Tue, 3 Dec 2024 at 04:05, Michael Paquier wrote: > > On Mon, Dec 02, 2024 at 01:50:57PM +0300, Nazir Bilal Yavuz wrote: > > On Mon, 2 Dec 2024 at 13:11, Peter Eisentraut wrote: > >> Is there any reason this was not done yet? > > > > This looks useful. I

Re: meson missing test dependencies

2024-12-03 Thread Nazir Bilal Yavuz
Hi, On Mon, 2 Dec 2024 at 22:27, Peter Eisentraut wrote: > > On 02.12.24 11:50, Nazir Bilal Yavuz wrote: > > I applied your patches and the cube example worked. But when I edited > > 'test_json_parser_perf.c' and 'test_json_parser_incremental.c', meson >

Re: Using read stream in autoprewarm

2024-12-02 Thread Nazir Bilal Yavuz
Hi, On Mon, 2 Dec 2024 at 16:30, Andrey M. Borodin wrote: > > > On 2 Dec 2024, at 16:16, Kirill Reshke wrote: > > > > I feel like we are ready to mark this as RFC, WDYT? > > +1 Done. -- Regards, Nazir Bilal Yavuz Microsoft

Re: meson missing test dependencies

2024-12-02 Thread Nazir Bilal Yavuz
x27;, meson did not rebuild. I used the 'meson test -C build --suite setup --suite test_json_parser' command to test test_json_parser. Did I do something wrong? -- Regards, Nazir Bilal Yavuz Microsoft

Re: Use streaming read API in pgstattuple.

2024-12-02 Thread Nazir Bilal Yavuz
Hi, On Fri, 29 Nov 2024 at 20:05, Matheus Alcantara wrote: > > Hi, > > > On 29/11/24 04:28, Nazir Bilal Yavuz wrote: > > -for (; blkno < nblocks; blkno++) > > +p.last_exclusive = nblocks; > > + > > +while (BufferIsValid(

Re: Using read stream in autoprewarm

2024-11-29 Thread Nazir Bilal Yavuz
it is worth adding: + Assert(p->pos > 0 && p->pos < p->max_pos); + + old_blk = &(p->block_info[p->pos - 1]); + cur_blk = &(p->block_info[p->pos]); v4 is attached. -- Regards, Nazir Bilal Yavuz Microsoft From ffd37cc5010d2253b3f843e891df0e75ee8d27a6 Mon

Re: plperl version on the meson setup summary screen

2024-11-29 Thread Nazir Bilal Yavuz
secret sauce to make the version number > appear colorized, either. Maybe some meson guru can tell us how. It seems that we can force ANSI colors: - 'plperl': [perl_dep, perlversion], + 'plperl': [perl_dep, '\033[1;36m@0@\033[0m'.format(perlversion)], But I think this is too hacky. I am not sure if that is the best way or worth it. -- Regards, Nazir Bilal Yavuz Microsoft

Re: Use streaming read API in pgstattuple.

2024-11-28 Thread Nazir Bilal Yavuz
ample at least the read_stream_next_buffer() calls in the collect_corrupt_items() function face the same issue. -- Regards, Nazir Bilal Yavuz Microsoft

Re: Using read stream in autoprewarm

2024-11-27 Thread Nazir Bilal Yavuz
ers. However, for the autoprewarm case; the callback function needs to do additional checks so another callback and the use of this variable are required. v3 is attached. -- Regards, Nazir Bilal Yavuz Microsoft From f5a8faa6ac3f836784c15b776633e7729d95378d Mon Sep 17 00:00:00 2001 From: Nazir Bi

Re: ci: Macos failures due to MacPorts behaviour change

2024-11-27 Thread Nazir Bilal Yavuz
https://cirrus-ci.com/task/4656279002546176 [PG 15] https://cirrus-ci.com/task/5192066743926784 - https://cirrus-ci.com/task/5033544097988608 -- Regards, Nazir Bilal Yavuz Microsoft From 6420ec51c8953c9746fe2eb77a91202f5ced4ed9 Mon Sep 17 00:00:00 2001 From: Nazir Bilal Yavuz Date: Wed, 27 Nov 2024 14:3

Re: Use streaming read API in pgstattuple.

2024-11-26 Thread Nazir Bilal Yavuz
s problem couldn't be found by the CI, you may want to add a test for the pgstat_index function. [1] CREATE EXTENSION pgstattuple; create table test (a int primary key, b int[]); create index test_hashidx on test using hash (b); select pgstattuple(oid) from pg_class where relname = 'test_hashidx'; -- Regards, Nazir Bilal Yavuz Microsoft

Re: meson and check-tests

2024-11-25 Thread Nazir Bilal Yavuz
gt; > accordingly. > > > > Didn't find a CF entry for this. Please create and update. It is added and marked as ready for the committer [1]. There are multiple 'Jian He' users in the commitfest app, so I couldn't add you as a reviewer; please add yourself. [1]

Re: Add a write_to_file member to PgStat_KindInfo

2024-11-20 Thread Nazir Bilal Yavuz
he file */ +if (!kind_info->write_to_file) +continue; + /* if not dropped the valid-entry refcount should exist */ Assert(pg_atomic_read_u32(&ps->refcount) > 0); === -- Regards, Nazir Bilal Yavuz Microsoft

Re: Add a write_to_file member to PgStat_KindInfo

2024-11-20 Thread Nazir Bilal Yavuz
the injection_stats_fixed.c file. Do you think that 'modifying existing functions to take an argument to differentiate whether the kind is default or no-write' would be better? -- Regards, Nazir Bilal Yavuz Microsoft

Re: meson and check-tests

2024-11-20 Thread Nazir Bilal Yavuz
Hi, On Tue, 12 Nov 2024 at 18:13, jian he wrote: > > also played around with v5-0002, works fine. > overall, v5-0001 and v5-0002 works as i expected. Thanks for checking it! -- Regards, Nazir Bilal Yavuz Microsoft

Re: meson and check-tests

2024-11-19 Thread Nazir Bilal Yavuz
Hi, Thanks for checking it! On Tue, 19 Nov 2024 at 15:19, Ashutosh Bapat wrote: > > On Thu, Oct 31, 2024 at 11:28 PM Nazir Bilal Yavuz wrote: > > > > Yes, that is exactly why we have both '--schedule' and '--tests' > > flags. Also, a comment is adde

Re: FW: Building Postgres 17.0 with meson

2024-11-15 Thread Nazir Bilal Yavuz
and compiles Postgres using meson [1]. I guess this may help you. [1] https://github.com/dpage/winpgbuild [1.1] https://github.com/dpage/winpgbuild/actions -- Regards, Nazir Bilal Yavuz Microsoft

  1   2   3   4   >