Re: Speed up JSON escape processing with SIMD plus other optimisations

quot;timestamp")); Wouldn't using escape_json_cstring() be better instead? IIUC there isn't much difference between escape_json() and escape_json_cstring(), right? We would avoid strlen() with escape_json_cstring(). Regards, -- Melih Mutlu Microsoft

Re: Parent/child context relation in pg_get_backend_memory_contexts()

of the frequently used cases, not everyone may agree with having an _including_children column for only total_bytes. I'm open to hear more opinions on this. Best Regards, -- Melih Mutlu Microsoft v6-0002-Add-total_bytes_including_children-column.patch Description: Binary data v6-0001-Add-path-column-into-pg_backend_memory_contexts.patch Description: Binary data

Re: Parent/child context relation in pg_get_backend_memory_contexts()

; Note that these IDs are unstable between multiple invocations of the > view. See the example query below for advice on how to use this > column effectively." > Done. > There are also a couple of white space issues with the patch. If > you're in a branch w

Re: Parent/child context relation in pg_get_backend_memory_contexts()

don't think we need this outer while loop. Appending to the end of a queue naturally results in top-to-bottom order anyway, keeping two lists, "queue" and "nextQueue", might not be necessary. I believe that it's safe to append to a list while iterating o

Re: Parent/child context relation in pg_get_backend_memory_contexts()

If we should, how do we explain that the IDs are transient and can change but also may not change if they're closer to TopMemoryContext? If it's better not to mention this in the documentation, does it really matter since most users would not be aware? I've been also thinking if we should still have the parent column, as finding out the parent is also possible via looking into the path. What do you think? Thanks, -- Melih Mutlu Microsoft

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

Hi, Amit Kapila , 6 Tem 2023 Per, 06:56 tarihinde şunu yazdı: > > On Wed, Jul 5, 2023 at 1:48 AM Melih Mutlu wrote: > > > > Hayato Kuroda (Fujitsu) , 4 Tem 2023 Sal, > > 08:42 tarihinde şunu yazdı: > > > > > But in the later patch the tablesync worker tr

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

cant when workers spend less time with actually copying data but more with other stuff like launching workers, opening connections etc. > * 0003 basically improved performance from first two patches Agree, 0003 is definitely a good addition which was missing earlier. Thanks, -- Melih Mutlu Microsoft

Re: Changing types of block and chunk sizes in memory contexts

id of. > > - Afaics AllocSet->keeper is unnecessary these days, as it is always allocated > together with the context itself. Saves 8 bytes. This seemed like a safe change and removed the keeper field in AllocSet and Generation contexts. It saves an additional 8 bytes. Best, -- Melih M

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

at even master beats design#2 in some cases though. Not sure if that is expected or there are some places to improve design#2 even more. What do you think? PS: I only attached the related patches and not the whole patch set. 0001 and 0002 may contain some of your earlier reviews, but I'll send a

Re: Changing types of block and chunk sizes in memory contexts

There's also > the fundamental limitation that MemoryChunk can't store block offsets > larger than 1GBs anyway, so things will go bad if we tried to have > blocks bigger than 1GB. > Right! I don't know why I cast them to Size. Thanks for the fix. Best, -- Melih Mutlu Microsoft

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

to the publisher. I'm not sure how that would also affect the performance if there were any writes. Thanks, -- Melih Mutlu Microsoft

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

.postgresql.org/message-id/CAGPVpCTvALKEXe0%3DN-%2BiMmVxVQ-%2BP8KZ_1qQ1KsSSZ-V9wJ5hw%40mail.gmail.com Thanks for the reminder. -- Melih Mutlu Microsoft

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

#x27;t it decided to use "logical replication worker" regardless of the worker's type [1]. That's why I made this change. If that's not the case here, I'll put it back. [1] https://www.postgresql.org/message-id/flat/CAHut%2BPt1xwATviPGjjtJy5L631SGf3qjV9XUCmxLu16cHamfgg%40mail.gmail.com Thanks, -- Melih Mutlu Microsoft

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

processes > after the streaming is done once. > ~ > Is this paragraph even needed? Since the connection is reused then it > already implies the other end (the Wlasender) is being reused, right? I actually see no harm in explaining this explicitly. Thanks, -- Melih Mutlu Microsoft v21-0001-Refactor-to-split-Apply-and-Tablesync-Workers.patch Description: Binary data v21-0002-Reuse-Tablesync-Workers.patch Description: Binary data v21-0003-Reuse-connection-when-tablesync-workers-change-t.patch Description: Binary data

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

ss ambiguous. > > > > +1. Also, note that they should be in the same order as they are in .c > files. > I did not realize the order is the same with .c files. Good to know. I'll fix it along with other comments. Thanks, -- Melih Mutlu Microsoft

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

tch first but keep all the new function names to > > follow _ style. > > > > Fixing the naming inconsistency will be more far-reaching than just a > few functions affected by these "reuse" patches. There are plenty of > existing functions already inconsistently named in the HEAD code. So > perhaps this topic should be moved to a separate thread? > +1 for moving it to a separate thread. This is not something particularly introduced by this patch. Thanks, -- Melih Mutlu Microsoft

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

Hi, Melih Mutlu , 21 Tem 2023 Cum, 12:47 tarihinde şunu yazdı: > I did not realize the order is the same with .c files. Good to know. I'll > fix it along with other comments. > Addressed the recent reviews and attached the updated patches. Thanks, -- Melih Mutlu Microsoft v22

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

also prefer to assert in the very beginning but am_tablesync_worker and am_leader_apply_worker require MyLogicalRepWorker to be not NULL. And MyLogicalRepWorker is assigned in logicalrep_worker_attach. I can change this if you think there is a better way to check the worker type. Thanks, -- Melih Mutlu Microsoft

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

I placed it into 0002 with a slight change as follows: - send_feedback(last_received, false, false); > + if (!MyLogicalRepWorker->relsync_completed) > + send_feedback(last_received, false, false); IMHO relsync_completed means simply the same with streaming_done, that's why I wanted to check

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

Hi, > PFA an updated version with some of the earlier reviews addressed. Forgot to include them in the previous email. Thanks, -- Melih Mutlu Microsoft v24-0003-Reuse-connection-when-tablesync-workers-change-t.patch Description: Binary data v24-0002-Reuse-Tablesync-Workers.patch Descript

Re: Adding a LogicalRepWorker type field

6,7 @@ logicalrep_worker_detach(void) > { > LogicalRepWorker *w = (LogicalRepWorker *) lfirst(lc); > > - if (isParallelApplyWorker(w)) > + if (is_worker_type(w, TYPE_PARALLEL_APPLY_WORKER)) > logicalrep_worker_stop_internal(w, SIGTERM); >} Regards, -- Melih Mutlu Microsoft

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

nks! Peter Smith , 3 Ağu 2023 Per, 12:06 tarihinde şunu yazdı: > Just to clarify my previous post, I meant we will need new v26* patches > Right. I attached the v26 as you asked. Thanks, -- Melih Mutlu Microsoft v26-0001-Reuse-Tablesync-Workers.patch Description: Binary data v26-0002

Re: Parent/child context relation in pg_get_backend_memory_contexts()

Hi hackers, Melih Mutlu , 16 Haz 2023 Cum, 17:03 tarihinde şunu yazdı: > With this change, here's a query to find how much space used by each > context including its children: > > > WITH RECURSIVE cte AS ( > > SELECT id, total_bytes, id as root, name

Separate memory contexts for relcache and catcache

used_bytes, sum(free_bytes) AS free_bytes, sum(total_bytes) AS total_bytes FROM pg_backend_memory_contexts WHERE name LIKE '%CacheMemoryContext%' OR parent LIKE '%CacheMemoryContext%' GROUP BY name ORDER BY total_bytes DESC; Thanks, -- Melih Mutlu Microsoft 0001-Separate-memory-contexts-for-relcache-and-catcache.patch Description: Binary data

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

t Vignesh shared earlier [1]. It makes sense that those issues can cause this problem here. It just takes a bit of time for me to figure out these things, but I'm working on it. [1] https://www.postgresql.org/message-id/CALDaNm1TA068E2niJFUR9ig%2BYz3-ank%3Dj5%3Dj-2UocbzaDnQPrA%40mail.gmail.com Thanks, -- Melih Mutlu Microsoft

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

hared? Do you think v26-0001 will perform 84% worse than HEAD, if you try again? I just want to be sure that it was not a random thing. Interestingly, I also don't see an improvement in above results as big as in your results when inserts/tx ratio is smaller. Even though it certainly is improved in such cases. Thanks, -- Melih Mutlu Microsoft

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

Hi Amit, Amit Kapila , 6 Ağu 2022 Cmt, 16:01 tarihinde şunu yazdı: > I think there is some basic flaw in slot reuse design. Currently, we > copy the table by starting a repeatable read transaction (BEGIN READ > ONLY ISOLATION LEVEL REPEATABLE READ) and create a slot that > establishes a snapshot

Re: Allow logical replication to copy tables in binary format

Euler Taveira , 11 Ağu 2022 Per, 20:16 tarihinde şunu yazdı: > My main concern is to break a scenario that was previously working (14 -> > 15) but after a subscriber upgrade > it won't (14 -> 16). > Fair concern. Some cases that might break the logical replication with version upgrade would be: 1-

Summary function for pg_buffercache

Hi hackers, Added a pg_buffercache_summary() function to retrieve an aggregated summary information with less cost. It's often useful to know only how many buffers are used, how many of them are dirty etc. for monitoring purposes. This info can already be retrieved by pg_buffercache. The extensio

Re: Mingw task for Cirrus CI

Justin Pryzby , 19 Ağu 2022 Cum, 05:34 tarihinde şunu yazdı: > Inline notes about changes since the last version. > > On Thu, Jul 28, 2022 at 05:44:28PM -0500, Justin Pryzby wrote: > > I think the "only_if" should allow separately running one but not both > of the > > windows instances, like: > >

Re: Allow logical replication to copy tables in binary format

Hello, Andres Freund , 2 Eyl 2022 Cum, 01:25 tarihinde şunu yazdı: > It'd be good to collect some performance numbers justifying this. I'd > expect > decent gains if there's e.g. a bytea or timestamptz column involved. Experimented the binary copy with a quick setup. - Created a "temp" table w

Re: Summary function for pg_buffercache

Hi hackers, I also added documentation changes into the patch. You can find it attached. I would appreciate any feedback about this pg_buffercache_summary function. Best, Melih From 82e92d217dd240a9b6c1184cf29d4718343558b8 Mon Sep 17 00:00:00 2001 From: Melih Mutlu Date: Tue, 9 Aug 2022 16:42

Re: Summary function for pg_buffercache

Hi Aleksander and Nathan, Thanks for your comments. Aleksander Alekseev , 9 Eyl 2022 Cum, 17:36 tarihinde şunu yazdı: > However I'm afraid you can't examine BufferDesc's without taking > locks. This is explicitly stated in buf_internals.h: > > """ > Buffer header lock (BM_LOCKED flag) must be he

Re: Summary function for pg_buffercache

Hello Aleksander, > I'm not sure about what undefined behaviour could harm this badly. > > You are right that in practice nothing wrong will (probably) happen on > x86/x64 architecture with (most?) modern C compilers. This is not true in > the general case though. It's up to the compiler to decide

Re: Allow logical replication to copy tables in binary format

Hi hackers, I just wanted to gently ping to hear what you all think about this patch. Appreciate any feedback/thougths. Thanks, Melih

Re: Summary function for pg_buffercache

Hi, Also I suggest changing the names of the columns in order to make them > consistent with the rest of the system. If you consider pg_stat_activity > and family [1] you will notice that the columns are named > (entity)_(property), e.g. backend_xid, backend_type, client_addr, etc. So > instead of

Re: Summary function for pg_buffercache

Aleksander Alekseev , 20 Eyl 2022 Sal, 13:57 tarihinde şunu yazdı: > There was a missing empty line in pg_buffercache.out which made the > tests fail. Here is a corrected v8 patch. > I was just sending a corrected patch without the missing line. Thanks a lot for all these reviews and the correct

Re: Summary function for pg_buffercache

Hi, Seems like cfbot tests are passing now: https://cirrus-ci.com/build/4727923671302144 Best, Melih Melih Mutlu , 20 Eyl 2022 Sal, 14:00 tarihinde şunu yazdı: > Aleksander Alekseev , 20 Eyl 2022 Sal, 13:57 > tarihinde şunu yazdı: > >> There was a missing empty line in pg_buffer

Re: Summary function for pg_buffercache

Hi Zhang, Those are two different locks. The locks that are taken in the patch are for buffer headers. This locks only the current buffer and makes that particular buffer's info consistent within itself. However, the lock mentioned in the doc is for buffer manager which would prevent changes on a

Re: Mingw task for Cirrus CI

Hi hackers, Justin Pryzby , 5 Eyl 2022 Pzt, 14:50 tarihinde şunu yazdı: > But cfbot should run the Mingw task for this patch's own commitfest > entry. But right now (because cfbot doesn't include the original commit > message/s), it doesn't get run :( > I've been thinking about how to make the

Removing unused parameter in SnapBuildGetOrBuildSnapshot

Hi hackers, Sharing a small patch to remove an unused parameter in SnapBuildGetOrBuildSnapshot function in snapbuild.c With commit 6c2003f8a1bbc7c192a2e83ec51581c018aa162f, SnapBuildBuildSnapshot no longer needs transaction id. This also makes the xid parameter in SnapBuildGetOrBuildSnapshot usel

Re: Summary function for pg_buffercache

Hi, Since header locks are removed again, I put some doc changes and comments back. Thanks, Melih v10-0001-Added-pg_buffercache_summary-function.patch Description: Binary data

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

-++ | patch | 1724.230 ms | 853.894 ms | 601.176 ms | 496.395 ms | ++-+-+-++ So yes, increasing the number of workers makes it faster. But reusing workers can still improve more. [1] https://www.postgresql.org/message-id/CAAKRu_YKGyF%2BsvRQqe1th-mG9xLdzneWgh9H1z1DtypBkawkkw%40mail.gmail.com [2] https://www.postgresql.org/message-id/CAGPVpCRWEVhXa7ovrhuSQofx4to7o22oU9iKtrOgAOtz_%3DY6vg%40mail.gmail.com [3] https://www.postgresql.org/message-id/CAGPVpCRzD-ZZEc9ienhyrVpCzd9AJ7fxE--OFFJBnBg3E0438w%40mail.gmail.com Best, -- Melih Mutlu Microsoft

Re: Refactor ssl tests to avoid using internal PostgreSQL::Test::Cluster methods

be nice to make it look like start()/stop(). What do you think? > command_fails( > [ 'pg_ctl', '-D', $node->data_dir, '-l', $node->logfile, 'restart' ], > 'restart fails with incorrect SSL protocol bounds'); There are two o

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

Hi Peter, Peter Smith , 26 May 2023 Cum, 10:30 tarihinde şunu yazdı: > > On Thu, May 25, 2023 at 6:59 PM Melih Mutlu wrote: > Yes, I was mostly referring to the same as point 1 below about patch > 0001. I guess I just found the concept of mixing A) launching TSW (via > apply

Parent/child context relation in pg_get_backend_memory_contexts()

ries similar to the one in case of total_bytes including children. Maybe, we can also consider adding such frequently used and/or useful information as new fields in pg_get_backend_memory_contexts() too. I appreciate any comment/feedback on this. Thanks, -- Melih Mutlu Microsoft 0001-Adding-id-parent_id-into-pg_backend_memory_contexts.patch Description: Binary data

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

ould be done only at the end of sync workers. How do you think? I tried to move the logicalrep_worker_wakeup call from clean_sync_worker (end of an iteration) to finish_sync_worker (end of sync worker). I made table sync much slower for some reason, then I reverted that change. Maybe I should look

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

mmon) worker.c. Was there some reason > why they cannot be put there? I'm not really against moving those functions to tablesync.c. But what's not clear to me is worker.c. Is it the places to put common functions for all log. rep. workers? Then, what about apply worker? Should we consider a separate file for apply worker too? Thanks, -- Melih Mutlu Microsoft

Changing types of block and chunk sizes in memory contexts

ontexts. I tried to find most of the places that needed to be changed to uint32, but I probably missed some. I can add more places if you feel like it. I would appreciate any feedback. Thanks, -- Melih Mutlu Microsoft 0001-Change-memory-context-fields-to-uint32.patch Description: Binary data

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

? Yeah, maybe going to the publisher for creating a slot or only a snapshot does not really make enough difference. I was hoping that creating only snapshot by an existing replication slot would help the performance. I guess I was either wrong or am missing something in the implementation. The tricky

Re: ANALYZE ONLY

ables and not care about inheritance, then naming this option to SKIP_PARTITIONS as Jelte suggested sounds fine. But that name wouldn't work if this option will affect inheritance tables. Thanks, -- Melih Mutlu Microsoft

Re: ANALYZE ONLY

Melih Mutlu , 20 Ağu 2024 Sal, 19:26 tarihinde şunu yazdı: > Hi Michael, > > Thanks for starting this thread. I've also spent a bit time on this after > reading your first thread on this issue [1] > Forgot to add the reference [1] [1] https://www.postg

Re: ANALYZE ONLY

. But that would require changes in grams.y and could complicate things. So it may not be necessary and we may be fine with just a warning. Regards, -- Melih Mutlu Microsoft

Re: ANALYZE ONLY

tance tables the same way you test ANALYZE ONLY. Lastly, the patch includes an unrelated file (compile_flags.txt) and has whitespace errors when I apply it. Regards, -- Melih Mutlu Microsoft

Re: ANALYZE ONLY

; ! i > nclude_children) There are also some issues with coding conventions in some places (e.g. the space between "!" and "include_children" abode). I think running pgindent would resolve such issue in most places. [1] https://www.postgresql.org/docs/16/sql-createpublication.html Regards, -- Melih Mutlu Microsoft

Re: Parent/child context relation in pg_get_backend_memory_contexts()

Hi David, David Rowley , 15 Tem 2024 Pzt, 14:38 tarihinde şunu yazdı: > On Sat, 13 Jul 2024 at 10:12, Melih Mutlu wrote: > > I updated documentation for path and level columns and also fixed the > tests as level starts from 1. > > Thanks for updating. > > + The path

Do we still need parent column in pg_backend_memory_context?

odesearch.debian.net/search?q=pg_backend_memory_context&literal=1&page=3 Regards, -- Melih Mutlu Microsoft v1-0001-Remove-parent-from-pg_backend_memory_context.patch Description: Binary data

Vectored IO in XLogWrite()

ce I've been struggling with coming up a good test case. Regards, -- Melih Mutlu Microsoft v1-0001-Use-pg_pwritev-in-XlogWrite.patch Description: Binary data

Re: Vectored IO in XLogWrite()

Hi Robert, Thanks for reviewing. Robert Haas , 6 Ağu 2024 Sal, 20:43 tarihinde şunu yazdı: > On Tue, Aug 6, 2024 at 5:36 AM Melih Mutlu wrote: > > I think that we don't have the "contiguous pages" constraint when > writing anymore as we can do vectored IO. It seems

Re: Do we still need parent column in pg_backend_memory_context?

David Rowley , 12 Ağu 2024 Pzt, 06:44 tarihinde şunu yazdı: > I made a few adjustments and pushed the patch. Let's see if anyone > complains. > Thanks David.

Re: Separate memory contexts for relcache and catcache

Hi Jeff, Jeff Davis , 30 Eki 2024 Çar, 01:00 tarihinde şunu yazdı: > On Wed, 2024-04-03 at 16:12 +0300, Melih Mutlu wrote: > > Rebased. PSA. > > Thank you. I missed your patch and came up with a similar patch over > here: > > > https://www.pos

Re: Separate memory contexts for relcache and catcache

AExHW5vLRUk%2B9ZxF4FgaqdfmU2e8JbWES7ijhA0Bd6_bekr%3DKw%40mail.gmail.com [2] https://www.postgresql.org/message-id/CAExHW5skNdLG-kDiKe5k0EHUjc9xumjogHOWtEJKgS_xMB2Vcg%40mail.gmail.com Thanks, -- Melih Mutlu Microsoft

Re: Separate memory contexts for relcache and catcache

t in the future. What are your thoughts? [1] https://www.postgresql.org/message-id/CAExHW5sH4NZnHi4S5ai0uFQgfS_R=rts_+lk5jeeq-dvzwk...@mail.gmail.com Thanks, -- Melih Mutlu Microsoft v5-0001-Separate-memory-contexts-for-caches.patch Description: Binary data v5-0002-Adjusting-cache-memory-cont

Re: speedup COPY TO for partitioned table.

s. I believe something like (PartitionDesc) partdesc->oid should give us the partition OIDs in order. Thanks, -- Melih Mutlu Microsoft

Re: speedup COPY TO for partitioned table.

with the patch. What do you think? Thanks, -- Melih Mutlu Microsoft

Re: Align memory context level numbering in pg_log_backend_memory_contexts()

Hi, torikoshia , 17 Nis 2025 Per, 12:35 tarihinde şunu yazdı: > I guess few people would notice this difference, but I think it's better > to avoid changing it unless there's a good reason to do so. > Personally, I also feel the original formatting better -- especially > because the "xx more chil

<    1   2