Re: Align memory context level numbering in pg_log_backend_memory_contexts()

Hi, torikoshia , 17 Nis 2025 Per, 12:35 tarihinde şunu yazdı: > I guess few people would notice this difference, but I think it's better > to avoid changing it unless there's a good reason to do so. > Personally, I also feel the original formatting better -- especially > because the "xx more chil

Re: speedup COPY TO for partitioned table.

s. I believe something like (PartitionDesc) partdesc->oid should give us the partition OIDs in order. Thanks, -- Melih Mutlu Microsoft

Re: Separate memory contexts for relcache and catcache

t in the future. What are your thoughts? [1] https://www.postgresql.org/message-id/CAExHW5sH4NZnHi4S5ai0uFQgfS_R=rts_+lk5jeeq-dvzwk...@mail.gmail.com Thanks, -- Melih Mutlu Microsoft v5-0001-Separate-memory-contexts-for-caches.patch Description: Binary data v5-0002-Adjusting-cache-memory-cont

Re: speedup COPY TO for partitioned table.

with the patch. What do you think? Thanks, -- Melih Mutlu Microsoft

Re: Separate memory contexts for relcache and catcache

AExHW5vLRUk%2B9ZxF4FgaqdfmU2e8JbWES7ijhA0Bd6_bekr%3DKw%40mail.gmail.com [2] https://www.postgresql.org/message-id/CAExHW5skNdLG-kDiKe5k0EHUjc9xumjogHOWtEJKgS_xMB2Vcg%40mail.gmail.com Thanks, -- Melih Mutlu Microsoft

Re: Separate memory contexts for relcache and catcache

Hi Jeff, Jeff Davis , 30 Eki 2024 Çar, 01:00 tarihinde şunu yazdı: > On Wed, 2024-04-03 at 16:12 +0300, Melih Mutlu wrote: > > Rebased. PSA. > > Thank you. I missed your patch and came up with a similar patch over > here: > > > https://www.pos

Re: ANALYZE ONLY

; ! i > nclude_children) There are also some issues with coding conventions in some places (e.g. the space between "!" and "include_children" abode). I think running pgindent would resolve such issue in most places. [1] https://www.postgresql.org/docs/16/sql-createpublication.html Regards, -- Melih Mutlu Microsoft

Re: ANALYZE ONLY

tance tables the same way you test ANALYZE ONLY. Lastly, the patch includes an unrelated file (compile_flags.txt) and has whitespace errors when I apply it. Regards, -- Melih Mutlu Microsoft

Re: ANALYZE ONLY

. But that would require changes in grams.y and could complicate things. So it may not be necessary and we may be fine with just a warning. Regards, -- Melih Mutlu Microsoft

Re: ANALYZE ONLY

Melih Mutlu , 20 Ağu 2024 Sal, 19:26 tarihinde şunu yazdı: > Hi Michael, > > Thanks for starting this thread. I've also spent a bit time on this after > reading your first thread on this issue [1] > Forgot to add the reference [1] [1] https://www.postg

Re: ANALYZE ONLY

ables and not care about inheritance, then naming this option to SKIP_PARTITIONS as Jelte suggested sounds fine. But that name wouldn't work if this option will affect inheritance tables. Thanks, -- Melih Mutlu Microsoft

Re: Do we still need parent column in pg_backend_memory_context?

David Rowley , 12 Ağu 2024 Pzt, 06:44 tarihinde şunu yazdı: > I made a few adjustments and pushed the patch. Let's see if anyone > complains. > Thanks David.

Re: Vectored IO in XLogWrite()

Hi Robert, Thanks for reviewing. Robert Haas , 6 Ağu 2024 Sal, 20:43 tarihinde şunu yazdı: > On Tue, Aug 6, 2024 at 5:36 AM Melih Mutlu wrote: > > I think that we don't have the "contiguous pages" constraint when > writing anymore as we can do vectored IO. It seems

Vectored IO in XLogWrite()

ce I've been struggling with coming up a good test case. Regards, -- Melih Mutlu Microsoft v1-0001-Use-pg_pwritev-in-XlogWrite.patch Description: Binary data

Do we still need parent column in pg_backend_memory_context?

odesearch.debian.net/search?q=pg_backend_memory_context&literal=1&page=3 Regards, -- Melih Mutlu Microsoft v1-0001-Remove-parent-from-pg_backend_memory_context.patch Description: Binary data

Re: Parent/child context relation in pg_get_backend_memory_contexts()

Hi David, David Rowley , 15 Tem 2024 Pzt, 14:38 tarihinde şunu yazdı: > On Sat, 13 Jul 2024 at 10:12, Melih Mutlu wrote: > > I updated documentation for path and level columns and also fixed the > tests as level starts from 1. > > Thanks for updating. > > + The path

Re: Parent/child context relation in pg_get_backend_memory_contexts()

If we should, how do we explain that the IDs are transient and can change but also may not change if they're closer to TopMemoryContext? If it's better not to mention this in the documentation, does it really matter since most users would not be aware? I've been also thinking if we should still have the parent column, as finding out the parent is also possible via looking into the path. What do you think? Thanks, -- Melih Mutlu Microsoft

Re: Parent/child context relation in pg_get_backend_memory_contexts()

don't think we need this outer while loop. Appending to the end of a queue naturally results in top-to-bottom order anyway, keeping two lists, "queue" and "nextQueue", might not be necessary. I believe that it's safe to append to a list while iterating o

Re: Parent/child context relation in pg_get_backend_memory_contexts()

; Note that these IDs are unstable between multiple invocations of the > view. See the example query below for advice on how to use this > column effectively." > Done. > There are also a couple of white space issues with the patch. If > you're in a branch w

Re: Parent/child context relation in pg_get_backend_memory_contexts()

of the frequently used cases, not everyone may agree with having an _including_children column for only total_bytes. I'm open to hear more opinions on this. Best Regards, -- Melih Mutlu Microsoft v6-0002-Add-total_bytes_including_children-column.patch Description: Binary data v6-0001-Add-path-column-into-pg_backend_memory_contexts.patch Description: Binary data

Re: Speed up JSON escape processing with SIMD plus other optimisations

quot;timestamp")); Wouldn't using escape_json_cstring() be better instead? IIUC there isn't much difference between escape_json() and escape_json_cstring(), right? We would avoid strlen() with escape_json_cstring(). Regards, -- Melih Mutlu Microsoft

Re: Flushing large data immediately in pqcomm

David Rowley , 6 Nis 2024 Cmt, 04:34 tarihinde şunu yazdı: > Does anyone else want to try the attached script on the v5 patch to > see if their numbers are better? > I'm seeing the below results with your script on my machine (). I ran it several times, results were almost similar each time. mas

Re: Flushing large data immediately in pqcomm

Jelte Fennema-Nio , 4 Nis 2024 Per, 16:34 tarihinde şunu yazdı: > On Thu, 4 Apr 2024 at 13:08, Melih Mutlu wrote: > > I changed internal_flush() to an inline function, results look better > this way. > > It seems you also change internal_flush_buffer to be inline (but on

Re: Flushing large data immediately in pqcomm

Hi, Melih Mutlu , 28 Mar 2024 Per, 22:44 tarihinde şunu yazdı: > > On Wed, Mar 27, 2024 at 14:39 David Rowley wrote: >> >> On Fri, 22 Mar 2024 at 12:46, Melih Mutlu wrote: >> can you confirm if the test was done in debug with casserts on? If >> so, it would be m

Re: Parent/child context relation in pg_get_backend_memory_contexts()

that particular name, is actually included in the path. I couldn't find any other information that is unique to each context. Thanks, -- Melih Mutlu Microsoft

Re: Separate memory contexts for relcache and catcache

vignesh C , 27 Oca 2024 Cmt, 06:01 tarihinde şunu yazdı: > On Wed, 3 Jan 2024 at 16:56, Melih Mutlu wrote: > CFBot shows that the patch does not apply anymore as in [1]: > === Applying patches on top of PostgreSQL commit ID > 729439607ad210dbb446e31754e8627d7e3f7dda === > ===

Re: Flushing large data immediately in pqcomm

On Wed, Mar 27, 2024 at 18:54 Robert Haas wrote: > On Wed, Mar 27, 2024 at 7:39 AM David Rowley wrote: > > Robert, I understand you'd like a bit more from this patch. I'm > > wondering if you planning on blocking another committer from going > > ahead with this? Or if you have a reason why the c

Re: Flushing large data immediately in pqcomm

On Wed, Mar 27, 2024 at 14:39 David Rowley wrote: > On Fri, 22 Mar 2024 at 12:46, Melih Mutlu wrote: > > I did all of the above changes and it seems like those resolved the > regression issue. > > Thanks for adjusting the patch. The numbers do look better, but on > lo

Re: Flushing large data immediately in pqcomm

Hi, PSA v3. Jelte Fennema-Nio , 21 Mar 2024 Per, 12:58 tarihinde şunu yazdı: > On Thu, 21 Mar 2024 at 01:24, Melih Mutlu wrote: > > What if I do a simple comparison like PqSendStart == PqSendPointer > instead of calling pq_is_send_pending() > > Yeah, that sounds worth tryi

Re: Flushing large data immediately in pqcomm

Heikki Linnakangas , 14 Mar 2024 Per, 15:46 tarihinde şunu yazdı: > On 14/03/2024 13:22, Melih Mutlu wrote: > > @@ -1282,14 +1283,32 @@ internal_putbytes(const char *s, size_t len) > > if (internal_flush()) > >

Re: Flushing large data immediately in pqcomm

David Rowley , 21 Mar 2024 Per, 00:54 tarihinde şunu yazdı: > On Fri, 15 Mar 2024 at 02:03, Jelte Fennema-Nio > wrote: > > > > On Thu, 14 Mar 2024 at 13:12, Robert Haas wrote: > > > > > > On Thu, Mar 14, 2024 at 7:22 AM Melih Mutlu > wrote: > > &

Re: Flushing large data immediately in pqcomm

buffering at all starts performing better than HEAD. Similarly the patch performs better too as it stops buffering if data does not fit into the buffer. [1] https://www.postgresql.org/message-id/CAGECzQTYUhnC1bO%3DzNiSpUgCs%3DhCYxVHvLD2doXNx3My6ZAC2w%40mail.gmail.com Thanks, -- Melih Mutlu

Re: Flushing large data immediately in pqcomm

he cases. That may even help us to find a better approach than what's already proposed. Just to clarify, I don't think anyone here suggests that the bar should be at "if it can't lose relative to today, it's good enough". IMHO "a change that improves some cases, but regresses nowhere" does not translate to that. Thanks, -- Melih Mutlu Microsoft

Re: Flushing large data immediately in pqcomm

tmessage_noblock() without blocking and copy the data into PqSendBuffer only if the socket would block and cannot send it. Unfortunately, I don't have strong numbers to demonstrate any improvement in perf or timing yet. But I still like to know what would you think about it? Thanks, -- Melih Mutlu Microsoft

Re: Flushing large data immediately in pqcomm

, name, time) SELECT i AS id, repeat('dummy', 1) AS name, NOW() AS time FROM generate_series(1, 100) AS i; [2] rm /tmp/dummy && echo 3 | sudo tee /proc/sys/vm/drop_caches && time psql -d postgres -c "COPY test TO STDOUT;" > /tmp/dummy Thanks, -- Melih Mutlu Microsoft

Re: Improve WALRead() to suck data directly from WAL buffers when possible

age boundary because of the concurrent + * insertions. + */ I'm not sure about pasting this into the most places we call WalRead(). Wouldn't it be better if we mention this somewhere around WALRead() only once? Best, -- Melih Mutlu Microsoft

Re: Parent/child context relation in pg_get_backend_memory_contexts()

o next > to them. > > If we do this, orders of columns in the system view should be the same, > I think. > I've done what you suggested. Also moved "total_bytes_including_children" right after "total_bytes". 14dd0f27d have introduced new macro foreach_int. > It seems to be able to make the code a bit simpler and the commit log > says this macro is primarily intended for use in new code. For example: > Makes sense. Done. Thanks, -- Melih Mutlu Microsoft v5-0001-Adding-id-parent_id-into-pg_backend_memory_contex.patch Description: Binary data

Re: Parent/child context relation in pg_get_backend_memory_contexts()

pg_backend_memory_contex.patch > > ../patch/pg_backend_memory_context_refine/v3-0001-Adding-id-parent_id-into-pg_backend_memory_contex.patch:282: > > trailing whitespace. >select count(*) > 0 > > ../patch/pg_backend_memory_context_refine/v3-0001-Adding-id-parent_id-into-pg_

Re: Separate memory contexts for relcache and catcache

this check everywhere... > That would be nice. Do you have a suggestion about where that right place would be? I'd just delete these comments, they're just pointlessly restating the code. > Done. Thanks, -- Melih Mutlu Microsoft v2-0001-Separate-memory-contexts-for-relcache-and-catcach.patch Description: Binary data

Flushing large data immediately in pqcomm

#x27;dummy', 100) AS name, NOW() AS time FROM generate_series(1, 1) AS i; Thanks, -- Melih Mutlu Microsoft 0001-Flush-large-data-immediately-in-pqcomm.patch Description: Binary data

Re: Parent/child context relation in pg_get_backend_memory_contexts()

( > ( > (SELECT sum(total_bytes) FROM contexts WHERE ARRAY[a.id] <@ path) > + total_bytes > ), > total_bytes > ) AS total_bytes_including_children > FROM contexts a; > I added a "total_bytes_including_children" column as you suggest

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

hared? Do you think v26-0001 will perform 84% worse than HEAD, if you try again? I just want to be sure that it was not a random thing. Interestingly, I also don't see an improvement in above results as big as in your results when inserts/tx ratio is smaller. Even though it certainly is improved in such cases. Thanks, -- Melih Mutlu Microsoft

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

t Vignesh shared earlier [1]. It makes sense that those issues can cause this problem here. It just takes a bit of time for me to figure out these things, but I'm working on it. [1] https://www.postgresql.org/message-id/CALDaNm1TA068E2niJFUR9ig%2BYz3-ank%3Dj5%3Dj-2UocbzaDnQPrA%40mail.gmail.com Thanks, -- Melih Mutlu Microsoft

Separate memory contexts for relcache and catcache

used_bytes, sum(free_bytes) AS free_bytes, sum(total_bytes) AS total_bytes FROM pg_backend_memory_contexts WHERE name LIKE '%CacheMemoryContext%' OR parent LIKE '%CacheMemoryContext%' GROUP BY name ORDER BY total_bytes DESC; Thanks, -- Melih Mutlu Microsoft 0001-Separate-memory-contexts-for-relcache-and-catcache.patch Description: Binary data

Re: Parent/child context relation in pg_get_backend_memory_contexts()

Hi hackers, Melih Mutlu , 16 Haz 2023 Cum, 17:03 tarihinde şunu yazdı: > With this change, here's a query to find how much space used by each > context including its children: > > > WITH RECURSIVE cte AS ( > > SELECT id, total_bytes, id as root, name

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

nks! Peter Smith , 3 Ağu 2023 Per, 12:06 tarihinde şunu yazdı: > Just to clarify my previous post, I meant we will need new v26* patches > Right. I attached the v26 as you asked. Thanks, -- Melih Mutlu Microsoft v26-0001-Reuse-Tablesync-Workers.patch Description: Binary data v26-0002

Re: Adding a LogicalRepWorker type field

6,7 @@ logicalrep_worker_detach(void) > { > LogicalRepWorker *w = (LogicalRepWorker *) lfirst(lc); > > - if (isParallelApplyWorker(w)) > + if (is_worker_type(w, TYPE_PARALLEL_APPLY_WORKER)) > logicalrep_worker_stop_internal(w, SIGTERM); >} Regards, -- Melih Mutlu Microsoft

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

Hi, > PFA an updated version with some of the earlier reviews addressed. Forgot to include them in the previous email. Thanks, -- Melih Mutlu Microsoft v24-0003-Reuse-connection-when-tablesync-workers-change-t.patch Description: Binary data v24-0002-Reuse-Tablesync-Workers.patch Descript

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

I placed it into 0002 with a slight change as follows: - send_feedback(last_received, false, false); > + if (!MyLogicalRepWorker->relsync_completed) > + send_feedback(last_received, false, false); IMHO relsync_completed means simply the same with streaming_done, that's why I wanted to check

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

also prefer to assert in the very beginning but am_tablesync_worker and am_leader_apply_worker require MyLogicalRepWorker to be not NULL. And MyLogicalRepWorker is assigned in logicalrep_worker_attach. I can change this if you think there is a better way to check the worker type. Thanks, -- Melih Mutlu Microsoft

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

Hi, Melih Mutlu , 21 Tem 2023 Cum, 12:47 tarihinde şunu yazdı: > I did not realize the order is the same with .c files. Good to know. I'll > fix it along with other comments. > Addressed the recent reviews and attached the updated patches. Thanks, -- Melih Mutlu Microsoft v22

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

tch first but keep all the new function names to > > follow _ style. > > > > Fixing the naming inconsistency will be more far-reaching than just a > few functions affected by these "reuse" patches. There are plenty of > existing functions already inconsistently named in the HEAD code. So > perhaps this topic should be moved to a separate thread? > +1 for moving it to a separate thread. This is not something particularly introduced by this patch. Thanks, -- Melih Mutlu Microsoft

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

ss ambiguous. > > > > +1. Also, note that they should be in the same order as they are in .c > files. > I did not realize the order is the same with .c files. Good to know. I'll fix it along with other comments. Thanks, -- Melih Mutlu Microsoft

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

processes > after the streaming is done once. > ~ > Is this paragraph even needed? Since the connection is reused then it > already implies the other end (the Wlasender) is being reused, right? I actually see no harm in explaining this explicitly. Thanks, -- Melih Mutlu Microsoft v21-0001-Refactor-to-split-Apply-and-Tablesync-Workers.patch Description: Binary data v21-0002-Reuse-Tablesync-Workers.patch Description: Binary data v21-0003-Reuse-connection-when-tablesync-workers-change-t.patch Description: Binary data

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

#x27;t it decided to use "logical replication worker" regardless of the worker's type [1]. That's why I made this change. If that's not the case here, I'll put it back. [1] https://www.postgresql.org/message-id/flat/CAHut%2BPt1xwATviPGjjtJy5L631SGf3qjV9XUCmxLu16cHamfgg%40mail.gmail.com Thanks, -- Melih Mutlu Microsoft

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

.postgresql.org/message-id/CAGPVpCTvALKEXe0%3DN-%2BiMmVxVQ-%2BP8KZ_1qQ1KsSSZ-V9wJ5hw%40mail.gmail.com Thanks for the reminder. -- Melih Mutlu Microsoft

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

to the publisher. I'm not sure how that would also affect the performance if there were any writes. Thanks, -- Melih Mutlu Microsoft

Re: Changing types of block and chunk sizes in memory contexts

There's also > the fundamental limitation that MemoryChunk can't store block offsets > larger than 1GBs anyway, so things will go bad if we tried to have > blocks bigger than 1GB. > Right! I don't know why I cast them to Size. Thanks for the fix. Best, -- Melih Mutlu Microsoft

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

at even master beats design#2 in some cases though. Not sure if that is expected or there are some places to improve design#2 even more. What do you think? PS: I only attached the related patches and not the whole patch set. 0001 and 0002 may contain some of your earlier reviews, but I'll send a

Re: Changing types of block and chunk sizes in memory contexts

id of. > > - Afaics AllocSet->keeper is unnecessary these days, as it is always allocated > together with the context itself. Saves 8 bytes. This seemed like a safe change and removed the keeper field in AllocSet and Generation contexts. It saves an additional 8 bytes. Best, -- Melih M

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

cant when workers spend less time with actually copying data but more with other stuff like launching workers, opening connections etc. > * 0003 basically improved performance from first two patches Agree, 0003 is definitely a good addition which was missing earlier. Thanks, -- Melih Mutlu Microsoft

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

Hi, Amit Kapila , 6 Tem 2023 Per, 06:56 tarihinde şunu yazdı: > > On Wed, Jul 5, 2023 at 1:48 AM Melih Mutlu wrote: > > > > Hayato Kuroda (Fujitsu) , 4 Tem 2023 Sal, > > 08:42 tarihinde şunu yazdı: > > > > > But in the later patch the tablesync worker tr

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

? Yeah, maybe going to the publisher for creating a slot or only a snapshot does not really make enough difference. I was hoping that creating only snapshot by an existing replication slot would help the performance. I guess I was either wrong or am missing something in the implementation. The tricky

Changing types of block and chunk sizes in memory contexts

ontexts. I tried to find most of the places that needed to be changed to uint32, but I probably missed some. I can add more places if you feel like it. I would appreciate any feedback. Thanks, -- Melih Mutlu Microsoft 0001-Change-memory-context-fields-to-uint32.patch Description: Binary data

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

mmon) worker.c. Was there some reason > why they cannot be put there? I'm not really against moving those functions to tablesync.c. But what's not clear to me is worker.c. Is it the places to put common functions for all log. rep. workers? Then, what about apply worker? Should we consider a separate file for apply worker too? Thanks, -- Melih Mutlu Microsoft

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

ould be done only at the end of sync workers. How do you think? I tried to move the logicalrep_worker_wakeup call from clean_sync_worker (end of an iteration) to finish_sync_worker (end of sync worker). I made table sync much slower for some reason, then I reverted that change. Maybe I should look

Parent/child context relation in pg_get_backend_memory_contexts()

ries similar to the one in case of total_bytes including children. Maybe, we can also consider adding such frequently used and/or useful information as new fields in pg_get_backend_memory_contexts() too. I appreciate any comment/feedback on this. Thanks, -- Melih Mutlu Microsoft 0001-Adding-id-parent_id-into-pg_backend_memory_contexts.patch Description: Binary data

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

Hi Peter, Peter Smith , 26 May 2023 Cum, 10:30 tarihinde şunu yazdı: > > On Thu, May 25, 2023 at 6:59 PM Melih Mutlu wrote: > Yes, I was mostly referring to the same as point 1 below about patch > 0001. I guess I just found the concept of mixing A) launching TSW (via > apply

Re: Refactor ssl tests to avoid using internal PostgreSQL::Test::Cluster methods

be nice to make it look like start()/stop(). What do you think? > command_fails( > [ 'pg_ctl', '-D', $node->data_dir, '-l', $node->logfile, 'restart' ], > 'restart fails with incorrect SSL protocol bounds'); There are two o

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

-++ | patch | 1724.230 ms | 853.894 ms | 601.176 ms | 496.395 ms | ++-+-+-++ So yes, increasing the number of workers makes it faster. But reusing workers can still improve more. [1] https://www.postgresql.org/message-id/CAAKRu_YKGyF%2BsvRQqe1th-mG9xLdzneWgh9H1z1DtypBkawkkw%40mail.gmail.com [2] https://www.postgresql.org/message-id/CAGPVpCRWEVhXa7ovrhuSQofx4to7o22oU9iKtrOgAOtz_%3DY6vg%40mail.gmail.com [3] https://www.postgresql.org/message-id/CAGPVpCRzD-ZZEc9ienhyrVpCzd9AJ7fxE--OFFJBnBg3E0438w%40mail.gmail.com Best, -- Melih Mutlu Microsoft

Re: Subscription statistics are not dropped at DROP SUBSCRIPTION in some cases

transaction fails for some reason, as you explained. IMO, the patch will be OK after commit message is added. Thanks, -- Melih Mutlu Microsoft

Re: pg_stat_io for the startup process

ProcInterrupts take a parameter to decide whether the timer needs to be armed? Or need to add an additional global flag to rearm the timer? Any thoughts? [1] https://api.cirrus-ci.com/v1/artifact/task/5282291971260416/testrun/build/testrun/recovery/010_logical_decoding_timelines/log/010_logical_decoding_timelines_replica.log Best, -- Melih Mutlu Microsoft v2-0001-Add-timeout-to-flush-stats-during-startup-s-main-replay-loop.patch Description: Binary data

Re: pg_stat_io for the startup process

ense to use PGSTAT_MIN/MAX/IDLE_INTERVAL instead? 2- I'm also not sure if this timeout should be registered at the beginning of StartupProcessMain, or does it even make any difference. I tried to do this just before the main redo loop in PerformWalRecovery, but that made the CI red. Best, -- Melih Mutlu Microsoft 0001-Add-timeout-to-flush-stats-during-startup-s-main-replay-loop.patch Description: Binary data

Re: Allow logical replication to copy tables in binary format

Hi, Amit Kapila , 23 Mar 2023 Per, 08:48 tarihinde şunu yazdı: > Pushed the 0001. It may be better to start a separate thread for 0002. > Great! Thanks. Best, -- Melih Mutlu Microsoft

Re: Allow logical replication to copy tables in binary format

it to list_make1. Amit Kapila , 21 Mar 2023 Sal, 12:27 tarihinde şunu yazdı: > > Explaining the issue explicitly with a comment seems better to me than > the trick of changing order of table creation just for some test cases. > > But I'm also ok with removing the use of

Re: Allow logical replication to copy tables in binary format

again. I feel it would be better without using > disable_on_error option in these tests. > While this change would make the test inconsistent, I think it also would make more confusing. Explaining the issue explicitly with a comment seems better to me than the trick of changing order of table creation just for some test cases. But I'm also ok with removing the use of disable_on_error if that's what you agree on. Thanks, -- Melih Mutlu Microsoft

Re: Allow logical replication to copy tables in binary format

Hi, Please see the attached patch. vignesh C , 18 Mar 2023 Cmt, 12:03 tarihinde şunu yazdı: > On Fri, 17 Mar 2023 at 17:55, Melih Mutlu wrote: > 1) Currently we refer the link to the beginning of create subscription > page, this can be changed to refer to binary option contents

Re: Allow logical replication to copy tables in binary format

how didn't include those lines into the patch. Thanks for noticing. Fixed them now. Thanks, -- Melih Mutlu Microsoft v17-0001-Allow-logical-replication-to-copy-table-in-binary.patch Description: Binary data

Re: Allow logical replication to copy tables in binary format

I am fine with supporting this for >= > v16. > Upgrading the subscriber to v16 and keeping the subscriber in v14 could break existing subscriptions. I don't know how likely such a case is. I don't have a strong preference on this. What do you think? Should we change it >=v16 or keep it as it is? Best, -- Melih Mutlu Microsoft

Re: Allow logical replication to copy tables in binary format

would be better to pass the log offset when using wait_for_log, > because otherwise it will check the whole log file to find the target > message, > This might not be a big problem, but it has a risk of getting unexpected > log message > which was generated by previous commands. >

Re: Allow logical replication to copy tables in binary format

d data on the subscriber > +$result = $node_subscriber->safe_psql('postgres', 'SELECT a,b FROM > +public.test_col_order;'); > + > +is( $result, '1|2 > +3|4', 'check synced data on subscriber for different column order'); > Right, it needs to be

Re: Allow logical replication to copy tables in binary format

Amit Kapila , 15 Mar 2023 Çar, 12:31 tarihinde şunu yazdı: > On Tue, Mar 14, 2023 at 4:32 PM Melih Mutlu > wrote: > > What purpose does this test serve w.r.t this patch? Before checking > the sync for different column orders, the patch has already changed > binary to false,

Re: Allow logical replication to copy tables in binary format

introduced until v14 it was decided to check using the later version. > Changed it as suggested here [2]. [1] https://www.postgresql.org/message-id/CAGPVpCTaXYctCUp3z%3D_BstonHiZcC5Jj7584i7B8jeZQq4RJkw%40mail.gmail.com [2] https://www.postgresql.org/message-id/CAA4eK1%2BC7ykvdBxh_t1BdbX5Da1bM1BgsE%3D-i2koPkd3pSid0A%40mail.gmail.com Thanks, -- Melih Mutlu Microsoft

Re: Allow logical replication to copy tables in binary format

th "subscription" for example. Kindly please fix it. > Changed this with Amit's suggestion [1]. [1] https://www.postgresql.org/message-id/CAA4eK1%2BC7ykvdBxh_t1BdbX5Da1bM1BgsE%3D-i2koPkd3pSid0A%40mail.gmail.com Thanks, -- Melih Mutlu Microsoft v14-0001-Allow-logical-replication-to-copy-table-in-binary.patch Description: Binary data

Re: Allow logical replication to copy tables in binary format

d those tests into 014_binary.pl. Also added the case with different column order as Peter suggested. Best, -- Melih Mutlu Microsoft v13-0001-Allow-logical-replication-to-copy-table-in-binary.patch Description: Binary data

Re: Allow logical replication to copy tables in binary format

Hi, Attached v12 with a unified option. Setting binary = true now allows the initial sync to happen in binary format. Thanks, -- Melih Mutlu Microsoft v12-0001-Allow-logical-replication-to-copy-table-in-binary.patch Description: Binary data

Re: Allow logical replication to copy tables in binary format

e version checks? I removed any kind of check since it’s currently a different option. Should we check publisher version before doing binary copy to ensure that the publisher node supports binary option of COPY command? Thanks, -- Melih Mutlu Microsoft

Re: Allow logical replication to copy tables in binary format

is more likely happen to > fail > the initial copy user should enable binary format after synchronization if > tables have original datatype. > I tried to explain when binary copy can cause failures in the doc. What exactly do you think is missing? Best, -- Melih Mutlu Microsoft

Re: Allow logical replication to copy tables in binary format

ll good now with the binary_copy option which is not tied to anything, explanations in the doc and separate tests etc. But I also agree that binary=true should make everything in binary and binary=false should do them in text format. It makes more sense. Best, -- Melih Mutlu Microsoft

Re: Allow logical replication to copy tables in binary format

e no way of testing that the option took effect. > > Another way to test that BINARY is enabled could be to trigger one > of the failure cases. Yes, there is already a failure case for binary copy which resolves with swithcing binary_copy to false. But I also added checks for publisher logs now too. [1] https://www.postgresql.org/docs/devel/sql-copy.html Thanks, -- Melih Mutlu Microsoft v11-0001-Allow-logical-replication-to-copy-table-in-binary.patch Description: Binary data

Re: Allow logical replication to copy tables in binary format

k7Fow%40mail.gmail.com Thanks, -- Melih Mutlu Microsoft v10-0001-Allow-logical-replication-to-copy-table-in-binar.patch Description: Binary data

Re: Allow logical replication to copy tables in binary format

hould allow setting copy_format even if copy_data=false. Another way to deal with this issue could be expecting the user to specify format every time copy_format is needed, similar to the case for copy_data, and moving on with text (default) format if it's not specified for the current CREATE/ALTER SUBSCRIPTION execution. But I don't think this would make things easier. Best, -- Melih Mutlu Microsoft

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

nsistencies between "CREATE_REPLICATION_SLOT" and "CREATE_REPLICATION_SNAPSHOT" are expected since one creates a new replication slot and the other does not. CreateDecodingContext is also used in other places as well. Not sure how this change would affect those places. I'

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

Hi Shveta, Thanks for reviewing. Please see attached patches. shveta malik , 2 Şub 2023 Per, 14:31 tarihinde şunu yazdı: > On Wed, Feb 1, 2023 at 5:37 PM Melih Mutlu wrote: > for (int64 i = 1; i <= lastusedid; i++) > { > charori

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

Hi, Melih Mutlu , 16 Şub 2023 Per, 14:37 tarihinde şunu yazdı: > I see that setting originname in the catalog before actually creating it > causes issues. My concern with setting originname when setting the state to > FINISHEDCOPY is that if worker waits until FINISHEDCOPY to upda

Re: Allow logical replication to copy tables in binary format

Amit Kapila , 16 Şub 2023 Per, 15:47 tarihinde şunu yazdı: > On Mon, Jan 30, 2023 at 4:19 PM Melih Mutlu > wrote: > >> 8. Note that the COPY binary format isn't portable across platforms > >> (Windows to Linux for instance) or major versions > >> htt

Re: Allow logical replication to copy tables in binary format

Hi, Hayato Kuroda (Fujitsu) , 20 Şub 2023 Pzt, 10:12 tarihinde şunu yazdı: > Dear Melih, > > Thank you for updating the patch. Before reviewing, I found that > cfbot have not accepted v8 patch [1]. > Thanks for letting me know. Attached the fixed version of the patch. Best,

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

s limit in case of such failures between origin creation and FINISHEDCOPY. One fix I can think is to update the catalog right after creating a new origin. But this would also require commiting the current transaction to actually persist the originname. I guess this action of commiting the transaction in the middle of initial sync could hurt the copy process. What do you think? Also; working on an updated patch to address your other comments. Thanks again. Best, -- Melih Mutlu Microsoft

Re: Allow logical replication to copy tables in binary format

ou and other reviewers. > > Kindly let me know your status. > Sorry for the delay. This patch is currently one of my priorities. Hopefully I will share quicker updates from now on. [1] https://www.postgresql.org/message-id/CAGPVpCQYi9AYQSS%3DRmGgVNjz5ZEnLB8mACwd9aioVhLmbgiAMA%40mail.gmail.com Thanks, -- Melih Mutlu Microsoft

Re: Allow logical replication to copy tables in binary format

Hi, Please see the attached patch for following changes. Bharath Rupireddy , 30 Oca 2023 Pzt, 15:34 tarihinde şunu yazdı: > On Mon, Jan 30, 2023 at 4:19 PM Melih Mutlu > wrote: It'd be better and clearer to have a separate TAP test file IMO since > the focus of the feature here

  1   2   >