Re: [HACKERS] logical decoding of two-phase transactions

2021-02-25 Thread Amit Kapila
no restart then it would be cheap to check in the hash table but I don't think it is guaranteed. -- With Regards, Amit Kapila.

Re: [HACKERS] logical decoding of two-phase transactions

2021-02-26 Thread Amit Kapila
On Fri, Feb 26, 2021 at 9:56 AM Amit Kapila wrote: > > On Thu, Feb 25, 2021 at 12:32 PM Peter Smith wrote: > > > > 5. You need to write/sync the spool file at prepare time because after > restart between prepare and commit prepared the changes can be lost > and won'

Re: [PATCH] Note effect of max_replication_slots on subscriber side in documentation.

2021-02-26 Thread Amit Kapila
On Fri, Feb 26, 2021 at 1:53 AM Paul Martinez wrote: > > On Thu, Feb 25, 2021 at 5:31 AM Amit Kapila wrote: > > > > For docs only patch, I have few suggestions: > > 1. On page [1], it is not very clear that we are suggesting to set > > max_replication_slots for orig

Re: repeated decoding of prepared transactions

2021-02-26 Thread Amit Kapila
mns in pg_replication_slots view. > I have recommended above to change this name to initial_consistency_at because there are times when we don't export snapshot and we still set this like when creating slots with CRS_NOEXPORT_SNAPSHOT or when creating via SQL APIs. I am not sure why Ajin neither changed the name nor responded to that comment. What is your opinion? -- With Regards, Amit Kapila.

Re: [HACKERS] logical decoding of two-phase transactions

2021-02-26 Thread Amit Kapila
On Sat, Feb 27, 2021 at 7:31 AM Peter Smith wrote: > > On Fri, Feb 26, 2021 at 9:58 PM Amit Kapila wrote: > > 6. > > + * XXX - Is there a potential timing problem here - e.g. if signal arrives > > + * while executing this then maybe we will set table_states_valid without

Re: repeated decoding of prepared transactions

2021-02-26 Thread Amit Kapila
that LSN there won't be any WAL record or maybe WAL would be overwritten. I think this is primarily for our internal use so let's not expose it. I intend to remove it from the patch unless you have some reason for exposing this to the user. -- With Regards, Amit Kapila.

Re: repeated decoding of prepared transactions

2021-02-26 Thread Amit Kapila
_consistent_point. 3. Ran pgindent, minor changes in comments, and modified the commit message. Let me know what you think about these changes. Next, I'll review your second patch which allows the 2PC option to be enabled only at slot creation time. -- With Regards, Amit Kapila. From b4d4504b644

Re: [PATCH] Note effect of max_replication_slots on subscriber side in documentation.

2021-02-27 Thread Amit Kapila
On Sat, Feb 27, 2021 at 2:47 AM Paul Martinez wrote: > > On Fri, Feb 26, 2021 at 5:22 AM Amit Kapila wrote: > > > > https://www.postgresql.org/docs/devel/logical-replication-config.html > > > > Ah, yep. I added a clause to the end of the sentenc

Re: repeated decoding of prepared transactions

2021-02-27 Thread Amit Kapila
On Sat, Feb 27, 2021 at 11:38 AM Amit Kapila wrote: > > On Fri, Feb 26, 2021 at 4:13 PM Ajin Cherian wrote: > > > > On Fri, Feb 26, 2021 at 7:47 PM Ajin Cherian wrote: > > > > > I've updated snapshot_was_exported_at_ member to pg_replication_slots as &g

Re: locking [user] catalog tables vs 2pc vs logical rep

2021-02-28 Thread Amit Kapila
On Tue, Feb 23, 2021 at 12:00 PM Amit Kapila wrote: > > On Tue, Feb 23, 2021 at 9:33 AM Andres Freund wrote: > > > > On 2021-02-23 09:24:18 +0530, Amit Kapila wrote: > > > Okay, so is it sufficient to add comments in code, or do we want to > > > add something

Re: Update docs of logical replication for commit ce0fdbfe97.

2021-02-28 Thread Amit Kapila
On Fri, Feb 26, 2021 at 8:29 AM Amit Kapila wrote: > > We forgot to update the logical replication configuration settings > page in commit ce0fdbfe97. After commit ce0fdbfe97, table > synchronization workers also started using replication origins to > track the progress and the

Re: repeated decoding of prepared transactions

2021-02-28 Thread Amit Kapila
On Mon, Mar 1, 2021 at 7:23 AM Ajin Cherian wrote: > Pushed, the first patch in the series. -- With Regards, Amit Kapila.

Re: repeated decoding of prepared transactions

2021-03-01 Thread Amit Kapila
t you made similar change in CreateInitDecodingContext when I already suggested the same in my previous email? If we don't make that change then during slot initialization two_phase will always be true even though user passed in as false. It looks inconsistent and even though there is no direct problem due to that but it could be cause of possible problem in future. -- With Regards, Amit Kapila.

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-01 Thread Amit Kapila
On Fri, Feb 26, 2021 at 4:28 PM Amit Kapila wrote: > > On Fri, Feb 26, 2021 at 9:56 AM Amit Kapila wrote: > > > > On Thu, Feb 25, 2021 at 12:32 PM Peter Smith wrote: > > > > > > > 5. You need to write/sync the spool file at prepare time because after

Re: [PATCH] Note effect of max_replication_slots on subscriber side in documentation.

2021-03-01 Thread Amit Kapila
On Sat, Feb 27, 2021 at 2:35 PM Amit Kapila wrote: > > On Sat, Feb 27, 2021 at 2:47 AM Paul Martinez wrote: > > > > On Fri, Feb 26, 2021 at 5:22 AM Amit Kapila wrote: > > > > > > https://www.postgresql.org/docs/devel/logical-replication-config.html > >

Re: repeated decoding of prepared transactions

2021-03-01 Thread Amit Kapila
in the code. Apart from that ran pgindent. The patch looks good to me now. Let me know what do you think? -- With Regards, Amit Kapila. v9-0001-Add-option-to-enable-two_phase-commits-via-pg_cre.patch Description: Binary data

Re: repeated decoding of prepared transactions

2021-03-01 Thread Amit Kapila
On Tue, Mar 2, 2021 at 10:38 AM Ajin Cherian wrote: > > On Tue, Mar 2, 2021 at 3:03 PM Amit Kapila wrote: > > One minor comment: > + > + > + True if the slot is enabled for decoding prepared transactions. > Always > +

Re: Parallel INSERT (INTO ... SELECT ...)

2021-03-02 Thread Amit Kapila
- + Insert on testdef + -> Seq Scan on test_data +(2 rows) + +insert into testdef(a,c,d) select a,a*4,a*8 from test_data; +select * from testdef order by a; + a | b | c | d ++---++ + 1 | 5 | 4 | 8 + 2 | 5 | 8 | 16 + 3 | 5 | 12 | 24 + 4 | 5 | 16 | 32 + 5 | 5 | 20 | 40 + 6 | 5 | 24 | 48 + 7 | 5 | 28 | 56 + 8 | 5 | 32 | 64 + 9 | 5 | 36 | 72 + 10 | 5 | 40 | 80 +(10 rows) + -- With Regards, Amit Kapila.

Re: Track replica origin progress for Rollback Prepared

2021-03-02 Thread Amit Kapila
in, it is good if we get this in as well. -- With Regards, Amit Kapila. v2-0001-Track-replication-origin-progress-for-rollbacks.patch Description: Binary data

Re: repeated decoding of prepared transactions

2021-03-02 Thread Amit Kapila
On Tue, Mar 2, 2021 at 12:43 PM vignesh C wrote: > > On Tue, Mar 2, 2021 at 9:33 AM Amit Kapila wrote: > > > > On Tue, Mar 2, 2021 at 8:20 AM vignesh C wrote: > > > > > > > > > I have a minor comment regarding the below: > >

Re: [PATCH] Note effect of max_replication_slots on subscriber side in documentation.

2021-03-03 Thread Amit Kapila
On Mon, Mar 1, 2021 at 5:32 PM Amit Kapila wrote: > > On Sat, Feb 27, 2021 at 2:35 PM Amit Kapila wrote: > > > > On Sat, Feb 27, 2021 at 2:47 AM Paul Martinez wrote: > > > > > > On Fri, Feb 26, 2021 at 5:22 AM Amit Kapila > > > wrote: > >

Re: Parallel INSERT (INTO ... SELECT ...)

2021-03-03 Thread Amit Kapila
On Wed, Mar 3, 2021 at 12:52 PM Greg Nancarrow wrote: > > On Tue, Mar 2, 2021 at 11:19 PM Amit Kapila wrote: > > > > On Mon, Mar 1, 2021 at 9:08 AM Greg Nancarrow wrote: > > > 2. > > /* > > + * Prepare for entering parallel mode by assigning a > >

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-03 Thread Amit Kapila
ARE messages. Those should be handled by there corresponding apply_handle_* functions. Before processing the messages remote_final_lsn needs to be set as commit_prepared's commit_lsn (aka prepare_data.prepare_lsn) -- With Regards, Amit Kapila.

Re: Parallel INSERT (INTO ... SELECT ...)

2021-03-03 Thread Amit Kapila
g doesn't sound unreasonable to me. So, even, if we want to extend it later by making some checks specific to Inserts/Updates, we can do it at that time. The comments you have at that place are sufficient to tell that in the future we can use those checks for Updates as well. They will need some adjustment if we remove that check but the intent is clear. -- With Regards, Amit Kapila.

Re: Parallel INSERT (INTO ... SELECT ...)

2021-03-04 Thread Amit Kapila
at two places, one in plannedstmt->relationOids and the other in plannedstmt->partitionOids. I guess you have to do that because, in AcquireExecutorLocks, we can't find which relationOids are corresponding to partitionOids, am I right? If so, why not just maintain them in plannedstmt->partitionOids and then make PlanCacheRelCallback consider it? Also, in standard_planner, we should add these partitionOids only for parallel-mode. Thoughts? -- With Regards, Amit Kapila.

Re: Parallel INSERT (INTO ... SELECT ...)

2021-03-04 Thread Amit Kapila
On Thu, Mar 4, 2021 at 4:37 PM Amit Kapila wrote: > > On Fri, Feb 26, 2021 at 10:37 AM Amit Langote wrote: > > > > I realized that there is a race condition that will allow a concurrent > > backend to invalidate a parallel plan (for insert into a partitioned > >

Re: Parallel INSERT (INTO ... SELECT ...)

2021-03-04 Thread Amit Kapila
On Thu, Mar 4, 2021 at 5:24 PM Greg Nancarrow wrote: > > On Thu, Mar 4, 2021 at 10:07 PM Amit Kapila wrote: > > > > >Also, in > > standard_planner, we should add these partitionOids only for > > parallel-mode. > > > > It is doing that in v20 patch (

Re: Track replica origin progress for Rollback Prepared

2021-03-04 Thread Amit Kapila
On Thu, Mar 4, 2021 at 1:40 PM Michael Paquier wrote: > > On Wed, Mar 03, 2021 at 09:06:11AM +0530, Amit Kapila wrote: > That's perhaps not enough to be an actual objection, but I am not > really comfortable with the concept of pushing into the tree code that > would remain un

Re: Track replica origin progress for Rollback Prepared

2021-03-04 Thread Amit Kapila
On Thu, Mar 4, 2021 at 8:03 PM Amit Kapila wrote: > > Having said that, I think we use > replication origins to test it. For example: > > Create Table t1(c1 int); > > SELECT pg_replication_origin_create('regress'); > SELECT pg_replication_origin_

Re: Parallel INSERT (INTO ... SELECT ...)

2021-03-04 Thread Amit Kapila
On Fri, Mar 5, 2021 at 5:07 AM Greg Nancarrow wrote: > > On Thu, Mar 4, 2021 at 11:05 PM Amit Kapila wrote: > > > > On Thu, Mar 4, 2021 at 5:24 PM Greg Nancarrow wrote: > > > > > > On Thu, Mar 4, 2021 at 10:07 PM Amit Kapila > > > wrote: > >

Re: Track replica origin progress for Rollback Prepared

2021-03-04 Thread Amit Kapila
On Fri, Mar 5, 2021 at 7:11 AM Amit Kapila wrote: > > On Thu, Mar 4, 2021 at 8:03 PM Amit Kapila wrote: > > > > I have just checked via code coverage that we don't seem to have tests > for recovery of replication origin advance for commit [1], see > function x

Re: Parallel INSERT (INTO ... SELECT ...)

2021-03-05 Thread Amit Kapila
rds, Amit Kapila.

Re: Parallel INSERT (INTO ... SELECT ...)

2021-03-05 Thread Amit Kapila
On Fri, Mar 5, 2021 at 6:34 PM Greg Nancarrow wrote: > > On Fri, Mar 5, 2021 at 9:35 PM Amit Kapila wrote: > > > > On Fri, Mar 5, 2021 at 8:24 AM Greg Nancarrow wrote: > > > > > > > In patch v21-0003-Add-new-parallel-dml-GUC-and-table-options, we are &

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-06 Thread Amit Kapila
ty prepare" problem. + * + * The following code has a 2-part fix for that scenario. No need to describe it in terms of problem and fix. You can say something like: "This can lead to "empty prepare". We avoid this by " -- With Regards, Amit Kapila.

Re: Parallel INSERT (INTO ... SELECT ...)

2021-03-06 Thread Amit Kapila
y, thanks for bringing this up. -- With Regards, Amit Kapila.

Re: Tablesync early exit

2021-03-06 Thread Amit Kapila
his thread. -- With Regards, Amit Kapila.

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-06 Thread Amit Kapila
onality for 0002. I understand that you have kept them for easier review but I guess at this stage it is better to merge them so that the complete functionality can be reviewed. -- With Regards, Amit Kapila.

Re: Parallel INSERT (INTO ... SELECT ...)

2021-03-06 Thread Amit Kapila
two commits in this function (cfffe83b and 45639a05) also didn't bump the catversion. -- With Regards, Amit Kapila.

Re: Track replica origin progress for Rollback Prepared

2021-03-07 Thread Amit Kapila
On Thu, Mar 4, 2021 at 1:40 PM Michael Paquier wrote: > > On Wed, Mar 03, 2021 at 09:06:11AM +0530, Amit Kapila wrote: > > Michael, I want to push this patch soon unless you have any concerns. > > No concerns from me... > Pushed! I have tried but couldn't come up

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-07 Thread Amit Kapila
On Mon, Mar 8, 2021 at 10:04 AM Peter Smith wrote: > > On Sun, Mar 7, 2021 at 3:00 PM Amit Kapila wrote: > > > > On Sun, Mar 7, 2021 at 7:35 AM Peter Smith wrote: > > > > > > Please find attached the latest patch set v51* > > > > > >

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-08 Thread Amit Kapila
On Mon, Mar 8, 2021 at 1:26 PM Peter Smith wrote: > > On Mon, Mar 8, 2021 at 4:19 PM Amit Kapila wrote: > > > > On Mon, Mar 8, 2021 at 10:04 AM Peter Smith wrote: > > > > > > On Sun, Mar 7, 2021 at 3:00 PM Amit Kapila > > > wrote: > > >

Re: Parallel INSERT (INTO ... SELECT ...)

2021-03-08 Thread Amit Kapila
On Mon, Mar 8, 2021 at 12:55 PM Amit Langote wrote: > > Hi Amit, Greg, > > Sorry, I hadn't noticed last week that some questions were addressed to me. > > On Sat, Mar 6, 2021 at 7:19 PM Amit Kapila wrote: > > Thanks, your changes look good to me. I went ahead and c

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-08 Thread Amit Kapila
; pg_ctl: server does not shut down > > This is because the following process does not exit: > postgres: walsender vignesh 127.0.0.1(41550) START_REPLICATION > > It continuously loops at the below: > What happens if you don't set the two_phase option? If that also leads to the same error then can you please also check this case on the HEAD? -- With Regards, Amit Kapila.

Re: Parallel INSERT (INTO ... SELECT ...)

2021-03-08 Thread Amit Kapila
On Mon, Mar 8, 2021 at 3:54 PM Greg Nancarrow wrote: > > I've attached an updated set of patches with the suggested locking changes. > Amit L, others, do let me know if you have still more comments on 0001* patch or if you want to review it further? -- With Regards, Amit Kapila.

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-08 Thread Amit Kapila
s", subid, gid); I feel it is better to create these in pg_logical/twophase as that is where we store other logical replication related files. -- With Regards, Amit Kapila.

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-08 Thread Amit Kapila
ures, we will have to start > > adding more message types, using meaningful characters might become > > difficult. Should we start using numeric instead for the new feature > > getting added? > > This may or may not become a problem sometime in the future, but I > think the feedback is not really specific to the current patch set so > I am skipping it at this time. > +1. -- With Regards, Amit Kapila.

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-08 Thread Amit Kapila
On Mon, Mar 8, 2021 at 8:09 PM vignesh C wrote: > > On Mon, Mar 8, 2021 at 6:25 PM Amit Kapila wrote: > > > > I think in case of two_phase option, replicatedPtr and sentPtr never > becomes the same which causes this process to hang. > The reason is that because on subsc

Re: Make stream_prepare an optional callback

2021-03-08 Thread Amit Kapila
I also don't see a reason why we need to make this a necessary callback. Some plugin authors might just want 2PC without streaming support. Markus, others working on logical decoding plugins, do you have any opinion on this? -- With Regards, Amit Kapila.

Re: Make stream_prepare an optional callback

2021-03-09 Thread Amit Kapila
On Tue, Mar 9, 2021 at 1:55 PM Markus Wanner wrote: > > On 09.03.21 07:40, Amit Kapila wrote: > > Sounds reasonable to me. I also don't see a reason why we need to make > > this a necessary callback. Some plugin authors might just want 2PC > > without streaming s

Re: Make stream_prepare an optional callback

2021-03-09 Thread Amit Kapila
On Tue, Mar 9, 2021 at 2:23 PM Markus Wanner wrote: > > On 09.03.21 09:39, Amit Kapila wrote: > > It is attached with the initial email. > > Oh, sorry, I looked up the initial email, but still didn't see the patch. > > > I think so. The behavior has to be sim

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-09 Thread Amit Kapila
copy, it won't wait in that state because the tablesync worker will still be in SUBREL_STATE_DATASYNC state and we wait for SUBREL_STATE_SYNCDONE state only after the initial copy is finished. So, I think it is a good idea to call CHECK_FOR_INTERRUPTS in this loop. -- With Regards, Amit Kapila.

Re: Make stream_prepare an optional callback

2021-03-09 Thread Amit Kapila
On Tue, Mar 9, 2021 at 3:41 PM Markus Wanner wrote: > > On 09.03.21 10:37, Amit Kapila wrote: > I guess I don't quite understand the initial motivation for the patch. > It states: "This allows plugins to not allow the enabling of streaming > and two_phase at the same ti

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-09 Thread Amit Kapila
be also used from prepare_spoolfile_replay_messages. 6. I think we should also write some commentary about prepared transactions atop of worker.c as we have done for streamed transactions. -- With Regards, Amit Kapila.

Re: pgsql: Enable parallel SELECT for "INSERT INTO ... SELECT ...".

2021-03-10 Thread Amit Kapila
On Wed, Mar 10, 2021 at 9:07 AM Tom Lane wrote: > > Amit Kapila writes: > > Enable parallel SELECT for "INSERT INTO ... SELECT ...". > > skink (valgrind) is unhappy: > > creating configuration files ... ok > running bootstrap script ... ok > performing pos

Re: Parallel INSERT (INTO ... SELECT ...)

2021-03-10 Thread Amit Kapila
On Mon, Mar 8, 2021 at 7:19 PM Amit Langote wrote: > > Hi Amit > > On Mon, Mar 8, 2021 at 10:18 PM Amit Kapila wrote: > > On Mon, Mar 8, 2021 at 3:54 PM Greg Nancarrow wrote: > > > I've attached an updated set of patches with the suggested locking > > >

Re: [PATCH] Provide more information to filter_prepare

2021-03-10 Thread Amit Kapila
be > nice, I think. > How the proposed 'xid' parameter can be useful? What exactly plugins want to do with it? -- With Regards, Amit Kapila.

Re: [PATCH] Provide more information to filter_prepare

2021-03-10 Thread Amit Kapila
On Wed, Mar 10, 2021 at 4:26 PM Markus Wanner wrote: > > On 10.03.21 11:18, Amit Kapila wrote: > > On Tue, Mar 9, 2021 at 2:14 PM Markus Wanner > > wrote: > >> currently, only the gid is passed on to the filter_prepare callback. > >> While we probably should

Re: [Patch] Optimize dropping of relation buffers using dlist

2021-03-11 Thread Amit Kapila
hange the variable type to uint64. Any better ideas? -- With Regards, Amit Kapila.

Re: [Patch] Optimize dropping of relation buffers using dlist

2021-03-12 Thread Amit Kapila
> OK, attached, to be prepared for the distant future when NBuffers becomes > 64-bit. > Thanks for the patch. Pushed! -- With Regards, Amit Kapila.

Re: Parallel INSERT (INTO ... SELECT ...)

2021-03-12 Thread Amit Kapila
ing out these problems. > > I have tested the triggerdesc bugfix patch with CLOBBER_CACHE_ALWAYS flag. > It passed the testset where is fail in buildfarm (foreign_key, foreign_data). > Thanks for the patch and review. It looks good to me as well and passes the tests (foreign_key, foreign_data) with CLOBBER_CACHE_ALWAYS flag. I'll review the second patch of Greg. -- With Regards, Amit Kapila.

Re: Do we support upgrade of logical replication?

2021-03-12 Thread Amit Kapila
be tricky. So users need to probably recreate the slots and then perform the tablesync again via Alter Subscription ... Refresh Publication. Also, that might require truncating the previous data. I am also not very sure about the procedure but maybe someone else can correct me or add more to it. -- With Regards, Amit Kapila.

Re: Parallel INSERT (INTO ... SELECT ...)

2021-03-12 Thread Amit Kapila
atch on the table which can prove or disprove that theory. The only thing that we can see that even if parallel updates/deletes have overhead, it might not be due to similar reasons. Also, I guess the parallel-copy might need somewhat similar parallel-safety checking w.r.t partitioned tables and I feel the current proposed guc/reloption can be used for the same as it is quite a similar operation. -- With Regards, Amit Kapila.

Re: Parallel INSERT (INTO ... SELECT ...)

2021-03-12 Thread Amit Kapila
ng inserts in parallel-mode. The others seem to be related to update/delete, so we have not done anything, but maybe it is better to mark them as 'unsafe' now, and later when we support the update/delete in parallel-mode, we can see if any of those can be executed in parallel-mode. But OTOH, we can keep them as it is if they don't impact the current operation we have supported in parallel-mode. -- With Regards, Amit Kapila.

Re: [PATCH] Provide more information to filter_prepare

2021-03-13 Thread Amit Kapila
On Thu, Mar 11, 2021 at 2:44 PM Markus Wanner wrote: > > On 11.03.21 04:58, Amit Kapila wrote: > > But this happens when we are decoding prepare, so it is clear that the > > transaction is prepared, why any additional check? > > An output plugin cannot assume the transacti

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-14 Thread Amit Kapila
he state to 'enable' so that future restarts won't send this option again. Now on the publisher side, if this option is present, it will change the value of two_phase_at in the slot to start_decoding_at. I think something on these lines should be much easier than the spool-file implementation unless we see any problem with this idea. -- With Regards, Amit Kapila.

Re: Parallel INSERT (INTO ... SELECT ...)

2021-03-14 Thread Amit Kapila
mments or suggestions for this patch. -- With Regards, Amit Kapila. v28-0001-Add-a-new-GUC-and-a-reloption-to-enable-inserts-.patch Description: Binary data

Re: Consider Parallelism While Planning For REFRESH MATERIALIZED VIEW

2021-03-14 Thread Amit Kapila
fresh materialize view function so that it would be easier for future readers to understand why we have not enabled parallelism for this case. -- With Regards, Amit Kapila.

Re: Parallel INSERT (INTO ... SELECT ...)

2021-03-14 Thread Amit Kapila
On Mon, Mar 15, 2021 at 11:34 AM Justin Pryzby wrote: > > On Mon, Mar 15, 2021 at 11:25:26AM +0530, Amit Kapila wrote: > > On Fri, Mar 12, 2021 at 3:01 PM houzj.f...@fujitsu.com wrote: > > > > > > Attaching new version patch with this change. > > > >

Re: subscriptionCheck failures

2021-03-15 Thread Amit Kapila
On Mon, Mar 15, 2021 at 6:00 PM Thomas Munro wrote: > > Hi, > > This seems to be a new low frequency failure, I didn't see it mentioned > already: > Thanks for reporting, I'll look into it. -- With Regards, Amit Kapila.

Re: subscriptionCheck failures

2021-03-16 Thread Amit Kapila
On Tue, Mar 16, 2021 at 9:00 AM Amit Kapila wrote: > > On Mon, Mar 15, 2021 at 6:00 PM Thomas Munro wrote: > > > > Hi, > > > > This seems to be a new low frequency failure, I didn't see it mentioned > > already: > > > > Thanks for reporting

Re: pg_subscription - substream column?

2021-03-16 Thread Amit Kapila
r someone else is interested in providing a patch for this. -- With Regards, Amit Kapila.

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-16 Thread Amit Kapila
On Mon, Mar 15, 2021 at 6:14 PM Ajin Cherian wrote: > > On Mon, Mar 15, 2021 at 2:04 PM Amit Kapila wrote: >> >> I think something on these lines should be much >> easier than the spool-file implementation unless we see any problem >> with this idea. >> > &

Re: pg_subscription - substream column?

2021-03-16 Thread Amit Kapila
On Tue, Mar 16, 2021 at 5:27 PM Peter Smith wrote: > > On Tue, Mar 16, 2021 at 7:20 PM Amit Kapila wrote: > > > > On Tue, Mar 16, 2021 at 3:35 AM Peter Smith wrote: > > > > > > I noticed that the PG docs [1] for the catalog pg_subscription doesn't >

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-16 Thread Amit Kapila
On Tue, Mar 16, 2021 at 6:22 PM vignesh C wrote: > > On Mon, Mar 15, 2021 at 6:14 PM Ajin Cherian wrote: > > > > On Mon, Mar 15, 2021 at 2:04 PM Amit Kapila wrote: > >> > > 2) table_states_not_ready global variable is used immediately after > c

Re: Resetting spilled txn statistics in pg_stat_replication

2020-10-26 Thread Amit Kapila
On Tue, Oct 27, 2020 at 8:51 AM Justin Pryzby wrote: > > On Fri, Oct 23, 2020 at 10:45:34AM +0530, Amit Kapila wrote: > > On Fri, Oct 23, 2020 at 8:59 AM Amit Kapila wrote: > > > > While updating the streaming stats patch, it occurred to me that we > > can write a

Re: Parallel INSERT (INTO ... SELECT ...)

2020-10-27 Thread Amit Kapila
On Thu, Oct 22, 2020 at 9:47 AM Greg Nancarrow wrote: > > On Fri, Oct 16, 2020 at 9:26 PM Amit Kapila wrote: > > > > Posting an update to the smaller patch (Parallel SELECT for INSERT > INTO...SELECT...). > > Most of this patch feeds into the larger Parallel INSERT p

Re: deferred primary key and logical replication

2020-10-27 Thread Amit Kapila
On Sun, Oct 25, 2020 at 9:39 PM Euler Taveira wrote: > > On Mon, 5 Oct 2020 at 08:34, Amit Kapila wrote: >> >> On Mon, May 11, 2020 at 2:41 AM Euler Taveira >> wrote: >> > >> > Hi, >> > >> > While looking at an old wal2json issue, I st

Re: Resetting spilled txn statistics in pg_stat_replication

2020-10-27 Thread Amit Kapila
On Wed, Oct 28, 2020 at 5:26 AM Tomas Vondra wrote: > > On Tue, Oct 27, 2020 at 06:19:42PM -0500, Justin Pryzby wrote: > >On Tue, Oct 27, 2020 at 09:17:43AM +0530, Amit Kapila wrote: > >> On Tue, Oct 27, 2020 at 8:51 AM Justin Pryzby wrote: > >> > > >>

Re: Track statistics for streaming of in-progress transactions

2020-10-27 Thread Amit Kapila
On Fri, Oct 23, 2020 at 10:24 AM Amit Kapila wrote: > > On Thu, Oct 22, 2020 at 2:09 PM Amit Kapila wrote: > > > > I have fixed the above comment and rebased the patch. I have changed > the docs a bit to add more explanation about the counters. Let me know > if you have a

Re: Parallel copy

2020-10-28 Thread Amit Kapila
read. Please see if you can make these and similar macros as functions unless they are doing few memory instructions. Having functions makes it easier to debug the code as well. [1] - https://www.postgresql.org/message-id/CAJcOf-cgfjj0NfYPrNFGmQJxsnNW102LTXbzqxQJuziar1EKfQ%40mail.gmail.com -- With Regards, Amit Kapila.

Re: Parallel copy

2020-10-29 Thread Amit Kapila
On Thu, Oct 29, 2020 at 11:45 AM Amit Kapila wrote: > > On Tue, Oct 27, 2020 at 7:06 PM vignesh C wrote: > > > [latest version] > > I think the parallel-safety checks in this patch > (v9-0002-Allow-copy-from-command-to-process-data-from-file) are > incomplete and wrong

Re: Track statistics for streaming of in-progress transactions

2020-10-29 Thread Amit Kapila
On Thu, Oct 29, 2020 at 5:16 AM Tomas Vondra wrote: > > On Wed, Oct 28, 2020 at 08:54:53AM +0530, Amit Kapila wrote: > >On Fri, Oct 23, 2020 at 10:24 AM Amit Kapila wrote: > >> > >> On Thu, Oct 22, 2020 at 2:09 PM Amit Kapila > >> wrote: > >>

Re: [HACKERS] logical decoding of two-phase transactions

2020-10-29 Thread Amit Kapila
lbacks.prepare_cb != NULL) || + (ctx->callbacks.commit_prepared_cb != NULL) || + (ctx->callbacks.rollback_prepared_cb != NULL) || + (ctx->callbacks.filter_prepare_cb != NULL); + I think stream_prepare_cb should be checked for the 'twophase' flag because we won't use this unless two-phase is enabled. Am I missing something? -- With Regards, Amit Kapila.

Re: Enumize logical replication message actions

2020-10-29 Thread Amit Kapila
On Fri, Oct 23, 2020 at 6:26 PM Ashutosh Bapat wrote: > > > > On Fri, 23 Oct 2020 at 18:23, Amit Kapila wrote: >> >> On Fri, Oct 23, 2020 at 11:50 AM Kyotaro Horiguchi >> wrote: >> > >> > At Thu, 22 Oct 2020 22:31:41 -0300, Alvaro Herrera >

Re: Parallel INSERT (INTO ... SELECT ...)

2020-10-29 Thread Amit Kapila
On Fri, Oct 30, 2020 at 6:09 AM Greg Nancarrow wrote: > > On Tue, Oct 27, 2020 at 8:56 PM Amit Kapila wrote: > > > > IIUC, below is code for this workaround: > > > > +MaxRelParallelHazardForModify(Oid relid, > > + CmdType commandType, > &

Re: Enumize logical replication message actions

2020-10-29 Thread Amit Kapila
On Fri, Oct 30, 2020 at 10:37 AM Peter Smith wrote: > > On Fri, Oct 30, 2020 at 2:46 PM Amit Kapila wrote: > > Hi Amit > > > You mentioned in the beginning that you prefer to use Enum instead of > > define so that switch cases can detect any remaining items but I have

Re: Enumize logical replication message actions

2020-10-30 Thread Amit Kapila
On Fri, Oct 30, 2020 at 11:50 AM Amit Kapila wrote: > > On Fri, Oct 30, 2020 at 10:37 AM Peter Smith wrote: > > > > IIUC getting rid of the default from the switch can make the missing > > enum detection easier because then you can use -Wswitch option to > >

Re: [HACKERS] logical decoding of two-phase transactions

2020-10-30 Thread Amit Kapila
On Fri, Oct 30, 2020 at 2:46 PM Ajin Cherian wrote: > > On Thu, Oct 29, 2020 at 11:19 PM Amit Kapila wrote: > > > > > > 6. > > +pg_decode_stream_prepare(LogicalDecodingContext *ctx, > > + ReorderBufferTXN *txn, > > + XLogRecPtr prepare_lsn)

Re: Enumize logical replication message actions

2020-10-30 Thread Amit Kapila
On Fri, Oct 30, 2020 at 5:05 PM Ashutosh Bapat wrote: > > > > On Fri, 30 Oct 2020 at 09:16, Amit Kapila wrote >> >> I think we can simply use 'return apply_handle_begin;' instead of >> adding return in another line. Again, I think we changed this handli

Re: Parallel copy

2020-11-01 Thread Amit Kapila
esql.org/message-id/20200413201633.cki4nsptynq7blhg%40alap3.anarazel.de [2] - https://www.postgresql.org/message-id/20200415181913.4gjqcnuzxfzbbzxa%40alap3.anarazel.de [3] - https://www.postgresql.org/message-id/78C0107E-62F2-4F76-BFD8-34C73B716944%40anarazel.de -- With Regards, Amit Kapila.

Re: Enumize logical replication message actions

2020-11-02 Thread Amit Kapila
On Fri, Oct 30, 2020 at 5:52 PM Ashutosh Bapat wrote: > > On Fri, 30 Oct 2020 at 17:37, Amit Kapila wrote: >> >> >> >> Other than that the patch looks good to me. >> > > Patch with updated commit message and also the list of reviewers > Thanks, pushed! -- With Regards, Amit Kapila.

Re: [HACKERS] logical decoding of two-phase transactions

2020-11-02 Thread Amit Kapila
execute invalidations (which are present in top-level txn) for the reasons mentioned in ReorderBufferForget. Also, if we do this, don't forget to update the comment atop ReorderBufferImmediateInvalidation. 3. + /* This is a PREPARED transaction, part of a two-phase commit. + * The full cleanup will happen as part of the COMMIT PREPAREDs, so now + * just truncate txn by removing changes and tuple_cids + */ + ReorderBufferTruncateTXN(rb, txn, true); The first line in the multi-line comment should be empty. -- With Regards, Amit Kapila.

Re: Fix typo in xlogreader.h and procarray.c

2020-11-02 Thread Amit Kapila
h deleted tuples must > > - * Callers supply a page_read callback if they want to to call > + * Callers supply a page_read callback if they want to call > LGTM. I'll push this in some time, thanks. -- With Regards, Amit Kapila.

Re: Fix typo in xlogreader.h and procarray.c

2020-11-02 Thread Amit Kapila
On Tue, Nov 3, 2020 at 8:27 AM Amit Kapila wrote: > > On Tue, Nov 3, 2020 at 7:09 AM Hou, Zhijie wrote: > > > > Hi > > > > I found some possible typo in procarray.c and xlogreader.h > > > > - * For VACUUM separate horizons (used to to decide which d

Re: [HACKERS] logical decoding of two-phase transactions

2020-11-02 Thread Amit Kapila
On Mon, Nov 2, 2020 at 4:11 PM Amit Kapila wrote: > Few Comments on v15-0003-Support-2PC-txn-pgoutput === 1. This patch needs to be rebased after commit 644f0d7cc9 and requires some adjustments accordingly. 2. if (flags != 0) elog(ER

Re: Parallel copy

2020-11-03 Thread Amit Kapila
On Mon, Nov 2, 2020 at 12:40 PM Heikki Linnakangas wrote: > > On 02/11/2020 08:14, Amit Kapila wrote: > > On Fri, Oct 30, 2020 at 10:11 PM Heikki Linnakangas wrote: > >> > >> In this design, you don't need to keep line boundaries in shared memory, > >>

Re: Parallel INSERT (INTO ... SELECT ...)

2020-11-03 Thread Amit Kapila
+} > > Thanks, at least for INSERT, it's not needed, so I'll remove it. > Or you might want to consider moving the check related to IsModifySupportedInParallelMode() inside PrepareParallelModeForModify(). That way the code might look a bit cleaner. -- With Regards, Amit Kapila.

Re: Some doubious code in pgstat.c

2020-11-04 Thread Amit Kapila
at the given "char > *name" has the length of NAMEDATALEN. It actually is, but that > assumption seems a bit bogus. I think it should use strlcpy instead. > Agreed. Your patch looks good to me. -- With Regards, Amit Kapila.

Re: [HACKERS] logical decoding of two-phase transactions

2020-11-04 Thread Amit Kapila
On Wed, Nov 4, 2020 at 3:01 PM Ajin Cherian wrote: > > On Mon, Nov 2, 2020 at 9:40 PM Amit Kapila wrote: > > > > On Wed, Oct 28, 2020 at 10:50 AM Peter Smith wrote: > > 2. > > +DecodePrepare(LogicalDecodingContext *ctx, XLogRecordBuffer *buf, > >

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-11-04 Thread Amit Kapila
SCAN_THRESHOLD then you would anyway have to scan the entire shared buffers so the work done in optimized path for other two relations will add some over head. Also, as written, I think you need to remove the nodes for which you have invalidated the buffers via optimized path, no. -- With Regards, Amit Kapila.

<    1   2   3   4   5   6   7   8   9   10   >