Re: display offset along with block number in vacuum errors

2020-08-25 Thread Amit Kapila
On Wed, Aug 26, 2020 at 8:54 AM Mahendra Singh Thalor wrote: > > On Thu, 20 Aug 2020 at 17:42, Amit Kapila wrote: > > > > Attached are both the patches. The first one is to improve existing > > error context information, so I think we should back-patch to 13. The

Re: Implement UNLOGGED clause for COPY FROM

2020-08-26 Thread Amit Kapila
se again to setup 'replica' as they can't continue replication from the previous point (consider update on a page for which previously WAL was not written). -- With Regards, Amit Kapila.

Re: Parallel copy

2020-08-26 Thread Amit Kapila
xes. > Nice improvement and I think you are right that with larger load data we will get even better improvement. -- With Regards, Amit Kapila.

Re: Implement UNLOGGED clause for COPY FROM

2020-08-26 Thread Amit Kapila
On Thu, Aug 27, 2020 at 7:04 AM tsunakawa.ta...@fujitsu.com wrote: > > From: Amit Kapila > > So you want your users to shutdown and restart the server before Copy > > because that would be required if you want to change the wal_level. > > Yes. They seem to be fine wi

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-08-26 Thread Amit Kapila
On Wed, Aug 26, 2020 at 11:22 PM Jeff Janes wrote: > > > On Tue, Aug 25, 2020 at 8:58 AM Amit Kapila wrote: > >> >> I am planning >> to push the first patch (v53-0001-Extend-the-BufFile-interface) in >> this series tomorrow unless you have any comments on th

Re: Parallel copy

2020-08-27 Thread Amit Kapila
On Thu, Aug 27, 2020 at 4:56 PM vignesh C wrote: > > On Thu, Aug 27, 2020 at 8:24 AM Amit Kapila wrote: > > > > On Thu, Aug 27, 2020 at 8:04 AM Greg Nancarrow wrote: > > > > > > > I have attached new set of patches with the fixes. > > > > Th

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-08-29 Thread Amit Kapila
thing equivalent for uint32? If so, I think we have a couple of options (a) use lcons method and accordingly append the pointer to xid, I think we need to allocate memory for xid if we want to use this idea or (b) use an array instead. What do you think? -- With Regards, Amit Kapila. v54-0001-Fix-the

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-08-30 Thread Amit Kapila
On Sun, Aug 30, 2020 at 2:43 PM Dilip Kumar wrote: > > On Sat, Aug 29, 2020 at 5:18 PM Amit Kapila wrote: > > > > > One more comment for which I haven't done anything yet. > > +static void > > +set_schema_sent_in_streamed_txn(RelationSy

Use T_IntList for uint32

2020-08-31 Thread Amit Kapila
FiTN-u_4uvGjAPO536m-YsR%2Bk9J-%3Dwqx2K9CtrFOHcJPa7Szg%40mail.gmail.com -- With Regards, Amit Kapila.

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-08-31 Thread Amit Kapila
On Mon, Aug 31, 2020 at 1:24 PM Amit Kapila wrote: > > > 2. > 009_stream_simple.pl > +# Insert, update and delete enough rows to exceed the 64kB limit. > +$node_publisher->safe_psql('postgres', q{ > +BEGIN; > +INSERT INTO test_tab SELECT i, md5(i::text)

Re: Use T_IntList for uint32

2020-08-31 Thread Amit Kapila
On Mon, Aug 31, 2020 at 6:58 PM Tom Lane wrote: > > Amit Kapila writes: > > Currently pg_list.h doesn't have a variant for uint32 list (like > > T_UIntList), is there a reason other than that that we don't need it > > till now? > > I'm not in favor

Re: Use T_IntList for uint32

2020-08-31 Thread Amit Kapila
On Mon, Aug 31, 2020 at 5:44 PM Ashutosh Bapat wrote: > > On Mon, Aug 31, 2020 at 4:59 PM Amit Kapila wrote: > > > Now, as far as I can see there is no problem in using T_IntList in > > such usage because we are not going to fetch stored unsigned value as > > a signed

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-08-31 Thread Amit Kapila
On Mon, Aug 31, 2020 at 7:28 PM Dilip Kumar wrote: > > On Mon, Aug 31, 2020 at 1:24 PM Amit Kapila wrote: > > > > On Mon, Aug 31, 2020 at 10:49 AM Amit Kapila > > wrote: > > > > > > On Sun, Aug 30, 2020 at 2:43 PM Dilip Kumar

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-08-31 Thread Amit Kapila
ta discrepancies were observed, attached are the test scenarios > verified. > Thanks, I have pushed the fix (https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=4ab77697f67aa5b90b032b9175b46901859da6d7). -- With Regards, Amit Kapila.

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-09-01 Thread Amit Kapila
On Wed, Sep 2, 2020 at 7:01 AM Kyotaro Horiguchi wrote: > > Hello. > > At Tue, 1 Sep 2020 13:02:28 +, "k.jami...@fujitsu.com" > wrote in > > On Tuesday, August 18, 2020 3:05 PM (GMT+9), Amit Kapila wrote: > > > Today, again thinking about

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-09-01 Thread Amit Kapila
On Tue, Sep 1, 2020 at 8:33 PM Amit Kapila wrote: > > I have fixed all the comments except .. > 3. +# Change the local values of the extra columns on the subscriber, > +# update publisher, and check that subscriber retains the expected > +# values > +$node_subscriber-&g

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-09-02 Thread Amit Kapila
On Wed, Sep 2, 2020 at 9:17 AM Tom Lane wrote: > > Amit Kapila writes: > > Even if the relation is locked, background processes like checkpointer > > can still touch the relation which might cause problems. Consider a > > case where we extend the relation but didn

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-09-02 Thread Amit Kapila
On Wed, Sep 2, 2020 at 3:41 PM Dilip Kumar wrote: > > On Wed, Sep 2, 2020 at 10:55 AM Amit Kapila wrote: > > > > > > > > > We can combine the tests in 015_stream_simple.pl and > > 020_stream_binary.pl as I can't see a good reason to keep them >

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-09-03 Thread Amit Kapila
static void changes_filename(char *path, Oid subid, TransactionId xid); > +static inline void changes_filename(char *path, Oid subid, TransactionId > xid); > Thanks for the report, I'll take care of this. I think the nearby similar function subxact_filename() should also be inline. -- With Regards, Amit Kapila.

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-09-05 Thread Amit Kapila
On Tue, Sep 1, 2020 at 8:33 PM Amit Kapila wrote: > > On Tue, Sep 1, 2020 at 9:28 AM Amit Kapila wrote: > > I have fixed all the comments except the below comments. > 1. verify the size of various tests to ensure that it is above > logical_decoding_work_mem. > 2. I have c

Re: Resetting spilled txn statistics in pg_stat_replication

2020-09-06 Thread Amit Kapila
and 'R' message is fine because once we read GlobalStats, we can return the stats_timestamp. So the other two stands corrected. I understand that this is not directly related to this patch but if you agree we can do this as a separate patch. -- With Regards, Amit Kapila.

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-09-06 Thread Amit Kapila
%40mail.gmail.com -- With Regards, Amit Kapila.

Re: [HACKERS] logical decoding of two-phase transactions

2020-09-07 Thread Amit Kapila
we set ctx->streaming and you can use a similar way to set this parameter. -- With Regards, Amit Kapila.

Re: Transactions involving multiple postgres foreign servers, take 2

2020-09-07 Thread Amit Kapila
are just trying to bypass them to commit whatever we have in the current patch which I find quite surprising. [1] - https://www.postgresql.org/message-id/07b2c899-4ed0-4c87-1327-23c750311248%40postgrespro.ru -- With Regards, Amit Kapila.

Re: Transactions involving multiple postgres foreign servers, take 2

2020-09-07 Thread Amit Kapila
On Tue, Sep 8, 2020 at 8:05 AM Fujii Masao wrote: > > On 2020/09/08 10:34, Amit Kapila wrote: > > On Mon, Sep 7, 2020 at 2:29 PM Fujii Masao > > wrote: > >> > >> IMO it's not easy to commit this 2PC patch at once because it's still large > &

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-09-07 Thread Amit Kapila
On Mon, Sep 7, 2020 at 1:33 PM k.jami...@fujitsu.com wrote: > > On Wednesday, September 2, 2020 5:49 PM, Amit Kapila wrote: > > On Wed, Sep 2, 2020 at 9:17 AM Tom Lane wrote: > > > > > > Amit Kapila writes: > > > > Even if the relation is locked, back

Inconsistency in determining the timestamp of the db statfile.

2020-09-07 Thread Amit Kapila
gresql.org/message-id/CAA4eK1JBqQh9cBKjO-nKOOE%3D7f6ONDCZp0TJZfn4VsQqRZ%2BuYA%40mail.gmail.com -- With Regards, Amit Kapila. v1-0001-Fix-inconsistency-in-determining-the-timestamp-of.patch Description: Binary data

Re: Resetting spilled txn statistics in pg_stat_replication

2020-09-07 Thread Amit Kapila
On Tue, Sep 8, 2020 at 7:53 AM Masahiko Sawada wrote: > > On Mon, 7 Sep 2020 at 15:24, Amit Kapila wrote: > > I'm still going to work on this patch although I might be slow > response this month. > This is a quite fast response. Thanks for staying on top of it. >

Re: Resetting spilled txn statistics in pg_stat_replication

2020-09-08 Thread Amit Kapila
On Tue, Sep 8, 2020 at 7:53 AM Masahiko Sawada wrote: > > On Mon, 7 Sep 2020 at 15:24, Amit Kapila wrote: > > > > This patch needs a rebase. I don't see this patch in the CF app. I > > hope you are still interested in working on this. > > Thank you for review

Re: Inconsistency in determining the timestamp of the db statfile.

2020-09-08 Thread Amit Kapila
On Tue, Sep 8, 2020 at 7:03 PM Magnus Hagander wrote: > > On Tue, Sep 8, 2020 at 3:11 PM Fujii Masao > wrote: >> >> >> >> On 2020/09/08 19:28, Magnus Hagander wrote: >> > >> > >> > On Tue, Sep 8, 2020 at 8:10 AM Amit Kapila > >

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-09-08 Thread Amit Kapila
On Mon, Sep 7, 2020 at 12:57 PM Dilip Kumar wrote: > > On Mon, Sep 7, 2020 at 12:00 PM Amit Kapila wrote: >> >> On Sat, Sep 5, 2020 at 8:55 PM Dilip Kumar wrote: >> > >> > >> > I have reviewed the changes and looks fine to me. >> > >&g

Re: INSERT INTO SELECT, Why Parallelism is not selected?

2020-09-08 Thread Amit Kapila
On Thu, Jul 30, 2020 at 6:42 PM Amit Kapila wrote: > > On Thu, Jul 30, 2020 at 12:02 PM Amit Kapila wrote: > > > > On Wed, Jul 29, 2020 at 7:18 PM Robert Haas wrote: > > > > > > I still don't agree with this as proposed. > > > > > > +

Re: Inconsistency in determining the timestamp of the db statfile.

2020-09-09 Thread Amit Kapila
On Wed, Sep 9, 2020 at 10:54 AM Fujii Masao wrote: > > On 2020/09/09 12:04, Amit Kapila wrote: > > > > No, before patch as well, if we can't read the DB entry say because > > the file is corrupt, we return true and use timestamp of global stats > > file and

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-09-09 Thread Amit Kapila
e of the comments > > Patch attached. > LGTM. -- With Regards, Amit Kapila.

Re: Resetting spilled txn statistics in pg_stat_replication

2020-09-09 Thread Amit Kapila
On Tue, Sep 8, 2020 at 7:02 PM Amit Kapila wrote: > > Comments on the latest patch: > = > Apart from the comments I gave yesterday, another thing I was wondering is how to write some tests for this patch. The two ideas I could think of are as follows: 1.

Re: Inconsistency in determining the timestamp of the db statfile.

2020-09-09 Thread Amit Kapila
On Wed, Sep 9, 2020 at 3:15 PM Magnus Hagander wrote: > > On Wed, Sep 9, 2020 at 5:04 AM Amit Kapila wrote: >> > > Though in fact the one inconsistent place in the code now is that if it is > corrupt in the db entry part of the file it returns true and the global >

Re: Inconsistency in determining the timestamp of the db statfile.

2020-09-09 Thread Amit Kapila
On Wed, Sep 9, 2020 at 3:17 PM Magnus Hagander wrote: > > On Wed, Sep 9, 2020 at 9:11 AM Amit Kapila wrote: >> >> On Wed, Sep 9, 2020 at 10:54 AM Fujii Masao >> wrote: >> > >> > On 2020/09/09 12:04, Amit Kapila wrote: >> > > >> >

Re: Fix for parallel BTree initialization bug

2020-09-09 Thread Amit Kapila
ion > instance.) > > I think I can understand what you are pointing out here but it would be great if you can have a reproducible test case because that will make it apparent and we might want to include that in the regression tests if possible. -- With Regards, Amit Kapila.

Re: Resetting spilled txn statistics in pg_stat_replication

2020-09-09 Thread Amit Kapila
On Wed, Sep 9, 2020 at 3:20 PM Amit Kapila wrote: > > On Tue, Sep 8, 2020 at 7:02 PM Amit Kapila wrote: > > > > Comments on the latest patch: > > = > > > > Apart from the comments I gave yesterday, another thing I was > wonder

Re: Inconsistency in determining the timestamp of the db statfile.

2020-09-09 Thread Amit Kapila
On Wed, Sep 9, 2020 at 9:37 PM Fujii Masao wrote: > > On 2020/09/09 22:57, Magnus Hagander wrote: > > On Wed, Sep 9, 2020 at 3:56 PM Tomas Vondra > <mailto:tomas.von...@2ndquadrant.com>> wrote: > > > > On Wed, Sep 09, 2020 at 03:53:40PM +0530, Amit Kapila

Bug in logical decoding of in-progress transactions

2020-09-09 Thread Amit Kapila
mp;dt=2020-09-09+03%3A42%3A19 [2] - https://www.postgresql.org/message-id/118303.1599691636%40sss.pgh.pa.us -- With Regards, Amit Kapila.

Re: Bug in logical decoding of in-progress transactions

2020-09-09 Thread Amit Kapila
On Thu, Sep 10, 2020 at 11:42 AM Dilip Kumar wrote: > > On Thu, Sep 10, 2020 at 11:29 AM Amit Kapila wrote: >> >> Hi, >> >> There is a recent build farm failure [1] in one of the test_decoding >> tests as pointed by Tom Lane [2]. The failure report

Re: Inconsistency in determining the timestamp of the db statfile.

2020-09-10 Thread Amit Kapila
sg("corrupted statistics file \"%s\"", > statfile))); > goto done; > makes sense, attached find the updated patch. -- With Regards, Amit Kapila. v3-0001-Fix-inconsistency-in-determining-the-timestamp-of.patch Description: Binary data

Re: Inconsistency in determining the timestamp of the db statfile.

2020-09-10 Thread Amit Kapila
On Thu, Sep 10, 2020 at 1:03 PM Magnus Hagander wrote: > > On Thu, Sep 10, 2020 at 9:05 AM Amit Kapila wrote: >> >> On Thu, Sep 10, 2020 at 11:52 AM Masahiko Sawada >> wrote: >> > >> > >> > Regarding the v2 patch, I think we

Re: Bug in logical decoding of in-progress transactions

2020-09-10 Thread Amit Kapila
u", txn->xid); @@ -601,16 +610,15 @@ pg_decode_stream_start(LogicalDecodingContext *ctx, OutputPluginWrite(ctx, true); In this API, we need to use 'last_write' in OutputPluginPrepareWrite() and OutputPluginWrite(). The attached patch fixes both these comments. -- With Regards, Amit Kapila.

Re: Fix for parallel BTree initialization bug

2020-09-10 Thread Amit Kapila
se workers will also update their scan key and move forward from there. So, I am not seeing how this could create a problem. -- With Regards, Amit Kapila.

Re: Inconsistency in determining the timestamp of the db statfile.

2020-09-10 Thread Amit Kapila
On Thu, Sep 10, 2020 at 6:42 PM Alvaro Herrera wrote: > > On 2020-Sep-10, Amit Kapila wrote: > > > On Thu, Sep 10, 2020 at 1:03 PM Magnus Hagander wrote: > > > The comments already say what you said in the second suggestion:"The > > caller must rely on times

Re: Added schema level support for publication.

2021-10-14 Thread Amit Kapila
> ---++-+-+-+---+-- > gregn | f | t | t | t | t | f > Tables: > "sch1.sale_201901" > "sch1.sale_201902" > Tables from schemas: > "sch" > I don't see any problem with this. Do you have a specific problem in mind due to this? -- With Regards, Amit Kapila.

Re: Added schema level support for publication.

2021-10-17 Thread Amit Kapila
On Fri, Oct 15, 2021 at 6:45 AM Greg Nancarrow wrote: > > On Thu, Oct 14, 2021 at 9:59 PM Amit Kapila wrote: > > > > > If partitions belong to a different schema than the parent partitioned > > > table, then the current patch implementation allows the parti

Re: pgsql: Document XLOG_INCLUDE_XID a little better

2021-10-17 Thread Amit Kapila
On Thu, Oct 7, 2021 at 10:46 AM Amit Kapila wrote: > > On Sat, Oct 2, 2021 at 4:16 PM Dilip Kumar wrote: > > > > On Fri, Oct 1, 2021 at 6:24 PM Alvaro Herrera > > wrote: > > > > > > On 2021-Oct-01, Amit Kapila wrote: > > > > > I think

Re: Data is copied twice when specifying both child and parent table in publication

2021-10-17 Thread Amit Kapila
. We might want to consider the case where a user has toggled the "publish_via_partition_root" parameter. It seems both these behaviours/problems exist since commit 17b9e7f9 (Support adding partitioned tables to publication). Adding Amit L and Peter E (people involved in this work) to know their opinion? -- With Regards, Amit Kapila.

Re: [BUG] Unexpected action when publishing partition tables

2021-10-17 Thread Amit Kapila
On Wed, Oct 13, 2021 at 9:11 AM Amit Langote wrote: > > Hi Amit, > > On Fri, Oct 8, 2021 at 12:47 PM Amit Kapila wrote: > > On Thu, Oct 7, 2021 at 12:39 PM Amit Langote > > wrote: > > > Sorry that I didn't comment on this earlier, but I think either >

Re: [BUG] Unexpected action when publishing partition tables

2021-10-17 Thread Amit Kapila
On Mon, Oct 18, 2021 at 12:24 PM Amit Kapila wrote: > > On Wed, Oct 13, 2021 at 9:11 AM Amit Langote wrote: > > Anyway, I'll think a bit more about the possible hazards of not doing > > the locking and will reply again if there's indeed a problem(s) that > >

Re: Skipping logical replication transactions on subscriber side

2021-10-18 Thread Amit Kapila
COMMIT instead. I > > think that this happens even by using ‘SET (‘streaming’ = off)’. > You are right. Yes, I checked that the apply worker did exit > and the new apply worker process dealt with the INSERT in the above case. > Also, setting streaming = false was same. > I think you can additionally verify that temporary streaming files get removed after restart. -- With Regards, Amit Kapila.

Re: Skipping logical replication transactions on subscriber side

2021-10-18 Thread Amit Kapila
transaction ids in > another installation is different. > This is a xid of publisher which subscriber wants to skip. So, even if one restores the subscriber data in a different installation why would it matter till it points to the same publisher? Either way, can't we handle this in pg_dump? -- With Regards, Amit Kapila.

Re: Data is copied twice when specifying both child and parent table in publication

2021-10-18 Thread Amit Kapila
On Mon, Oct 18, 2021 at 2:32 PM Amit Langote wrote: > > On Mon, Oct 18, 2021 at 3:00 PM Amit Kapila wrote: > > On Mon, Oct 18, 2021 at 8:27 AM Greg Nancarrow wrote: > > > > > > On Sat, Oct 16, 2021 at 5:30 PM houzj.f...@fujitsu.com > > > wrote: > >

Re: Failed transaction statistics to measure the logical replication progress

2021-10-18 Thread Amit Kapila
On Thu, Oct 14, 2021 at 9:23 AM houzj.f...@fujitsu.com wrote: > > On Thursday, September 30, 2021 12:15 PM Amit Kapila > > > > These all views are related to untransmitted to the collector but what > > we really need is a view similar to pg_stat_archiver or > >

Re: relation OID in ReorderBufferToastReplace error message

2021-10-18 Thread Amit Kapila
On Fri, Oct 15, 2021 at 3:40 AM Bossart, Nathan wrote: > > On 9/23/21, 11:26 AM, "Alvaro Herrera" wrote: > > On 2021-Sep-23, Jeremy Schneider wrote: > > > >> On 9/22/21 20:11, Amit Kapila wrote: > >> > > >> > On Thu, Sep 23, 2021 at

Re: Data is copied twice when specifying both child and parent table in publication

2021-10-18 Thread Amit Kapila
On Mon, Oct 18, 2021 at 2:58 PM Amit Kapila wrote: > > On Mon, Oct 18, 2021 at 2:32 PM Amit Langote wrote: > > > > Looking at this command in Hou-san's email: > > > > create publication pub for table tbl1, tbl1_part1 with > > (publish_via_partition_ro

Re: Skipping logical replication transactions on subscriber side

2021-10-18 Thread Amit Kapila
On Tue, Oct 19, 2021 at 8:23 AM Masahiko Sawada wrote: > > On Mon, Oct 18, 2021 at 6:07 PM Amit Kapila wrote: > > > > On Mon, Oct 11, 2021 at 1:00 PM Masahiko Sawada > > wrote: > > > > > > On Sun, Oct 10, 2021 at 11:04 PM Peter Eisentraut >

Re: Data is copied twice when specifying both child and parent table in publication

2021-10-18 Thread Amit Kapila
On Tue, Oct 19, 2021 at 8:45 AM Amit Kapila wrote: > > On Mon, Oct 18, 2021 at 2:58 PM Amit Kapila wrote: > > > > On Mon, Oct 18, 2021 at 2:32 PM Amit Langote > > wrote: > > > > > > Looking at this command in Hou-san's email: > > > > &

Re: Added schema level support for publication.

2021-10-18 Thread Amit Kapila
> was > not assigned yet. I think we should check it later. > It seems this got broken in yesterday's patch version. Do you think it is a good idea to add a test for this case? -- With Regards, Amit Kapila.

Re: Added schema level support for publication.

2021-10-19 Thread Amit Kapila
the tables present in + the schema "production": + +CREATE PUBLICATION production_publication FOR TABLE users, departments, ALL TABLES IN SCHEMA production; + + + + + Create a publication that publishes all changes for all the tables present in + the schemas "marketing" and "sales": It is better to use before and after schema names in above descriptions. -- With Regards, Amit Kapila.

Re: Added schema level support for publication.

2021-10-20 Thread Amit Kapila
be some timing-dependent tests in that file. [1] - https://www.postgresql.org/message-id/OS0PR01MB57167F45D481F78CDC5986F794B99%40OS0PR01MB5716.jpnprd01.prod.outlook.com -- With Regards, Amit Kapila. change_partitions_schema_interaction_1.patch Description: Binary data

Re: Data is copied twice when specifying both child and parent table in publication

2021-10-20 Thread Amit Kapila
On Wed, Oct 20, 2021 at 1:32 PM Dilip Kumar wrote: > > On Wed, Oct 20, 2021 at 12:44 PM Greg Nancarrow wrote: > > > > On Mon, Oct 18, 2021 at 5:00 PM Amit Kapila wrote: > > > > > > > I have not debugged it yet to find out why, but with the patch >

Re: LogicalChanges* and LogicalSubxact* wait events are never reported

2021-10-20 Thread Amit Kapila
le interface and forgot to remove these events which were required by the previous versions of the patch. I'll take care of this. Thanks for the report and patch! -- With Regards, Amit Kapila.

Re: Data is copied twice when specifying both child and parent table in publication

2021-10-20 Thread Amit Kapila
On Wed, Oct 20, 2021 at 3:03 PM Greg Nancarrow wrote: > > On Wed, Oct 20, 2021 at 7:59 PM Amit Kapila wrote: > > > > > > Actually, at least with the scenario I gave steps for, after looking > > > > at it again and debugging, I think that the behavior is

Re: pgsql: Document XLOG_INCLUDE_XID a little better

2021-10-20 Thread Amit Kapila
On Wed, Oct 20, 2021 at 10:25 AM Dilip Kumar wrote: > > On Mon, Oct 18, 2021 at 10:48 AM Amit Kapila wrote: > > > > > Today, I have looked at this patch again and slightly changed a > > comment, one of the function name and variable name. Do, let me know > > if y

Re: Question about building an exportable snapshop

2021-10-20 Thread Amit Kapila
still see any problem? -- With Regards, Amit Kapila.

Re: pgsql: Document XLOG_INCLUDE_XID a little better

2021-10-20 Thread Amit Kapila
se is in the critical section and other is not. OTOH, we can move MarkCurrentTransactionIdLoggedIfAny() out of the critical section in this patch itself but that appears like an unrelated change and we may or may not want to back-patch the same. -- With Regards, Amit Kapila.

Re: Failed transaction statistics to measure the logical replication progress

2021-10-21 Thread Amit Kapila
On Thu, Oct 21, 2021 at 6:49 AM Masahiko Sawada wrote: > > On Mon, Oct 18, 2021 at 7:03 PM Amit Kapila wrote: > > > > On Thu, Oct 14, 2021 at 9:23 AM houzj.f...@fujitsu.com > > wrote: > > > > > > On Thursday, September 30, 2021 12:15 PM Amit Kapila

Re: LogicalChanges* and LogicalSubxact* wait events are never reported

2021-10-21 Thread Amit Kapila
On Wed, Oct 20, 2021 at 3:46 PM Masahiro Ikeda wrote: > > On 2021/10/20 18:17, Amit Kapila wrote: > > On Wed, Oct 20, 2021 at 10:50 AM Michael Paquier > > wrote: > >> > >> On Wed, Oct 20, 2021 at 02:12:20PM +0900, Masahiro Ikeda wrote: > >>> If

Re: relation OID in ReorderBufferToastReplace error message

2021-10-21 Thread Amit Kapila
On Mon, Oct 18, 2021 at 3:59 PM Amit Kapila wrote: > > On Fri, Oct 15, 2021 at 3:40 AM Bossart, Nathan wrote: > > > > On 9/23/21, 11:26 AM, "Alvaro Herrera" wrote: > > > On 2021-Sep-23, Jeremy Schneider wrote: > > > > > >> On 9/22/21

Re: Data is copied twice when specifying both child and parent table in publication

2021-10-21 Thread Amit Kapila
On Wed, Oct 20, 2021 at 7:11 PM Greg Nancarrow wrote: > > On Wed, Oct 20, 2021 at 9:19 PM Amit Kapila wrote: > > > > I don't see why data need to be replicated again even in that case. > > Can you see any such duplicate data replicated for non-partitioned > &g

Re: Added schema level support for publication.

2021-10-21 Thread Amit Kapila
ake the handling of T_PublicationObjSpec before T_PublicationTable in equal() function as that is the way nodes are defined. -- With Regards, Amit Kapila.

Re: logical decoding/replication: new functions pg_ls_logicaldir and pg_ls_replslotdir

2021-10-22 Thread Amit Kapila
ING-PG-STAT-REPLICATION-SLOTS-VIEW -- With Regards, Amit Kapila.

Re: Added schema level support for publication.

2021-10-24 Thread Amit Kapila
ng for the "For Table" case as well and it seems to be clarified by "publish_via_partition_root" definition in Create Publication docs [1]. [1] - https://www.postgresql.org/docs/devel/sql-createpublication.html -- With Regards, Amit Kapila.

Re: Added schema level support for publication.

2021-10-24 Thread Amit Kapila
need the second and third parameters in pg_dump.c/getPublicationNamespaces() as those are not getting used. -- With Regards, Amit Kapila.

Re: Added schema level support for publication.

2021-10-25 Thread Amit Kapila
On Mon, Oct 25, 2021 at 1:11 PM vignesh C wrote: > > On Mon, Oct 25, 2021 at 10:52 AM Amit Kapila wrote: > > > > On Fri, Oct 22, 2021 at 8:56 PM vignesh C wrote: > > > > > > > I am getting a compilation error in the latest patch on HEAD. I think > &

Re: pgsql: Document XLOG_INCLUDE_XID a little better

2021-10-25 Thread Amit Kapila
On Thu, Oct 21, 2021 at 11:20 AM Dilip Kumar wrote: > > On Thu, Oct 21, 2021 at 9:11 AM Amit Kapila wrote: > > > > v5-0001, incorporates all the comment fixes suggested by Alvaro. and > 0001 is an additional patch which moves > MarkCurrentTransactionIdLoggedIfAny(), out

Re: Next Steps with Hash Indexes

2021-10-25 Thread Amit Kapila
ases before we allow multiple columns for some cases. The user will > already get to compare btree-vs-hash before they use them in a > particular use case. The perfect should not be the enemy of the good. > > Storing multiple hashes uses more space and is more complex. > I agree that storing trailing columns (except the first one) as payload uses more space but it will save heap fetches in many cases. Apart from search, even for unique key insertion, we need to do heap fetches as we can only verify the other values after fetching the row from the heap. Now, here I feel the question is do we want to save random heap I/O or save extra space in a hash? I think both approaches have pros and cons but probably saving heap I/O appears to be important especially for unique index checks where we need to hold bucket lock as well. -- With Regards, Amit Kapila.

Re: Next Steps with Hash Indexes

2021-10-25 Thread Amit Kapila
length. > > Since hash indexes never store Nulls, and the hash is always fixed > length, ISTM that we can compress the hash index entries down to > ItemPointerData (6 bytes) plus any hashes. > Nice observation but we use INDEX_AM_RESERVED_BIT (as INDEX_MOVED_BY_SPLIT_MASK) for hash indexes, so we need to take care of that in some way. -- With Regards, Amit Kapila.

Re: pgsql: Remove unused wait events.

2021-10-25 Thread Amit Kapila
to back-patch it. > > > > Yeah, exactly. I don't see any benefit that's commensurate with > > even a small risk of breaking extensions --- and apparently, in > > this case that's not a risk but a certainty. > > +1. > I agree with the points raised here and will revert this for v14. -- With Regards, Amit Kapila.

Re: pgsql: Document XLOG_INCLUDE_XID a little better

2021-10-25 Thread Amit Kapila
On Mon, Oct 25, 2021 at 4:21 PM Amit Kapila wrote: > > On Thu, Oct 21, 2021 at 11:20 AM Dilip Kumar wrote: > > > > On Thu, Oct 21, 2021 at 9:11 AM Amit Kapila wrote: > > > > > > > v5-0001, incorporates all the comment fixes suggested by Alvaro. and >

Re: Skipping logical replication transactions on subscriber side

2021-10-25 Thread Amit Kapila
blems if not used carefully so maybe it is better to provide it only by special functions. Having said that, I still feel we should do it via Alter Subscription in some way as that will be convenient to use. [1] - https://www.postgresql.org/message-id/CAA4eK1%2BBOHXC%3D0S2kA7GkErWq3-QKj34oQvwAPfuTHq%3Depf34w%40mail.gmail.com -- With Regards, Amit Kapila.

Re: Gather performance analysis

2021-10-26 Thread Amit Kapila
e patch helps, I have committed > the patch. > Can we mark the corresponding CF entry [1] as committed? [1] - https://commitfest.postgresql.org/35/3304/ -- With Regards, Amit Kapila.

Re: Skipping logical replication transactions on subscriber side

2021-10-26 Thread Amit Kapila
On Tue, Oct 26, 2021 at 2:27 PM Greg Nancarrow wrote: > > On Tue, Oct 26, 2021 at 5:16 PM Amit Kapila wrote: > > > > I agree that we will need a separate syntax for conflict resolution > > but there is some similarity in what I proposed above (On > > Error/Conflict

Re: pgsql: Remove unused wait events.

2021-10-26 Thread Amit Kapila
On Tue, Oct 26, 2021 at 7:34 PM Markus Wanner wrote: > > On 26.10.21 04:20, Amit Kapila wrote: > > I agree with the points raised here and will revert this for v14. > > Thanks, Amit. I appreciate the revert. > > Note that the removed events were almost at the end of Wa

Re: Skipping logical replication transactions on subscriber side

2021-10-26 Thread Amit Kapila
On Wed, Oct 27, 2021 at 8:32 AM Masahiko Sawada wrote: > > On Tue, Oct 26, 2021 at 7:29 PM Amit Kapila wrote: > > > > > > You have a point. The other alternatives on this line could be: > > > > Alter Subscription SKIP ( subscription_parameter [=v

Re: Skipping logical replication transactions on subscriber side

2021-10-26 Thread Amit Kapila
On Wed, Oct 27, 2021 at 10:43 AM Masahiko Sawada wrote: > > On Wed, Oct 27, 2021 at 12:35 PM Amit Kapila wrote: > > > > On Wed, Oct 27, 2021 at 8:32 AM Masahiko Sawada > > wrote: > > > > > > On Tue, Oct 26, 2021 at 7:29 PM Amit Kapila > > &

Re: Skipping logical replication transactions on subscriber side

2021-10-27 Thread Amit Kapila
sync worker is finished? 8. +# Create subscriptions. The table sync for test_tab2 on tap_sub will enter to +# infinite error due to violating the unique constraint. The second sentence of the comment can be written as: "The table sync for test_tab2 on tap_sub will enter into infinite error loop due to violating the unique constraint." -- With Regards, Amit Kapila.

Re: Skipping logical replication transactions on subscriber side

2021-10-27 Thread Amit Kapila
On Wed, Oct 27, 2021 at 8:32 AM Masahiko Sawada wrote: > > On Tue, Oct 26, 2021 at 7:29 PM Amit Kapila wrote: > > > > On Tue, Oct 26, 2021 at 2:27 PM Greg Nancarrow wrote: > > > > > > On Tue, Oct 26, 2021 at 5:16 PM Amit Kapila > > > wrote: > &g

Re: Next Steps with Hash Indexes

2021-10-27 Thread Amit Kapila
today. I guess this is the reason Simon took an example of unique data. -- With Regards, Amit Kapila.

Re: Next Steps with Hash Indexes

2021-10-27 Thread Amit Kapila
On Wed, Oct 27, 2021 at 4:55 PM Matthias van de Meent wrote: > > On Wed, 27 Oct 2021 at 12:58, Amit Kapila wrote: > > > > On Wed, Oct 27, 2021 at 2:32 AM Robert Haas wrote: > > > > > > On Tue, Oct 5, 2021 at 6:50 AM Simon Riggs > > > wrote

Re: Added schema level support for publication.

2021-10-27 Thread Amit Kapila
On Mon, Oct 25, 2021 at 3:09 PM Amit Kapila wrote: > > On Mon, Oct 25, 2021 at 1:11 PM vignesh C wrote: > > > > I have fixed this in the v47 version attached. > > > > Thanks, the first patch in the series "Allow publishing the tables of > schema.&qu

Re: pgsql: Document XLOG_INCLUDE_XID a little better

2021-10-27 Thread Amit Kapila
On Wed, Oct 27, 2021 at 4:39 PM Dilip Kumar wrote: > > On Tue, Oct 26, 2021 at 9:19 AM Amit Kapila wrote: > > > > > > > > Thanks, both your patches look good to me except that we need to > > > remove the sentence related to the revert of ade24dab97 fr

Re: Skipping logical replication transactions on subscriber side

2021-10-27 Thread Amit Kapila
On Thu, Oct 28, 2021 at 7:49 AM Masahiko Sawada wrote: > > On Wed, Oct 27, 2021 at 2:43 PM Amit Kapila wrote: > > > > On Wed, Oct 27, 2021 at 10:43 AM Masahiko Sawada > > wrote: > > > > > > > > BTW how useful is specifying LSN instead of XID

Re: Skipping logical replication transactions on subscriber side

2021-10-27 Thread Amit Kapila
On Wed, Oct 27, 2021 at 4:07 PM Amit Kapila wrote: > > On Wed, Oct 27, 2021 at 8:32 AM Masahiko Sawada wrote: > > > > On Tue, Oct 26, 2021 at 7:29 PM Amit Kapila wrote: > > > > > > You have a point. The other alternatives on this line could be:

Re: Skipping logical replication transactions on subscriber side

2021-10-28 Thread Amit Kapila
On Thu, Oct 28, 2021 at 10:56 AM Masahiko Sawada wrote: > > On Thu, Oct 28, 2021 at 1:29 PM Amit Kapila wrote: > > > > On Wed, Oct 27, 2021 at 4:07 PM Amit Kapila wrote: > > > > > > On Wed, Oct 27, 2021 at 8:32 AM Masahiko Sawada > > > wrote: &g

Re: Skipping logical replication transactions on subscriber side

2021-10-28 Thread Amit Kapila
On Thu, Oct 28, 2021 at 10:48 AM Masahiko Sawada wrote: > > On Thu, Oct 28, 2021 at 1:05 PM Amit Kapila wrote: > > > > On Thu, Oct 28, 2021 at 7:49 AM Masahiko Sawada > > wrote: > > > > > > > > > > > Either from the error messages in th

<    5   6   7   8   9   10   11   12   13   14   >