Re: block-level incremental backup

2019-08-30 Thread Robert Haas
On Thu, Aug 29, 2019 at 10:41 AM Jeevan Ladhe wrote: > Due to the inherent nature of pg_basebackup, the incremental backup also > allows taking backup in tar and compressed format. But, pg_combinebackup > does not understand how to restore this. I think we should either make > pg_combinebackup sup

Re: base backup client as auxiliary backend process

2019-08-30 Thread Peter Eisentraut
> Attached is a very hackish patch to implement this. It works like this: > > # (assuming you have a primary already running somewhere) > initdb -D data2 --replica > $EDITOR data2/postgresql.conf # set primary_conninfo > pg_ctl -D data2 start Attached is an updated patch for thi

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2019-08-30 Thread Konstantin Knizhnik
FWIW my understanding is that the speedup comes mostly from elimination of the serialization to a file. That however requires savepoints to handle aborts of subtransactions - I'm pretty sure I'd be trivial to create a workload where this will be much slower (with many aborts of large subtran

Re: pg_upgrade: Error out on too many command-line arguments

2019-08-30 Thread Fabien COELHO
Is this issue *really* worth expending test cycles on forevermore? With this argument consistently applied, postgres code coverage is consistently weak, with 25% of the code never executed, and 15% of functions never called. "psql" is abysmal, "libpq" is really weak. It's all a question of

Re: refactoring - share str2*int64 functions

2019-08-30 Thread Fabien COELHO
Michaël, For symmetry and efficiency, ISTM that uint16 mul overflow could use uint32 and uint32 could use uint64, and the division-based method be dropped in these cases. Yes, the division would be worse than the other. What do you think about using the previous module I sent and measure how

Re: pg_upgrade: Error out on too many command-line arguments

2019-08-30 Thread Tom Lane
Fabien COELHO writes: >> Is this issue *really* worth expending test cycles on forevermore? > With this argument consistently applied, postgres code coverage is > consistently weak, with 25% of the code never executed, and 15% of > functions never called. "psql" is abysmal, "libpq" is really we

Re: pg_upgrade: Error out on too many command-line arguments

2019-08-30 Thread Fabien COELHO
Hello Tom, Could we maintain coverage by adding a TAP test? See 1 liner attached. Is this issue *really* worth expending test cycles on forevermore? With this argument consistently applied, postgres code coverage is consistently weak, with 25% of the code never executed, and 15% of functi

Re: pg_upgrade: Error out on too many command-line arguments

2019-08-30 Thread Tom Lane
Fabien COELHO writes: > Could we maintain coverage by adding a TAP test? See 1 liner attached. Is this issue *really* worth expending test cycles on forevermore? Test cycles are not free, and I see zero reason to think that a check of this sort would ever catch any bugs. Now, if you had a way t

Re: refactoring - share str2*int64 functions

2019-08-30 Thread Michael Paquier
On Fri, Aug 30, 2019 at 10:06:11AM +0200, Fabien COELHO wrote: > Patch applies cleanly, compiles, "make check" ok, but the added functions > are not used (yet). Thanks. > I think that factoring out comments is a good move. > > For symmetry and efficiency, ISTM that uint16 mul overflow could use

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

2019-08-30 Thread Nikhil Sontakke
Hi, > > This patch has been around for some time now, the last version fails to > > apply cleanly and in-depth reviews have happened. I am moving that to > > the next CF, waiting on its author. > > Unfortunately, nothing was changed since then, so there is already some amount > of unaddressed rev

Re: block-level incremental backup

2019-08-30 Thread Jeevan Ladhe
Here are some comments: +/* The reference XLOG position for the incremental backup. */ +static XLogRecPtr refptr; As Robert already pointed we may want to pass this as parameter around instead of a global variable. Also, can be renamed to something like: incr_backup_refptr. I see in your earlie

Re: block-level incremental backup

2019-08-30 Thread Rajkumar Raghuwanshi
Hi, I am doing some testing on pg_basebackup and pg_combinebackup patches. I have also tried to create tap test for pg_combinebackup by taking reference from pg_basebackup tap cases. Attaching first draft test patch. I have done some testing with compression options, both -z and -Z level is worki

Re: WIP: Generic functions for Node types using generated metadata

2019-08-30 Thread Fabien COELHO
Hallo Andres, There've been various calls for automating their generation, but no actual patches that I am aware of. I started something a while back I have found this thread: https://www.postgresql.org/message-id/flat/E1cq93r-0004ey-Mp%40gemulon.postgresql.org It seems that comments from

Re: basebackup.c's sendFile() ignores read errors

2019-08-30 Thread Jeevan Ladhe
> > Fixed both comments in the attached patch. > Thanks, the patch looks good to me. Regards, Jeevan Ladhe

Re: WIP: Generic functions for Node types using generated metadata

2019-08-30 Thread Fabien COELHO
Hello Andres, Just my 0.02 €: There's been a lot of complaints over the years about how annoying it is to keep the out/read/copy/equalfuncs.c functions in sync with the actual underlying structs. There've been various calls for automating their generation, but no actual patches that I am awar

Re: Yet another fast GiST build

2019-08-30 Thread Andrey Borodin
> 30 авг. 2019 г., в 3:47, Alexander Korotkov > написал(а): > > 1) Binary search in non-leaf pages instead of probing each key is much faster. That's a neat idea, but key union breaks ordering, even for z-order. for two sets of tuples X and Y if for any i,o from N, Xi < Yo does not guaranty u

Re: [HACKERS] CLUSTER command progress monitor

2019-08-30 Thread Masahiko Sawada
On Thu, Aug 15, 2019 at 12:48 PM Tatsuro Yamada wrote: > > Hi Michael, Alvaro and Robert! > > On 2019/08/14 11:52, Michael Paquier wrote: > > On Wed, Aug 14, 2019 at 11:38:01AM +0900, Tatsuro Yamada wrote: > >> On 2019/08/13 14:40, Tatsuro Yamada wrote: > >>> On 2019/08/02 3:43, Alvaro Herrera wro

Re: basebackup.c's sendFile() ignores read errors

2019-08-30 Thread Jeevan Chalke
On Thu, Aug 29, 2019 at 3:17 PM Jeevan Ladhe wrote: > Hi Jeevan > > I had a look at the patch and this seems correct to me. > Thanks, Jeevan Ladhe. > > Few minor comments: > > + /* Check fread() error. */ > + CHECK_FREAD_ERROR(fp, pathbuf); > + > > The comments above the macro call at both the

Re: POC: Cleaning up orphaned files using undo logs

2019-08-30 Thread Kuntal Ghosh
Hello Thomas, I was doing some testing for the scenario where the undo written by a transaction overflows to multiple undo logs. For that I've modified the following macro: #define UndoLogMaxSize (1024 * 1024) /* 1MB undo log size */ (I should have used the provided pg_force_switch_undo though..)

Re: refactoring - share str2*int64 functions

2019-08-30 Thread Fabien COELHO
Michaël, attached is a first cut that I would like to commit separately which adds all the compatibility overflow routines to int.h for uint16, uint32 and uint64 with all the fallback implementations (int128-based method added as well if available). I have also grouped at the top of the file

Re: refactoring - share str2*int64 functions

2019-08-30 Thread Michael Paquier
On Thu, Aug 29, 2019 at 08:14:54AM +0200, Fabien COELHO wrote: > Attached v7 does create uint64 overflow inline functions. The stuff yet is > not moved to "common/int.c", a file which does not exists, even if > "common/int.h" does. Thanks for the new version. I have begun reviewing your patch, an

Re: pg_upgrade: Error out on too many command-line arguments

2019-08-30 Thread Fabien COELHO
Bonjour Michaël, I don't see why not. Perhaps this could be done for pgbench and oid2name as well? This is for pgbench. I did not found a TAP test in pg_upgrade, I do not think that it is worth creating one for that purpose. The "test.sh" script does not seem appropriate for this kind of