Re: Read access for pg_monitor to pg_replication_origin_status view

2020-06-14 Thread Michael Paquier
On Mon, Jun 15, 2020 at 12:44:02AM -0400, Tom Lane wrote: > FWIW, I'd have included a catversion bump in this, to enforce that > the modified backend functions are used with matching pg_proc entries. > It's not terribly important at this phase of the devel cycle, but still > somebody might wonder w

Re: TAP tests and symlinks on Windows

2020-06-14 Thread Michael Paquier
On Sat, Jun 13, 2020 at 03:00:54PM +0900, Michael Paquier wrote: > My take would be to actually enforce that as a requirement for 14~ if > that works reliably, and of course not backpatch that change as that's > clearly an improvement and not a bug fix. It would be good to check > the status of ea

Re: Transactions involving multiple postgres foreign servers, take 2

2020-06-14 Thread Amit Kapila
On Sun, Jun 14, 2020 at 2:21 PM Tatsuo Ishii wrote: > > >> Won't it create an inconsistency in viewing the data from the > >> different servers? Say, such a transaction inserts one row into a > >> local server and another into the foreign server. Now, if we follow > >> the above protocol, the us

Re: hashagg slowdown due to spill changes

2020-06-14 Thread Jeff Davis
On Sun, 2020-06-14 at 11:14 -0700, Andres Freund wrote: > I'm somewhat inclined to think that we should revert 4cad2534da6 and > then look at how precisely to tackle this in 14. I'm fine with that. > It'd probably make sense to request small tlists when the number of > estimated groups is large,

Re: [PATCH] Initial progress reporting for COPY command

2020-06-14 Thread Bharath Rupireddy
> I'm using ftell to get current position in file to populate > file_bytes_processed without error handling (ftell can return -1L and also > populate errno on problems). > > 1. Is that a good way to get progress of file processing? IMO, it's better to handle the error cases. One possible case wh

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-06-14 Thread Ashutosh Bapat
Thanks Andrey for the patch. I am glad that the patch has taken care of some corner cases already but there exist still more. COPY command constructed doesn't take care of dropped columns. There is code in deparseAnalyzeSql which constructs list of columns for a given foreign relation. 0002 patch

Re: Resetting spilled txn statistics in pg_stat_replication

2020-06-14 Thread Amit Kapila
On Sat, Jun 13, 2020 at 5:07 PM Fujii Masao wrote: > > > On 2020/06/13 14:23, Amit Kapila wrote: > > On Fri, Jun 12, 2020 at 6:11 PM Magnus Hagander wrote: > >> > >> On Fri, Jun 12, 2020 at 10:23 AM Amit Kapila > >> wrote: > >>> > >> > >> > >> The problem with "lifetime of a process" is that it

Re: min_safe_lsn column in pg_replication_slots view

2020-06-14 Thread Michael Paquier
On Mon, Jun 15, 2020 at 12:40:03PM +0900, Fujii Masao wrote: > BTW, I just wonder why each row in pg_replication_slots needs to have > min_safe_lsn column? Basically min_safe_lsn should be the same between > every replication slots. Indeed, that's confusing in its current shape. I would buy putti

Re: Read access for pg_monitor to pg_replication_origin_status view

2020-06-14 Thread Tom Lane
Michael Paquier writes: > On Wed, Jun 10, 2020 at 12:35:49PM +0900, Michael Paquier wrote: >> OK, thanks. Then let's wait a couple of days to see if anybody has >> any objections with the removal of the hardcoded superuser check >> for those functions. > Committed the part removing the superuser

Re: Review for GetWALAvailability()

2020-06-14 Thread Kyotaro Horiguchi
At Sat, 13 Jun 2020 01:38:49 +0900, Fujii Masao wrote in > Hi, > > The document explains that "lost" value that > pg_replication_slots.wal_status reports means > > some WAL files are definitely lost and this slot cannot be used to > resume replication anymore. > > However, I observed

Re: [PATCH] Initial progress reporting for COPY command

2020-06-14 Thread Fujii Masao
On 2020/06/14 21:32, Josef Šimánek wrote: Hello, as proposed by Pavel Stěhule and discussed on local czech PostgreSQL maillist (https://groups.google.com/d/msgid/postgresql-cz/CAFj8pRCZ42CBCa1bPHr7htffSV%2BNAcgcHHG0dVqOog4bsu2LFw%40mail.gmail.com?utm_medium=email&utm_source=footer), I have

Re: vacuum verbose: show pages marked allvisible/frozen/hintbits

2020-06-14 Thread Masahiko Sawada
On Sun, 26 Jan 2020 at 23:13, Justin Pryzby wrote: > > I'm forking this thread since it's separate topic, and since keeping in a > single branch hasn't made maintaining the patches any easier. > https://www.postgresql.org/message-id/CAMkU%3D1xAyWnwnLGORBOD%3Dpyv%3DccEkDi%3DwKeyhwF%3DgtB7QxLBwQ%40m

Re: valgrind versus pg_atomic_init()

2020-06-14 Thread Tom Lane
Andres Freund writes: > On 2020-06-14 22:30:25 -0400, Tom Lane wrote: >> Perhaps it'd be worth putting a memory barrier at the end of the _init >> function(s)? As you say, this is hypothetical right now, but that'd be >> a cheap improvement. > I don't think it'd be that cheap for some cases. The

Re: valgrind versus pg_atomic_init()

2020-06-14 Thread Andres Freund
Hi, On 2020-06-14 22:30:25 -0400, Tom Lane wrote: > Noah Misch writes: > > On Sun, Jun 07, 2020 at 12:23:35AM -0400, Tom Lane wrote: > >> ... But on thinking more about this, it seems like > >> generic.h's version of pg_atomic_init_u64_impl is just fundamentally > >> misguided. Why isn't it sim

Re: valgrind versus pg_atomic_init()

2020-06-14 Thread Andres Freund
Hi, On 2020-06-14 18:55:27 -0700, Noah Misch wrote: > Does something guarantee the write will be globally-visible by the time the > first concurrent accessor shows up? The function comments say: * * Has to be done before any concurrent usage.. * * No barrier semantics. > (If not, one could

Re: create database with template doesn't copy database ACL

2020-06-14 Thread Tom Lane
Bruce Momjian writes: > Well, I thought we copied everything except things tha can be specified > as different in CREATE DATABASE, though I can see why we would not copy > them. Should we document this or issue a notice about not copying > non-default database attributes? We do not need a notice

Re: Asynchronous Append on postgres_fdw nodes.

2020-06-14 Thread Andrey V. Lepikhov
The patch has a problem with partitionwise aggregates. Asynchronous append do not allow the planner to use partial aggregates. Example you can see in attachment. I can't understand why: costs of partitionwise join are less. Initial script and explains of the query with and without the patch you

Re: doc examples for pghandler

2020-06-14 Thread Mark Wong
On Sat, Jun 13, 2020 at 01:19:17PM +0900, Michael Paquier wrote: > On Fri, Jun 12, 2020 at 10:13:41PM -0400, Tom Lane wrote: > > On second thought, contrib/ is not quite the right place, because we > > typically expect modules there to actually get installed, meaning they > > have to have at least

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-06-14 Thread Dilip Kumar
On Fri, Jun 12, 2020 at 4:35 PM Amit Kapila wrote: > > On Fri, Jun 12, 2020 at 11:38 AM Dilip Kumar wrote: > > > > - Currently, while reading/writing the streaming/subxact files we are > > reporting the wait event for example > > 'pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_WRITE);', but

min_safe_lsn column in pg_replication_slots view

2020-06-14 Thread Fujii Masao
Hi, Per the docs, pg_replication_slots.min_safe_lsn inedicates "the minimum LSN currently available for walsenders". When I executed pg_walfile_name() with min_safe_lsn, the function returned the name of the last removed WAL file instead of minimum available WAL file name. This happens because mi

Re: create database with template doesn't copy database ACL

2020-06-14 Thread Bruce Momjian
On Sun, Jun 14, 2020 at 11:24:56PM -0400, Tom Lane wrote: > Bruce Momjian writes: > > I am unclear if we should be copying the CONNECT and TEMPORARY > > attributes or documenting that CREATE DATABASE does not copy them. > > We should absolutely not copy them. > > As an example, it'd make sense f

Re: create database with template doesn't copy database ACL

2020-06-14 Thread Tom Lane
Bruce Momjian writes: > I am unclear if we should be copying the CONNECT and TEMPORARY > attributes or documenting that CREATE DATABASE does not copy them. We should absolutely not copy them. As an example, it'd make sense for an admin to revoke CONNECT on a template database, just to help ensur

Re: create database with template doesn't copy database ACL

2020-06-14 Thread Bruce Momjian
On Sun, Jun 14, 2020 at 07:26:13AM +, Joseph Nahmias wrote: > On Fri, Jun 12, 2020 at 05:29:51PM -0400, Bruce Momjian wrote: > > On Fri, Jun 5, 2020 at 02:31:34PM +, PG Doc comments form wrote: > > > The following documentation comment has been logged on the website: > > > > > > Page: htt

Re: Postgresql13_beta1 (could not rename temporary statistics file) Windows 64bits

2020-06-14 Thread Justin Pryzby
On Fri, Jun 12, 2020 at 03:15:52PM -0300, Ranier Vilela wrote: > Posgres13_beta1, is consistently writing to the logs, "could not rename > temporary statistics file". > When analyzing the source that writes the log, I simplified the part that > writes the logs a little. What windows version and co

Re: valgrind versus pg_atomic_init()

2020-06-14 Thread Tom Lane
Noah Misch writes: > On Sun, Jun 07, 2020 at 12:23:35AM -0400, Tom Lane wrote: >> ... But on thinking more about this, it seems like >> generic.h's version of pg_atomic_init_u64_impl is just fundamentally >> misguided. Why isn't it simply assigning the value with an ordinary >> unlocked write?

Re: Postgresql13_beta1 (could not rename temporary statistics file) Windows 64bits

2020-06-14 Thread Michael Paquier
On Fri, Jun 12, 2020 at 03:15:52PM -0300, Ranier Vilela wrote: > Posgres13_beta1, is consistently writing to the logs, "could not rename > temporary statistics file". > When analyzing the source that writes the log, I simplified the part that > writes the logs a little. FWIW, I have been running a

Re: valgrind versus pg_atomic_init()

2020-06-14 Thread Noah Misch
On Sun, Jun 07, 2020 at 12:23:35AM -0400, Tom Lane wrote: > I experimented with running "make check" on ARM64 under a reasonably > bleeding-edge valgrind (3.16.0). One thing I ran into is that > regress.c's test_atomic_ops fails; valgrind shows the stack trace > >fun:__aarch64_cas8_acq_rel >

Re: Include access method in listTables output

2020-06-14 Thread vignesh C
On Tue, Jun 9, 2020 at 6:45 PM Georgios wrote: > > > Please add it to the commitfest at https://commitfest.postgresql.org/28/ > > Thank you very much for your time. Added to the commitfest as suggested. Patch applies cleanly, make check & make check-world passes. Few comments: + if (pset.sversi

Re: [PATCH] Initial progress reporting for COPY command

2020-06-14 Thread Michael Paquier
Hi Josef, On Sun, Jun 14, 2020 at 02:32:33PM +0200, Josef Šimánek wrote: > Hello, as proposed by Pavel Stěhule and discussed on local czech PostgreSQL > maillist ( > https://groups.google.com/d/msgid/postgresql-cz/CAFj8pRCZ42CBCa1bPHr7htffSV%2BNAcgcHHG0dVqOog4bsu2LFw%40mail.gmail.com?utm_medium=em

Re: hashagg slowdown due to spill changes

2020-06-14 Thread Andres Freund
Hi, On 2020-06-12 15:29:08 -0700, Jeff Davis wrote: > On Fri, 2020-06-12 at 14:37 -0700, Andres Freund wrote: > > I don't see why it's ok to force an additional projection in the very > > common case of hashaggs over a few rows. So I think we need to > > rethink > > 4cad2534da6. > > One possibili

Re: Postgresql13_beta1 (could not rename temporary statistics file) Windows 64bits

2020-06-14 Thread Ranier Vilela
posix rename, "renames a file, moving it between directories if required". pgrename, win32 port uses MoveFileEx, to support rename files at Windows side, but, actually don't allow "renames a file, moving it between directories if required". To match the same characteristics as posix rename, we n

Re: problem with RETURNING and update row movement

2020-06-14 Thread Amit Langote
On Sun, Jun 14, 2020 at 4:23 PM Etsuro Fujita wrote: > Hi Amit-san, > > On Thu, Jun 11, 2020 at 6:10 PM Amit Langote wrote: > > Reproduction steps: > > > > create table foo (a int, b int) partition by list (a); > > create table foo1 (c int, b int, a int); > > alter table foo1 drop c; > > alter ta

[PATCH] Initial progress reporting for COPY command

2020-06-14 Thread Josef Šimánek
Hello, as proposed by Pavel Stěhule and discussed on local czech PostgreSQL maillist ( https://groups.google.com/d/msgid/postgresql-cz/CAFj8pRCZ42CBCa1bPHr7htffSV%2BNAcgcHHG0dVqOog4bsu2LFw%40mail.gmail.com?utm_medium=email&utm_source=footer), I have prepared an initial patch for COPY command progre

Re: Internal key management system

2020-06-14 Thread Fabien COELHO
Hello Masahiko-san, * The external place needs to manage more encryption keys than the current patch does. Why? If the external place is just a separate process on the same host, probably it would manage the very same amount as what your patch. In the current patch, the external place need

Re: Internal key management system

2020-06-14 Thread Masahiko Sawada
On Sat, 13 Jun 2020 at 05:59, Fabien COELHO wrote: > > > Hello Masahiko-san, > > > Summarizing the discussed points so far, I think that the major > > advantage points of your idea comparing to the current patch's > > architecture are: > > > > * More secure. Because it never loads KEK in postgres

Re: Transactions involving multiple postgres foreign servers, take 2

2020-06-14 Thread Tatsuo Ishii
>> Won't it create an inconsistency in viewing the data from the >> different servers? Say, such a transaction inserts one row into a >> local server and another into the foreign server. Now, if we follow >> the above protocol, the user will be able to see the row from the >> local server but not

Re: problem with RETURNING and update row movement

2020-06-14 Thread Etsuro Fujita
Hi Amit-san, On Thu, Jun 11, 2020 at 6:10 PM Amit Langote wrote: > Reproduction steps: > > create table foo (a int, b int) partition by list (a); > create table foo1 (c int, b int, a int); > alter table foo1 drop c; > alter table foo attach partition foo1 for values in (1); > create table foo2 pa

Re: Recording test runtimes with the buildfarm

2020-06-14 Thread Fabien COELHO
Hello Tom, I have in the past scraped the latter results and tried to make sense of them. They are *mighty* noisy, even when considering just one animal that I know to be running on a machine with little else to do. Maybe averaging across the whole buildfarm could reduce the noise level, b

Re: extensible options syntax for replication parser?

2020-06-14 Thread Fabien COELHO
Hello Robert, My 0.02 €: It seems to me that we're making the same mistake with the replication parser that we've made in various placesin the regular parser: using a syntax for options that requires that every potential option be a keyword, and every potential option requires modification of