Re: Online verification of checksums

2019-03-29 Thread Stephen Frost
Greetings, * Andres Freund (and...@anarazel.de) wrote: > On 2019-03-29 11:30:15 -0400, Stephen Frost wrote: > > * Magnus Hagander (mag...@hagander.net) wrote: > > > On Thu, Mar 28, 2019 at 10:19 PM Tomas Vondra > > > > > > wrote: > > > > On Thu, Mar 28, 2019 at 01:11:40PM -0700, Andres Freund wr

Re: Online verification of checksums

2019-03-29 Thread Andres Freund
Hi, On 2019-03-29 11:38:02 -0400, Stephen Frost wrote: > The server-side function would essentially lock the page against i/o, > re-read it off disk into an independent location, unlock the page, then > calculate the checksum and report back? Right. I think there's a few minor variations of how t

Re: PostgreSQL pollutes the file system

2019-03-29 Thread Tom Lane
Christoph Berg writes: > What might possibly make sense is to add options to psql to > facilitate common tasks: > psql --createdb foo > psql --createuser bar --superuser > psql --reindex foo That's a thought. Or perhaps better, allow pg_ctl to grow new subcommands for those tasks?

Re: PostgreSQL pollutes the file system

2019-03-29 Thread Daniel Gustafsson
On Friday, March 29, 2019 4:41 PM, Tom Lane wrote: > Christoph Berg m...@debian.org writes: > > > What might possibly make sense is to add options to psql to > > facilitate common tasks: > > > psql --createdb foo > > psql --createuser bar --superuser > > psql --reindex foo > > That's a thought. O

Re: REINDEX CONCURRENTLY 2.0

2019-03-29 Thread Robert Treat
On Fri, Mar 29, 2019 at 3:28 AM Peter Eisentraut wrote: > > On 2019-03-28 09:07, Sergei Kornilov wrote: > > Unfortunately patch does not apply due recent commits. Any chance this can > > be fixed (and even committed in pg12)? > > Committed :) > Given this has been committed I've probably missed

Re: REINDEX CONCURRENTLY 2.0

2019-03-29 Thread Andres Freund
Hi, On 2019-03-29 11:47:10 -0400, Robert Treat wrote: > On Fri, Mar 29, 2019 at 3:28 AM Peter Eisentraut > wrote: > > > > On 2019-03-28 09:07, Sergei Kornilov wrote: > > > Unfortunately patch does not apply due recent commits. Any chance this > > > can be fixed (and even committed in pg12)? > >

Re: PostgreSQL pollutes the file system

2019-03-29 Thread Joe Conway
On 3/29/19 11:44 AM, Daniel Gustafsson wrote: > On Friday, March 29, 2019 4:41 PM, Tom Lane wrote: > >> Christoph Berg m...@debian.org writes: >> >> > What might possibly make sense is to add options to psql to >> > facilitate common tasks: >> >> > psql --createdb foo >> > psql --createuser bar -

Re: PostgreSQL pollutes the file system

2019-03-29 Thread Andres Freund
Hi, On 2019-03-29 11:41:26 -0400, Tom Lane wrote: > Or perhaps better, allow pg_ctl to grow new subcommands for those > tasks? We'd need to be careful to somehow delineate commands that need access to the data directory / run locally on the server from the ones that just needs a client connection

Re: Online verification of checksums

2019-03-29 Thread Magnus Hagander
On Fri, Mar 29, 2019 at 4:30 PM Stephen Frost wrote: > Greetings, > > * Magnus Hagander (mag...@hagander.net) wrote: > > On Thu, Mar 28, 2019 at 10:19 PM Tomas Vondra < > tomas.von...@2ndquadrant.com> > > wrote: > > > > > On Thu, Mar 28, 2019 at 01:11:40PM -0700, Andres Freund wrote: > > > >Hi, >

Re: REINDEX CONCURRENTLY 2.0

2019-03-29 Thread Bossart, Nathan
I noticed a very small typo in the documentation for this feature. diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index ccabb330cb..e45bf86c8d 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -349,7 +349,7 @@ REINDEX [ ( VERBOSE ) ] { IN

Re: pgsql: Compute XID horizon for page level index vacuum on primary.

2019-03-29 Thread Simon Riggs
On Fri, 29 Mar 2019 at 15:29, Andres Freund wrote: > On 2019-03-29 09:37:11 +, Simon Riggs wrote: > > > While trying to understand this, I see there is an even better way to > > optimize this. Since we are removing dead index tuples, we could alter > the > > killed index tuple interface so

Re: REINDEX CONCURRENTLY 2.0

2019-03-29 Thread Justin Pryzby
On Fri, Mar 29, 2019 at 03:53:05PM +, Bossart, Nathan wrote: > I noticed a very small typo in the documentation for this feature. I submit a bunch more changes for consideration, attached. >From dafdb15fb3e7c69de82a2206c9bf07588b5665ce Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Fri, 2

Re: Pluggable Storage - Andres's take

2019-03-29 Thread Andres Freund
On 2019-03-29 18:38:46 +1100, Haribabu Kommi wrote: > As I see that your are fixing some typos of the code that is committed, > I just want to share some more corrections that I found in the patches > that are committed till now. Pushed both, thanks!

Re: pgsql: Compute XID horizon for page level index vacuum on primary.

2019-03-29 Thread Andres Freund
Hi, On 2019-03-29 15:58:14 +, Simon Riggs wrote: > On Fri, 29 Mar 2019 at 15:29, Andres Freund wrote: > > That's far from a trivial feature imo. It seems quite possible that we'd > > end up with increased overhead, because the current logic can get away > > with only doing hint bit style writ

Re: pgsql: Compute XID horizon for page level index vacuum on primary.

2019-03-29 Thread Simon Riggs
On Fri, 29 Mar 2019 at 16:12, Andres Freund wrote: > On 2019-03-29 15:58:14 +, Simon Riggs wrote: > > On Fri, 29 Mar 2019 at 15:29, Andres Freund wrote: > > > That's far from a trivial feature imo. It seems quite possible that > we'd > > > end up with increased overhead, because the current

Re: pgsql: Compute XID horizon for page level index vacuum on primary.

2019-03-29 Thread Peter Geoghegan
On Fri, Mar 29, 2019 at 9:12 AM Andres Freund wrote: > But even so, you can't have unlogged changes that you then rely on. Even > if there's no torn page issue. Currently BTP_HAS_GARBAGE and > ItemIdMarkDead() are treated as hints - if we want to guarantee all > these are accurate, I don't quite

Re: PostgreSQL pollutes the file system

2019-03-29 Thread Tom Lane
Andres Freund writes: > On 2019-03-29 11:41:26 -0400, Tom Lane wrote: >> Or perhaps better, allow pg_ctl to grow new subcommands for those >> tasks? > We'd need to be careful to somehow delineate commands that need access > to the data directory / run locally on the server from the ones that > ju

Re: New vacuum option to do only freezing

2019-03-29 Thread Masahiko Sawada
On Fri, Mar 29, 2019 at 10:46 PM Robert Haas wrote: > > On Fri, Mar 29, 2019 at 2:16 AM Masahiko Sawada wrote: > > Attached updated patches. These patches are applied on top of 0001 > > patch on parallel vacuum thread[1]. > > +bool index_cleanup = true; /* by default */ > > I think we should

Re: partitioned tables referenced by FKs

2019-03-29 Thread Alvaro Herrera
On 2019-Mar-29, Jesper Pedersen wrote: > Thanks ! > > Maybe the "(" / ")" in the CASCADE description should be removed from > ref/drop_table.sgml as part of this patch. I'm not sure what text you propose to remove? > Should catalogs.sgml be updated for this case ? I'm not adding any new depen

Re: speeding up planning with partitions

2019-03-29 Thread Tom Lane
I wrote: > Amit Langote writes: >> About the XXX: I think resetting inh flag is unnecessary, so we should >> just remove the line. > Possibly. I hadn't had time to follow up the XXX annotation. Now I have ... Yeah, it seems we can just drop that and leave the flag alone. We'll end up running

Re: pgsql: Compute XID horizon for page level index vacuum on primary.

2019-03-29 Thread Andres Freund
On 2019-03-29 16:20:54 +, Simon Riggs wrote: > On Fri, 29 Mar 2019 at 16:12, Andres Freund wrote: > > > > On 2019-03-29 15:58:14 +, Simon Riggs wrote: > > > On Fri, 29 Mar 2019 at 15:29, Andres Freund wrote: > > > > That's far from a trivial feature imo. It seems quite possible that > >

Re: PostgreSQL pollutes the file system

2019-03-29 Thread Alvaro Herrera
On 2019-Mar-29, Tom Lane wrote: > Christoph Berg writes: > > What might possibly make sense is to add options to psql to > > facilitate common tasks: > > > psql --createdb foo > > psql --createuser bar --superuser > > psql --reindex foo > > That's a thought. Or perhaps better, allow pg_ctl to

Proposal: autovacuum_max_queue_depth

2019-03-29 Thread Chris Travers
Hi everyone. I would like to flesh this out in terms fo feedback before creating a patch. The Problem In large production systems often you can have problems when autovacuum is not tuned aggressively enough. This leads to long autovacuum runs when they happen, and autovacuum ends up eventually

Re: partitioned tables referenced by FKs

2019-03-29 Thread Jesper Pedersen
Hi, On 3/29/19 12:29 PM, Alvaro Herrera wrote: On 2019-Mar-29, Jesper Pedersen wrote: Maybe the "(" / ")" in the CASCADE description should be removed from ref/drop_table.sgml as part of this patch. I'm not sure what text you propose to remove? Just the attached. Should catalogs.sgml be

Re: monitoring CREATE INDEX [CONCURRENTLY]

2019-03-29 Thread Alvaro Herrera
On 2019-Mar-29, Alvaro Herrera wrote: > So, CLUSTER and ALTER TABLE rewrites only do non-concurrent index > builds; and REINDEX can reuse pretty much the same wait-for metrics > columns as CIC. So I think it's okay if I move only the metrics that > conflict for index_build. The attached version

Re: speeding up planning with partitions

2019-03-29 Thread Tom Lane
Amit Langote writes: > On 2019/03/29 7:38, Tom Lane wrote: >> 2. I seriously dislike what's been done in joinrels.c, too. That >> really seems like a kluge (and I haven't had time to study it >> closely). > Those hunks account for the fact that pruned partitions, for which we no > longer create

Re: PostgreSQL pollutes the file system

2019-03-29 Thread Christoph Berg
Re: Tom Lane 2019-03-29 <19517.1553876...@sss.pgh.pa.us> > >> Or perhaps better, allow pg_ctl to grow new subcommands for those > >> tasks? > > > We'd need to be careful to somehow delineate commands that need access > > to the data directory / run locally on the server from the ones that > > just

Re: PostgreSQL pollutes the file system

2019-03-29 Thread Pavel Stehule
pá 29. 3. 2019 v 19:50 odesílatel Christoph Berg napsal: > Re: Tom Lane 2019-03-29 <19517.1553876...@sss.pgh.pa.us> > > >> Or perhaps better, allow pg_ctl to grow new subcommands for those > > >> tasks? > > > > > We'd need to be careful to somehow delineate commands that need access > > > to the

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-03-29 Thread Shaun Thomas
Another ping on this Incremental Sort patch. Alexander, you'd noted that you would try to get it into subsequent Commit Fests with improvements you've been considering, but I don't see it in anything but 2018-11. Have you abandoned this as a maintainer? If so, it would be nice to know so someone e

Re: monitoring CREATE INDEX [CONCURRENTLY]

2019-03-29 Thread Robert Haas
On Fri, Mar 29, 2019 at 2:16 PM Alvaro Herrera wrote: > I think we should consider a new column of an array type, where we could > put things like the list of PIDs to be waited for, the list of OIDs of > index to rebuild, or the list of partitions to build the index on. This has to work with a fi

Re: Enable data checksums by default

2019-03-29 Thread Bernd Helmle
Am Freitag, den 29.03.2019, 23:10 +0900 schrieb Michael Paquier: > > I can't really believe that many people set up shared_buffers at > 128kB > which would cause such a large number of page evictions, but I can > believe that many users have shared_buffers set to its default value > and that we ar

Re: monitoring CREATE INDEX [CONCURRENTLY]

2019-03-29 Thread Alvaro Herrera
On 2019-Mar-29, Robert Haas wrote: > On Fri, Mar 29, 2019 at 2:16 PM Alvaro Herrera > wrote: > > I think we should consider a new column of an array type, where we could > > put things like the list of PIDs to be waited for, the list of OIDs of > > index to rebuild, or the list of partitions to

Re: PostgreSQL pollutes the file system

2019-03-29 Thread Joe Conway
On 3/29/19 3:01 PM, Pavel Stehule wrote: > But psql has safe escaping via :"xxx" notation. So some like > > psql -c 'create role :"role"' -v role='my role' ... > > But what I know the psql variables are not evaluated for -c query You can do this: echo "create role :\"role\"" | psql -v role='my r

Re: Enable data checksums by default

2019-03-29 Thread Christoph Berg
Re: Bernd Helmle 2019-03-29 <3586bb9345a59bfc8d13a50a7c729be1ee6759fd.ca...@oopsware.de> > Am Freitag, den 29.03.2019, 23:10 +0900 schrieb Michael Paquier: > > > > I can't really believe that many people set up shared_buffers at > > 128kB > > which would cause such a large number of page eviction

Re: PostgreSQL pollutes the file system

2019-03-29 Thread Christoph Berg
Re: Pavel Stehule 2019-03-29 > > Other idea: If we don't want to reinvent a new tool, how about > > supporting prepared statements in psql? > > > > psql -c 'create user %i' --args 'bob w. space' > > > > Prepared statements cannot be DDL commands. "Prepared" in the sense of what format() does.

Re: partitioned tables referenced by FKs

2019-03-29 Thread Jesper Pedersen
Hi Alvaro, On 3/28/19 2:59 PM, Alvaro Herrera wrote: I ended up revising the dependencies that we give to the constraint in the partition -- instead of giving it partition-type dependencies, we give it an INTERNAL dependency. Now when you request to drop the partition, it says this: create tab

Re: PostgreSQL pollutes the file system

2019-03-29 Thread Chapman Flack
On 3/29/19 3:32 PM, Joe Conway wrote: > pg_util > > Of course that does not lend itself to symlinking for backward > compatibility, does it? If there is a way I am not familiar with it. On Unix-like systems, you can have pg_util look at argv[0] to see if it was called createuser or what not.

Re: PostgreSQL pollutes the file system

2019-03-29 Thread Christoph Berg
Re: Joe Conway 2019-03-29 <48e5efaf-7ea2-ed70-a803-949bbfec8...@joeconway.com> > echo "\password :\"role\"" | psql -v role='my role' > Enter new password: > Enter it again: > > That said, this is kind of off the topic of this thread. It is on-topic because the reason we can't just tell people to

Re: New vacuum option to do only freezing

2019-03-29 Thread Robert Haas
On Fri, Mar 29, 2019 at 12:27 PM Masahiko Sawada wrote: > Yeah, but since multiple relations might be specified in VACUUM > command we need to process index_cleanup option after opened each > relations. Maybe we need to process all options except for > INDEX_CLEANUP in ExecVacuum() and pass Vacuum

Re: PostgreSQL pollutes the file system

2019-03-29 Thread Joe Conway
On 3/29/19 3:43 PM, Christoph Berg wrote: > Re: Joe Conway 2019-03-29 <48e5efaf-7ea2-ed70-a803-949bbfec8...@joeconway.com> >> echo "\password :\"role\"" | psql -v role='my role' >> Enter new password: >> Enter it again: >> >> That said, this is kind of off the topic of this thread. > > It is on-t

Re: monitoring CREATE INDEX [CONCURRENTLY]

2019-03-29 Thread Robert Haas
On Fri, Mar 29, 2019 at 3:28 PM Alvaro Herrera wrote: > On 2019-Mar-29, Robert Haas wrote: > > On Fri, Mar 29, 2019 at 2:16 PM Alvaro Herrera > > wrote: > > > I think we should consider a new column of an array type, where we could > > > put things like the list of PIDs to be waited for, the lis

Re: PostgreSQL pollutes the file system

2019-03-29 Thread Christoph Berg
Re: Joe Conway 2019-03-29 > >> echo "\password :\"role\"" | psql -v role='my role' > > > > It is on-topic because the reason we can't just tell people to replace > > createuser $foo > > with > > psql -c "create user $foo" > > is because $foo might need escaping. > > > > IMHO if we find an wa

Unix socket dir, an idea

2019-03-29 Thread Danylo Hlynskyi
Before writing a patch, I'd like to hear discussion first. I've searched archives first and read following previous discussions on this topic: - https://www.postgresql.org/message-id/4FCF6040.5030408%40redhat.com - https://www.postgresql.org/message-id/14899.974513046%40sss.pgh.pa.us The problem

Re: PostgreSQL pollutes the file system

2019-03-29 Thread Andrew Dunstan
On 3/29/19 11:41 AM, Tom Lane wrote: > Christoph Berg writes: >> What might possibly make sense is to add options to psql to >> facilitate common tasks: >> psql --createdb foo >> psql --createuser bar --superuser >> psql --reindex foo > That's a thought. Or perhaps better, allow pg_ctl to grow

Re: explain plans with information about (modified) gucs

2019-03-29 Thread Tomas Vondra
On Wed, Mar 27, 2019 at 09:06:04AM +0100, Rafia Sabih wrote: On Tue, 26 Mar 2019 at 21:04, Tomas Vondra wrote: On Mon, Mar 18, 2019 at 11:31:48AM +0100, Rafia Sabih wrote: >On Sun, 24 Feb 2019 at 00:06, Tomas Vondra wrote: >> >> Hi, >> >> attached is an updated patch, fixing and slightly twea

Re: Online verification of checksums

2019-03-29 Thread Michael Banck
Hi, Am Freitag, den 29.03.2019, 16:52 +0100 schrieb Magnus Hagander: > On Fri, Mar 29, 2019 at 4:30 PM Stephen Frost wrote: > > * Magnus Hagander (mag...@hagander.net) wrote: > > > On Thu, Mar 28, 2019 at 10:19 PM Tomas Vondra > > > > > > wrote: > > > > On Thu, Mar 28, 2019 at 01:11:40PM -0700,

Re: REINDEX CONCURRENTLY 2.0

2019-03-29 Thread Peter Eisentraut
On 2019-03-29 16:53, Bossart, Nathan wrote: > I noticed a very small typo in the documentation for this feature. fixed -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: REINDEX CONCURRENTLY 2.0

2019-03-29 Thread Peter Eisentraut
On 2019-03-29 17:01, Justin Pryzby wrote: > On Fri, Mar 29, 2019 at 03:53:05PM +, Bossart, Nathan wrote: >> I noticed a very small typo in the documentation for this feature. > > I submit a bunch more changes for consideration, attached. fixed, thanks -- Peter Eisentraut http:/

Re: patch to allow disable of WAL recycling

2019-03-29 Thread Jerry Jelinek
On Thu, Mar 28, 2019 at 6:10 PM Thomas Munro wrote: > On Fri, Mar 29, 2019 at 10:47 AM Thomas Munro > wrote: > > On Fri, Mar 29, 2019 at 8:59 AM Robert Haas > wrote: > > > On Tue, Mar 26, 2019 at 3:24 PM Jerry Jelinek < > jerry.jeli...@joyent.com> wrote: > > > > The latest patch is rebased, bui

Re: speeding up planning with partitions

2019-03-29 Thread Tom Lane
Amit Langote writes: > On 2019/03/29 7:38, Tom Lane wrote: >> 2. I seriously dislike what's been done in joinrels.c, too. That >> really seems like a kluge (and I haven't had time to study it >> closely). > Those hunks account for the fact that pruned partitions, for which we no > longer create

Re: Unix socket dir, an idea

2019-03-29 Thread Tom Lane
Danylo Hlynskyi writes: > The problem (as I see it) is that everybody would like to move `/tmp` > socket dir to `/var/run`, or even `/var/run/postgresql` (or even > `/run/postgresql`), but compatibility with old clients (which connect to > /tmp by default) is a concern. *Some* people would like t

Re: pgsql: Improve autovacuum logging for aggressive and anti-wraparound ru

2019-03-29 Thread Michael Paquier
On Fri, Mar 29, 2019 at 11:22:55AM -0300, Alvaro Herrera wrote: > Yeah, that looks good to me too. I wonder if we really need it as LOG > though; we don't say anything for actions unless they take more than the > min duration, so why say something for a no-op that takes almost no time? > Maybe mak

Re: REINDEX CONCURRENTLY 2.0

2019-03-29 Thread Michael Paquier
On Fri, Mar 29, 2019 at 03:10:23PM +, Shinoda, Noriyoshi (PN Japan A&PS Delivery) wrote: > I tried this great feature for partition index. > The first time the REINDEX TABLE CONCURRENTLY statement is executed > to the partition, then an error occurs. Yes, that's a problem. I am adding an op

Re: REINDEX CONCURRENTLY 2.0

2019-03-29 Thread Michael Paquier
On Fri, Mar 29, 2019 at 08:48:03AM -0700, Andres Freund wrote: > Yes, it increases the total runtime quite considerably. And it adds new > failure modes with partially built invalid indexes hanging around that > need to be dropped manually. On top of that CONCURRENTLY needs multiple transactions t

Re: clean up pg_checksums.sgml

2019-03-29 Thread Michael Paquier
On Fri, Mar 29, 2019 at 09:32:10AM -0500, Justin Pryzby wrote: > PFA patch with minor improvements to documentation. Patch does not apply, and I have reworded the last paragraph about failures while operating. > Also, what do you think about changing user-facing language from > "check checksum" t

Re: speeding up planning with partitions

2019-03-29 Thread Amit Langote
Thanks for the new patches. On Sat, Mar 30, 2019 at 9:17 AM Tom Lane wrote: > > Amit Langote writes: > > On 2019/03/29 7:38, Tom Lane wrote: > >> 2. I seriously dislike what's been done in joinrels.c, too. That > >> really seems like a kluge (and I haven't had time to study it > >> closely). >

Re: pgbench - add pseudo-random permutation function

2019-03-29 Thread Hironobu SUZUKI
On 2019/03/21 17:27, David Steele wrote: Hi Hironobu, Sorry for the late reply. I reviewed this patch. Function nbits(), which was previously discussed, has been simplified by using the function pg_popcount64(). By adding the mathematical explanation, it has been easier to understand the

Re: monitoring CREATE INDEX [CONCURRENTLY]

2019-03-29 Thread Alvaro Herrera
On 2019-Mar-29, Robert Haas wrote: > On Fri, Mar 29, 2019 at 3:28 PM Alvaro Herrera > wrote: > > Maybe we can consider using dynamic shmem for that, and include a > > pointer to it in the fixed-size chunk. (It's a bit too late to be > > writing this code, mind; I'm just proposing this for a fu

Indexscan failed assert caused by using index without lock

2019-03-29 Thread 高增琦
Following example can reproduce the problem: ``` create table d(a int); create index di on d(a); set enable_seqscan=off; set enable_bitmapscan to off; prepare p as delete from d where a=3; execute p; execute p; ``` The reason is that: ExecInitIndexScan will not lock index because it thinks InitPl

Column lookup in a row performance

2019-03-29 Thread Павлухин Иван
Hi PostgresSQL developers, I asked my question already on pgsql-general list and did not find an explanation. Below is the question mainly copied from [0]. I am learning deeply how tuples are organized and column values are accessed in different databases. As far as undertood postgres does n

Speed up build on Windows by generating symbol definition in batch

2019-03-29 Thread Peifeng Qiu
Hi, hackers. Build process on Windows includes compiling all source into object files, linking them to binaries, and generating export symbol definitions, etc. When I watched the whole build process with a task manager, I discovered that a lot of time was spent on generating export symbol definiti

<    1   2