Re: Online verification of checksums

2019-03-19 Thread Andres Freund
Hi, On 2019-03-19 22:39:16 +0100, Michael Banck wrote: > Am Dienstag, den 19.03.2019, 13:00 -0700 schrieb Andres Freund: > > a) checks that the page is all zeroes if PageIsNew() (like > >PageIsVerified() does for the backend). That avoids missing cases > >where corruption just zeroed out t

Re: Proposal to suppress errors thrown by to_reg*()

2019-03-19 Thread Tatsuo Ishii
>> I (and Hoshiai-san) concern about following case: >> >> # revoke usage on schema s1 from foo; >> REVOKE >> : >> [connect as foo] >> test=> select to_regclass('s1.t1')::oid; >> ERROR: permission denied for schema s1 > > That works in a transaction. It looks right that the actually > revoked sc

Re: Offline enabling/disabling of data checksums

2019-03-19 Thread Michael Paquier
On Tue, Mar 19, 2019 at 09:47:17AM -0700, Andres Freund wrote: > I'm not sure it needs to be this patch's responsibility to come up with > a scheme here at all however. pg_rewind, pg_resetwal, pg_upgrade all > don't really have a lockout mechanism, and it hasn't caused a ton of > problems. I think

Re: Making all nbtree entries unique by having heap TIDs participate in comparisons

2019-03-19 Thread Peter Geoghegan
On Mon, Mar 18, 2019 at 10:17 AM Peter Geoghegan wrote: > The big difference is that you make the possible call to > _bt_stepright() conditional on this being a checkingunique index -- > the duplicate code is indented in that branch of _bt_findsplitloc(). > Whereas I break early in the loop when "

Re: Compressed TOAST Slicing

2019-03-19 Thread Paul Ramsey
> On Mar 19, 2019, at 4:47 AM, Stephen Frost wrote: > > Greetings, > > * Paul Ramsey (pram...@cleverelephant.ca) wrote: >>> On Mar 18, 2019, at 7:34 AM, Robert Haas wrote: >>> +1. I think Paul had it right originally. >> >> In that spirit, here is a “one pglz_decompress function, new param

Re: Fwd: Add tablespace tap test to pg_rewind

2019-03-19 Thread Michael Paquier
On Tue, Mar 19, 2019 at 08:16:21PM +0800, Shaoqi Bai wrote: > Thanks for your advice, sorry for taking so long to give update in the > thread, because I am stuck in modifing Perl script, knowing little about > Perl language. No problem. It is true that using perl for the first time can be a certa

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2019-03-19 Thread Julien Rouhaud
On Tue, Mar 19, 2019 at 8:38 PM legrand legrand wrote: > > Would it make sense to add it in auto explain ? > I don't know for explain itself, but maybe ... I'd think that people interested in getting the queryid in the logs would configure the log_line_prefix to display it consistently rather tha

Re: Online verification of checksums

2019-03-19 Thread Michael Paquier
On Tue, Mar 19, 2019 at 02:44:52PM -0700, Andres Freund wrote: > That's *PRECISELY* my point. I think it's a bad idea to do online > checksumming from outside the backend. It needs to be inside the > backend, and if there's any verification failures on a block, it needs > to acquire the IO lock on

Re: Proposal to suppress errors thrown by to_reg*()

2019-03-19 Thread Kyotaro HORIGUCHI
At Tue, 19 Mar 2019 19:09:59 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20190319.190959.25783254.horiguchi.kyot...@lab.ntt.co.jp> > That works in a transaction. It looks right that the actually > revoked schema cannot be accessed. >From another viewpoint, the behavior really doesn

RE: [survey] New "Stable" QueryId based on normalized query text

2019-03-19 Thread Tsunakawa, Takayuki
From: legrand legrand [mailto:legrand_legr...@hotmail.com] > There are many projects that use alternate QueryId > distinct from the famous pg_stat_statements jumbling algorithm. I'd like to welcome the standard QueryID that DBAs and extension developers can depend on. Are you surveying the needs

Re: Proposal to suppress errors thrown by to_reg*()

2019-03-19 Thread Kyotaro HORIGUCHI
At Wed, 20 Mar 2019 07:13:28 +0900 (JST), Tatsuo Ishii wrote in <20190320.071328.48576044685486.t-is...@sraoss.co.jp> > >> I (and Hoshiai-san) concern about following case: > >> > >> # revoke usage on schema s1 from foo; > >> REVOKE > >> : > >> [connect as foo] > >> test=> select to_regclass

Re: speeding up planning with partitions

2019-03-19 Thread Robert Haas
On Fri, Mar 8, 2019 at 4:18 AM Amit Langote wrote: > Maybe you know that range_table_mutator() spends quite a long time if > there are many target children, but I realized there's no need for > range_table_mutator() to copy/mutate child target RTEs. First, there's > nothing to translate in their

Re: speeding up planning with partitions

2019-03-19 Thread Amit Langote
On 2019/03/20 9:49, Robert Haas wrote: > On Fri, Mar 8, 2019 at 4:18 AM Amit Langote > wrote: >> Maybe you know that range_table_mutator() spends quite a long time if >> there are many target children, but I realized there's no need for >> range_table_mutator() to copy/mutate child target RTEs. F

Re: Sparse bit set data structure

2019-03-19 Thread Heikki Linnakangas
On 14/03/2019 17:37, Julien Rouhaud wrote: On Wed, Mar 13, 2019 at 8:18 PM Heikki Linnakangas wrote: I started to consider rewriting the data structure into something more like B-tree. Then I remembered that I wrote a data structure pretty much like that last year already! We discussed that on

Re: [survey] New "Stable" QueryId based on normalized query text

2019-03-19 Thread Kyotaro HORIGUCHI
At Wed, 20 Mar 2019 00:23:30 +, "Tsunakawa, Takayuki" wrote in <0A3221C70F24FB45833433255569204D1FBE20A4@G01JPEXMBYT05> > From: legrand legrand [mailto:legrand_legr...@hotmail.com] > > There are many projects that use alternate QueryId > > distinct from the famous pg_stat_statements jumbling

Re: BUG #15668: Server crash in transformPartitionRangeBounds

2019-03-19 Thread Michael Paquier
On Thu, Mar 14, 2019 at 01:23:08PM +0900, Michael Paquier wrote: > I actually think that what you propose here makes more sense than what > HEAD does because the most inner expression gets evaluated first. > This for example generates the same error as on HEAD: > =# create table foo (a int default

Re: Special role for subscriptions

2019-03-19 Thread Andrey Borodin
Hi! > 13 марта 2019 г., в 22:55, Evgeniy Efimkin > написал(а): > > I've prepare patch with new system role, i'm not sure about name, called it > "pg_subscription_users". > In that patch we don't check permissions on target tables, i don't know, > should we check it? Currently, user with pg_s

RE: Willing to fix a PQexec() in libpq module

2019-03-19 Thread Wu, Fei
Hi, thanks for all replies. According to all your discussions, Maybe the problems is that 1) keep modifications just in client side; 2) modifications VS client current applications Maybe we could create a new function(May called PQexecSafe() ) just likes PQexec() but with additional input argumen

RE: speeding up planning with partitions

2019-03-19 Thread Imai, Yoshikazu
Amit-san, On Wed, Mar 20, 2019 at 0:42 AM, Amit Langote wrote: > On 2019/03/19 20:13, Imai, Yoshikazu wrote: > > Thanks for new patches. > > I looked over them and there are little comments. > > > > ... > > > > I have no more comments about codes other than above :) > > I have fixed all. Attach

Re: Willing to fix a PQexec() in libpq module

2019-03-19 Thread Andres Freund
Hi, On 2019-03-20 02:19:54 +, Wu, Fei wrote: > Hi, thanks for all replies. > According to all your discussions, Maybe the problems is that > 1) keep modifications just in client side; > 2) modifications VS client current applications > > Maybe we could create a new function(May called PQexecS

RE: Transaction commits VS Transaction commits (with parallel) VS query mean time

2019-03-19 Thread Jamison, Kirk
I tried to confirm the patch with the following configuration: max_parallel_workers_per_gather = 2 autovacuum = off postgres=# BEGIN; BEGIN postgres=# select xact_commit from pg_stat_database where datname = 'postgres'; xact_commit - 118 (1 row) postgres=# explain analyz

Re: speeding up planning with partitions

2019-03-19 Thread Amit Langote
Imai-san, On 2019/03/20 11:21, Imai, Yoshikazu wrote: > (4) > We expect the performance does not depend on the number of partitions after > applying all patches, if possible. > > num of partTPS > --- - > 1024 7,257 (7274, 7246, 7252) > 2048 6,718 (6627, 6780, 674

Re: Sparse bit set data structure

2019-03-19 Thread Andrey Borodin
Hi! Great job! > 20 марта 2019 г., в 9:10, Heikki Linnakangas написал(а): > > Please review, if you have a chance. > > - Heikki > <0001-Add-IntegerSet-to-hold-large-sets-of-64-bit-ints-eff.patch> I'm looking into the code and have few questions: 1. I'm not sure it is the best interface for i

RE: speeding up planning with partitions

2019-03-19 Thread Imai, Yoshikazu
Amit-san, On Wed, Mar 20, 2019 at 2:34 AM, Amit Langote wrote: > On 2019/03/20 11:21, Imai, Yoshikazu wrote: > > (4) > > We expect the performance does not depend on the number of partitions > after applying all patches, if possible. > > > > num of partTPS > > --- - > > 1024

Re: [HACKERS] generated columns

2019-03-19 Thread Michael Paquier
On Mon, Mar 18, 2019 at 03:14:09PM +0100, Pavel Stehule wrote: > postgres=# update foo set name = 'bbbxx' where id = 1; -- error > ERROR: no generation expression found for column number 3 of table > "foo" Yes I can see the problem after adding a generated column and dropping it on an INSERT quer

Re: Automated way to find actual COMMIT LSN of subxact LSN

2019-03-19 Thread Kyotaro HORIGUCHI
At Tue, 19 Mar 2019 12:16:34 -0500, Jeremy Finzel wrote in > I want to build automation to recover a database to a specific LSN > *inclusive*, even if that LSN is from a subtransaction. The problem I am > facing is that I know what specific LSN wrote a row on a remote system, but > if I create

Re: speeding up planning with partitions

2019-03-19 Thread Amit Langote
On 2019/03/20 11:51, Imai, Yoshikazu wrote: > Amit-san, > > On Wed, Mar 20, 2019 at 2:34 AM, Amit Langote wrote: >> On 2019/03/20 11:21, Imai, Yoshikazu wrote: >>> (4) >>> We expect the performance does not depend on the number of partitions >> after applying all patches, if possible. >>> >>> num

RE: [survey] New "Stable" QueryId based on normalized query text

2019-03-19 Thread Tsunakawa, Takayuki
From: Kyotaro HORIGUCHI [mailto:horiguchi.kyot...@lab.ntt.co.jp] > > > needs.1: stable accross different databases, > > > > Does this mean different database clusters, not different databases in > a single database cluster? > > Does this mean you want different QueryID for the same-looking > query

RE: speeding up planning with partitions

2019-03-19 Thread Imai, Yoshikazu
Amit-san, On Wed, Mar 20, 2019 at 3:01 PM, Amit Langote wrote: > > On Wed, Mar 20, 2019 at 2:34 AM, Amit Langote wrote: > >> On 2019/03/20 11:21, Imai, Yoshikazu wrote: > >>> (4) > >>> We expect the performance does not depend on the number of > >>> partitions > >> after applying all patches, if p

Re: Special role for subscriptions

2019-03-19 Thread Euler Taveira
Em qui, 14 de mar de 2019 às 00:03, Stephen Frost escreveu: > > I view that as the first step towards building a more granular privilege > system for subscription creation, and that was the second half of what I > was trying to say before- I do think there's value in having something > more granul

Re: [HACKERS] Block level parallel vacuum

2019-03-19 Thread Masahiko Sawada
On Tue, Mar 19, 2019 at 7:29 PM Kyotaro HORIGUCHI wrote: > > At Tue, 19 Mar 2019 17:51:32 +0900, Masahiko Sawada > wrote in > > On Tue, Mar 19, 2019 at 10:39 AM Haribabu Kommi > > wrote: > > > The performance results are good. Do we want to add the recommended > > > size in the document for th

selecting from partitions and constraint exclusion

2019-03-19 Thread Amit Langote
Hi, While looking at a partition pruning bug [1], I noticed something that started to feel like a regression: Setup: create table p (a int) partition by list (a); create table p1 partition of p for values in (1); In PG 10: set constraint_exclusion to on; explain select * from p1 where a = 2;

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-03-19 Thread Paul Guo
On Tue, Mar 19, 2019 at 2:18 PM Michael Paquier wrote: > On Tue, Mar 19, 2019 at 02:09:03PM +0800, Paul Guo wrote: > > The first patch adds an option to automatically generate recovery conf > > contents in related files, following pg_basebackup. In the patch, > > GenerateRecoveryConf(), WriteReco

Re: Proposal to suppress errors thrown by to_reg*()

2019-03-19 Thread Takuma Hoshiai
On Wed, 20 Mar 2019 09:48:59 +0900 (Tokyo Standard Time) Kyotaro HORIGUCHI wrote: > At Wed, 20 Mar 2019 07:13:28 +0900 (JST), Tatsuo Ishii > wrote in <20190320.071328.48576044685486.t-is...@sraoss.co.jp> > > >> I (and Hoshiai-san) concern about following case: > > >> > > >> # revoke usage

Re: Offline enabling/disabling of data checksums

2019-03-19 Thread Michael Paquier
On Wed, Mar 20, 2019 at 08:09:07AM +0900, Michael Paquier wrote: > In short, you keep the main feature with: > - No tweaks with postmaster.pid. > - Rely just on the control file indicating an instance shutdown > cleanly. > - No tweaks with the system ID. > - No renaming of the control file. FWIW,

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-03-19 Thread Michael Paquier
On Wed, Mar 20, 2019 at 12:48:52PM +0800, Paul Guo wrote: > This is a good suggestion also. Will do it. Please note also that we don't care about recovery.conf since v12 as recovery parameters are now GUCs. I would suggest appending those extra parameters to postgresql.auto.conf, which is what pg

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-03-19 Thread Paul Guo
On Wed, Mar 20, 2019 at 1:20 PM Michael Paquier wrote: > On Wed, Mar 20, 2019 at 12:48:52PM +0800, Paul Guo wrote: > > This is a good suggestion also. Will do it. > > Please note also that we don't care about recovery.conf since v12 as > recovery parameters are now GUCs. I would suggest appendin

Re: current_logfiles not following group access and instead follows log_file_mode permissions

2019-03-19 Thread Michael Paquier
On Fri, Mar 15, 2019 at 06:51:37PM +1100, Haribabu Kommi wrote: > IMO, this update is just a recommendation to the user, and sometimes it is > still possible that there may be strict permissions for the log file > even the data directory is allowed for the group access. So I feel > it is still bett

Re: Proposal to suppress errors thrown by to_reg*()

2019-03-19 Thread Pavel Stehule
st 20. 3. 2019 v 5:55 odesílatel Takuma Hoshiai napsal: > On Wed, 20 Mar 2019 09:48:59 +0900 (Tokyo Standard Time) > Kyotaro HORIGUCHI wrote: > > > At Wed, 20 Mar 2019 07:13:28 +0900 (JST), Tatsuo Ishii < > is...@sraoss.co.jp> wrote in < > 20190320.071328.48576044685486.t-is...@sraoss.co.jp>

Re: [Patch] pg_rewind: options to use restore_command from recovery.conf or command line

2019-03-19 Thread Andrey Borodin
Hi! > 7 марта 2019 г., в 20:27, Alexey Kondratov > написал(а): > > I'm a bit confused by by console output routines. E.g. in pg_rewind's main() you call pg_fatal()s, and printf(), and pg_log() with various levels. Shouldn't we use all the pg_* functions? But most of this printing usages we

RE: Libpq support to connect to standby server as priority

2019-03-19 Thread Tsunakawa, Takayuki
From: Robert Haas [mailto:robertmh...@gmail.com] > I really dislike having both target_sesion_attrs and > target_server_type. It doesn't solve any actual problem. master, > slave, prefer-save, or whatever you like could be put in > target_session_attrs just as easily, and then we wouldn't end up

RE: [PATCH] get rid of StdRdOptions, use individual binary reloptions representation for each relation kind instead

2019-03-19 Thread Iwata, Aya
Hi, > hio.c: > > -saveFreeSpace = RelationGetTargetPageFreeSpace(relation, > - > HEAP_DEFAULT_FILLFACTOR); > +if (IsToastRelation(relation)) > +saveFreeSpace = ToastGetTargetPageFreeSpace(); > +else > +saveFreeSpace = HeapGetTargetPageFreeSpace(relation); > > This loc

Re: Offline enabling/disabling of data checksums

2019-03-19 Thread Fabien COELHO
Hallo Andres, And you're basically adding it because Fabien doesn't like postmaster.pid and wants to invent another lockout mechanism in this thread. I did not suggest to rename the control file, but as it is already done by another command it did not look like a bad idea in itself, or at le

<    1   2