Re: [HACKERS] Assertion failure in REL9_5_STABLE

2016-08-10 Thread Michael Paquier
On Thu, Aug 11, 2016 at 8:09 AM, Marko Tiikkaja wrote: > On 2016-08-11 12:09 AM, Alvaro Herrera wrote: >> >> BTW this is not a regression, right? It's been broken all along. Or am >> I mistaken? > > > You're probably right. I just hadn't realized I could run our app against > 9.5 with --enable-

Re: [HACKERS] new autovacuum criterion for visible pages

2016-08-10 Thread Michael Paquier
On Thu, Aug 11, 2016 at 5:39 AM, Jeff Janes wrote: > I wanted to create a new relopt named something like > autovacuum_vacuum_pagevisible_factor which would cause autovacuum to > vacuum a table once less than a certain fraction of the relation's > pages are marked allvisible. Interesting idea. >

Re: [HACKERS] regression test for extended query protocol

2016-08-10 Thread Michael Paquier
On Thu, Aug 11, 2016 at 5:33 AM, Alvaro Herrera wrote: > Michael Paquier wrote: >> On Fri, Aug 5, 2016 at 12:21 AM, Alvaro Herrera >> wrote: >> > If somebody had some spare time to devote to this, I would suggest to >> > implement something in core that can be used to specify a list of >> > comma

Re: [HACKERS] Surprising behaviour of \set AUTOCOMMIT ON

2016-08-10 Thread Venkata Balaji N
On Thu, Aug 11, 2016 at 2:58 PM, Venkata Balaji N wrote: > > On Tue, Aug 9, 2016 at 1:02 AM, Robert Haas wrote: > >> On Mon, Aug 8, 2016 at 10:10 AM, Rahila Syed >> wrote: >> > Thank you for inputs everyone. >> > >> > The opinions on this thread can be classified into following >> > 1. Commit >

Re: [HACKERS] multivariate statistics (v19)

2016-08-10 Thread Michael Paquier
On Thu, Aug 11, 2016 at 3:34 AM, Tomas Vondra wrote: > On 08/10/2016 02:23 PM, Michael Paquier wrote: >> >> On Wed, Aug 10, 2016 at 8:33 PM, Tomas Vondra >> wrote: >>> The idea is that the syntax should work even for statistics built on >>> multiple tables, e.g. to provide better statistics for j

Re: [HACKERS] Heap WARM Tuples - Design Draft

2016-08-10 Thread Amit Kapila
On Mon, Aug 8, 2016 at 9:56 PM, Bruce Momjian wrote: > On Mon, Aug 8, 2016 at 06:34:46PM +0530, Amit Kapila wrote: >> I think here expensive part would be recheck for the cases where the >> index value is changed to a different value (value which doesn't exist >> in WARM chain). You anyway have

Re: [HACKERS] Surprising behaviour of \set AUTOCOMMIT ON

2016-08-10 Thread Venkata Balaji N
On Tue, Aug 9, 2016 at 1:02 AM, Robert Haas wrote: > On Mon, Aug 8, 2016 at 10:10 AM, Rahila Syed > wrote: > > Thank you for inputs everyone. > > > > The opinions on this thread can be classified into following > > 1. Commit > > 2. Rollback > > 3. Error > > 4. Warning > > > > As per opinion upth

Re: [HACKERS] Btree Index on PostgreSQL and Wiredtiger (MongoDB3.2)

2016-08-10 Thread Kisung Kim
Thank you for your information. Here is the result: After insertions: ycsb=# select * from pgstatindex('usertable_pkey'); version | tree_level | index_size | root_block_no | internal_pages | leaf_pages | empty_pages | deleted_pages | avg_leaf_density | leaf_fragmentation -++-

Re: [HACKERS] phrase search TS_phrase_execute code readability patch

2016-08-10 Thread Robert Haas
On Tue, Aug 9, 2016 at 3:35 PM, David G. Johnston wrote: > I don't follow why LposStart is needed so I removed it... That doesn't seem very reasonable. > Not compiled or in any way tested... Please do not bother submitting patches that you aren't prepared to compile and test. -- Robert Haas E

Re: [HACKERS] No longer possible to query catalogs for index capabilities?

2016-08-10 Thread Robert Haas
On Wed, Aug 10, 2016 at 6:14 PM, Tom Lane wrote: > Kevin Grittner writes: >> That one seems like it should either be at the AM level or not >> included at all. Where it would be interesting to know is if you >> are a hacker looking for an AM to enhance with support, or (when >> there is more tha

Re: [HACKERS] new pgindent run before branch?

2016-08-10 Thread Robert Haas
On Wed, Aug 10, 2016 at 10:23 PM, Tom Lane wrote: >> Great. Are you handling creating the new branch? > > Yeah, I guess. I've made most of them lately. I'll do it if you want. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hacke

Re: [HACKERS] new pgindent run before branch?

2016-08-10 Thread Bruce Momjian
On Wed, Aug 10, 2016 at 10:35:44PM -0400, Tom Lane wrote: > Bruce Momjian writes: > > It sounds like you are saying that the branch is to happen before the > > pgindent run. Am I missing something? > > I read it as pgindent then branch. OK, good. -- Bruce Momjian http://momjian.us

Re: [HACKERS] new pgindent run before branch?

2016-08-10 Thread Tom Lane
Bruce Momjian writes: > It sounds like you are saying that the branch is to happen before the > pgindent run. Am I missing something? I read it as pgindent then branch. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make cha

Re: [HACKERS] new pgindent run before branch?

2016-08-10 Thread Bruce Momjian
On Wed, Aug 10, 2016 at 10:31:35PM -0400, Tom Lane wrote: > Bruce Momjian writes: > > I am confused --- are you pgindenting just HEAD and not 9.6? Why? > > The point is to pgindent while they're still the same. So I read this: > >> +1, I was planning to do that myself. > > > Great. Are you h

Re: [HACKERS] new pgindent run before branch?

2016-08-10 Thread Tom Lane
Bruce Momjian writes: > I am confused --- are you pgindenting just HEAD and not 9.6? Why? The point is to pgindent while they're still the same. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscriptio

Re: [HACKERS] new pgindent run before branch?

2016-08-10 Thread Bruce Momjian
On Wed, Aug 10, 2016 at 10:23:14PM -0400, Tom Lane wrote: > Robert Haas writes: > > On Wed, Aug 10, 2016 at 5:04 PM, Tom Lane wrote: > >> Robert Haas writes: > >>> Would anyone mind too much if I refreshed typedefs.list and > >>> re-indented the whole tree before we branch? > > >> +1, I was pla

Re: [HACKERS] new pgindent run before branch?

2016-08-10 Thread Tom Lane
Robert Haas writes: > On Wed, Aug 10, 2016 at 5:04 PM, Tom Lane wrote: >> Robert Haas writes: >>> Would anyone mind too much if I refreshed typedefs.list and >>> re-indented the whole tree before we branch? >> +1, I was planning to do that myself. > Great. Are you handling creating the new br

Re: [HACKERS] new pgindent run before branch?

2016-08-10 Thread Robert Haas
On Wed, Aug 10, 2016 at 5:04 PM, Tom Lane wrote: > Robert Haas writes: >> Would anyone mind too much if I refreshed typedefs.list and >> re-indented the whole tree before we branch? > > +1, I was planning to do that myself. Great. Are you handling creating the new branch? If so, it probably ma

Re: [HACKERS] Btree Index on PostgreSQL and Wiredtiger (MongoDB3.2)

2016-08-10 Thread Lukas Fittl
On Wed, Aug 10, 2016 at 4:24 PM, Kisung Kim wrote: > > When I used the index bloating estimation script in > https://github.com/ioguix/pgsql-bloat-estimation, > the result is as follows: > Regardless of the issue at hand, it might make sense to verify these statistics using pgstattuple - those bl

Re: [HACKERS] Set log_line_prefix and application name in test drivers

2016-08-10 Thread Peter Eisentraut
On 8/10/16 5:18 PM, Tom Lane wrote: > Or in short: I don't want to be seeing one prefix format in some buildfarm > logs and a different format in others. Sure. My patch has log_line_prefix = '%t [%p]: [%l] %qapp=%a ' which is modeled after the pgfouine recommendation, which is I believe a wide-

Re: Improved ICU patch - WAS: [HACKERS] Implementing full UTF-8 support (aka supporting 0x00)

2016-08-10 Thread Peter Geoghegan
On Wed, Aug 10, 2016 at 1:42 PM, Palle Girgensohn wrote: > They've been used for the FreeBSD ports since 2005, and have served us well. > I have of course updated them regularly. In this latest version, I've removed > support for other encodings beside UTF-8, mostly since I don't know how to >

[HACKERS] Btree Index on PostgreSQL and Wiredtiger (MongoDB3.2)

2016-08-10 Thread Kisung Kim
Hi, I've run YCSB(Yahoo! Cloud Service Benchmark) on PostgreSQL and MongoDB with WiredTiger. And I found some interesting results and some issues(maybe) on Btree index of PostgreSQL. Here is my experiments and results. YCSB is for document store benchmark and I build following schema in PG. CREA

Re: [HACKERS] Assertion failure in REL9_5_STABLE

2016-08-10 Thread Marko Tiikkaja
On 2016-08-11 12:09 AM, Alvaro Herrera wrote: BTW this is not a regression, right? It's been broken all along. Or am I mistaken? You're probably right. I just hadn't realized I could run our app against 9.5 with --enable-cassert until last week. .m -- Sent via pgsql-hackers mailing lis

Re: [HACKERS] Assertion failure in REL9_5_STABLE

2016-08-10 Thread Marko Tiikkaja
On 2016-08-10 11:01 PM, Alvaro Herrera wrote: Oh, I see ... so there's an update chain, and you're updating a earlier tuple. But the later tuple has a multixact and one of the members is the current transaction. I wonder how can you lock a tuple that's not the latest, where that update chain wa

Re: [HACKERS] No longer possible to query catalogs for index capabilities?

2016-08-10 Thread Tom Lane
Andrew Gierth writes: > So these properties (I've changed all the names here, suggestions > welcome for better ones) I think should be testable on the AM, each with > an example of why: > can_order > can_unique > can_multi_col > can_exclude Check, flags that indicate what you can put in

Re: [HACKERS] No longer possible to query catalogs for index capabilities?

2016-08-10 Thread Tom Lane
Kevin Grittner writes: > That one seems like it should either be at the AM level or not > included at all. Where it would be interesting to know is if you > are a hacker looking for an AM to enhance with support, or (when > there is more than just btree supported, so it is not so easy to > rememb

Re: [HACKERS] Assertion failure in REL9_5_STABLE

2016-08-10 Thread Alvaro Herrera
BTW this is not a regression, right? It's been broken all along. Or am I mistaken? -- Álvaro Herrerahttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make

Re: [HACKERS] Parallel tuplesort, partitioning, merging, and the future

2016-08-10 Thread Peter Geoghegan
On Wed, Aug 10, 2016 at 11:59 AM, Robert Haas wrote: > I think that last part is a very important property; my intuition is > that dividing up the work between cooperating processes in a way that > should come out equal will often fail to do so, either due to the > operating system scheduler or du

Re: [HACKERS] per-statement-level INSTEAD OF triggers

2016-08-10 Thread Emre Hasegeli
> It might be more useful after we get the infrastructure that Kevin's been > working on to allow collecting all the updates into a tuplestore that > could be passed to a statement-level trigger. Right now I tend to agree > that there's little point. Maybe, this can be used to re-implement FOREIG

Re: [HACKERS] Set log_line_prefix and application name in test drivers

2016-08-10 Thread Tom Lane
Peter Eisentraut writes: > On 8/9/16 12:16 PM, Tom Lane wrote: >> Peter Eisentraut writes: >>> Here is a small patch that sets log_line_prefix and application name in >>> pg_regress and the TAP tests, to make analyzing the server log output >>> easier. >> How would this interact with the buildfa

Re: [HACKERS] Parallel tuplesort, partitioning, merging, and the future

2016-08-10 Thread Peter Geoghegan
On Wed, Aug 10, 2016 at 12:08 PM, Claudio Freire wrote: > I think it's a great design, but for that, per-worker final tapes have > to always be random-access. Thanks. I don't think I need to live with the randomAccess restriction, because I can be clever about reading only the first tuple on each

Re: [HACKERS] new pgindent run before branch?

2016-08-10 Thread Tom Lane
Robert Haas writes: > Would anyone mind too much if I refreshed typedefs.list and > re-indented the whole tree before we branch? +1, I was planning to do that myself. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make change

Re: [HACKERS] Assertion failure in REL9_5_STABLE

2016-08-10 Thread Alvaro Herrera
Marko Tiikkaja wrote: > On 2016-08-10 19:28, Alvaro Herrera wrote: > >Umm. AFAICS HeapTupleSatisfiesUpdate() only returns SelfUpdated after > >already calling HeapTupleHeaderGetCmax() (which obviously hasn't caught > >the same assertion). Something is odd there ... > > HeapTupleSatisfiesUpdate()

Re: [HACKERS] Parallel tuplesort, partitioning, merging, and the future

2016-08-10 Thread Peter Geoghegan
On Wed, Aug 10, 2016 at 11:59 AM, Robert Haas wrote: > My view on this - currently anyway - is that we shouldn't conflate the > tuplesort with the subsequent index generation, but that we should try > to use parallelism within the tuplesort itself to the greatest extent > possible. If there is a

Re: [HACKERS] Assertion failure in REL9_5_STABLE

2016-08-10 Thread Marko Tiikkaja
On 2016-08-10 19:28, Alvaro Herrera wrote: Umm. AFAICS HeapTupleSatisfiesUpdate() only returns SelfUpdated after already calling HeapTupleHeaderGetCmax() (which obviously hasn't caught the same assertion). Something is odd there ... HeapTupleSatisfiesUpdate() returns HeapTupleBeingUpdated in

Re: [HACKERS] new pgindent run before branch?

2016-08-10 Thread Bruce Momjian
On Wed, Aug 10, 2016 at 04:02:27PM -0400, Robert Haas wrote: > Hi, > > Would anyone mind too much if I refreshed typedefs.list and > re-indented the whole tree before we branch? There's not too much > churn at the moment, and I feel like it would be nice to start the > cycle in as clean a state a

Improved ICU patch - WAS: [HACKERS] Implementing full UTF-8 support (aka supporting 0x00)

2016-08-10 Thread Palle Girgensohn
> 4 aug. 2016 kl. 02:40 skrev Bruce Momjian : > > On Thu, Aug 4, 2016 at 08:22:25AM +0800, Craig Ringer wrote: >> Yep, it does. But we've made little to no progress on integration of ICU >> support and AFAIK nobody's working on it right now. > > Uh, this email from July says Peter Eisentraut wi

Re: [HACKERS] Wait events monitoring future development

2016-08-10 Thread Robert Haas
On Tue, Aug 9, 2016 at 12:07 AM, Tsunakawa, Takayuki wrote: > As another idea, we can stand on the middle ground. Interestingly, MySQL > also enables their event monitoring (Performance Schema) by default, but not > all events are collected. I guess highly encountered events are not > collect

[HACKERS] new autovacuum criterion for visible pages

2016-08-10 Thread Jeff Janes
I wanted to create a new relopt named something like autovacuum_vacuum_pagevisible_factor which would cause autovacuum to vacuum a table once less than a certain fraction of the relation's pages are marked allvisible. I wanted some feedback on some things. 1) One issue is that pg_class.relpages a

Re: [HACKERS] Slowness of extended protocol

2016-08-10 Thread Vladimir Sitnikov
Shay> it's important to note that query parsing and rewriting isn't an "inevitable evil". Ok, I stay corrected. ADO.NET have raw mode in the API. That's interesting. Let's say "lots of heavily used languages do have their own notion of bind placeholders". And for the reset, it is still not that h

Re: [HACKERS] regression test for extended query protocol

2016-08-10 Thread Alvaro Herrera
Michael Paquier wrote: > On Fri, Aug 5, 2016 at 12:21 AM, Alvaro Herrera > wrote: > > If somebody had some spare time to devote to this, I would suggest to > > implement something in core that can be used to specify a list of > > commands to run, and a list of files-to-be-saved-in-bf-log emitted b

[HACKERS] new pgindent run before branch?

2016-08-10 Thread Robert Haas
Hi, Would anyone mind too much if I refreshed typedefs.list and re-indented the whole tree before we branch? There's not too much churn at the moment, and I feel like it would be nice to start the cycle in as clean a state as possible. Current results of this attached. -- Robert Haas Enterpris

Re: [HACKERS] Heap WARM Tuples - Design Draft

2016-08-10 Thread Claudio Freire
On Wed, Aug 10, 2016 at 4:37 PM, Simon Riggs wrote: > On 10 August 2016 at 03:45, Pavan Deolasee wrote: >> >> >> On Tue, Aug 9, 2016 at 12:06 AM, Claudio Freire >> wrote: >>> >>> On Mon, Aug 8, 2016 at 2:52 PM, Pavan Deolasee >>> wrote: >>> >>> > Some heuristics and limits on amount of work don

Re: [HACKERS] Slowness of extended protocol

2016-08-10 Thread Shay Rojansky
Some comments... For the record, I do find implicit/transparent driver-level query preparation interesting and potentially useful, and have opened https://github.com/npgsql/npgsql/issues/1237 to think about it - mostly based on arguments on this thread. One big question mark I have is whether this

Re: [HACKERS] Heap WARM Tuples - Design Draft

2016-08-10 Thread Simon Riggs
On 10 August 2016 at 03:45, Pavan Deolasee wrote: > > > On Tue, Aug 9, 2016 at 12:06 AM, Claudio Freire > wrote: >> >> On Mon, Aug 8, 2016 at 2:52 PM, Pavan Deolasee >> wrote: >> >> > Some heuristics and limits on amount of work done to detect duplicate >> > index >> > entries will help too. >>

Re: [HACKERS] Set log_line_prefix and application name in test drivers

2016-08-10 Thread Peter Eisentraut
On 8/9/16 12:16 PM, Tom Lane wrote: > Peter Eisentraut writes: >> > Here is a small patch that sets log_line_prefix and application name in >> > pg_regress and the TAP tests, to make analyzing the server log output >> > easier. > How would this interact with the buildfarm's existing policies > on

Re: [HACKERS] Parallel tuplesort, partitioning, merging, and the future

2016-08-10 Thread Claudio Freire
On Mon, Aug 8, 2016 at 4:44 PM, Peter Geoghegan wrote: > The basic idea I have in mind is that we create runs in workers in the > same way that the parallel CREATE INDEX patch does (one output run per > worker). However, rather than merging in the leader, we use a > splitting algorithm to determin

Re: [HACKERS] Parallel tuplesort, partitioning, merging, and the future

2016-08-10 Thread Robert Haas
On Mon, Aug 8, 2016 at 3:44 PM, Peter Geoghegan wrote: > I don't think partitioning is urgent for CREATE INDEX, and may be > inappropriate for CREATE INDEX under any circumstances, because: > > * Possible problems with parallel infrastructure and writes. > * Unbalanced B-Trees (or the risk thereof

Re: [HACKERS] Proposal for CSN based snapshots

2016-08-10 Thread Alexander Korotkov
On Wed, Aug 10, 2016 at 8:26 PM, Greg Stark wrote: > On Wed, Aug 10, 2016 at 5:54 PM, Alexander Korotkov > wrote: > > Oh, I found that I underestimated complexity of async commit... :) > > Indeed. I think Tom's attitude was right even if the specific > conclusion was wrong. While I don't think

Re: [HACKERS] Proposal for CSN based snapshots

2016-08-10 Thread Kevin Grittner
On Wed, Aug 10, 2016 at 12:26 PM, Greg Stark wrote: > Complexity like this makes it hard to implement other features such as > CSNs. IIRC this already bit hot standby as well. I think it would be a > big improvement if we had a clear, well defined commit order that was > easy to explain and easy

Re: [HACKERS] multivariate statistics (v19)

2016-08-10 Thread Tomas Vondra
On 08/10/2016 02:23 PM, Michael Paquier wrote: On Wed, Aug 10, 2016 at 8:33 PM, Tomas Vondra wrote: On 08/10/2016 06:41 AM, Michael Paquier wrote: Patch 0001: there have been comments about that before, and you have put the checks on RestrictInfo in a couple of variables of pull_varnos_walker,

Re: [HACKERS] pg_ctl promote wait

2016-08-10 Thread Peter Eisentraut
On 8/7/16 9:44 PM, Michael Paquier wrote: >>> This is not a good >>> >> idea, and the idea of putting a wait argument in get_controlfile does >>> >> not seem a good interface to me. I'd rather see get_controlfile be >>> >> extended with a flag saying no_error_on_failure and keep the wait >>> >> log

Re: [HACKERS] Declarative partitioning - another take

2016-08-10 Thread Robert Haas
On Wed, Aug 10, 2016 at 7:09 AM, Amit Langote wrote: > Attached is the latest set of patches to implement declarative > partitioning. Cool. I would encourage you to give some thought to what is the least committable subset of these patches, and think about whether it can be reorganized to make t

Re: [HACKERS] multivariate statistics (v19)

2016-08-10 Thread Tomas Vondra
On 08/10/2016 02:24 PM, Michael Paquier wrote: On Wed, Aug 10, 2016 at 8:50 PM, Petr Jelinek wrote: On 10/08/16 13:33, Tomas Vondra wrote: On 08/10/2016 06:41 AM, Michael Paquier wrote: On Wed, Aug 3, 2016 at 10:58 AM, Tomas Vondra 2) combining multiple statistics I think the ability to

Re: [HACKERS] multivariate statistics (v19)

2016-08-10 Thread Tomas Vondra
On 08/10/2016 03:29 PM, Ants Aasma wrote: On Wed, Aug 3, 2016 at 4:58 AM, Tomas Vondra wrote: 2) combining multiple statistics I think the ability to combine multivariate statistics (covering different subsets of conditions) is important and useful, but I'm starting to think that the current i

Re: [HACKERS] Is there a way around function search_path killing SQL function inlining?

2016-08-10 Thread Regina Obe
> Michael Banck writes: >> As I've been bitten by this problem recently, I thought I'd take a >> look at editing the PostGIS extension SQL file to this end, but >> contrary to the above, the @extschema@ feature only applies to >> non-relocatable extensions, from src/backend/commands/extension.

Re: [HACKERS] Heap WARM Tuples - Design Draft

2016-08-10 Thread Claudio Freire
On Tue, Aug 9, 2016 at 11:39 PM, Jim Nasby wrote: > On 8/9/16 6:44 PM, Claudio Freire wrote: >> >> Since we can lookup all occurrences of k1=a index=0 and k2=a index=0, >> and in fact we probably did so already as part of the update logic > > > That's a change from what currently happens, right? >

Re: [HACKERS] No longer possible to query catalogs for index capabilities?

2016-08-10 Thread Kevin Grittner
On Wed, Aug 10, 2016 at 12:31 PM, Andrew Gierth wrote: > These could be limited to being testable only on a specified index, and > not AM-wide: > predicate_locks (? maybe) That one seems like it should either be at the AM level or not included at all. Where it would be interesting to know is

Re: [HACKERS] No longer possible to query catalogs for index capabilities?

2016-08-10 Thread Andrew Gierth
> "Tom" == Tom Lane writes: >> - this still has everything in amapi.c rather than creating any new >> files. Also, the regression tests are in create_index.sql for lack >> of any obviously better place. Tom> This more than doubles the size of amapi.c, so it has a definite Tom> feel of t

Re: [HACKERS] Proposal for CSN based snapshots

2016-08-10 Thread Ants Aasma
On Wed, Aug 10, 2016 at 7:54 PM, Alexander Korotkov wrote: > Oh, I found that I underestimated complexity of async commit... :) > > Do I understand right that now async commit right as follows? > 1) Async transaction confirms commit before flushing WAL. > 2) Other transactions sees effect of asyn

Re: [HACKERS] Assertion failure in REL9_5_STABLE

2016-08-10 Thread Alvaro Herrera
Marko Tiikkaja wrote: > Hi, > > Running one specific test from our application against REL9_5_STABLE > (current as of today) gives me this: > > #2 0x7effb59595be in ExceptionalCondition ( > conditionName=conditionName@entry=0x7effb5b27a88 "!(CritSectionCount > 0 > || TransactionIdIsCurre

Re: [HACKERS] Slowness of extended protocol

2016-08-10 Thread Vladimir Sitnikov
Stephen> While it may have good results in many cases, it's not accurate to say that using prepared statements will always be faster than not. There's no silver bullet. <-- that is accurate, but it is useless for end-user applications I've never claimed that "server prepared statement" is a silver

Re: [HACKERS] Proposal for CSN based snapshots

2016-08-10 Thread Greg Stark
On Wed, Aug 10, 2016 at 5:54 PM, Alexander Korotkov wrote: > Oh, I found that I underestimated complexity of async commit... :) Indeed. I think Tom's attitude was right even if the specific conclusion was wrong. While I don't think removing async commit is viable I think it would be a laudable g

Re: [HACKERS] Proposal for CSN based snapshots

2016-08-10 Thread Heikki Linnakangas
On 08/10/2016 07:54 PM, Alexander Korotkov wrote: Do I understand right that now async commit right as follows? 1) Async transaction confirms commit before flushing WAL. Yes. 2) Other transactions sees effect of async transaction only when its WAL flushed. No. Other transactions also see th

Re: [HACKERS] Slowness of extended protocol

2016-08-10 Thread Stephen Frost
* Vladimir Sitnikov (sitnikov.vladi...@gmail.com) wrote: > 3) "suddently get slow the 6th time" is a PostgreSQL bug that both fails to > estimate cardinality properly, and it does not provide administrator a way > to disable the feature (generic vs specific plan). Dropping and recreating the prepa

Re: [HACKERS] Proposal for CSN based snapshots

2016-08-10 Thread Alexander Korotkov
On Wed, Aug 10, 2016 at 6:09 PM, Heikki Linnakangas wrote: > On 08/10/2016 05:51 PM, Tom Lane wrote: > >> Heikki Linnakangas writes: >> >>> On 08/10/2016 05:09 PM, Tom Lane wrote: >>> Uh, what? That's not the semantics we have today, and I don't see why it's necessary or a good idea.

Re: [HACKERS] Slowness of extended protocol

2016-08-10 Thread Vladimir Sitnikov
Stephen>I encourage you to look through the archives The thing is pl/pgsql suffers from exactly the same problem. pl/pgsql is not a typical language of choice (e.g. see Tiobe index and alike), so the probability of running into "prepared statement issues" was low. As more languages would use serv

Re: [HACKERS] Assertion failure in REL9_5_STABLE

2016-08-10 Thread Tom Lane
Marko Tiikkaja writes: > The failure is a number of levels down a call stack of PL/PgSQL > functions, but I can reproduce it at will by calling the function. I > unfortunately can't narrow it down to a smaller test case, but attached > is an xlogdump of the session. The query that finally bre

Re: [HACKERS] Proposal for CSN based snapshots

2016-08-10 Thread Ants Aasma
On Wed, Aug 10, 2016 at 6:09 PM, Heikki Linnakangas wrote: > Hmm. There's one more possible way this could all work. Let's have CSN == > LSN, also for asynchronous commits. A snapshot is the current insert > position, but also make note of the current flush position, when you take a > snapshot. No

[HACKERS] Assertion failure in REL9_5_STABLE

2016-08-10 Thread Marko Tiikkaja
Hi, Running one specific test from our application against REL9_5_STABLE (current as of today) gives me this: #2 0x7effb59595be in ExceptionalCondition ( conditionName=conditionName@entry=0x7effb5b27a88 "!(CritSectionCount > 0 || TransactionIdIsCurrentTransactionId(( (!((tup)->t_inf

Re: [HACKERS] Proposal for CSN based snapshots

2016-08-10 Thread Joshua D. Drake
On 08/10/2016 09:04 AM, Stephen Frost wrote: * Joshua D. Drake (j...@commandprompt.com) wrote: +1 for Robert here, removing async commit is a non-starter. It is PostgreSQL performance 101 that you disable synchronous commit unless you have a specific data/business requirement that needs it. Spec

Re: [HACKERS] Slowness of extended protocol

2016-08-10 Thread Vladimir Sitnikov
Robert>But that makes it the job of every driver to implement some sort of cache, which IMHO isn't a very reasonable position Let's wait what Shay decides on implementing query cache in npgsql ? Here's the issue: https://github.com/npgsql/npgsql/issues/1237 He could change his mind when it comes

Re: [HACKERS] Is there a way around function search_path killing SQL function inlining?

2016-08-10 Thread Tom Lane
Michael Banck writes: > As I've been bitten by this problem recently, I thought I'd take a look > at editing the PostGIS extension SQL file to this end, but contrary to > the above, the @extschema@ feature only applies to non-relocatable > extensions, from src/backend/commands/extension.c: > *

Re: [HACKERS] Slowness of extended protocol

2016-08-10 Thread Stephen Frost
* Vladimir Sitnikov (sitnikov.vladi...@gmail.com) wrote: > It works completely transparent to the application, and it does use > server-prepared statements even though application builds "brand new" sql > text every time. And is the source of frequent complaints on various mailing lists along the

Re: [HACKERS] Proposal for CSN based snapshots

2016-08-10 Thread Stephen Frost
* Joshua D. Drake (j...@commandprompt.com) wrote: > +1 for Robert here, removing async commit is a non-starter. It is > PostgreSQL performance 101 that you disable synchronous commit > unless you have a specific data/business requirement that needs it. > Specifically because of how much faster Pg i

Re: [HACKERS] Slowness of extended protocol

2016-08-10 Thread Robert Haas
On Wed, Aug 10, 2016 at 11:50 AM, Tom Lane wrote: > Robert Haas writes: >> Sure, but I don't want the application to have to know about that, and >> I don't really think the driver should need to know about that either. >> Your point, as I understand it, is that sufficiently good query >> caching

Re: [HACKERS] Is there a way around function search_path killing SQL function inlining?

2016-08-10 Thread Michael Banck
On Thu, Mar 10, 2016 at 11:48:41AM -0500, Tom Lane wrote: > If you're worried about preserving relocatability of an extension > containing such functions, the @extschema@ feature might help. As I've been bitten by this problem recently, I thought I'd take a look at editing the PostGIS extension SQ

Re: [HACKERS] Proposal for CSN based snapshots

2016-08-10 Thread Joshua D. Drake
On 08/10/2016 08:28 AM, Robert Haas wrote: On Wed, Aug 10, 2016 at 11:09 AM, Heikki Linnakangas wrote: Still, having to invent CSNs seems like a huge loss for this design. Personally I'd give up async commit first. If we had only sync commit, the rule could be "xact LSN less than snapshot thre

Re: [HACKERS] Slowness of extended protocol

2016-08-10 Thread Tom Lane
Robert Haas writes: > Sure, but I don't want the application to have to know about that, and > I don't really think the driver should need to know about that either. > Your point, as I understand it, is that sufficiently good query > caching in the driver can ameliorate the problem, and I agree wi

Re: [HACKERS] No longer possible to query catalogs for index capabilities?

2016-08-10 Thread Tom Lane
Andrew Gierth writes: > - this still has everything in amapi.c rather than creating any new > files. Also, the regression tests are in create_index.sql for lack > of any obviously better place. This more than doubles the size of amapi.c, so it has a definite feel of tail-wags-dog for me

Re: [HACKERS] Slowness of extended protocol

2016-08-10 Thread Robert Haas
On Tue, Aug 9, 2016 at 5:07 PM, Vladimir Sitnikov wrote: > I do not buy that "dynamically generated queries defeat server-prepared > usage" argument. It is just not true (see below). > > Do you mean "in language X, where X != Java it is impossible to implement a > query cache"? > That is just ridi

Re: [HACKERS] Proposal for CSN based snapshots

2016-08-10 Thread Robert Haas
On Wed, Aug 10, 2016 at 11:09 AM, Heikki Linnakangas wrote: >> Still, having to invent CSNs seems like a huge loss for this design. >> Personally I'd give up async commit first. If we had only sync commit, >> the rule could be "xact LSN less than snapshot threshold and less than >> WAL flush posi

Re: [HACKERS] Proposal for CSN based snapshots

2016-08-10 Thread Heikki Linnakangas
On 08/10/2016 05:51 PM, Tom Lane wrote: Heikki Linnakangas writes: On 08/10/2016 05:09 PM, Tom Lane wrote: Uh, what? That's not the semantics we have today, and I don't see why it's necessary or a good idea. Once the commit is in the WAL stream, any action taken on the basis of seeing the co

Re: [HACKERS] No longer possible to query catalogs for index capabilities?

2016-08-10 Thread Andrew Gierth
Updated patch. Changes: - returns NULL rather than "cache lookup failed" - added pg_index_column_has_property (incl. docs) - added regression tests Not changed / need consideration: - this still has everything in amapi.c rather than creating any new files. Also, the regression test

Re: [HACKERS] Proposal for CSN based snapshots

2016-08-10 Thread Tom Lane
Heikki Linnakangas writes: > On 08/10/2016 05:09 PM, Tom Lane wrote: >> Uh, what? That's not the semantics we have today, and I don't see why >> it's necessary or a good idea. Once the commit is in the WAL stream, >> any action taken on the basis of seeing the commit must be later in >> the WAL

Re: [HACKERS] Wait events monitoring future development

2016-08-10 Thread Bruce Momjian
On Wed, Aug 10, 2016 at 11:37:36PM +0900, Satoshi Nagayasu wrote: > Agreed. > > If people are facing with some difficult situation in terms of performance, > they may accept some (one-time) overhead to resolve the issue. > But if they don't have (recognize) any issue, they may not. > > That's one

Re: [HACKERS] Wait events monitoring future development

2016-08-10 Thread Satoshi Nagayasu
2016/08/10 23:22 "Bruce Momjian" : > > On Wed, Aug 10, 2016 at 05:14:52PM +0300, Alexander Korotkov wrote: > > On Tue, Aug 9, 2016 at 5:37 AM, Bruce Momjian wrote: > > > > On Tue, Aug 9, 2016 at 02:06:40AM +, Tsunakawa, Takayuki wrote: > > > I hope wait event monitoring will be on by

Re: [HACKERS] Proposal for CSN based snapshots

2016-08-10 Thread Heikki Linnakangas
On 08/10/2016 05:09 PM, Tom Lane wrote: Heikki Linnakangas writes: Imagine that you have a stream of normal, synchronous, commits. They get assigned LSNs: 1, 2, 3, 4. They become visible to other transactions in that order. The way I described this scheme in the first emails on this thread,

Re: [HACKERS] Wait events monitoring future development

2016-08-10 Thread Bruce Momjian
On Wed, Aug 10, 2016 at 05:14:52PM +0300, Alexander Korotkov wrote: > On Tue, Aug 9, 2016 at 5:37 AM, Bruce Momjian wrote: > > On Tue, Aug  9, 2016 at 02:06:40AM +, Tsunakawa, Takayuki wrote: > > I hope wait event monitoring will be on by default even if the overhead > is not >

Re: [HACKERS] Wait events monitoring future development

2016-08-10 Thread Alexander Korotkov
On Tue, Aug 9, 2016 at 5:37 AM, Bruce Momjian wrote: > On Tue, Aug 9, 2016 at 02:06:40AM +, Tsunakawa, Takayuki wrote: > > I hope wait event monitoring will be on by default even if the overhead > is not > > almost zero, because the data needs to be readily available for faster > > troublesh

Re: [HACKERS] Proposal for CSN based snapshots

2016-08-10 Thread Tom Lane
Heikki Linnakangas writes: > Imagine that you have a stream of normal, synchronous, commits. They get > assigned LSNs: 1, 2, 3, 4. They become visible to other transactions in > that order. > The way I described this scheme in the first emails on this thread, was > to use the current WAL inser

Re: [HACKERS] Wait events monitoring future development

2016-08-10 Thread Alexander Korotkov
On Tue, Aug 9, 2016 at 12:47 AM, Ilya Kosmodemiansky < ilya.kosmodemian...@postgresql-consulting.com> wrote: > On Mon, Aug 8, 2016 at 7:03 PM, Bruce Momjian wrote: > > It seems asking users to run pg_test_timing before deploying to check > > the overhead would be sufficient. > > I'am not sure. Ti

Re: [HACKERS] Proposal for CSN based snapshots

2016-08-10 Thread Heikki Linnakangas
On 08/10/2016 04:34 PM, Alexander Korotkov wrote: On Tue, Aug 9, 2016 at 3:16 PM, Heikki Linnakangas wrote: I switched to using a separate counter for CSNs. CSN is no longer the same as the commit WAL record's LSN. While I liked the conceptual simplicity of CSN == LSN a lot, and the fact that

Re: [HACKERS] Curing plpgsql's memory leaks for statement-lifespan values

2016-08-10 Thread Pavel Stehule
2016-08-10 11:25 GMT+02:00 Pavel Stehule : > Hi > > 2016-07-27 16:49 GMT+02:00 Tom Lane : > >> Robert Haas writes: >> > On Mon, Jul 25, 2016 at 6:04 PM, Tom Lane wrote: >> >> I suppose that a fix based on putting PG_TRY blocks into all the >> affected >> >> functions might be simple enough that

Re: [HACKERS] Proposal for CSN based snapshots

2016-08-10 Thread Alexander Korotkov
On Tue, Aug 9, 2016 at 3:16 PM, Heikki Linnakangas wrote: > (Reviving an old thread) > > I spent some time dusting off this old patch, to implement CSN snapshots. > Attached is a new patch, rebased over current master, and with tons of > comments etc. cleaned up. There's more work to be done here

Re: [HACKERS] Proposal for CSN based snapshots

2016-08-10 Thread Alexander Korotkov
On Wed, Aug 10, 2016 at 2:10 PM, Heikki Linnakangas wrote: > Yeah, if the csnlog access turns out to be too expensive, we could do > something like this. In theory, you can always convert a CSN snapshot into > an old-style list of XIDs, by scanning the csnlog between the xmin and > xmax. That cou

Re: [HACKERS] multivariate statistics (v19)

2016-08-10 Thread Ants Aasma
On Wed, Aug 3, 2016 at 4:58 AM, Tomas Vondra wrote: > 2) combining multiple statistics > > I think the ability to combine multivariate statistics (covering different > subsets of conditions) is important and useful, but I'm starting to think > that the current implementation may not be the correct

Re: [HACKERS] multivariate statistics (v19)

2016-08-10 Thread Michael Paquier
On Wed, Aug 10, 2016 at 8:50 PM, Petr Jelinek wrote: > On 10/08/16 13:33, Tomas Vondra wrote: >> >> On 08/10/2016 06:41 AM, Michael Paquier wrote: >>> >>> On Wed, Aug 3, 2016 at 10:58 AM, Tomas Vondra 2) combining multiple statistics I think the ability to combine multivar

Re: [HACKERS] multivariate statistics (v19)

2016-08-10 Thread Michael Paquier
On Wed, Aug 10, 2016 at 8:33 PM, Tomas Vondra wrote: > On 08/10/2016 06:41 AM, Michael Paquier wrote: >> Patch 0001: there have been comments about that before, and you have >> put the checks on RestrictInfo in a couple of variables of >> pull_varnos_walker, so nothing to say from here. >> > > I d

Re: [HACKERS] multivariate statistics (v19)

2016-08-10 Thread Petr Jelinek
On 10/08/16 13:33, Tomas Vondra wrote: On 08/10/2016 06:41 AM, Michael Paquier wrote: On Wed, Aug 3, 2016 at 10:58 AM, Tomas Vondra 2) combining multiple statistics I think the ability to combine multivariate statistics (covering different subsets of conditions) is important and useful, but I'

  1   2   >