Re: a misbehavior of partition row movement (?)

2021-02-19 Thread Masahiko Sawada
On Mon, Feb 15, 2021 at 10:37 PM Amit Langote wrote: > > Thank you for looking at this. > > On Mon, Feb 15, 2021 at 4:12 PM Masahiko Sawada wrote: > > On Wed, Jan 20, 2021 at 7:04 PM Amit Langote > > wrote: > > > This shows that the way we've made these triggers behave in general > > > can caus

Re: Problem with accessing TOAST data in stored procedures

2021-02-19 Thread Konstantin Knizhnik
On 19.02.2021 10:47, Pavel Stehule wrote: pá 19. 2. 2021 v 8:39 odesílatel Konstantin Knizhnik mailto:k.knizh...@postgrespro.ru>> napsal: On 19.02.2021 10:14, Pavel Stehule wrote: pá 19. 2. 2021 v 7:51 odesílatel Konstantin Knizhnik mailto:k.knizh...@postgrespro.ru>> n

Re: Problem with accessing TOAST data in stored procedures

2021-02-19 Thread Pavel Stehule
pá 19. 2. 2021 v 9:08 odesílatel Konstantin Knizhnik < k.knizh...@postgrespro.ru> napsal: > > > On 19.02.2021 10:47, Pavel Stehule wrote: > > > > pá 19. 2. 2021 v 8:39 odesílatel Konstantin Knizhnik < > k.knizh...@postgrespro.ru> napsal: > >> >> >> On 19.02.2021 10:14, Pavel Stehule wrote: >> >> >

Re: ERROR: "ft1" is of the wrong type.

2021-02-19 Thread Kyotaro Horiguchi
At Thu, 18 Feb 2021 17:17:37 +0900 (JST), Kyotaro Horiguchi wrote in > I can add some regression tests to cover all the live cases. That > could reveal no-longer-used combinations. The attached is that. ATT_VIEW is used for "CREATE OR REPLACE view" and checked against earlier in DefineVirtualR

Re: [PoC] Non-volatile WAL buffer

2021-02-19 Thread Konstantin Knizhnik
Thank you for your feedback. On 19.02.2021 6:25, Tomas Vondra wrote: On 1/22/21 5:04 PM, Konstantin Knizhnik wrote: ... I have heard from several DBMS experts that appearance of huge and cheap non-volatile memory can make a revolution in database system architecture. If all database can fit in

Re: Extensibility of the PostgreSQL wire protocol

2021-02-19 Thread Kuntal Ghosh
On Thu, Feb 18, 2021 at 9:32 PM Jan Wieck wrote: > And, here is how it looks with the following configuration: telnet_srv.port = 1433 telnet_srv.listen_addresses = '*' telnet localhost 1433   master Trying 127.0.0.1... Connected to localhost. Escape character is '

Re: [WIP] UNNEST(REFCURSOR): allowing SELECT to consume data from a REFCURSOR

2021-02-19 Thread Dent John
Hi Massimo, Happy to help. And actually, end user (i.e., developer) feedback on the feature’s usefulness is probably one of the more important contributions. d. > On 10 Feb 2021, at 08:57, Massimo Fidanza wrote: > > Hi John, > > I never build postgresql from source, so I must get some inform

Re: Extend more usecase for planning time partition pruning and init partition pruning.

2021-02-19 Thread Andy Fan
On Mon, Feb 8, 2021 at 3:43 PM Andy Fan wrote: > > > On Mon, Jan 25, 2021 at 10:21 AM Andy Fan > wrote: > >> >> >> On Sun, Jan 24, 2021 at 6:34 PM Andy Fan >> wrote: >> >>> Hi: >>> >>> I recently found a use case like this. SELECT * FROM p, q WHERE >>> p.partkey = >>> q.colx AND (q.colx = $1

Re: Parallel INSERT (INTO ... SELECT ...)

2021-02-19 Thread Amit Kapila
On Thu, Feb 18, 2021 at 11:05 AM Amit Langote wrote: > > > > It also occurred to me that we can avoid pointless adding of > > > partitions if the final plan won't use parallelism. For that, the > > > patch adds checking glob->parallelModeNeeded, which seems to do the > > > trick though I don't kn

Re: Parallel INSERT (INTO ... SELECT ...)

2021-02-19 Thread Amit Kapila
On Fri, Feb 19, 2021 at 10:13 AM tsunakawa.ta...@fujitsu.com wrote: > > From: Greg Nancarrow > -- > On Mon, Jan 25, 2021 at 10:23 AM tsunakawa.ta...@fujitsu.com > wrote: > > (8) > > + /* > > +* If the trigger type is

RE: libpq debug log

2021-02-19 Thread iwata....@fujitsu.com
Hi all, I update patch to v18. It has been fixed in response to Tsunakawa san's review. > From: Tsunakawa, Takayuki/綱川 貴之 > Sent: Friday, February 12, 2021 1:53 PM > > (48) > > void PQtrace(PGconn *conn, FILE *stream); > > > > + > + Calls PQtraceSetFlags to output with or

Re: Extensibility of the PostgreSQL wire protocol

2021-02-19 Thread Damir Simunic
> On 11 Feb 2021, at 16:06, Tom Lane wrote: > > Maybe there is some useful thing that can be accomplished here, but we > need to consider the bigger picture rather than believing (without proof) > that a few hook variables will be enough to do anything. > > regards, tom l

[PATCH] Present all committed transaction to the output plugin

2021-02-19 Thread Markus Wanner
Hi, attached is a patch that I think is cleaning up the API between Postgres and the logical decoding plugin. Up until now, not only transactions rolled back, but also some committed transactions were filtered and not presented to the output plugin. While it is documented that aborted trans

Re: Some regular-expression performance hacking

2021-02-19 Thread Joel Jacobson
On Thu, Feb 18, 2021, at 19:53, Tom Lane wrote: >(Having said that, I can't help noticing that a very large fraction >of those usages look like, eg, "[\w\W]". It seems to me that that's >a very expensive and unwieldy way to spell ".". Am I missing >something about what that does in Javascript?)

Re: Extensibility of the PostgreSQL wire protocol

2021-02-19 Thread Jan Wieck
Thank you Kuntal, On Fri, Feb 19, 2021 at 4:36 AM Kuntal Ghosh wrote: > On Thu, Feb 18, 2021 at 9:32 PM Jan Wieck wrote: > > > Few comments in the extension code (although experimental): > > 1. In telnet_srv.c, > + static intpe_port; > .. > + DefineCustomIntVariable("telnet_srv.po

Re: Extensibility of the PostgreSQL wire protocol

2021-02-19 Thread Heikki Linnakangas
On 19/02/2021 14:29, Damir Simunic wrote: On 11 Feb 2021, at 16:06, Tom Lane wrote: Maybe there is some useful thing that can be accomplished here, but we need to consider the bigger picture rather than believing (without proof) that a few hook variables will be enough to do anything. Plugg

Re: Extensibility of the PostgreSQL wire protocol

2021-02-19 Thread Jonah H. Harris
On Fri, Feb 19, 2021 at 8:48 AM Heikki Linnakangas wrote: > With the hooks that exist today, would it possible to write a background > worker that listens on a port, instead of postmaster? Can you launch > backends from a background worker? And communicate the backend processes > using a shared m

Re: repeated decoding of prepared transactions

2021-02-19 Thread Markus Wanner
Ajin, Amit, thank you both a lot for thinking this through and even providing a patch. The changes in expectation for twophase.out matches exactly with what I prepared. And the switch with pg_logical_slot_get_changes indeed is something I had not yet considered, either. On 19.02.21 03:50, A

Re: A reloption for partitioned tables - parallel_workers

2021-02-19 Thread Seamus Abshere
hi Amit, Thanks so much for doing this. I had created https://commitfest.postgresql.org/32/2987/ and it looks like it now shows your patch as the one to use. Let me know if I should do anything else. Best, Seamus On Fri, Feb 19, 2021, at 2:30 AM, Amit Langote wrote: > On Tue, Feb 16, 2021 at

Re: Finding cause of test fails on the cfbot site

2021-02-19 Thread Andrew Dunstan
On 2/18/21 11:01 AM, Andrew Dunstan wrote: > On 2/17/21 3:42 PM, Thomas Munro wrote: >> On Thu, Feb 18, 2021 at 9:18 AM Andrew Dunstan wrote: >>> On 2/17/21 11:06 AM, Tom Lane wrote: Peter Smith writes: > I saw that one of our commitfest entries (32/2914) is recently > reporting a f

Re: pg_config_h.in not up-to-date

2021-02-19 Thread Tom Lane
Michael Paquier writes: > On Fri, Feb 19, 2021 at 02:21:21AM -0500, Tom Lane wrote: >> +1, but I think the first period in this comment is redundant: >> + AC_DEFINE([USE_OPENSSL], 1, [Define to 1 to build with OpenSSL support. >> (--with-ssl=openssl).]) > I guess that you mean the second period

Re: Extensibility of the PostgreSQL wire protocol

2021-02-19 Thread Jan Wieck
On 2/19/21 8:48 AM, Heikki Linnakangas wrote: I can see value in supporting different protocols. I don't like the approach discussed in this thread, however. For example, there has been discussion elsewhere about integrating connection pooling into the server itself. For that, you want to have a

Re: Problem with accessing TOAST data in stored procedures

2021-02-19 Thread Konstantin Knizhnik
On 19.02.2021 11:12, Pavel Stehule wrote: pá 19. 2. 2021 v 9:08 odesílatel Konstantin Knizhnik mailto:k.knizh...@postgrespro.ru>> napsal: On 19.02.2021 10:47, Pavel Stehule wrote: pá 19. 2. 2021 v 8:39 odesílatel Konstantin Knizhnik mailto:k.knizh...@postgrespro.ru>> n

Re: Some regular-expression performance hacking

2021-02-19 Thread Tom Lane
"Joel Jacobson" writes: > On Thu, Feb 18, 2021, at 19:53, Tom Lane wrote: >> (Having said that, I can't help noticing that a very large fraction >> of those usages look like, eg, "[\w\W]". It seems to me that that's >> a very expensive and unwieldy way to spell ".". Am I missing >> something abo

Re: Problem with accessing TOAST data in stored procedures

2021-02-19 Thread Pavel Stehule
pá 19. 2. 2021 v 16:19 odesílatel Konstantin Knizhnik < k.knizh...@postgrespro.ru> napsal: > > > On 19.02.2021 11:12, Pavel Stehule wrote: > > > > pá 19. 2. 2021 v 9:08 odesílatel Konstantin Knizhnik < > k.knizh...@postgrespro.ru> napsal: > >> >> >> On 19.02.2021 10:47, Pavel Stehule wrote: >> >>

Re: PATCH: Attempt to make dbsize a bit more consistent

2021-02-19 Thread gkokolatos
‐‐‐ Original Message ‐‐‐ On Monday, February 1, 2021 1:18 PM, Masahiko Sawada wrote: > On Thu, Nov 12, 2020 at 2:54 AM Soumyadeep Chakraborty > soumyadeep2...@gmail.com wrote: > > > Hey Georgios, > > On Tue, Nov 10, 2020 at 6:20 AM gkokola...@pm.me wrote: > > > > > ‐‐‐ Original

Re: [HACKERS] Custom compression methods

2021-02-19 Thread Dilip Kumar
On Thu, Feb 11, 2021 at 1:37 AM Robert Haas wrote: > > On Wed, Feb 10, 2021 at 9:52 AM Dilip Kumar wrote: > > [ new patches ] > > I think that in both varattrib_4b and toast_internals.h it would be > better to pick a less generic field name. In toast_internals.h it's > just info; in postgres.h it

Re: PoC Refactor AM analyse API

2021-02-19 Thread Zhihong Yu
Denis: Thanks for considering my suggestion. For #1, I didn't take your example into account. Thanks for pointing that out. Cheers On Thu, Feb 18, 2021 at 11:59 PM Denis Smirnov wrote: > Hello, Zhihong. > > Thanks for your comments. > > 1. I am afraid that an equivalence of "floor(val + 0.5)"

Re: [HACKERS] Custom compression methods

2021-02-19 Thread Dilip Kumar
On Thu, Feb 11, 2021 at 3:26 AM Robert Haas wrote: > > In CompareCompressionMethodAndDecompress, I think this is still > playing a bit fast and loose with the rules around slots. I think we > can do better. Suppose that at the point where we discover that we > need to decompress at least one attri

Re: Extensibility of the PostgreSQL wire protocol

2021-02-19 Thread Damir Simunic
> On 19 Feb 2021, at 14:48, Heikki Linnakangas wrote: > > For example, there has been discussion elsewhere about integrating connection > pooling into the server itself. For that, you want to have a custom process > that listens for incoming connections, and launches backends independently of

Re: Extensibility of the PostgreSQL wire protocol

2021-02-19 Thread Jan Wieck
On 2/19/21 12:18 PM, Damir Simunic wrote: On 19 Feb 2021, at 14:48, Heikki Linnakangas wrote: For example, there has been discussion elsewhere about integrating connection pooling into the server itself. For that, you want to have a custom process that listens for incoming connections, and

Re: PATCH: Batch/pipelining support for libpq

2021-02-19 Thread Alvaro Herrera
Hello, thanks for looking at this patch. On 2021-Feb-16, Zhihong Yu wrote: > + if (querymode == QUERY_SIMPLE) > + { > + commandFailed(st, "startpipeline", "cannot use pipeline mode > with the simple query protocol"); > + st->state = CSTATE_ABORTED; > + re

Re: PATCH: Batch/pipelining support for libpq

2021-02-19 Thread Alvaro Herrera
On 2021-Jan-21, Zhihong Yu wrote: > It seems '\\gset or \\aset is not ' would correspond to the check more > closely. > > + if (my_command->argc != 1) > + syntax_error(source, lineno, my_command->first_line, > my_command->argv[0], > > It is possible that my_command->argc == 0 (wh

Re: Extensibility of the PostgreSQL wire protocol

2021-02-19 Thread Damir Simunic
> On 19 Feb 2021, at 19:30, Jan Wieck wrote: > > An "extended" libpq protocol could allow the pool to give clients a unique > ID. The protocol handler would then maintain maps with the SQL of prepared > statements and what the client thinks their prepared statement name is. Or, the connecti

Re: PATCH: Batch/pipelining support for libpq

2021-02-19 Thread Zhihong Yu
Hi, +static int pqBatchProcessQueue(PGconn *conn); I was suggesting changing: +int +PQexitBatchMode(PGconn *conn) to: +static int +PQexitBatchMode(PGconn *conn) Cheers On Fri, Feb 19, 2021 at 10:43 AM Alvaro Herrera wrote: > On 2021-Jan-21, Zhihong Yu wrote: > > > It seems '\\gset or \\aset

Re: Extensibility of the PostgreSQL wire protocol

2021-02-19 Thread Álvaro Hernández
On 19/2/21 14:48, Heikki Linnakangas wrote: > [...] > > I can see value in supporting different protocols. I don't like the > approach discussed in this thread, however. > > For example, there has been discussion elsewhere about integrating > connection pooling into the server itself. For that,

Re: Extensibility of the PostgreSQL wire protocol

2021-02-19 Thread Álvaro Hernández
On 19/2/21 19:30, Jan Wieck wrote: > [...] > > I also am not sure if building a connection pool into a background > worker or postmaster is a good idea to begin with. One of the > important features of a pool is to be able to suspend traffic and make > the server completely idle to for example b

Re: PATCH: Batch/pipelining support for libpq

2021-02-19 Thread Alvaro Herrera
Hi, On 2021-Feb-19, Zhihong Yu wrote: > Hi, > +static int pqBatchProcessQueue(PGconn *conn); > > I was suggesting changing: > > +int > +PQexitBatchMode(PGconn *conn) > > to: > > +static int > +PQexitBatchMode(PGconn *conn) I understand that, but what I'm saying is that it doesn't work. pqBat

Re: PATCH: Batch/pipelining support for libpq

2021-02-19 Thread Zhihong Yu
Hi, Thanks for the response. On Fri, Feb 19, 2021 at 12:46 PM Alvaro Herrera wrote: > Hi, > > On 2021-Feb-19, Zhihong Yu wrote: > > > Hi, > > +static int pqBatchProcessQueue(PGconn *conn); > > > > I was suggesting changing: > > > > +int > > +PQexitBatchMode(PGconn *conn) > > > > to: > > > > +sta

Re: [HACKERS] Custom compression methods

2021-02-19 Thread Robert Haas
On Fri, Feb 19, 2021 at 11:12 AM Dilip Kumar wrote: > I had an off list discussion with Robert and based on his suggestion > and a poc patch, I have come up with an updated version for handling > the composite type. Basically, the problem was that ExecEvalRow we > are first forming the tuple and

Re: Finding cause of test fails on the cfbot site

2021-02-19 Thread Peter Smith
Here is another related question about the cfbot error reporting - The main cfbot "status page" [1] still shows a couple of fails for the 32/2914 (for freebsd & linux). But looking more closely, those fails are not from the latest run. e.g. I also found this execution "history" page [2] for our pa

Re: Finding cause of test fails on the cfbot site

2021-02-19 Thread Thomas Munro
On Sat, Feb 20, 2021 at 10:31 AM Peter Smith wrote: > Here is another related question about the cfbot error reporting - > > The main cfbot "status page" [1] still shows a couple of fails for the > 32/2914 (for freebsd & linux). But looking more closely, those fails > are not from the latest run.

Re: [HACKERS] Custom compression methods

2021-02-19 Thread Robert Haas
On Fri, Feb 19, 2021 at 4:21 PM Robert Haas wrote: > What makes me a bit uncomfortable about that approach is that it > presupposes that everything that uses expanded records has some other > defense against those tuples getting written to disk without first > expanding any external datums. And it

Re: PATCH: Batch/pipelining support for libpq

2021-02-19 Thread Craig Ringer
On Wed, 17 Feb 2021, 07:13 Alvaro Herrera, wrote: > Here's a new version, where I've renamed everything to "pipeline". Wow. Thanks. I > think the docs could use some additional tweaks now in order to make a > coherent story on pipeline mode, how it can be used in a batched > fashion, etc. > I

Re: pg_config_h.in not up-to-date

2021-02-19 Thread Michael Paquier
On Fri, Feb 19, 2021 at 09:57:22AM -0500, Tom Lane wrote: > Hm. It should be consistent with the rest, for sure. Personally I'd put > the only period at the end, but I suppose we should stick with the > prevailing style if there is one. Thanks. I have just used the same style as XML, LDAP and L

Re: PATCH: Batch/pipelining support for libpq

2021-02-19 Thread Alvaro Herrera
On 2021-Feb-20, Craig Ringer wrote: > On Wed, 17 Feb 2021, 07:13 Alvaro Herrera, wrote: > > I > > think the docs could use some additional tweaks now in order to make a > > coherent story on pipeline mode, how it can be used in a batched > > fashion, etc. > > I'll do that soon and send an updat

Re: progress reporting for partitioned REINDEX

2021-02-19 Thread Michael Paquier
On Fri, Feb 19, 2021 at 12:12:54AM -0600, Justin Pryzby wrote: > Looks fine. Thanks, applied then to clarify things. > Also, I noticed that vacuum recurses into partition heirarchies since v10, but > pg_stat_progress_vacuum also doesn't show anything about the parent table or > the progress of re

Re: A reloption for partitioned tables - parallel_workers

2021-02-19 Thread Amit Langote
On Fri, Feb 19, 2021 at 11:54 PM Seamus Abshere wrote: > On Fri, Feb 19, 2021, at 2:30 AM, Amit Langote wrote: > > Here is an updated version of the Seamus' patch that takes into > > account these and other comments received on this thread so far. > > Maybe warrants adding some tests too but I hav

UniqueKey on Partitioned table.

2021-02-19 Thread Andy Fan
Suppose we have partitioned table defined as below: P(a, b, c, d) partition by (a) p1 (a=1) p2 (a=2) p3 (a=3) Since in PG, we can define different indexes among partitions, so each child may have different UniqueKeys, and some of them might be invalidated in parent's level. For example,

Re: progress reporting for partitioned REINDEX

2021-02-19 Thread Justin Pryzby
On Sat, Feb 20, 2021 at 10:37:08AM +0900, Michael Paquier wrote: > > Also, I noticed that vacuum recurses into partition heirarchies since v10, > > but > > pg_stat_progress_vacuum also doesn't show anything about the parent table or > > the progress of recursing through the hierarchy. > > Yeah, t

Re: repeated decoding of prepared transactions

2021-02-19 Thread Amit Kapila
On Fri, Feb 19, 2021 at 8:23 PM Markus Wanner wrote: > > With that line of thinking, the point in time (or in WAL) of the COMMIT > PREPARED does not matter at all to reason about the decoding of the > PREPARE operation. Instead, there are only exactly two cases to consider: > > a) the PREPARE hap

Re: A reloption for partitioned tables - parallel_workers

2021-02-19 Thread Laurenz Albe
On Fri, 2021-02-19 at 16:30 +0900, Amit Langote wrote: > On Tue, Feb 16, 2021 at 1:35 AM Seamus Abshere wrote: > > > Here we go, my first patch... solves > > > https://www.postgresql.org/message-id/7d6fdc20-857c-4cbe-ae2e-c0ff9520e...@www.fastmail.com > > Here is an updated version of the Seamus

Re: repeated decoding of prepared transactions

2021-02-19 Thread Amit Kapila
On Fri, Feb 19, 2021 at 8:21 AM Ajin Cherian wrote: > > Based on this suggestion, I have created a patch on HEAD which now > does not allow repeated decoding > of prepared transactions. For this, the code now enforces > full_snapshot if two-phase decoding is enabled. > Do have a look at the patch

Re: [HACKERS] Custom compression methods

2021-02-19 Thread Dilip Kumar
On Sat, Feb 20, 2021 at 2:51 AM Robert Haas wrote: > > On Fri, Feb 19, 2021 at 11:12 AM Dilip Kumar wrote: > I think that these performance tests aren't really exercising the > expanded-record stuff, just the ExecEvalRow changes. We need to test > that test case, and I tend to suspect there's goi

Re: New Table Access Methods for Multi and Single Inserts

2021-02-19 Thread Bharath Rupireddy
On Wed, Feb 17, 2021 at 12:46 PM Bharath Rupireddy wrote: > Hi, > > I addressed the following review comments and attaching v3 patch set. > > 1) ExecClearTuple happens before ExecCopySlot in heap_multi_insert_v2 > and this allowed us to remove clear_mi_slots flag from > TableInsertState. > 2) I re

Re: Improvements and additions to COPY progress reporting

2021-02-19 Thread Bharath Rupireddy
On Fri, Feb 19, 2021 at 2:34 AM Tomas Vondra wrote: > > On Mon, 15 Feb 2021 at 17:07, Tomas Vondra > > wrote: > >> > >> - The blocks in copyfrom.cc/copyto.c should be reworked - I don't think > >> we do this in our codebase. > > > > I saw this being used in (re)index progress reporting, that's wh

Re: Asynchronous Append on postgres_fdw nodes.

2021-02-19 Thread Etsuro Fujita
On Thu, Feb 18, 2021 at 3:16 PM Kyotaro Horiguchi wrote: > At Thu, 18 Feb 2021 11:51:59 +0900, Etsuro Fujita > wrote in > > I noticed that this doesn’t work for cases where ForeignScans are > > executed inside functions, and I don’t have any simple solution for > > Ah, concurrent fetches in diff

Re: Improvements and additions to COPY progress reporting

2021-02-19 Thread Michael Paquier
On Sat, Feb 20, 2021 at 11:39:22AM +0530, Bharath Rupireddy wrote: > Actually in the code base the style of that variable declaration and > usage of pgstat_progress_update_multi_param is a mix. For instance, in > lazy_scan_heap, ReindexRelationConcurrently, the variables are > declared at the start

Re: New Table Access Methods for Multi and Single Inserts

2021-02-19 Thread Bharath Rupireddy
On Sat, Feb 20, 2021 at 12:53 PM Zhihong Yu wrote: > > Hi, > bq. case 3 - 2 integer(of 4 bytes each) columns, tuple size 32 bytes > > Is there some other column(s) per row apart from the integer columns ? Since > the 2 integer columns only occupy 8 bytes. I wonder where the other 32-8=24 > bytes