Re: Parallel worker hangs while handling errors.

2020-07-27 Thread vignesh C
Thanks for your comments Bharath. On Mon, Jul 27, 2020 at 10:13 AM Bharath Rupireddy wrote: > 1. Do we need "worker" as a function argument in > update_parallel_worker_sigmask(BackgroundWorker *worker, ? Since > MyBgworkerEntry is a global variable, can't we have a local variable > instead?

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-07-27 Thread Amit Kapila
On Sun, Jul 26, 2020 at 11:04 AM Dilip Kumar wrote: > > > Today, I have again looked at the first patch > > (v42-0001-Extend-the-logical-decoding-output-plugin-API-wi) and didn't > > find any more problems with it so planning to commit the same unless > > you or someone else want to add more to it

Re: HashAgg's batching counter starts at 0, but Hash's starts at 1.

2020-07-27 Thread David Rowley
On Tue, 28 Jul 2020 at 15:21, Peter Geoghegan wrote: > Separately, I wonder what your opinion is about what should happen for > the partial sort related EXPLAIN ANALYZE format open item, described > here: > > https://www.postgresql.org/message-id/flat/20200619040358.GZ17995%40telsasoft.com#b20bd20

Re: HashAgg's batching counter starts at 0, but Hash's starts at 1.

2020-07-27 Thread David Rowley
On Tue, 28 Jul 2020 at 15:01, Justin Pryzby wrote: > > On Tue, Jul 28, 2020 at 12:54:35PM +1200, David Rowley wrote: > > On Mon, 27 Jul 2020 at 14:54, Justin Pryzby wrote: > > > It's unrelated to hashAgg vs hashJoin, but I also noticed that this is > > > shown > > > only conditionally: > > > > >

Re: HashAgg's batching counter starts at 0, but Hash's starts at 1.

2020-07-27 Thread Justin Pryzby
On Mon, Jul 27, 2020 at 08:20:45PM -0700, Peter Geoghegan wrote: > On Mon, Jul 27, 2020 at 5:54 PM David Rowley wrote: > > hmm. I'm not sure. I think this should follow the same logic as what > > "Disk Usage" follows, and right now we don't show Disk Usage unless we > > spill. Since we only use pa

Re: stress test for parallel workers

2020-07-27 Thread Tom Lane
Thomas Munro writes: > Hehe, the dodgy looking magic numbers *were* wrong: > - * The kernel signal delivery code writes up to about 1.5kB > + * The kernel signal delivery code writes a bit over 4KB > https://patchwork.ozlabs.org/project/linuxppc-dev/patch/20200724092528.1578671-2-...@ellerman.id.a

Re: HashAgg's batching counter starts at 0, but Hash's starts at 1.

2020-07-27 Thread Peter Geoghegan
On Mon, Jul 27, 2020 at 5:54 PM David Rowley wrote: > hmm. I'm not sure. I think this should follow the same logic as what > "Disk Usage" follows, and right now we don't show Disk Usage unless we > spill. Since we only use partitions when spilling, I don't think it > makes sense to show the estima

Re: HashAgg's batching counter starts at 0, but Hash's starts at 1.

2020-07-27 Thread Justin Pryzby
On Tue, Jul 28, 2020 at 12:54:35PM +1200, David Rowley wrote: > On Mon, 27 Jul 2020 at 14:54, Justin Pryzby wrote: > > It's unrelated to hashAgg vs hashJoin, but I also noticed that this is shown > > only conditionally: > > > > if (es->format != EXPLAIN_FORMAT_TEXT) > > { > >

Re: Online checksums patch - once again

2020-07-27 Thread Justin Pryzby
On Thu, Jun 25, 2020 at 11:43:00AM +0200, Daniel Gustafsson wrote: > The attached v19 fixes a few doc issues I had missed. + They can also be enabled or disabled at a later timne, either as an offline => time +* awaiting shutdown, but we can continue turning off checksums anyway => a wa

Re: max_slot_wal_keep_size and wal_keep_segments

2020-07-27 Thread Fujii Masao
On 2020/07/20 21:21, David Steele wrote: On 7/20/20 6:02 AM, Fujii Masao wrote: On 2020/07/20 13:48, Fujii Masao wrote: On 2020/07/17 20:24, David Steele wrote: On 7/17/20 5:11 AM, Fujii Masao wrote: On 2020/07/14 20:30, David Steele wrote: On 7/14/20 12:00 AM, Fujii Masao wrote:

printing oid with %d

2020-07-27 Thread Justin Pryzby
+JsonEncodeDateTime(char *buf, Datum value, Oid typid) ... + elog(ERROR, "unknown jsonb value datetime type oid %d", typid); I think this should be %u. commit cc4feded0a31d2b732d4ea68613115cb720e624e Author: Andrew Dunstan Date: Tue Jan 16 19:07:13 2018 -0500 Centra

Re: stress test for parallel workers

2020-07-27 Thread Thomas Munro
On Wed, Dec 11, 2019 at 3:22 PM Thomas Munro wrote: > On Tue, Oct 15, 2019 at 4:50 AM Tom Lane wrote: > > > Filed at > > > https://bugzilla.kernel.org/show_bug.cgi?id=205183 > > For the curious-and-not-subscribed, there's now a kernel patch > proposed for this. We guessed pretty close, but the p

Re: expose parallel leader in CSV and log_line_prefix

2020-07-27 Thread Michael Paquier
On Sun, Jul 26, 2020 at 01:54:27PM -0500, Justin Pryzby wrote: > + > + %P > + For a parallel worker, this is the Process ID of its > leader > + process. > + > + no > + Let's be a maximum simple and consistent wit

Re: Default setting for enable_hashagg_disk

2020-07-27 Thread Peter Geoghegan
On Mon, Jul 27, 2020 at 5:10 PM Jeff Davis wrote: > I noticed that one of the conditionals, "cheapest_total_path != NULL", > was already redundant with the outer conditional before your patch. I > guess that was just a mistake which your patch corrects along the way? Makes sense. > Anyway, the p

Re: HashAgg's batching counter starts at 0, but Hash's starts at 1.

2020-07-27 Thread David Rowley
On Mon, 27 Jul 2020 at 14:54, Justin Pryzby wrote: > It's unrelated to hashAgg vs hashJoin, but I also noticed that this is shown > only conditionally: > > if (es->format != EXPLAIN_FORMAT_TEXT) > { > if (es->costs && aggstate->hash_planned_partitions > 0) >

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-07-27 Thread Andrey Lepikhov
27.07.2020 21:34, Alexey Kondratov пишет: Hi Andrey, On 2020-07-23 09:23, Andrey V. Lepikhov wrote: On 7/16/20 2:14 PM, Amit Langote wrote: Amit Langote EnterpriseDB: http://www.enterprisedb.com Version 5 of the patch. With changes caused by Amit's comments. Just got a segfault with yo

Re: Default setting for enable_hashagg_disk

2020-07-27 Thread Jeff Davis
On Mon, 2020-07-27 at 11:30 -0700, Peter Geoghegan wrote: > The v4-0001-Remove-hashagg_avoid_disk_plan-GUC.patch changes are > surprisingly complicated. It would be nice if you could take a look > at > that aspect (or confirm that it's included in your review). I noticed that one of the conditiona

Re: deferred primary key and logical replication

2020-07-27 Thread Euler Taveira
On Fri, 24 Jul 2020 at 05:16, Ajin Cherian wrote: > The patch no longer applies, because of additions in the test source. > Otherwise, I have tested the patch and confirmed that updates and deletes > on tables with deferred primary keys work with logical replication. > > The new status of this pa

Re: Default setting for enable_hashagg_disk

2020-07-27 Thread Peter Geoghegan
On Mon, Jul 27, 2020 at 12:52 PM Alvaro Herrera wrote: > On 2020-Jul-27, Peter Geoghegan wrote: > > The v4-0001-Remove-hashagg_avoid_disk_plan-GUC.patch changes are > > surprisingly complicated. It would be nice if you could take a look at > > that aspect (or confirm that it's included in your rev

Re: Default setting for enable_hashagg_disk

2020-07-27 Thread Alvaro Herrera
On 2020-Jul-27, Peter Geoghegan wrote: > The v4-0001-Remove-hashagg_avoid_disk_plan-GUC.patch changes are > surprisingly complicated. It would be nice if you could take a look at > that aspect (or confirm that it's included in your review). I think you mean "it replaces surprisingly complicated c

Re: Default setting for enable_hashagg_disk

2020-07-27 Thread Peter Geoghegan
On Mon, Jul 27, 2020 at 11:24 AM Alvaro Herrera wrote: > > Are you proposing that I just put the prototype in miscadmin.h, while > > leaving the implementation where it is (in nodeHash.c)? > > Yes, that's in the part of my reply you didn't quote: > > : It remains strange to have the function in ex

Re: Default setting for enable_hashagg_disk

2020-07-27 Thread Alvaro Herrera
On 2020-Jul-27, Peter Geoghegan wrote: > On Mon, Jul 27, 2020 at 10:30 AM Alvaro Herrera > wrote: > > On 2020-Jul-23, Peter Geoghegan wrote: > > I notice you put the prototype for get_hash_mem in nodeHash.h. This > > would be fine if not for the fact that optimizer needs to call the > > function

Re: Default setting for enable_hashagg_disk

2020-07-27 Thread Peter Geoghegan
On Mon, Jul 27, 2020 at 10:30 AM Alvaro Herrera wrote: > On 2020-Jul-23, Peter Geoghegan wrote: > I notice you put the prototype for get_hash_mem in nodeHash.h. This > would be fine if not for the fact that optimizer needs to call the > function too, which means now optimizer have to include exec

Re: Default setting for enable_hashagg_disk

2020-07-27 Thread Alvaro Herrera
On 2020-Jul-23, Peter Geoghegan wrote: > Attached is v3 of the hash_mem_multiplier patch series, which now has > a preparatory patch that removes hashagg_avoid_disk_plan. I notice you put the prototype for get_hash_mem in nodeHash.h. This would be fine if not for the fact that optimizer needs to

Re: [BUG] Error in BRIN summarization

2020-07-27 Thread Alvaro Herrera
On 2020-Jul-27, Anastasia Lubennikova wrote: > Here is the updated version of the fix. > The problem can be reproduced on all supported versions, so I suggest to > backpatch it. > Code slightly changed in v12, so here are two patches: one for versions 9.5 > to 11 and another for versions from 12 t

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-07-27 Thread Alexey Kondratov
Hi Andrey, On 2020-07-23 09:23, Andrey V. Lepikhov wrote: On 7/16/20 2:14 PM, Amit Langote wrote: Amit Langote EnterpriseDB: http://www.enterprisedb.com Version 5 of the patch. With changes caused by Amit's comments. Just got a segfault with your v5 patch by deleting from a foreign table.

Re: Should we remove a fallback promotion? take 2

2020-07-27 Thread Fujii Masao
On 2020/07/27 17:53, Hamid Akhtar wrote: Applying the patch to the current master branch throws 9 hunks. AFAICT, the patch is good otherwise. So you think that the patch can be marked as Ready for Committer? Regards, -- Fujii Masao Advanced Computing Technology Center Research and Develop

Re: Display individual query in pg_stat_activity

2020-07-27 Thread Dave Page
On Mon, Jul 27, 2020 at 4:28 PM Jeremy Schneider wrote: > On 7/27/20 07:57, Dave Page wrote: > > I'm not sure I'd want that to happen, as it could make it much harder to > track the activity back to a query in the application layer or server logs. > > Perhaps a separate field could be added for t

Re: Default setting for enable_hashagg_disk

2020-07-27 Thread Peter Geoghegan
On Sun, Jul 26, 2020 at 11:34 AM Tomas Vondra wrote: > That's 1.6GB, if I read it right. Which is more than 200MB ;-) Sigh. That solves that "mystery": the behavior that my sorted vs random example exhibited is a known limitation in hash aggs that spill (and an acceptable one). The memory usage i

Re: hashagg slowdown due to spill changes

2020-07-27 Thread Peter Geoghegan
On Sun, Jul 26, 2020 at 4:17 PM Jeff Davis wrote: > I saw less of an improvement than you or Andres (perhaps just more > noise). But given that both you and Andres are reporting a measurable > improvement, then I went ahead and committed it. Thank you. Thanks! -- Peter Geoghegan

Re: [BUG] Error in BRIN summarization

2020-07-27 Thread Anastasia Lubennikova
On 23.07.2020 20:39, Anastasia Lubennikova wrote: One of our clients caught an error "failed to find parent tuple for heap-only tuple at (50661,130) in table "tbl'" in PostgreSQL v12. Steps to reproduce (REL_12_STABLE): 1) Create table with primary key, create brin index, fill table with some

Re: Display individual query in pg_stat_activity

2020-07-27 Thread Dave Page
Hi On Mon, Jul 27, 2020 at 3:40 PM Drouvot, Bertrand wrote: > Hi hackers, > > I've attached a patch to display individual query in the pg_stat_activity > query field when multiple SQL statements are currently displayed. > > *Motivation:* > > When multiple statements are displayed then we don’t k

Re: display offset along with block number in vacuum errors

2020-07-27 Thread Justin Pryzby
On Fri, Jul 24, 2020 at 11:18:43PM +0530, Mahendra Singh Thalor wrote: > Hi hackers, > We discussed in another email thread[1], that it will be helpful if we can > display offset along with block number in vacuum error. Here, proposing a > patch to add offset along with block number in vacuum error

Re: shared tempfile was not removed on statement_timeout

2020-07-27 Thread Justin Pryzby
On Mon, Jul 27, 2020 at 08:00:46PM +1200, Thomas Munro wrote: > On Tue, Jul 21, 2020 at 4:33 PM Justin Pryzby wrote: > > /* > > * clean up a spool structure and its substructures. > > */ > > static void > > _bt_spooldestroy(BTSpool *btspool) > > { > > + void *fileset = tuplesort_shar

Re: Index Skip Scan (new UniqueKeys)

2020-07-27 Thread Dmitry Dolgov
> On Tue, Jul 21, 2020 at 04:35:55PM -0700, Peter Geoghegan wrote: > > > Well, it's obviously wrong, thanks for noticing. What is necessary is to > > compare two index tuples, the start and the next one, to test if they're > > the same (in which case if I'm not mistaken probably we can compare item

Re: Should we remove a fallback promotion? take 2

2020-07-27 Thread Hamid Akhtar
Applying the patch to the current master branch throws 9 hunks. AFAICT, the patch is good otherwise. On Wed, Jun 3, 2020 at 3:20 PM Fujii Masao wrote: > > > On 2020/06/03 12:06, Kyotaro Horiguchi wrote: > > At Wed, 3 Jun 2020 09:43:17 +0900, Fujii Masao < > masao.fu...@oss.nttdata.com> wrote in

Re: shared tempfile was not removed on statement_timeout

2020-07-27 Thread Thomas Munro
On Tue, Jul 21, 2020 at 4:33 PM Justin Pryzby wrote: > /* > * clean up a spool structure and its substructures. > */ > static void > _bt_spooldestroy(BTSpool *btspool) > { > + void *fileset = tuplesort_shared_fileset(btspool->sortstate); > + if (fileset) > + Share

Re: display offset along with block number in vacuum errors

2020-07-27 Thread Mahendra Singh Thalor
Thanks Michael for looking into this. On Sat, 25 Jul 2020 at 15:02, Michael Paquier wrote: > > On Fri, Jul 24, 2020 at 11:18:43PM +0530, Mahendra Singh Thalor wrote: > > In commit b61d161(Introduce vacuum errcontext to display additional > > information), we added vacuum errcontext to display add

Re: display offset along with block number in vacuum errors

2020-07-27 Thread Masahiko Sawada
On Sat, 25 Jul 2020 at 02:49, Mahendra Singh Thalor wrote: > > Hi hackers, > We discussed in another email thread[1], that it will be helpful if we can > display offset along with block number in vacuum error. Here, proposing a > patch to add offset along with block number in vacuum errors. > >

Re: Postgres-native method to identify if a tuple is frozen

2020-07-27 Thread Lawrence Jones
Thanks for the help. I'd seen the heap_page_items functions, but wanted to avoid the superuser requirement and wondered if this was going to be a performant method of finding the freeze column (we're scanning some billions of rows). Fwiw, we think we'll probably go with a tiny extension that expos

Re: Transactions involving multiple postgres foreign servers, take 2

2020-07-27 Thread Masahiko Sawada
On Thu, 23 Jul 2020 at 22:51, Muhammad Usama wrote: > > > > On Wed, Jul 22, 2020 at 12:42 PM Masahiko Sawada > wrote: >> >> On Sat, 18 Jul 2020 at 01:55, Fujii Masao >> wrote: >> > >> > >> > >> > On 2020/07/16 14:47, Masahiko Sawada wrote: >> > > On Tue, 14 Jul 2020 at 11:19, Fujii Masao >>