Re: using extended statistics to improve join estimates

2021-12-13 Thread Tomas Vondra
On 11/6/21 11:03, Andy Fan wrote: Hi Tomas: This is the exact patch I want, thanks for the patch! Good to hear. On Thu, Oct 7, 2021 at 3:33 AM Tomas Vondra wrote: 3) estimation by join pairs At the moment, the estimates are calculated for pairs of relations, so for example given a

Re: extended stats on partitioned tables

2021-12-13 Thread Tomas Vondra
On 12/13/21 14:48, Justin Pryzby wrote: > On Sun, Dec 12, 2021 at 10:29:39PM +0100, Tomas Vondra wrote: >>> In your 0003 patch, the "if inh: break" isn't removed from >>> examine_variable(), >>> but the corresponding thing is removed everywhere else.

Re: extended stats on partitioned tables

2021-12-13 Thread Tomas Vondra
On 12/13/21 14:53, Justin Pryzby wrote: > On Sun, Dec 12, 2021 at 11:23:19PM +0100, Tomas Vondra wrote: >> On 12/12/21 22:32, Justin Pryzby wrote: >>> On Sun, Dec 12, 2021 at 05:17:10AM +0100, Tomas Vondra wrote: >>>> The one thing bugging me a bit is that the

Re: Column Filtering in Logical Replication

2021-12-14 Thread Tomas Vondra
't know what are the complications you mentioned. regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: Column Filtering in Logical Replication

2021-12-14 Thread Tomas Vondra
On 12/14/21 20:35, Alvaro Herrera wrote: On 2021-Dec-14, Tomas Vondra wrote: Yeah. I think it's not clear if this should behave more like an index or a view. When an indexed column gets dropped we simply drop the index. But if you drop a column referenced by a view, we fail with an err

Re: Column Filtering in Logical Replication

2021-12-14 Thread Tomas Vondra
the offset), so maybe let's move that to a function with sensible name. Also, some places don't check indattrs - seems a bit suspicious. regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL CompanyFrom fb5ce02d36b46f92ab01c9a823cc4e315c

Re: logical decoding and replication of sequences

2021-12-15 Thread Tomas Vondra
On 12/15/21 14:20, Amit Kapila wrote: > On Tue, Dec 14, 2021 at 7:02 AM Tomas Vondra > wrote: >> >> Hi, >> >> here's an updated version of the patches, dealing with almost all of the >> issues (at least in the 0001 and 0002 parts). The main changes: >&

Re: logical decoding and replication of sequences

2021-12-15 Thread Tomas Vondra
On 12/15/21 14:51, Tomas Vondra wrote: > On 12/15/21 14:20, Amit Kapila wrote: >> On Tue, Dec 14, 2021 at 7:02 AM Tomas Vondra >> wrote: >>> >>> Hi, >>> >>> here's an updated version of the patches, dealing with almost all of the &g

Re: logical decoding and replication of sequences

2021-12-15 Thread Tomas Vondra
On 12/15/21 17:42, Alvaro Herrera wrote: > Looking at 0003, > > On 2021-Dec-14, Tomas Vondra wrote: > >> diff --git a/doc/src/sgml/ref/alter_publication.sgml >> b/doc/src/sgml/ref/alter_publication.sgml >> index bb4ef5e5e22..4d166ad3f9c 100644 >> --- a/doc

Re: logical decoding and replication of sequences

2021-12-16 Thread Tomas Vondra
On 12/16/21 12:59, Amit Kapila wrote: > On Wed, Dec 15, 2021 at 7:21 PM Tomas Vondra > wrote: >> >> On 12/15/21 14:20, Amit Kapila wrote: >>> On Tue, Dec 14, 2021 at 7:02 AM Tomas Vondra >>> wrote: >>>> >>>> Hi, >>>> &

Re: Use generation context to speed up tuplesorts

2021-12-16 Thread Tomas Vondra
the minimum block size, with about the same effect, no? I would like to try to implement some dynamic glibc malloc tuning, if that is something we don't reject on principle from the get go. +1 to that regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: incremental sort vs. gather paths

2021-12-16 Thread Tomas Vondra
get, subpath->pathkeys, NULL, rowsp); add_path(rel, &path->path); } i.e. it's doing the same (rows * parallel_workers) calculation. It may not not be the right way to estimate this, of course. But I'd say if ba3e76cc57

Re: Use generation context to speed up tuplesorts

2021-12-17 Thread Tomas Vondra
On 12/17/21 09:08, Ronan Dunklau wrote: > Le jeudi 16 décembre 2021, 18:00:56 CET Tomas Vondra a écrit : >> On 12/16/21 17:03, Ronan Dunklau wrote: >>> Le jeudi 16 décembre 2021, 11:56:15 CET Ronan Dunklau a écrit : >>>> I will follow up with a benchmark of t

Re: Column Filtering in Logical Replication

2021-12-17 Thread Tomas Vondra
thing. Does that mean we currently sync all the columns in the initial sync, and only start filtering columns later while decoding transactions? regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: WIP: WAL prefetch (another approach)

2021-12-17 Thread Tomas Vondra
7:51:51.761 EST [50955] LOG: all server processes terminated; reinitializing Interesting. In my experience SIGBUS on PPC tends to be due to incorrect alignment, but I'm not sure how that works with the emulation. Can you get a backtrace? regards -- Tomas Vondra EnterpriseDB: http://www.

sequences vs. synchronous replication

2021-12-17 Thread Tomas Vondra
s like one. My proposal is to fix this by tracking the lsn of the last LSN for a sequence increment, and then check that LSN in RecordTransactionCommit() before calling XLogFlush(). regards [1] https://www.postgresql.org/message-id/ae3cab67-c31e-b527-dd73-08f196999ad4%40enterprisedb

Re: logical decoding and replication of sequences

2021-12-17 Thread Tomas Vondra
On 12/15/21 14:51, Tomas Vondra wrote: On 12/15/21 14:20, Amit Kapila wrote: On Tue, Dec 14, 2021 at 7:02 AM Tomas Vondra wrote: Hi, here's an updated version of the patches, dealing with almost all of the issues (at least in the 0001 and 0002 parts). The main changes: 1) I'

Re: Column Filtering in Logical Replication

2021-12-17 Thread Tomas Vondra
On 12/18/21 02:34, Alvaro Herrera wrote: On 2021-Dec-17, Tomas Vondra wrote: On 12/17/21 22:07, Alvaro Herrera wrote: So I've been thinking about this as a "security" item (you can see my comments to that effect sprinkled all over this thread), in the sense that if a pub

Re: sequences vs. synchronous replication

2021-12-17 Thread Tomas Vondra
On 12/18/21 05:52, Tom Lane wrote: Tomas Vondra writes: The problem is exactly the same as in [1] - the aborted transaction generated WAL, but RecordTransactionAbort() ignores that and does not update LogwrtResult.Write, with the reasoning that aborted transactions do not matter. But

Re: sequences vs. synchronous replication

2021-12-18 Thread Tomas Vondra
On 12/18/21 07:00, Tomas Vondra wrote: On 12/18/21 05:52, Tom Lane wrote: Tomas Vondra writes: The problem is exactly the same as in [1] - the aborted transaction generated WAL, but RecordTransactionAbort() ignores that and does not update LogwrtResult.Write, with the reasoning that

Re: sequences vs. synchronous replication

2021-12-18 Thread Tomas Vondra
On 12/18/21 22:27, Tom Lane wrote: Tomas Vondra writes: Here's a PoC demonstrating this idea. I'm not convinced it's the right way to deal with this - it surely seems more like a duct tape fix than a clean solution. But it does the trick. I was imagining something a wh

Re: sequences vs. synchronous replication

2021-12-20 Thread Tomas Vondra
On 12/20/21 15:31, Peter Eisentraut wrote: On 18.12.21 22:48, Tomas Vondra wrote: What do you mean by "not caching unused sequence numbers"? Reducing SEQ_LOG_VALS to 1, i.e. WAL-logging every sequence increment? That'd work, but I wonder how significant the impact will be. It&

Re: sequences vs. synchronous replication

2021-12-20 Thread Tomas Vondra
On 12/20/21 17:40, Tomas Vondra wrote: On 12/20/21 15:31, Peter Eisentraut wrote: On 18.12.21 22:48, Tomas Vondra wrote: What do you mean by "not caching unused sequence numbers"? Reducing SEQ_LOG_VALS to 1, i.e. WAL-logging every sequence increment? That'd work, but I wonder

Re: sequences vs. synchronous replication

2021-12-20 Thread Tomas Vondra
On 12/21/21 02:01, Tom Lane wrote: Tomas Vondra writes: OK, I did a quick test with two very simple benchmarks - simple select from a sequence, and 'pgbench -N' on scale 1. Benchmark was on current master, patched means SEQ_LOG_VALS was set to 1. But ... pgbench -N doesn't u

Re: sequences vs. synchronous replication

2021-12-21 Thread Tomas Vondra
On 12/19/21 04:03, Amit Kapila wrote: On Sat, Dec 18, 2021 at 7:24 AM Tomas Vondra wrote: while working on logical decoding of sequences, I ran into an issue with nextval() in a transaction that rolls back, described in [1]. But after thinking about it a bit more (and chatting with Petr

Re: sequences vs. synchronous replication

2021-12-22 Thread Tomas Vondra
On 12/22/21 05:56, Fujii Masao wrote: On 2021/12/22 10:57, Tomas Vondra wrote: On 12/19/21 04:03, Amit Kapila wrote: On Sat, Dec 18, 2021 at 7:24 AM Tomas Vondra wrote: while working on logical decoding of sequences, I ran into an issue with nextval() in a transaction that rolls

Re: sequences vs. synchronous replication

2021-12-22 Thread Tomas Vondra
On 12/21/21 03:49, Tomas Vondra wrote: On 12/21/21 02:01, Tom Lane wrote: Tomas Vondra writes: OK, I did a quick test with two very simple benchmarks - simple select from a sequence, and 'pgbench -N' on scale 1. Benchmark was on current master, patched means SEQ_LOG_VALS was set t

Re: sequences vs. synchronous replication

2021-12-22 Thread Tomas Vondra
On 12/22/21 18:50, Fujii Masao wrote: On 2021/12/22 21:11, Tomas Vondra wrote: Interesting idea, but I think it has a couple of issues :-( Thanks for the review! 1) We'd need to know the LSN of the last WAL record for any given sequence, and we'd need to communicate that betwee

Re: sequences vs. synchronous replication

2021-12-23 Thread Tomas Vondra
On 12/23/21 15:42, Fujii Masao wrote: On 2021/12/23 3:49, Tomas Vondra wrote: Attached is a patch tweaking WAL logging - in wal_level=minimal we do the same thing as now, in higher levels we log every sequence fetch. Thanks for the patch! With the patch, I found that the regression test

Re: sequences vs. synchronous replication

2021-12-23 Thread Tomas Vondra
On 12/24/21 06:37, Kyotaro Horiguchi wrote: At Thu, 23 Dec 2021 19:50:22 +0100, Tomas Vondra wrote in On 12/23/21 15:42, Fujii Masao wrote: On 2021/12/23 3:49, Tomas Vondra wrote: Attached is a patch tweaking WAL logging - in wal_level=minimal we do the same thing as now, in higher

Re: sequences vs. synchronous replication

2021-12-24 Thread Tomas Vondra
that position. regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: sequences vs. synchronous replication

2021-12-27 Thread Tomas Vondra
has other downsides. But that's not implemented either, and it's certainly out of scope for this patch. regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: using extended statistics to improve join estimates

2022-01-01 Thread Tomas Vondra
Hi, Here's an updated patch, rebased and fixing a couple typos reported by Justin Pryzby directly. regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL CompanyFrom 15d0fa5b565d9ae8b4f333c1d54745397964110d Mon Sep 17 00:00:00 2001 From: Tomas V

Re: PostgreSQL stops when adding a breakpoint in CLion

2022-01-03 Thread Tomas Vondra
uld affect debugging. Try redirecting the output to a log file, maybe that'll tell you what happened. regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: Collecting statistics about contents of JSONB columns

2022-01-06 Thread Tomas Vondra
similar to what + * jsonPathStatsGetLengthStats does? I think `jsonPathStatsGetTypeFreq(pstats, jbvArray, 0.0) <= 0.0` check should be added for jsonPathStatsGetArrayLengthStats(). To be continued ... OK. I'll see if Nikita has some ideas about the naming changes. regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: Collecting statistics about contents of JSONB columns

2022-01-06 Thread Tomas Vondra
quality? */ iseq = operator == JsonbLeOperator || operator == JsonbGeOperator, So I think this is correct. A comment explaining this would be nice. regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: Collecting statistics about contents of JSONB columns

2022-01-06 Thread Tomas Vondra
On 1/5/22 21:22, Simon Riggs wrote: On Fri, 31 Dec 2021 at 22:07, Tomas Vondra wrote: The patch does something far more elegant - it simply uses stavalues to store an array of JSONB documents, each describing stats for one path extracted from the sampled documents. Sounds good. I&#

Re: Use generation context to speed up tuplesorts

2022-01-07 Thread Tomas Vondra
eneration contexts, the default "-1" often outperforms "1MB" (but not the other options), while for aset it's pretty much "the higher value the better". Since this affects all memory allocations, I need to come up with other meaningful scenarios to benchmarks.

Re: sequences vs. synchronous replication

2022-01-11 Thread Tomas Vondra
On 12/22/21 18:50, Fujii Masao wrote: > > > On 2021/12/22 21:11, Tomas Vondra wrote: >> Interesting idea, but I think it has a couple of issues :-( > > Thanks for the review! > >> 1) We'd need to know the LSN of the last WAL record for any given >> se

Re: sequences vs. synchronous replication

2022-01-11 Thread Tomas Vondra
On 12/24/21 11:40, Tomas Vondra wrote: > > ... > > FWIW I plan to explore the idea of looking at sequence page LSN, and > flushing up to that position. > So, I explored the page LSN idea, and it seems to be working pretty nicely. There still is some impact on the workload

Re: More data files / forks

2022-01-11 Thread Tomas Vondra
e right the number of forks is fixed, and it's one of the places that's not extensible. I don't recall any proposals to change that, though, and even if we decided to do that, I doubt we'd allow the number of forks to be entirely dynamic. regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: PATCH: generate fractional cheapest paths in generate_orderedappend_path

2022-01-12 Thread Tomas Vondra
sort on both sides. I don't plan to pursue this further at this point, so I pushed the first part because that's a fairly simple improvement over what we have now. But it seems like a fairly promising area for improvements. Also, the non-intuitive effects of enabling partition-wi

Re: PATCH: generate fractional cheapest paths in generate_orderedappend_path

2022-01-13 Thread Tomas Vondra
FWIW this is now marked as committed. I've created a separate entry in the next CF for the incremental sort part. https://commitfest.postgresql.org/37/3513/ regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: PATCH: generate fractional cheapest paths in generate_orderedappend_path

2022-01-13 Thread Tomas Vondra
On 1/13/22 21:12, Arne Roland wrote: >  Hi! > >> From: Tomas Vondra >> Subject: Re: PATCH: generate fractional cheapest paths in > generate_orderedappend_path >>   >> test-# SELECT * FROM fract_t x LEFT JOIN fract_t y USING (id1, id2) >

minor bug in sort_inner_and_outer()

2022-01-13 Thread Tomas Vondra
_PathKey, length = 0, ...} Maybe it's some clever list-fu that I can't comprehend, but I guess it's a bug present since ~2004. It's benign because we only ever pass the front_pathkey to lcons() which does not really care. regards -- Tomas Vondra EnterpriseDB: http://www.ent

Re: Undocumented error

2022-01-14 Thread Tomas Vondra
ls, so use pg_waldump to dump that segment and show us a sufficiently large chunk from before that LSN. Not sure how much, because I don't know if you use subtransactions, how long the transactions are, etc. regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: extended stats on partitioned tables

2022-01-15 Thread Tomas Vondra
On 1/15/22 06:11, Justin Pryzby wrote: On Mon, Dec 13, 2021 at 09:40:09PM +0100, Tomas Vondra wrote: 1) If the table is a separate relation (not part of an inheritance tree), this should make no difference. -> OK 2) If the table is using "old" inheritance, this reverts back to p

Re: extended stats on partitioned tables

2022-01-15 Thread Tomas Vondra
On 1/15/22 19:35, Tomas Vondra wrote: On 1/15/22 06:11, Justin Pryzby wrote: On Mon, Dec 13, 2021 at 09:40:09PM +0100, Tomas Vondra wrote: 1) If the table is a separate relation (not part of an inheritance tree), this should make no difference. -> OK 2) If the table is using &

Re: sequences vs. synchronous replication

2022-01-15 Thread Tomas Vondra
On 1/15/22 06:12, Fujii Masao wrote: On 2022/01/12 1:07, Tomas Vondra wrote: I explored the idea of using page LSN a bit Many thanks! The patch from 22/12 simply checks if the change should/would wait for sync replica, and if yes it WAL-logs the sequence increment. There's a c

Re: extended stats on partitioned tables

2022-01-16 Thread Tomas Vondra
it'd make backpatching harder. So I'm not sure it'd be an improvement. In any case, I'm going to mark this as committed. Justin, if you think we should reconsider the ACL refactoring, please submit it as a separate patch. It seems mostly unrelated to the issue this thread w

Re: WIP: WAL prefetch (another approach)

2021-03-17 Thread Tomas Vondra
s just going to point out the FPIs are not necessary - what matters is the presence of long streaks of WAL records touching the same set of blocks. But people with workloads where this is common likely don't need the WAL prefetching at all - the replica can keep up just fine, because it doesn't need to do much I/O anyway (and if it can't then prefetching won't help much anyway). So just don't enable the prefetching, and there'll be no overhead. If it was up to me, I'd just get the patch committed as is. Delaying the feature because of concerns that it might have some negative effect in some cases, when that can be simply mitigated by disabling the feature, is not really beneficial for our users. regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: PoC/WIP: Extended statistics on expressions

2021-03-17 Thread Tomas Vondra
Hi, On 3/17/21 4:55 PM, Dean Rasheed wrote: > On Sun, 7 Mar 2021 at 21:10, Tomas Vondra > wrote: >> >> 2) ndistinct >> >> There's one thing that's bugging me, in how we handle "partial" matches. >> For each expression we track both the or

Re: Index Skip Scan (new UniqueKeys)

2021-03-17 Thread Tomas Vondra
On 3/17/21 6:02 PM, Dmitry Dolgov wrote: >> On Wed, Mar 17, 2021 at 03:28:00AM +0100, Tomas Vondra wrote: >> Hi, >> >> I took a look at the new patch series, focusing mostly on the uniquekeys >> part. It'd be a bit tedious to explain all the review comme

Re: PoC/WIP: Extended statistics on expressions

2021-03-17 Thread Tomas Vondra
On 3/17/21 7:54 PM, Dean Rasheed wrote: > On Wed, 17 Mar 2021 at 17:26, Tomas Vondra > wrote: >> >> My concern is that the current behavior (where we prefer expression >> stats over multi-column stats to some extent) works fine as long as the >> parts are

Re: WIP: BRIN multi-range indexes

2021-03-17 Thread Tomas Vondra
On 3/17/21 7:59 PM, John Naylor wrote: > On Thu, Mar 11, 2021 at 12:26 PM Tomas Vondra > mailto:tomas.von...@enterprisedb.com>> > wrote: >> >> Hi, >> >> Here is an updated version of the patch series. >> >> It fixes the assert failure (ju

Re: PoC/WIP: Extended statistics on expressions

2021-03-17 Thread Tomas Vondra
ctual = 30, current estimate = 120) But that's clearly a matter for a future patch, and I'm sure there are cases where this will produce worse estimates. Anyway, I plan to go over the patches one more time, and start pushing them sometime early next week. I don't want to leave it until the very last moment in the CF. regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: WIP: WAL prefetch (another approach)

2021-03-17 Thread Tomas Vondra
Hi, On 3/17/21 10:43 PM, Stephen Frost wrote: > Greetings, > > * Tomas Vondra (tomas.von...@enterprisedb.com) wrote: >> Right, I was just going to point out the FPIs are not necessary - what >> matters is the presence of long streaks of WAL records touching the same >>

Re: cleanup temporary files after crash

2021-03-18 Thread Tomas Vondra
On 3/17/21 2:34 AM, Euler Taveira wrote: > On Sun, Mar 14, 2021, at 11:01 PM, Thomas Munro wrote: >> On Wed, Mar 10, 2021 at 1:31 AM Michael Paquier > <mailto:mich...@paquier.xyz>> wrote: >> > On Tue, Mar 09, 2021 at 02:28:43AM +0100, Tomas Vondra wrote: >&g

Re: GROUP BY DISTINCT

2021-03-18 Thread Tomas Vondra
On 3/16/21 3:52 PM, Tomas Vondra wrote: > > > On 3/16/21 9:21 AM, Vik Fearing wrote: >> On 3/13/21 12:33 AM, Tomas Vondra wrote: >>> Hi Vik, >>> >>> The patch seems quite ready, I have just two comments. >> >> Thanks for taking a lo

Re: cleanup temporary files after crash

2021-03-18 Thread Tomas Vondra
ry? The other option is that the sleep in the TAP test is a bit too short, but those machines don't seem to be that slow. Anyway, TAP test relying on timing like this may not be the best idea, so I wonder how else to test this ... regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: cleanup temporary files after crash

2021-03-18 Thread Tomas Vondra
On 3/18/21 6:51 PM, Tomas Vondra wrote: > Hmm, > > crake and florican seem to have failed because of this commit, with this > error in the new TAP test: > > error running SQL: 'psql::1: ERROR: could not open directory > "base/pgsql_tmp": No such file or direc

Re: GROUP BY DISTINCT

2021-03-18 Thread Tomas Vondra
On 3/18/21 6:25 PM, Tomas Vondra wrote: > On 3/16/21 3:52 PM, Tomas Vondra wrote: >> >> >> On 3/16/21 9:21 AM, Vik Fearing wrote: >>> On 3/13/21 12:33 AM, Tomas Vondra wrote: >>>> Hi Vik, >>>> >>>> The patch seems quite

Re: cleanup temporary files after crash

2021-03-18 Thread Tomas Vondra
On 3/18/21 9:06 PM, Euler Taveira wrote: > On Thu, Mar 18, 2021, at 4:20 PM, Tomas Vondra wrote: >> I think a better way to test this would be to use a tuple lock: > I predicated such issues with this test. Your suggestion works for me. Maybe > you should use less rows in the

Re: GROUP BY DISTINCT

2021-03-18 Thread Tomas Vondra
On 3/18/21 10:02 PM, Thomas Munro wrote: > On Fri, Mar 19, 2021 at 8:27 AM Tomas Vondra > wrote: >> Hmmm, this seems to fail on lapwing with this error: >> >> parse_agg.c: In function 'expand_grouping_sets': >> parse_agg.c:1851:23: error: value compu

Re: cleanup temporary files after crash

2021-03-18 Thread Tomas Vondra
On 3/18/21 10:44 PM, Euler Taveira wrote: > On Thu, Mar 18, 2021, at 6:00 PM, Euler Taveira wrote: >> On Thu, Mar 18, 2021, at 5:51 PM, Tomas Vondra wrote: >>> OK. Can you prepare a patch with the proposed test approach? >> I'm on it. > What do you think about thi

Re: GROUP BY DISTINCT

2021-03-18 Thread Tomas Vondra
very other one. I'm not sure why > the compiler warnings are phrased just as they are, but the fix > I just pushed does make 4.5 happy. > Thanks! Yeah, that looks obvious. Funny the older compilers noticed this, not the new fancy ones. regards -- Tomas Vondra EnterpriseDB:

Re: cleanup temporary files after crash

2021-03-18 Thread Tomas Vondra
On 3/19/21 1:54 AM, Euler Taveira wrote: > On Thu, Mar 18, 2021, at 8:34 PM, Tomas Vondra wrote: >> Well, that's better, bit it still does not do the trick on the 32-bit >> machine - in that case a 1000 rows with int4 still fit into work_mem, so >> the temp file is not c

Re: WIP: WAL prefetch (another approach)

2021-03-18 Thread Tomas Vondra
On 3/18/21 1:54 AM, Thomas Munro wrote: > On Thu, Mar 18, 2021 at 12:00 PM Tomas Vondra > wrote: >> On 3/17/21 10:43 PM, Stephen Frost wrote: >>> Guess I'm just not a fan of pushing out a change that will impact >>> everyone by default, in a possibly negativ

Re: cleanup temporary files after crash

2021-03-18 Thread Tomas Vondra
On 3/19/21 2:21 AM, Tomas Vondra wrote: > On 3/19/21 1:54 AM, Euler Taveira wrote: >> On Thu, Mar 18, 2021, at 8:34 PM, Tomas Vondra wrote: >>> Well, that's better, bit it still does not do the trick on the 32-bit >>> machine - in that case a 1000 rows with int4 sti

Re: cleanup temporary files after crash

2021-03-18 Thread Tomas Vondra
On 3/19/21 3:57 AM, Tom Lane wrote: > Tomas Vondra writes: >> So crake and florican seem to be happy now. Not sure about lapwing yet. >> But interestingly enough, prion and curculio got unhappy. They worked >> fine with the older test, but now it fails with the "no

Re: cleanup temporary files after crash

2021-03-19 Thread Tomas Vondra
that does not seem to happen, so when the query gets executed before the lock is there, it results in infinite loop. In the attached patch I've simulated this by random() < 0.5. If I replace this with a wait loop in a plpgsql block, that works perfectly fine (no infinite loops). Tested bot

Re: cleanup temporary files after crash

2021-03-19 Thread Tomas Vondra
On 3/19/21 5:23 PM, Tomas Vondra wrote: > ... > > If I replace this with a wait loop in a plpgsql block, that works > perfectly fine (no infinite loops). Tested both on x86_64 and rpi. > For the record, here's the version with plpgsql block, which seems to be working

Re: cleanup temporary files after crash

2021-03-19 Thread Tomas Vondra
On 3/19/21 5:23 PM, Tomas Vondra wrote: > ... > > If I replace this with a wait loop in a plpgsql block, that works > perfectly fine (no infinite loops). Tested both on x86_64 and rpi. > For the record, here's the version with plpgsql block, which seems to be working

Re: cleanup temporary files after crash

2021-03-19 Thread Tomas Vondra
On 3/19/21 5:26 PM, Tomas Vondra wrote: > On 3/19/21 5:23 PM, Tomas Vondra wrote: >> ... >> >> If I replace this with a wait loop in a plpgsql block, that works >> perfectly fine (no infinite loops). Tested both on x86_64 and rpi. >> > > For the record

Re: [HACKERS] Custom compression methods

2021-03-19 Thread Tomas Vondra
e details. To get pkg-config, see <http://pkg-config.freedesktop.org/>. See `config.log' for more details I see xml2 also mentions pkg-config in configure (next to XML2_CFLAGS), but works fine without it. regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: [HACKERS] Custom compression methods

2021-03-20 Thread Tomas Vondra
opclasses (bloom and minmax-multi). Unfortunately, the only thing we have here is the type OID, so I guess the only option is using GetDefaultToastCompression(). Perhaps we might include that into BrinOpcInfo too, in the future. regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.c

Re: [HACKERS] Custom compression methods

2021-03-20 Thread Tomas Vondra
On 3/20/21 11:18 AM, Dilip Kumar wrote: > On Sat, Mar 20, 2021 at 3:05 PM Tomas Vondra > wrote: >> >> Hi, >> >> I think this bit in brin_tuple.c is wrong: >> >> ... >> Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc,

Re: [HACKERS] Custom compression methods

2021-03-20 Thread Tomas Vondra
On 3/20/21 11:45 AM, Tomas Vondra wrote: > > > On 3/20/21 11:18 AM, Dilip Kumar wrote: >> On Sat, Mar 20, 2021 at 3:05 PM Tomas Vondra >> wrote: >>> >>> Hi, >>> >>> I think this bit in brin_tuple.c is wrong: >>> >>&g

Re: [HACKERS] Custom compression methods

2021-03-20 Thread Tomas Vondra
On 3/20/21 4:21 PM, Justin Pryzby wrote: > On Sat, Mar 20, 2021 at 04:13:47PM +0100, Tomas Vondra wrote: >> +++ b/src/backend/access/brin/brin_tuple.c >> @@ -213,10 +213,20 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, >> BrinMemTuple *tuple, >>

Re: WIP: BRIN multi-range indexes

2021-03-22 Thread Tomas Vondra
On 3/22/21 6:27 AM, Tomas Vondra wrote: > > ... > > All the regression tests work fine, with the exception of minmax-multi > on a CIDR column. I don't know why, but the CREATE INDEX then fails like > this: > > ERROR: missing operator 1(650,650) in opfamily 16

Handling of opckeytype / CREATE OPERATOR CLASS (bug?)

2021-03-22 Thread Tomas Vondra
doing the first thing - this does the trick for me, but I'm not 100% sure it's the right fix. [1] https://www.postgresql.org/message-id/54b47e66-bd8a-d44a-2090-fd4b2f49abe6%40enterprisedb.com -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Compan

Re: Handling of opckeytype / CREATE OPERATOR CLASS (bug?)

2021-03-22 Thread Tomas Vondra
On 3/23/21 3:13 AM, Tom Lane wrote: > Tomas Vondra writes: >> while working on the new BRIN opclasses in [1], I stumbled on something >> I think is actually a bug in how CREATE OPERATOR CLASS handles the >> storage type. > > Hm. Both catalogs.sgml and pg_opcl

Re: Handling of opckeytype / CREATE OPERATOR CLASS (bug?)

2021-03-23 Thread Tomas Vondra
On 3/23/21 6:15 AM, Tom Lane wrote: > Tomas Vondra writes: >> On 3/23/21 3:13 AM, Tom Lane wrote: >>> Hm. Both catalogs.sgml and pg_opclass.h say specifically that >>> opckeytype should be zero if it's to be the same as the input >>> column type. I

Re: WIP: BRIN multi-range indexes

2021-03-23 Thread Tomas Vondra
On 3/23/21 2:36 PM, Alvaro Herrera wrote: > On 2021-Mar-22, Tomas Vondra wrote: > >> I don't know what's the right fix, but it seems like this patch has >> nothing to do with it. If we want to move the opclasses into an >> extension, we can comment out that on

Re: PoC/WIP: Extended statistics on expressions

2021-03-24 Thread Tomas Vondra
on (cast(null as uuid) < cast(null as uuid)) > left join pg_catalog.pg_aggregate as sample_1 tablesample > system (2.9) > on (sample_0.amprocnum = sample_1.aggnumdirectargs ) > inner join pg_catalog.pg_trigger as sample_2 tablesampl > -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: PoC/WIP: Extended statistics on expressions

2021-03-24 Thread Tomas Vondra
On 3/24/21 2:36 PM, Dean Rasheed wrote: > On Wed, 24 Mar 2021 at 10:22, Tomas Vondra > wrote: >> >> Thanks, it seems to be some thinko in handling in PlaceHolderVars, which >> seem to break the code's assumptions about varnos. This fixes it for me, >> bu

Re: PoC/WIP: Extended statistics on expressions

2021-03-24 Thread Tomas Vondra
lumns,\n" >> + " 'd' = any(stxkind) >> AS ndist_enabled,\n" >> + " 'f' = any(stxkind) >> AS deps_enabled,\n" >> + " 'm' = any(stxkind) >> AS mcv_enabled,\n"); >> + >> +if (pset.sversion >= 13) >> +appendPQExpBufferStr(&buf, " stxstattarget\n"); >> +else >> +appendPQExpBufferStr(&buf, " -1 AS >> stxstattarget\n"); > > >= 13 is fully determined by >= 14000 :) > Ah, right. >> + * type of the opclass, which is not interesting for our purposes. >> (Note: >> + * if we did anything with non-expression index columns, we'd need to > > index is wrong ? > Fixed > I mentioned a bunch of other references to "index" and "predicate" which are > still around: > Whooops, sorry. Fixed. I'll post a cleaned-up version of the patch addressing Dean's review comments too. regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: PoC/WIP: Extended statistics on expressions

2021-03-24 Thread Tomas Vondra
On 3/24/21 5:28 PM, Dean Rasheed wrote: > On Wed, 24 Mar 2021 at 14:48, Tomas Vondra > wrote: >> >> AFAIK the primary issue here is that the two places disagree. While >> estimate_num_groups does this >> >> varnos = pull_varnos(root, (Node *) varsher

Re: PoC/WIP: Extended statistics on expressions

2021-03-24 Thread Tomas Vondra
On 3/25/21 1:05 AM, Tomas Vondra wrote: > ... > > 0002 is an attempt to fix an issue I noticed today - we need to handle > type changes. Until now we did not have problems with that, because we > only had attnums - so we just reset the statistics (with the exception > of funct

Re: PoC/WIP: Extended statistics on expressions

2021-03-26 Thread Tomas Vondra
On 3/26/21 12:37 PM, Dean Rasheed wrote: > On Thu, 25 Mar 2021 at 19:59, Tomas Vondra > wrote: >> >> Attached is an updated patch series, with all the changes discussed >> here. I've cleaned up the ndistinct stuff a bit more (essentially >> reverting back

Re: WIP: BRIN multi-range indexes

2021-03-26 Thread Tomas Vondra
On 3/23/21 7:28 PM, Alvaro Herrera wrote: > On 2021-Mar-23, Tomas Vondra wrote: > >> FWIW there's yet another difference between the current BRIN opclass >> definition, compared to what CREATE OPERATOR CLASS would do. Or more >> precisely, how we'd define

Re: WIP: BRIN multi-range indexes

2021-03-26 Thread Tomas Vondra
bricks shy, because with (oi_regular_nulls == false) it should probably keep passing the NULL scan keys to the consistent function. I'll look into that and get it fixed. Thanks everyone who helped with those patches! -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: WIP: BRIN multi-range indexes

2021-03-26 Thread Tomas Vondra
On 3/26/21 2:08 PM, Tomas Vondra wrote: > Hi, > > I've pushed both the bloom and minmax-multi indexes today. > Hmmm, I see a couple buildfarm animals are upset about the minmax-multi patch. It does pass for me both on x86_64 and 32-bit ARM (rpi4), so I'm unable to repro

Re: WIP: BRIN multi-range indexes

2021-03-26 Thread Tomas Vondra
On 3/26/21 2:55 PM, Tom Lane wrote: > Tomas Vondra writes: >> I recall seeing "bus error" on sparc with other patches because of >> alignment issues, so I wonder if this is what's happening here. > > Try compiling with the address sanitizer enabled. Per c

Re: WIP: BRIN multi-range indexes

2021-03-26 Thread Tomas Vondra
On 3/26/21 3:04 PM, Tomas Vondra wrote: > On 3/26/21 2:55 PM, Tom Lane wrote: >> Tomas Vondra writes: >>> I recall seeing "bus error" on sparc with other patches because of >>> alignment issues, so I wonder if this is what's happening here. >> >

Re: WIP: BRIN multi-range indexes

2021-03-26 Thread Tomas Vondra
On 3/26/21 3:45 PM, Tomas Vondra wrote: > On 3/26/21 3:04 PM, Tomas Vondra wrote: >> On 3/26/21 2:55 PM, Tom Lane wrote: >>> Tomas Vondra writes: >>>> I recall seeing "bus error" on sparc with other patches because of >>>> alignment

Re: PoC/WIP: Extended statistics on expressions

2021-03-26 Thread Tomas Vondra
On 3/26/21 1:54 PM, Tomas Vondra wrote: > > > On 3/26/21 12:37 PM, Dean Rasheed wrote: >> On Thu, 25 Mar 2021 at 19:59, Tomas Vondra >> wrote: >>> >>> Attached is an updated patch series, with all the changes discussed >>> here. I've c

Showing applied extended statistics in explain

2021-03-26 Thread Tomas Vondra
ormed to AND list, which is wrong. This is easy to fix, I was lazy to do that in a PoC patch. 7) It does not show statistics for individual expressions. I suppose examine_variable could add it to the rel somehow, and maybe we could do that with index expressions too? regards --

Re: PoC/WIP: Extended statistics on expressions

2021-03-26 Thread Tomas Vondra
On 3/27/21 1:17 AM, Tomas Vondra wrote: > On 3/26/21 1:54 PM, Tomas Vondra wrote: >> >> >> On 3/26/21 12:37 PM, Dean Rasheed wrote: >>> On Thu, 25 Mar 2021 at 19:59, Tomas Vondra >>> wrote: >>>> >>>> Attached is an updated patch se

Re: WIP: BRIN multi-range indexes

2021-03-27 Thread Tomas Vondra
On 3/27/21 7:09 PM, Alvaro Herrera wrote: > On 2021-Mar-26, Tomas Vondra wrote: > >> Hi, >> >> I've pushed both the bloom and minmax-multi indexes today. > > One thing I've been wondering all along is how useful are these > BRIN-backed bloom indexes

<    1   2   3   4   5   6   7   8   9   10   >