Re: How to properly hash a row or set of columns.

2022-01-18 Thread Klaudie Willis
:34, Klaudie Willis wrote: > Hashing a row or set of columns is useful in some circumstances where you > need to compare the row with for instance incoming data. This is most > relevant when you do not control the source data yourself, or else you would > usually solve it by

How to properly hash a row or set of columns.

2022-01-18 Thread Klaudie Willis
Hashing a row or set of columns is useful in some circumstances where you need to compare the row with for instance incoming data. This is most relevant when you do not control the source data yourself, or else you would usually solve it by other means. Still, It would be great if you could do

Re: When Update balloons memory

2021-12-14 Thread Klaudie Willis
txid = txid + 1; Klaudie ‐‐‐ Original Message ‐‐‐ On Tuesday, December 14th, 2021 at 16:58, Vincent Veyron wrote: > On Tue, 14 Dec 2021 08:16:08 + > > Klaudie Willis klaudie.wil...@protonmail.com wrote: > > > CREATE INDEX partindx ON public.part_main USING btree ((ac

Re: When Update balloons memory

2021-12-14 Thread Klaudie Willis
te it. best regards Klaudie ‐‐‐ Original Message ‐‐‐ On Tuesday, December 14th, 2021 at 12:17 AM, Tom Lane wrote: > Klaudie Willis klaudie.wil...@protonmail.com writes: > > > So, it seems like the index is central cog here: > > > > > create index ind1 on alpha ((de

Re: When Update balloons memory

2021-12-13 Thread Klaudie Willis
can't vouch for what the system creates behind my back though. Is this a feature or a bug? -- Klaudie Sent with ProtonMail Secure Email. ‐‐‐ Original Message ‐‐‐ On Tuesday, December 7th, 2021 at 15:57, Tom Lane wrote: > Klaudie Willis klaudie.wil...@protonmail.com writes: > &g

When Update balloons memory

2021-12-07 Thread Klaudie Willis
About the system: Ubuntu 20.04, 64GB ram, 16GB shared buffer, 500 MB working mem, Postgresql 14.1 Core issue: The following statement below, when not divided up into chunks, but run across all 800M rows, did trigger an OOM-kill from the OS. I have looked into it by kernel logs as well as postgres

Fw: Re: Bitmap scan seem like such a strange choice when "limit 1"

2020-09-07 Thread Klaudie Willis
> t > '2020-01-01'::timestamp and t < '2021-01-01'::timestamp >Not at all important, but it seems odd to be exclusive of the start and end >both. I would >consider including the start with >= >Michael Lewis | Database Engineer >Entrata Michael, funny I was thinking that myself minutes after posti

Fw: Re: Bitmap scan seem like such a strange choice when "limit 1"

2020-09-07 Thread Klaudie Willis
‐‐‐ Original Message ‐‐‐ On Monday, September 7, 2020 9:04 AM, Klaudie Willis wrote: > Maybe the right way to improve this is to bypass add_paths_to_append_rel > entirely when there's exactly one surviving child rel, and make it > just use all the surviving paths

Re: Bitmap scan seem like such a strange choice when "limit 1"

2020-09-04 Thread Klaudie Willis
> PostgreSQL estimates that 2817675 rows satisfy the index condition and expects > that it will have to scan many of them before it finds one that satisfies the > filter condition. That turns out to be a wrong guess. > > You could create an index on (cars_ref, t), then PostgreSQL will certainly > p

Bitmap scan seem like such a strange choice when "limit 1"

2020-09-04 Thread Klaudie Willis
Postgres 13 beta3 set enable_bitmapscan=1; -- default explain (analyze,buffers) select * from bigtable where cars_ref = 1769854207 and t > '2020-01-01'::timestamp and t < '2021-01-01'::timestamp limit 1 Short story. Big table > 100M rows. b-tree index on cars_ref, the t constraints limits it to

Re: Performance of "distinct with limit"

2020-08-29 Thread Klaudie Willis
Thanks for your insight Jeff. Interesting read! K Sent with [ProtonMail](https://protonmail.com) Secure Email. ‐‐‐ Original Message ‐‐‐ On Saturday, August 29, 2020 6:23 PM, Jeff Janes wrote: > On Fri, Aug 28, 2020 at 8:34 AM Klaudie Willis > wrote: > >> No index

Re: Performance of "distinct with limit"

2020-08-28 Thread Klaudie Willis
uickly. Can you share the execution plan > for your query? > > ----------- > > De: "Klaudie Willis" > Para: "pgsql-general" > Enviadas: Sexta-feira, 28 de agosto de 2020 8:29:58 > Assunto: Performan

Performance of "distinct with limit"

2020-08-28 Thread Klaudie Willis
Hi, Ran into this under-optimized query execution. select distinct n from bigtable; -- Lets say this takes 2 minutes select distinct n from bigtable limit 2 -- This takes approximately the same time However, the latter should have the potential to be so much quicker. I checked the same query o

Re: n_distinct off by a factor of 1000

2020-06-27 Thread Klaudie Willis
> I am not sure whether I am misinterpreting something, or if it is a bug > (probably former) however, when using partitions, are not > n_distinct_inherited supposed to propagate to the child partitions? It does > not seem to do so. (Yes, I run Analyze after setting the variable) I had to > set

Re: n_distinct off by a factor of 1000

2020-06-27 Thread Klaudie Willis
On the topic of n_distinct. I am not sure whether I am misinterpreting something, or if it is a bug (probably former) however, when using partitions, are not n_distinct_inherited supposed to propagate to the child partitions? It does not seem to do so. (Yes, I run Analyze after setting the vari

Re: n_distinct off by a factor of 1000

2020-06-25 Thread Klaudie Willis
> If we could increase the sampling ratio beyond the hard coded 300x to get a > more representative sample and use that to estimate ndistinct (and also the > frequency of the most common values) but only actually stored the 100 MCVs > (or whatever the stats target is set to for the system or col

Re: n_distinct off by a factor of 1000

2020-06-24 Thread Klaudie Willis
ot; 0.9258158 113846 2107 5001 "bigtable_y2020" 0.875 -0.375 2 "bigtable_y2021" 0.92304045 118267 2204 5001 "bigtable_y2019" select count(distinct instrumentid_ref) from bigtable --> 33 385 922 Bigtables instrumentid_ref is underestimated by 300X even when statistics t

Re: n_distinct off by a factor of 1000

2020-06-23 Thread Klaudie Willis
Adrian, you are correct. My mistanke. K ‐‐‐ Original Message ‐‐‐ On Tuesday, June 23, 2020 4:14 PM, Adrian Klaver wrote: > On 6/23/20 7:05 AM, Fabio Pardi wrote: > > > On 23/06/2020 14:42, Klaudie Willis wrote: > > > > > I got my first hint of why this prob

Re: n_distinct off by a factor of 1000

2020-06-23 Thread Klaudie Willis
ote: > Maybe I missed it, but did you run "ANALYZE VERBOSE bigtable;"? > > On 6/23/20 7:42 AM, Klaudie Willis wrote: > >> Friends, >> >> I run Postgresql 12.3, on Windows. I have just discovered a pretty >> significant problem with Postgresql and my data.

n_distinct off by a factor of 1000

2020-06-23 Thread Klaudie Willis
Friends, I run Postgresql 12.3, on Windows. I have just discovered a pretty significant problem with Postgresql and my data. I have a large table, 500M rows, 50 columns. It is split in 3 partitions by Year. In addition to the primary key, one of the columns is indexed, and I do lookups on this.