Re: Partition column should be part of PK

2021-07-08 Thread Christophe Pettus
> On Jul 8, 2021, at 20:32, Nagaraj Raj wrote: > > My apologies for making confusion with new thread. Yes its same issue related > to earlier post. > > I was trying to figure out how to ensure unique values for columns > (billing_account_guid, ban). If i add partition key to constraint , i

Re: Partition column should be part of PK

2021-07-08 Thread Nagaraj Raj
My apologies for making confusion with new thread. Yes its same issue related to earlier post. I was trying to figure out  how to ensure unique values for columns (billing_account_guid, ban). If i add partition key to constraint , it wont be possible what im looking for. My use case as below  IN

Re: Strange execution plan

2021-07-08 Thread Manuel Weitzman
> On 08-07-2021, at 17:13, Manuel Weitzman wrote: > > Option A: Use a common table expression to "force" the usage of > test_json_data_idx > >WITH json_matching_rows AS ( >SELECT t.* >FROM test ti >WHERE t.json_data @> '{"book":{"title":"In Search of Lost Time"}}'

RE: Partition column should be part of PK

2021-07-08 Thread Mike Sofen
I believe this thread qualifies for the funniest thread of 2021 (so far). And yes, this is a top post. :-) Mike Sofen -Original Message- From: Alvaro Herrera Sent: Thursday, July 08, 2021 3:29 PM To: Justin Pryzby Cc: Nagaraj Raj ; pgsql-performa...@postgresql.org Subject: Re: Parti

Re: Partition column should be part of PK

2021-07-08 Thread Alvaro Herrera
On 2021-Jul-08, Justin Pryzby wrote: > If I'm not wrong, this is the same thing you asked 2 week ago. > > If so, why not continue the conversation on the same thread, and why not > reference the old thread ? > > I went to the effort to find the old conversation. > https://www.postgresql.org/mess

Re: Partition column should be part of PK

2021-07-08 Thread Justin Pryzby
If I'm not wrong, this is the same thing you asked 2 week ago. If so, why not continue the conversation on the same thread, and why not reference the old thread ? I went to the effort to find the old conversation. https://www.postgresql.org/message-id/20210625042228.gj29...@telsasoft.com If decl

temporary file log lines

2021-07-08 Thread MichaelDBA
Hi all, I got a question about PG log lines with temporary file info like this: case 1: log line with no contextual info 2021-07-07 20:28:15 UTC:10.100.11.95(50274):myapp@mydb:[35200]:LOG: temporary file: path "base/pgsql_tmp/pgsql_tmp35200.0", size 389390336 case 2: log line with contextual

Re: Strange execution plan

2021-07-08 Thread Manuel Weitzman
> On 08-07-2021, at 17:13, Manuel Weitzman wrote: > > I'd recommend you using cursors for pagination in general (I know it > may not be possible for you, just wanted to explain as it could be > useful). By the way, I mean cursor pagination as the general concept. I'm not talking about Postgre

Re: Strange execution plan

2021-07-08 Thread Manuel Weitzman
> On 08-07-2021, at 04:26, Joel Frid wrote: > > Hi, > > On my production environment (PostgreSQL 13.3), one of my queries runs very > slow, about 2 minutes. > I noticed that it does not use an execution plan that I anticapited it would. > > The query is : > > SELECT t.* > FROM test t > W

ERROR: there is no unique or exclusion constraint matching the ON CONFLICT specification

2021-07-08 Thread Nagaraj Raj
Hello everyone, I have a scenario where wanted to add PK on partition to make sure to monitor unique values for two columns values. but as PG required to partition column should be part of PK. How can we make sure actual two columns need to be unique values. and also while insert into table need

Re: ETL - sql orchestrator is stuck when there is not sleep() between queries

2021-07-08 Thread Allan Barrielle
fsync is off and full_page_writes is off because the script works one time. We create the db, we load the data, then we dump the data and kill the db. No need to handle servers crashed or anything like that. 0.1 vacuum_cleanup_index_scale_factor is the default value. On Thu, Jul 8, 2021 at 4:06 P

Re: ETL - sql orchestrator is stuck when there is not sleep() between queries

2021-07-08 Thread Allan Barrielle
On a different machine, we use 12.7. Still same issue On Thu, Jul 8, 2021 at 3:49 PM Allan Barrielle wrote: > Hello, > > > Is it true that the SELECTs have no joins in them ? > > Yes there is a lot of LEFT JOIN. > > > When it doesn't work, you could check SELECT * FROM pg_stat_activity, > and >

Re: ETL - sql orchestrator is stuck when there is not sleep() between queries

2021-07-08 Thread Allan Barrielle
Hello, > Is it true that the SELECTs have no joins in them ? Yes there is a lot of LEFT JOIN. > When it doesn't work, you could check SELECT * FROM pg_stat_activity, and >SELECT pg_blocking_pids(pid), * FROM pg_locks, to see what's going on. I can't see any blocking queries blocking pg_locks, p

Re: ETL - sql orchestrator is stuck when there is not sleep() between queries

2021-07-08 Thread Justin Pryzby
On Thu, Jul 08, 2021 at 03:49:12PM +0200, Allan Barrielle wrote: > > Is it true that the SELECTs have no joins in them ? > > Yes there is a lot of LEFT JOIN. > > > It'd be very useful to get "explain analyze" for a working query and for a > > stuck query. It sound like the stuck query never fini

Re: ETL - sql orchestrator is stuck when there is not sleep() between queries

2021-07-08 Thread Imre Samu
> We use different machines, different config, and different datasets. > ... > PostgreSQL 12.4 (Debian 12.4-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit Is It possible to upgrade and test with PG 12.7? IMHO: lot of changes: * https://www.postgresql.org/docs

Re: ETL - sql orchestrator is stuck when there is not sleep() between queries

2021-07-08 Thread Justin Pryzby
On Thu, Jul 08, 2021 at 01:00:28AM +0200, Allan Barrielle wrote: > All is fine, and can work great. > But sometimes, some queries that used to take about 20 secs to complete can > suddenly end in 5mins. > Important all queries have the same shape -> CREATE TABLE SELECT AS *(a bit > of transform) FR

Strange execution plan

2021-07-08 Thread Joel Frid
Hi, On my production environment (PostgreSQL 13.3), one of my queries runs very slow, about 2 minutes. I noticed that it does not use an execution plan that I anticapited it would. The query is : SELECT t.* FROM test t  WHERE  t."existe" IS true and  t.json_data" @> '{"book":{"title"

Strange execution plan

2021-07-08 Thread Joel Frid
Hi, On my production environment (PostgreSQL 13.3), one of my queries runs very slow, about 2 minutes. I noticed that it does not use an execution plan that I anticapited it would. The query is : SELECT t.* FROM test t  WHERE  t."existe" IS true and  t.json_data" @> '{"book":{"title"

ETL - sql orchestrator is stuck when there is not sleep() between queries

2021-07-08 Thread Allan Barrielle
Hello guys, I'm facing a problem. Currently I'm working on a Data transformation Pipeline on Postgres. The strategy is, We select every tables in a given schema ( 50 tables ), we apply some case when, translation, enum and load it into a different new schema with a CREATE TABLE SCHEMA_2.table_1 AS