Re: [External] Re: FDW, too long to run explain

2019-02-18 Thread Vijaykumar Jain
Oh Wow, i guess you are right. I just ran example where local runs make use of parallel setup, but not FDW. i have three servers 2 x pg10 1 x pg11 i run queries on coordinator node ( pg11 ) which makes calls to foreign server to do a simple count. the individual nodes run the query in parallel, t

Re: [External] Re: FDW, too long to run explain

2019-02-17 Thread Jeff Janes
On Sun, Feb 17, 2019 at 6:32 AM Vijaykumar Jain wrote: > I am yet to figure out the reason, what we have done is implement fake > columns to represent samples and giving them random numbers and keeping > other bulls to fake limit. > > Most of the queries that were impacted were the ones that did

Re: [External] Re: FDW, too long to run explain

2019-02-17 Thread Vijaykumar Jain
Ok. I’ll try to work on it this week and see if i am able to reproduce anything. On Mon, 18 Feb 2019 at 2:30 AM Jeff Janes wrote: > > > On Sun, Feb 17, 2019 at 2:37 PM Vijaykumar Jain > wrote: > >> >> Ok, i raked this from the logs where enabled log_min_duration_statement = >> 10s >> >> 2019-01

Re: [External] Re: FDW, too long to run explain

2019-02-17 Thread Jeff Janes
On Sun, Feb 17, 2019 at 2:37 PM Vijaykumar Jain wrote: > > Ok, i raked this from the logs where enabled log_min_duration_statement = > 10s > > 2019-01-31 12:48:18 UTC LOG: duration: 29863.311 ms statement: EXPLAIN > SELECT blah, FROM public.view WHERE ((scheduled_bdt >= '2019-01-20'::date)) > A

Re: FDW, too long to run explain

2019-02-17 Thread auxsvr
On Sunday, 17 February 2019 20:58:47 EET Jeff Janes wrote: > A question for the PostgreSQL hackers would be, Is it necessary and > desirable that the EXPLAIN be issued in the same transaction as the > eventual DECLARE and FETCHes? I don't think it is. I guess if the foreign > side table definitio

Re: [External] Re: FDW, too long to run explain

2019-02-17 Thread Vijaykumar Jain
Regards, Vijay On Mon, Feb 18, 2019 at 12:56 AM Jeff Janes wrote: > On Sun, Feb 17, 2019 at 1:52 PM Vijaykumar Jain > wrote: > >> Assuming your questions as 1,2,3, please find my answers below. >> >> 1)"explain" on foreign servers run as "idle in transactions". coz they >> were running very lo

Re: [External] Re: FDW, too long to run explain

2019-02-17 Thread Vijaykumar Jain
Hey Jeff, yes, we now relaxed the idle in transaction setting to 15 mins. i was hesitant to increase the settings as it blocked auto vaccum. We use hot_standby_feedback = true also as we split reads/writes and allow long running queries on read replicas, this too affects auto vaccum. so overall,

Re: [External] Re: FDW, too long to run explain

2019-02-17 Thread Jeff Janes
On Sun, Feb 17, 2019 at 1:52 PM Vijaykumar Jain wrote: > Assuming your questions as 1,2,3, please find my answers below. > > 1)"explain" on foreign servers run as "idle in transactions". coz they > were running very long (in the order of some minutes) , pgbouncer (in tx > level pooling) setting k

Re: FDW, too long to run explain

2019-02-17 Thread Tom Lane
Jeff Janes writes: > A question for the PostgreSQL hackers would be, Is it necessary and > desirable that the EXPLAIN be issued in the same transaction as the > eventual DECLARE and FETCHes? I don't think it is. It seems like a good idea to me. I certainly don't think "I've got an idle-in-trans

Re: FDW, too long to run explain

2019-02-17 Thread Jeff Janes
On Sun, Feb 17, 2019 at 12:41 PM Jeff Janes wrote: > On Mon, Feb 4, 2019 at 2:15 AM Vijaykumar Jain > wrote: > >> >> now we have some long running queries via FDW that take minutes and get >> killed explain runs as idle in transaction on remote servers. >> > > Are you saying the EXPLAIN itself g

Re: [External] Re: FDW, too long to run explain

2019-02-17 Thread Vijaykumar Jain
Assuming your questions as 1,2,3, please find my answers below. 1)"explain" on foreign servers run as "idle in transactions". coz they were running very long (in the order of some minutes) , pgbouncer (in tx level pooling) setting kill them (as idle in tx time limit exceeded of 5 mins) or else res

Re: FDW, too long to run explain

2019-02-17 Thread Jeff Janes
On Mon, Feb 4, 2019 at 2:15 AM Vijaykumar Jain wrote: > > now we have some long running queries via FDW that take minutes and get > killed explain runs as idle in transaction on remote servers. > Are you saying the EXPLAIN itself gets killed, or execution of the plan generated based on the EXPLA

Re: [External] Re: FDW, too long to run explain

2019-02-17 Thread Vijaykumar Jain
I am yet to figure out the reason, what we have done is implement fake columns to represent samples and giving them random numbers and keeping other bulls to fake limit. Most of the queries that were impacted were the ones that did not push order by and limit to foreign servers. I am also trying t

Re: FDW, too long to run explain

2019-02-17 Thread auxsvr
Related to this question: Postgresql cursors are in most cases I've tried extremely slow. The cause is as described in my previous answer, in my experience. Is there any plan or way to improve this situation? For example, for FDW one would expect the plan on the remote side to be similar, if no

Re: FDW, too long to run explain

2019-02-09 Thread auxsvr
On Monday, 4 February 2019 09:14:14 EET Vijaykumar Jain wrote: > Hi, Hi, > with pg v10.1 > we use writes directly to shards, and reads via FDW from all shards (RO) > our DB size is ~ 500GB each shard, and tables are huge too. > 1 table ~ 200GB, 1 ~55GB, 1 ~40GB and a lot of small tables, but lar