Re: Parallel threads in query

2018-11-01 Thread Tomas Vondra
On 11/01/2018 08:03 PM, Andres Freund wrote: > On 2018-11-01 19:57:17 +0100, Tomas Vondra wrote: I think that very much depends on how expensive the tasks handled by the threads are. It may still be cheaper than a reasonable IPC, and if you don't create/destroy threads, that also sav

Re: Parallel threads in query

2018-11-01 Thread Andres Freund
Hi, On 2018-11-01 09:17:56 -1000, Darafei "Komяpa" Praliaskouski wrote: > So, do I understand correctly that I need to start a parallel worker that > does nothing for each thread I launch to consume the parallel worker limit? No, I don't think that'd be reasonable. I think what we're saying is th

Re: Parallel threads in query

2018-11-01 Thread Komяpa
> > > Because you said "faster than reasonable IPC" - which to me implies that > you don't do full blown IPC. Which using threads in a bgworker is very > strongly implying. What you're proposing strongly implies multiple > context switches just to process a few results. Even before, but > especiall

Re: Parallel threads in query

2018-11-01 Thread Andres Freund
On 2018-11-01 19:57:17 +0100, Tomas Vondra wrote: > >> I think that very much depends on how expensive the tasks handled by the > >> threads are. It may still be cheaper than a reasonable IPC, and if you > >> don't create/destroy threads, that also saves quite a bit of time. > > > > I'm not follow

Re: Parallel threads in query

2018-11-01 Thread Tomas Vondra
On 11/01/2018 07:50 PM, Andres Freund wrote: > Hi, > > On 2018-11-01 19:44:54 +0100, Tomas Vondra wrote: >> On 11/01/2018 07:40 PM, Andres Freund wrote: >>> On 2018-11-01 19:33:39 +0100, Tomas Vondra wrote: In theory, simulating such global limit should be possible using a bit of shared

Re: Parallel threads in query

2018-11-01 Thread Andres Freund
Hi, On 2018-11-01 19:44:54 +0100, Tomas Vondra wrote: > On 11/01/2018 07:40 PM, Andres Freund wrote: > > On 2018-11-01 19:33:39 +0100, Tomas Vondra wrote: > >> In theory, simulating such global limit should be possible using a bit > >> of shared memory for the current total, per-process counter an

Re: Parallel threads in query

2018-11-01 Thread Tomas Vondra
On 11/01/2018 07:43 PM, Darafei "Komяpa" Praliaskouski wrote: > In theory, simulating such global limit should be possible using a bit > of shared memory for the current total, per-process counter and probably > some simple abort handling (say, just like contrib/openssl does using >

Re: Parallel threads in query

2018-11-01 Thread Tomas Vondra
On 11/01/2018 07:40 PM, Andres Freund wrote: > Hi, > > On 2018-11-01 19:33:39 +0100, Tomas Vondra wrote: >> In theory, simulating such global limit should be possible using a bit >> of shared memory for the current total, per-process counter and probably >> some simple abort handling (say, just li

Re: Parallel threads in query

2018-11-01 Thread Komяpa
> > In theory, simulating such global limit should be possible using a bit > of shared memory for the current total, per-process counter and probably > some simple abort handling (say, just like contrib/openssl does using > ResourceOwner). > I would expect that this limit is already available and

Re: Parallel threads in query

2018-11-01 Thread Andres Freund
Hi, On 2018-11-01 19:33:39 +0100, Tomas Vondra wrote: > In theory, simulating such global limit should be possible using a bit > of shared memory for the current total, per-process counter and probably > some simple abort handling (say, just like contrib/openssl does using > ResourceOwner). Right

Re: Parallel threads in query

2018-11-01 Thread Tomas Vondra
On 11/01/2018 06:15 PM, Andres Freund wrote: > On 2018-11-01 10:10:33 -0700, Paul Ramsey wrote: >> On Wed, Oct 31, 2018 at 2:11 PM Tom Lane wrote: >> >>> =?UTF-8?Q?Darafei_=22Kom=D1=8Fpa=22_Praliaskouski?= >>> writes: Question is, what's the best policy to allocate cores so we can play ni

Re: Parallel threads in query

2018-11-01 Thread Andres Freund
On 2018-11-01 10:10:33 -0700, Paul Ramsey wrote: > On Wed, Oct 31, 2018 at 2:11 PM Tom Lane wrote: > > > =?UTF-8?Q?Darafei_=22Kom=D1=8Fpa=22_Praliaskouski?= > > writes: > > > Question is, what's the best policy to allocate cores so we can play nice > > > with rest of postgres? > > > > > > Ther

Re: Parallel threads in query

2018-11-01 Thread Paul Ramsey
On Wed, Oct 31, 2018 at 2:11 PM Tom Lane wrote: > =?UTF-8?Q?Darafei_=22Kom=D1=8Fpa=22_Praliaskouski?= > writes: > > Question is, what's the best policy to allocate cores so we can play nice > > with rest of postgres? > > There is not, because we do not use or support multiple threads inside >

Re: Parallel threads in query

2018-11-01 Thread Konstantin Knizhnik
On 31.10.2018 22:07, Darafei "Komяpa" Praliaskouski wrote: Hi, I've tried porting some of PostGIS algorithms to utilize multiple cores via OpenMP to return faster. Question is, what's the best policy to allocate cores so we can play nice with rest of postgres? What I'd like to see is so

Re: Parallel threads in query

2018-10-31 Thread David Fetter
On Wed, Oct 31, 2018 at 09:07:43AM -1000, Darafei "Komяpa" Praliaskouski wrote: > Hi, > > I've tried porting some of PostGIS algorithms to utilize multiple cores via > OpenMP to return faster. Great! > Question is, what's the best policy to allocate cores so we can play nice > with rest of postg

Re: Parallel threads in query

2018-10-31 Thread Tom Lane
=?UTF-8?Q?Darafei_=22Kom=D1=8Fpa=22_Praliaskouski?= writes: > Question is, what's the best policy to allocate cores so we can play nice > with rest of postgres? > What I'd like to see is some function that I can call and get a number of > threads I'm allowed to run, that will also advise rest of