On 27.12.2017 13:08, Andres Freund wrote:
On December 27, 2017 11:05:52 AM GMT+01:00, james
wrote:
All threads are blocked in semaphores.
That they are blocked is inevitable - I guess the issue is that they
are
thrashing.
I guess it would be necessary to separate the internals to have some
On 27/12/2017 10:08, Andres Freund wrote:
Optimizing for this seems like a pointless exercise. If the goal is efficient
processing of 100k connections the solution is a session / connection
abstraction and a scheduler. Optimizing for this amount of concurrency just
will add complexity and sl
On December 27, 2017 11:05:52 AM GMT+01:00, james
wrote:
> > All threads are blocked in semaphores.
>That they are blocked is inevitable - I guess the issue is that they
>are
>thrashing.
>I guess it would be necessary to separate the internals to have some
>internal queueing and effectively r
> All threads are blocked in semaphores.
That they are blocked is inevitable - I guess the issue is that they are
thrashing.
I guess it would be necessary to separate the internals to have some
internal queueing and effectively reduce the number of actively
executing threads.
In effect make th
far from completion.
4. I have performed experiments with replacing synchronization
primitives used in Postgres with pthread analogues.
Unfortunately it has almost now influence on performance.
5. Handling large number of connections.
The maximal number of postgres connections is almost the
l regression tests now. But handling of
> errors is still far from completion.
>
> 4. I have performed experiments with replacing synchronization primitives
> used in Postgres with pthread analogues.
> Unfortunately it has almost now influence on performance.
>
> 5. Handling l
experiments with replacing synchronization
primitives used in Postgres with pthread analogues.
Unfortunately it has almost now influence on performance.
5. Handling large number of connections.
The maximal number of postgres connections is almost the same: 100k.
But memory footprint in case of pthreads
On 06/12/2017 17:26, Andreas Karlsson wrote:
An additional issue is that this could break a lot of extensions and
in a way that it is not apparent at compile time. This means we may
need to break all extensions to force extensions authors to check if
they are thread safe.
I do not like making
On Sat, Dec 9, 2017 at 1:09 AM, konstantin knizhnik <
k.knizh...@postgrespro.ru> wrote:
> I am not going to show stack traces of all 1000 threads.
> But you may notice that proc array lock really seems be be a bottleneck.
>
Yes, proc array lock easily becomes a bottleneck on multicore machine wit
On Dec 7, 2017, at 10:41 AM, Simon Riggs wrote:
>> But it is a theory. The main idea of this prototype was to prove or disprove
>> this expectation at practice.
>
>> But please notice that it is very raw prototype. A lot of stuff is not
>> working yet.
>
>> And supporting all of exited Postgres
On 8 December 2017 at 03:58, Andres Freund wrote:
> On 2017-12-07 11:26:07 +0800, Craig Ringer wrote:
> > PostgreSQL's architecture conflates "connection", "session" and
> "executor"
> > into one somewhat muddled mess.
>
> How is the executor entangled in the other two?
>
>
Executor in the postgr
Hi,
On 2017-12-07 20:48:06 +, Greg Stark wrote:
> But then I thought about it a bit and I do wonder. I don't know how
> well we test having multiple portals doing all kinds of different
> query plans with their execution interleaved.
Cursors test that pretty well.
> And I definitely have do
On 7 December 2017 at 19:58, Andres Freund wrote:
> On 2017-12-07 11:26:07 +0800, Craig Ringer wrote:
>> PostgreSQL's architecture conflates "connection", "session" and "executor"
>> into one somewhat muddled mess.
>
> How is the executor entangled in the other two?
I was going to ask the same qu
On 2017-12-07 11:26:07 +0800, Craig Ringer wrote:
> PostgreSQL's architecture conflates "connection", "session" and "executor"
> into one somewhat muddled mess.
How is the executor entangled in the other two?
Greetings,
Andres Freund
On Wed, Dec 6, 2017 at 10:20 PM, Craig Ringer wrote:
> Personally I think it's a pity we didn't land up here before the foundations
> for parallel query went in - DSM, shm_mq, DSA, etc. I know the EDB folks at
> least looked into it though, and presumably there were good reasons to go in
> this di
On 7 December 2017 at 19:55, Konstantin Knizhnik
wrote:
>
> Pros:
> 1. Simplified memory model: no need in DSM, shm_mq, DSA, etc
>
shm_mq would remain useful, and the others could only be dropped if you
also dropped process-model support entirely.
> 1. Breaks compatibility with existed extensi
On 07.12.2017 00:58, Thomas Munro wrote:
Using a ton of thread local variables may be a useful stepping stone,
but if we want to be able to separate threads/processes from sessions
eventually then I guess we'll want to model sessions as first class
objects and pass them around explicitly or usi
Hi
On 06.12.2017 20:08, Andres Freund wrote:
4. Rewrite file descriptor cache to be global (shared by all threads).
That one I'm very unconvinced of, that's going to add a ton of new
contention.
Do you mean lock contention because of mutex I used to synchronize
access to shared file descript
I want to thank everybody for feedbacks and a lot of useful notices.
I am very pleased with interest of community to this topic and will
continue research in this direction.
Some more comments from my side:
My original intention was to implement some king of built-in connection
pooling for Pos
> But it is a theory. The main idea of this prototype was to prove or disprove
> this expectation at practice.
> But please notice that it is very raw prototype. A lot of stuff is not
> working yet.
> And supporting all of exited Postgres functionality requires
> much more efforts (and even more
On 7 December 2017 at 11:44, Tsunakawa, Takayuki <
tsunakawa.ta...@jp.fujitsu.com> wrote:
> From: Craig Ringer [mailto:cr...@2ndquadrant.com]
> > I'd personally expect that an immediate conversion would result
> > in very
> > little speedup, a bunch of code deleted, a bunch of complexi
From: Craig Ringer [mailto:cr...@2ndquadrant.com]
> I'd personally expect that an immediate conversion would result
> in very
> little speedup, a bunch of code deleted, a bunch of complexity
> added. And it'd still be massively worthwhile, to keep medium to
> long
> term com
On 7 December 2017 at 05:58, Thomas Munro
wrote:
>
> Using a ton of thread local variables may be a useful stepping stone,
> but if we want to be able to separate threads/processes from sessions
> eventually then I guess we'll want to model sessions as first class
> objects and pass them around e
On 7 December 2017 at 01:17, Andres Freund wrote:
>
> > I think you've done us a very substantial service by pursuing
> > this far enough to get some quantifiable performance results.
> > But now that we have some results in hand, I think we're best
> > off sticking with the architecture we've g
On Thu, Dec 7, 2017 at 6:08 AM, Andres Freund wrote:
> On 2017-12-06 19:40:00 +0300, Konstantin Knizhnik wrote:
>> As far as I remember, several years ago when implementation of intra-query
>> parallelism was just started there was discussion whether to use threads or
>> leave traditional Postgres
Hi,
On 2017-12-06 12:28:29 -0500, Robert Haas wrote:
> > Possibly we even want to continue having various
> > processes around besides that, the most interesting cases involving
> > threads are around intra-query parallelism, and pooling, and for both a
> > hybrid model could be beneficial.
>
> I
On Wed, Dec 6, 2017 at 12:08 PM, Andres Freund wrote:
>> 4. Rewrite file descriptor cache to be global (shared by all threads).
>
> That one I'm very unconvinced of, that's going to add a ton of new
> contention.
It might be OK on systems where we can use pread()/pwrite().
Otherwise it's going to
On 12/06/2017 06:08 PM, Andres Freund wrote:
I think the biggest problem with doing this for real is that it's a huge
project, and that it'll take a long time.
An additional issue is that this could break a lot of extensions and in
a way that it is not apparent at compile time. This means we m
> "barely a 50% speedup" - Hah. I don't believe the numbers, but that'd be
> huge.
They are numbers derived from a benchmark that any sane person would
be using a connection pool for in a production environment, but
impressive if true none the less.
Hi,
On 2017-12-06 11:53:21 -0500, Tom Lane wrote:
> Konstantin Knizhnik writes:
> However, if I guess at which numbers are supposed to be what,
> it looks like even the best case is barely a 50% speedup.
"barely a 50% speedup" - Hah. I don't believe the numbers, but that'd be
huge.
> That woul
On Wed, Dec 6, 2017 at 11:53 AM, Tom Lane wrote:
> barely a 50% speedup.
I think that's an awfully strange choice of adverb. This is, by its
authors own admission, a rough cut at this, probably with very little
of the optimization that could ultimately done, and it's already
buying 50% on some t
Hi!
On 2017-12-06 19:40:00 +0300, Konstantin Knizhnik wrote:
> As far as I remember, several years ago when implementation of intra-query
> parallelism was just started there was discussion whether to use threads or
> leave traditional Postgres process architecture. The decision was made to
> leav
Here it is formatted a little better.
So a little over 50% performance improvement for a couple of the test cases.
On Wed, Dec 6, 2017 at 11:53 AM, Tom Lane wrote:
> Konstantin Knizhnik writes:
> > Below are some results (1000xTPS) of select-only (-S) pgbench with scale
> > 100 at my des
Konstantin Knizhnik writes:
> Below are some results (1000xTPS) of select-only (-S) pgbench with scale
> 100 at my desktop with quad-core i7-4770 3.40GHz and 16Gb of RAM:
> Connections Vanilla/default Vanilla/prepared
> pthreads/defaultpthreads/prepared
> 10 100 191
Hi hackers,
As far as I remember, several years ago when implementation of
intra-query parallelism was just started there was discussion whether to
use threads or leave traditional Postgres process architecture. The
decision was made to leave processes. So now we have bgworkers, shared
messag
35 matches
Mail list logo
