On 27.12.2017 13:08, Andres Freund wrote:
On December 27, 2017 11:05:52 AM GMT+01:00, james
wrote:
All threads are blocked in semaphores.
That they are blocked is inevitable - I guess the issue is that they
are
thrashing.
I guess it would be necessary to separate the internals to have some
On 27/12/2017 10:08, Andres Freund wrote:
Optimizing for this seems like a pointless exercise. If the goal is efficient
processing of 100k connections the solution is a session / connection
abstraction and a scheduler. Optimizing for this amount of concurrency just
will add complexity and sl
On December 27, 2017 11:05:52 AM GMT+01:00, james
wrote:
> > All threads are blocked in semaphores.
>That they are blocked is inevitable - I guess the issue is that they
>are
>thrashing.
>I guess it would be necessary to separate the internals to have some
>internal queueing and effectively r
> All threads are blocked in semaphores.
That they are blocked is inevitable - I guess the issue is that they are
thrashing.
I guess it would be necessary to separate the internals to have some
internal queueing and effectively reduce the number of actively
executing threads.
In effect make th
On 21.12.2017 16:25, Konstantin Knizhnik wrote:
I continue experiments with my pthread prototype.
Latest results are the following:
1. I have eliminated all (I hope) calls of non-reentrant functions
(getopt, setlocale, setitimer, localtime, ...). So now parallel tests
are passed.
2. I have
2017-12-21 14:25 GMT+01:00 Konstantin Knizhnik :
> I continue experiments with my pthread prototype.
> Latest results are the following:
>
> 1. I have eliminated all (I hope) calls of non-reentrant functions
> (getopt, setlocale, setitimer, localtime, ...). So now parallel tests are
> passed.
>
>
I continue experiments with my pthread prototype.
Latest results are the following:
1. I have eliminated all (I hope) calls of non-reentrant functions
(getopt, setlocale, setitimer, localtime, ...). So now parallel tests
are passed.
2. I have implemented deallocation of top memory context (at
On 06/12/2017 17:26, Andreas Karlsson wrote:
An additional issue is that this could break a lot of extensions and
in a way that it is not apparent at compile time. This means we may
need to break all extensions to force extensions authors to check if
they are thread safe.
I do not like making
On Sat, Dec 9, 2017 at 1:09 AM, konstantin knizhnik <
k.knizh...@postgrespro.ru> wrote:
> I am not going to show stack traces of all 1000 threads.
> But you may notice that proc array lock really seems be be a bottleneck.
>
Yes, proc array lock easily becomes a bottleneck on multicore machine wit
On Dec 7, 2017, at 10:41 AM, Simon Riggs wrote:
>> But it is a theory. The main idea of this prototype was to prove or disprove
>> this expectation at practice.
>
>> But please notice that it is very raw prototype. A lot of stuff is not
>> working yet.
>
>> And supporting all of exited Postgres
On 8 December 2017 at 03:58, Andres Freund wrote:
> On 2017-12-07 11:26:07 +0800, Craig Ringer wrote:
> > PostgreSQL's architecture conflates "connection", "session" and
> "executor"
> > into one somewhat muddled mess.
>
> How is the executor entangled in the other two?
>
>
Executor in the postgr
Hi,
On 2017-12-07 20:48:06 +, Greg Stark wrote:
> But then I thought about it a bit and I do wonder. I don't know how
> well we test having multiple portals doing all kinds of different
> query plans with their execution interleaved.
Cursors test that pretty well.
> And I definitely have do
On 7 December 2017 at 19:58, Andres Freund wrote:
> On 2017-12-07 11:26:07 +0800, Craig Ringer wrote:
>> PostgreSQL's architecture conflates "connection", "session" and "executor"
>> into one somewhat muddled mess.
>
> How is the executor entangled in the other two?
I was going to ask the same qu
On 2017-12-07 11:26:07 +0800, Craig Ringer wrote:
> PostgreSQL's architecture conflates "connection", "session" and "executor"
> into one somewhat muddled mess.
How is the executor entangled in the other two?
Greetings,
Andres Freund
On Wed, Dec 6, 2017 at 10:20 PM, Craig Ringer wrote:
> Personally I think it's a pity we didn't land up here before the foundations
> for parallel query went in - DSM, shm_mq, DSA, etc. I know the EDB folks at
> least looked into it though, and presumably there were good reasons to go in
> this di
On 7 December 2017 at 19:55, Konstantin Knizhnik
wrote:
>
> Pros:
> 1. Simplified memory model: no need in DSM, shm_mq, DSA, etc
>
shm_mq would remain useful, and the others could only be dropped if you
also dropped process-model support entirely.
> 1. Breaks compatibility with existed extensi
On 07.12.2017 00:58, Thomas Munro wrote:
Using a ton of thread local variables may be a useful stepping stone,
but if we want to be able to separate threads/processes from sessions
eventually then I guess we'll want to model sessions as first class
objects and pass them around explicitly or usi
Hi
On 06.12.2017 20:08, Andres Freund wrote:
4. Rewrite file descriptor cache to be global (shared by all threads).
That one I'm very unconvinced of, that's going to add a ton of new
contention.
Do you mean lock contention because of mutex I used to synchronize
access to shared file descript
I want to thank everybody for feedbacks and a lot of useful notices.
I am very pleased with interest of community to this topic and will
continue research in this direction.
Some more comments from my side:
My original intention was to implement some king of built-in connection
pooling for Pos
> But it is a theory. The main idea of this prototype was to prove or disprove
> this expectation at practice.
> But please notice that it is very raw prototype. A lot of stuff is not
> working yet.
> And supporting all of exited Postgres functionality requires
> much more efforts (and even more
On 7 December 2017 at 11:44, Tsunakawa, Takayuki <
tsunakawa.ta...@jp.fujitsu.com> wrote:
> From: Craig Ringer [mailto:cr...@2ndquadrant.com]
> > I'd personally expect that an immediate conversion would result
> > in very
> > little speedup, a bunch of code deleted, a bunch of complexi
From: Craig Ringer [mailto:cr...@2ndquadrant.com]
> I'd personally expect that an immediate conversion would result
> in very
> little speedup, a bunch of code deleted, a bunch of complexity
> added. And it'd still be massively worthwhile, to keep medium to
> long
> term com
On 7 December 2017 at 05:58, Thomas Munro
wrote:
>
> Using a ton of thread local variables may be a useful stepping stone,
> but if we want to be able to separate threads/processes from sessions
> eventually then I guess we'll want to model sessions as first class
> objects and pass them around e
On 7 December 2017 at 01:17, Andres Freund wrote:
>
> > I think you've done us a very substantial service by pursuing
> > this far enough to get some quantifiable performance results.
> > But now that we have some results in hand, I think we're best
> > off sticking with the architecture we've g
On Thu, Dec 7, 2017 at 6:08 AM, Andres Freund wrote:
> On 2017-12-06 19:40:00 +0300, Konstantin Knizhnik wrote:
>> As far as I remember, several years ago when implementation of intra-query
>> parallelism was just started there was discussion whether to use threads or
>> leave traditional Postgres
Hi,
On 2017-12-06 12:28:29 -0500, Robert Haas wrote:
> > Possibly we even want to continue having various
> > processes around besides that, the most interesting cases involving
> > threads are around intra-query parallelism, and pooling, and for both a
> > hybrid model could be beneficial.
>
> I
On Wed, Dec 6, 2017 at 12:08 PM, Andres Freund wrote:
>> 4. Rewrite file descriptor cache to be global (shared by all threads).
>
> That one I'm very unconvinced of, that's going to add a ton of new
> contention.
It might be OK on systems where we can use pread()/pwrite().
Otherwise it's going to
On 12/06/2017 06:08 PM, Andres Freund wrote:
I think the biggest problem with doing this for real is that it's a huge
project, and that it'll take a long time.
An additional issue is that this could break a lot of extensions and in
a way that it is not apparent at compile time. This means we m
> "barely a 50% speedup" - Hah. I don't believe the numbers, but that'd be
> huge.
They are numbers derived from a benchmark that any sane person would
be using a connection pool for in a production environment, but
impressive if true none the less.
Hi,
On 2017-12-06 11:53:21 -0500, Tom Lane wrote:
> Konstantin Knizhnik writes:
> However, if I guess at which numbers are supposed to be what,
> it looks like even the best case is barely a 50% speedup.
"barely a 50% speedup" - Hah. I don't believe the numbers, but that'd be
huge.
> That woul
On Wed, Dec 6, 2017 at 11:53 AM, Tom Lane wrote:
> barely a 50% speedup.
I think that's an awfully strange choice of adverb. This is, by its
authors own admission, a rough cut at this, probably with very little
of the optimization that could ultimately done, and it's already
buying 50% on some t
Hi!
On 2017-12-06 19:40:00 +0300, Konstantin Knizhnik wrote:
> As far as I remember, several years ago when implementation of intra-query
> parallelism was just started there was discussion whether to use threads or
> leave traditional Postgres process architecture. The decision was made to
> leav
Here it is formatted a little better.
So a little over 50% performance improvement for a couple of the test cases.
On Wed, Dec 6, 2017 at 11:53 AM, Tom Lane wrote:
> Konstantin Knizhnik writes:
> > Below are some results (1000xTPS) of select-only (-S) pgbench with scale
> > 100 at my des
Konstantin Knizhnik writes:
> Below are some results (1000xTPS) of select-only (-S) pgbench with scale
> 100 at my desktop with quad-core i7-4770 3.40GHz and 16Gb of RAM:
> Connections Vanilla/default Vanilla/prepared
> pthreads/defaultpthreads/prepared
> 10 100 191
34 matches
Mail list logo
