Re: [HACKERS] Experimental dynamic memory allocation of postgresql shared memory

2016-06-19 Thread Michael Paquier
On Mon, Jun 20, 2016 at 12:40 PM, Craig Ringer wrote: > On 18 June 2016 at 02:42, Robert Haas wrote: >> >> On Fri, Jun 17, 2016 at 2:23 PM, Aleksey Demakov >> wrote: >> > Essentially this is pessimizing for the lowest common denominator >> > among OSes. >> >> I totally agree. That's how we make

Re: [HACKERS] Experimental dynamic memory allocation of postgresql shared memory

2016-06-19 Thread Craig Ringer
On 18 June 2016 at 02:42, Robert Haas wrote: > On Fri, Jun 17, 2016 at 2:23 PM, Aleksey Demakov > wrote: > > Essentially this is pessimizing for the lowest common denominator > > among OSes. > > I totally agree. That's how we make the server portable. > > > Having a contiguous address space mak

Re: [HACKERS] Experimental dynamic memory allocation of postgresql shared memory

2016-06-17 Thread Aleksey Demakov
On Sat, Jun 18, 2016 at 3:43 AM, Tom Lane wrote: > DSM already exists, and for many purposes its lack of a > within-a-shmem-segment dynamic allocator is irrelevant; the same purpose > is served (with more speed, more reliability, and less code) by releasing > the whole DSM segment when no longer n

Re: [HACKERS] Experimental dynamic memory allocation of postgresql shared memory

2016-06-17 Thread Tom Lane
Aleksey Demakov writes: > On Sat, Jun 18, 2016 at 12:45 AM, Tom Lane wrote: >> You're right, but that doesn't mean that the community is going to take >> much interest in an unportable replacement for code that already exists. > Excuse me, what code already exists? As far as I understand, we > c

Re: [HACKERS] Experimental dynamic memory allocation of postgresql shared memory

2016-06-17 Thread Aleksey Demakov
On Sat, Jun 18, 2016 at 12:45 AM, Tom Lane wrote: > Aleksey Demakov writes: >> On Fri, Jun 17, 2016 at 10:54 PM, Robert Haas wrote: >>> In my opinion, that's not going to fly. If I thought otherwise, I >>> would not have developed the DSM facility in the first place. > >> Essentially this is pe

Re: [HACKERS] Experimental dynamic memory allocation of postgresql shared memory

2016-06-17 Thread Tom Lane
Aleksey Demakov writes: > On Fri, Jun 17, 2016 at 10:54 PM, Robert Haas wrote: >> On Fri, Jun 17, 2016 at 12:34 PM, Aleksey Demakov wrote: >>> I believe it would be perfectly okay to allocate huge amount of address >>> space with mmap on startup. If the pages are not touched, the OS VM >>> subs

Re: [HACKERS] Experimental dynamic memory allocation of postgresql shared memory

2016-06-17 Thread Robert Haas
On Fri, Jun 17, 2016 at 2:23 PM, Aleksey Demakov wrote: > Essentially this is pessimizing for the lowest common denominator > among OSes. I totally agree. That's how we make the server portable. > Having a contiguous address space makes things so > much simpler that considering this case, IMHO,

Re: [HACKERS] Experimental dynamic memory allocation of postgresql shared memory

2016-06-17 Thread David G. Johnston
On Fri, Jun 17, 2016 at 2:23 PM, Aleksey Demakov wrote: > On Fri, Jun 17, 2016 at 10:54 PM, Robert Haas > wrote: > > On Fri, Jun 17, 2016 at 12:34 PM, Aleksey Demakov > wrote: > >>> I expect that to be useful for parallel query and anything else where > >>> processes need to share variable-size

Re: [HACKERS] Experimental dynamic memory allocation of postgresql shared memory

2016-06-17 Thread Aleksey Demakov
On Sat, Jun 18, 2016 at 12:31 AM, Andres Freund wrote: > On 2016-06-18 00:23:14 +0600, Aleksey Demakov wrote: >> Finally, it's possible to repeatedly mmap >> and munmap on portions of a contiguous address space providing >> a given addr argument for both of them. The last option might, of >> cours

Re: [HACKERS] Experimental dynamic memory allocation of postgresql shared memory

2016-06-17 Thread Aleksey Demakov
Sorry for unclear language. Late Friday evening in my place is to blame. On Sat, Jun 18, 2016 at 12:23 AM, Aleksey Demakov wrote: > On Fri, Jun 17, 2016 at 10:54 PM, Robert Haas wrote: >> On Fri, Jun 17, 2016 at 12:34 PM, Aleksey Demakov wrote: I expect that to be useful for parallel query

Re: [HACKERS] Experimental dynamic memory allocation of postgresql shared memory

2016-06-17 Thread Andres Freund
On 2016-06-18 00:23:14 +0600, Aleksey Demakov wrote: > Finally, it's possible to repeatedly mmap > and munmap on portions of a contiguous address space providing > a given addr argument for both of them. The last option might, of > course, is susceptible to hijacking this portion of the address by

Re: [HACKERS] Experimental dynamic memory allocation of postgresql shared memory

2016-06-17 Thread Aleksey Demakov
On Fri, Jun 17, 2016 at 10:54 PM, Robert Haas wrote: > On Fri, Jun 17, 2016 at 12:34 PM, Aleksey Demakov wrote: >>> I expect that to be useful for parallel query and anything else where >>> processes need to share variable-size data. However, that's different >>> from this because ours can grown

Re: [HACKERS] Experimental dynamic memory allocation of postgresql shared memory

2016-06-17 Thread Robert Haas
On Fri, Jun 17, 2016 at 12:34 PM, Aleksey Demakov wrote: >> I expect that to be useful for parallel query and anything else where >> processes need to share variable-size data. However, that's different >> from this because ours can grown to arbitrary size and shrink again by >> allocating and fr

Re: [HACKERS] Experimental dynamic memory allocation of postgresql shared memory

2016-06-17 Thread Aleksey Demakov
On Fri, Jun 17, 2016 at 10:18 PM, Robert Haas wrote: > On Fri, Jun 17, 2016 at 11:30 AM, Tom Lane wrote: > But I'm a bit confused about where it gets the bytes it wants to > manage. There's no call to dsm_create() or ShmemAlloc() anywhere in > the code, at least not that I could find quickly. T

Re: [HACKERS] Experimental dynamic memory allocation of postgresql shared memory

2016-06-17 Thread Aleksey Demakov
On Fri, Jun 17, 2016 at 9:30 PM, Tom Lane wrote: > Aleksey Demakov writes: >> I have some very experimental code to enable dynamic memory allocation >> of shared memory for postgresql backend processes. > > Um ... what's this do that the existing DSM stuff doesn't do? > It operates over a single

Re: [HACKERS] Experimental dynamic memory allocation of postgresql shared memory

2016-06-17 Thread Robert Haas
On Fri, Jun 17, 2016 at 11:30 AM, Tom Lane wrote: > Aleksey Demakov writes: >> I have some very experimental code to enable dynamic memory allocation >> of shared memory for postgresql backend processes. > > Um ... what's this do that the existing DSM stuff doesn't do? It seems to be a full-fled

Re: [HACKERS] Experimental dynamic memory allocation of postgresql shared memory

2016-06-17 Thread Tom Lane
Aleksey Demakov writes: > I have some very experimental code to enable dynamic memory allocation > of shared memory for postgresql backend processes. Um ... what's this do that the existing DSM stuff doesn't do? regards, tom lane -- Sent via pgsql-hackers mailing list

[HACKERS] Experimental dynamic memory allocation of postgresql shared memory

2016-06-17 Thread Aleksey Demakov
Hi all, I have some very experimental code to enable dynamic memory allocation of shared memory for postgresql backend processes. The source code in the repository is not complete yet. Moreover it is not immediately useful by itself. However it might serve as the basis to implement higher-level fe