Robert Haas writes:
> I've often thought that 64-bit machines are so capable that there's no
> reason to go any higher. But lately I've started to wonder. There
> are already machines out there with >2^40 bytes of physical memory,
> and the number just keeps creeping up. When you reserve a coup
On Fri, Jan 10, 2014 at 1:35 PM, Claudio Freire wrote:
>>> You can map a segment at fork time, and unmap it after forking. That
>>> doesn't really use RAM, since it's supposed to be lazily allocated (it
>>> can be forced to be so, I believe, with PROT_NONE and MAP_NORESERVE,
>>> but I don't think
On Fri, Jan 10, 2014 at 3:23 PM, Robert Haas wrote:
> On Thu, Jan 9, 2014 at 12:46 PM, Claudio Freire
> wrote:
>> On Thu, Jan 9, 2014 at 2:22 PM, Robert Haas wrote:
>>> It would be nice to have better operating system support for this.
>>> For example, IIUC, 64-bit Linux has 128TB of address sp
On Thu, Jan 9, 2014 at 2:09 PM, Amit Kapila wrote:
> On Thu, Jan 9, 2014 at 12:21 AM, Robert Haas wrote:
>> On Tue, Jan 7, 2014 at 10:20 PM, Amit Kapila wrote:
>>> On Tue, Jan 7, 2014 at 2:46 AM, Robert Haas wrote:
Well, right now we just reopen the same object from all of the
pr
On Thu, Jan 9, 2014 at 12:46 PM, Claudio Freire wrote:
> On Thu, Jan 9, 2014 at 2:22 PM, Robert Haas wrote:
>> It would be nice to have better operating system support for this.
>> For example, IIUC, 64-bit Linux has 128TB of address space available
>> for user processes. When you clone(), it ca
On 1/9/14, 1:18 PM, knizhnik wrote:
So it is clear why do we need shared memory for parallel query execution. But
why it has to be dynamic? Why it can not be preallocated at start time as most
of other resources used by PostgreSQL?
That would limit us to doing something like allocating a fixe
On 01/09/2014 11:48 PM, Claudio Freire wrote:
On Thu, Jan 9, 2014 at 4:39 PM, knizhnik wrote:
At fork time I only wrote about reserving the address space. After
reserving it, all you have to do is implement an allocator that works
in shared memory (protected by a lwlock of course).
In essence,
On Thu, Jan 9, 2014 at 4:48 PM, Claudio Freire wrote:
> On Thu, Jan 9, 2014 at 4:39 PM, knizhnik wrote:
>>> At fork time I only wrote about reserving the address space. After
>>> reserving it, all you have to do is implement an allocator that works
>>> in shared memory (protected by a lwlock of c
On Thu, Jan 9, 2014 at 4:39 PM, knizhnik wrote:
>> At fork time I only wrote about reserving the address space. After
>> reserving it, all you have to do is implement an allocator that works
>> in shared memory (protected by a lwlock of course).
>>
>> In essence, a hypothetical pg_dsm_alloc(region
On 01/09/2014 11:30 PM, Claudio Freire wrote:
On Thu, Jan 9, 2014 at 4:24 PM, knizhnik wrote:
On 01/09/2014 09:46 PM, Claudio Freire wrote:
On Thu, Jan 9, 2014 at 2:22 PM, Robert Haas wrote:
It would be nice to have better operating system support for this.
For example, IIUC, 64-bit Linux ha
On Fri, Jan 10, 2014 at 1:00 AM, knizhnik wrote:
> On 01/09/2014 11:09 PM, Amit Kapila wrote:
>>
>>
>> Using DuplicateHandle(), we can make segment stick for Postmaster
>> lifetime. I have used below test (used dsm_demo module) to verify:
>
> As far as I understand DuplicateHandle() should really
On 01/09/2014 11:09 PM, Amit Kapila wrote:
On Thu, Jan 9, 2014 at 12:21 AM, Robert Haas wrote:
On Tue, Jan 7, 2014 at 10:20 PM, Amit Kapila wrote:
On Tue, Jan 7, 2014 at 2:46 AM, Robert Haas wrote:
Well, right now we just reopen the same object from all of the
processes, which seems to work
On Thu, Jan 9, 2014 at 4:24 PM, knizhnik wrote:
> On 01/09/2014 09:46 PM, Claudio Freire wrote:
>>
>> On Thu, Jan 9, 2014 at 2:22 PM, Robert Haas wrote:
>>>
>>> It would be nice to have better operating system support for this.
>>> For example, IIUC, 64-bit Linux has 128TB of address space availa
On 01/09/2014 09:46 PM, Claudio Freire wrote:
On Thu, Jan 9, 2014 at 2:22 PM, Robert Haas wrote:
It would be nice to have better operating system support for this.
For example, IIUC, 64-bit Linux has 128TB of address space available
for user processes. When you clone(), it can either share the
On 01/09/2014 09:22 PM, Robert Haas wrote:
On Wed, Jan 8, 2014 at 2:39 PM, knizhnik wrote:
I wonder what is the intended use case of dynamic shared memory?
Is is primarly oriented on PostgreSQL extensions or it will be used also in
PosatgreSQL core?
My main motivation is that I want to use it
On Thu, Jan 9, 2014 at 12:21 AM, Robert Haas wrote:
> On Tue, Jan 7, 2014 at 10:20 PM, Amit Kapila wrote:
>> On Tue, Jan 7, 2014 at 2:46 AM, Robert Haas wrote:
>>>
>>> Well, right now we just reopen the same object from all of the
>>> processes, which seems to work fine and doesn't require any o
On Thu, Jan 9, 2014 at 2:22 PM, Robert Haas wrote:
> It would be nice to have better operating system support for this.
> For example, IIUC, 64-bit Linux has 128TB of address space available
> for user processes. When you clone(), it can either share the entire
> address space (i.e. it's a thread
On Wed, Jan 8, 2014 at 2:39 PM, knizhnik wrote:
> I wonder what is the intended use case of dynamic shared memory?
> Is is primarly oriented on PostgreSQL extensions or it will be used also in
> PosatgreSQL core?
My main motivation is that I want to use it to support parallel query.
There is unf
On 01/08/2014 10:51 PM, Robert Haas wrote:
On Tue, Jan 7, 2014 at 10:20 PM, Amit Kapila wrote:
On Tue, Jan 7, 2014 at 2:46 AM, Robert Haas wrote:
On Mon, Jan 6, 2014 at 4:04 PM, james wrote:
The point remains that you need to duplicate it into every process that
might
want to use it subsequ
On Tue, Jan 7, 2014 at 10:20 PM, Amit Kapila wrote:
> On Tue, Jan 7, 2014 at 2:46 AM, Robert Haas wrote:
>> On Mon, Jan 6, 2014 at 4:04 PM, james wrote:
>>> The point remains that you need to duplicate it into every process that
>>> might
>>> want to use it subsequently, so it makes sense to Dup
On Tue, Jan 7, 2014 at 2:46 AM, Robert Haas wrote:
> On Mon, Jan 6, 2014 at 4:04 PM, james wrote:
>> The point remains that you need to duplicate it into every process that
>> might
>> want to use it subsequently, so it makes sense to DuplicateHandle into the
>> parent, and then to advertise that
On Mon, Jan 6, 2014 at 4:04 PM, james wrote:
> The point remains that you need to duplicate it into every process that
> might
> want to use it subsequently, so it makes sense to DuplicateHandle into the
> parent, and then to advertise that handle value publicly so that other
> child
> processes
On 06/01/2014 04:20, Amit Kapila wrote:
Duplicate handle should work, but we need to communicate the handle
to other process using IPC.
Only if the other process needs to use it. The IPC is not to transfer
the handle to
the other process, just to tell it which slot in its handle table
contains
On 06/01/2014 03:14, Robert Haas wrote:
That's up to the application. After calling dsm_create(), you call
dsm_segment_handle() to get the 32-bit integer handle for that
segment. Then you have to get that to the other process(es) somehow.
If you're trying to share a handle with a background wor
On Sun, Jan 5, 2014 at 11:04 PM, james wrote:
> On 05/01/2014 16:50, Robert Haas wrote:
>
> But on Windows, segments are *automatically*
> destroyed *by the operating system* when the last process unmaps them,
> so it's not quite so clear to me how we can allow it there. The main
> shared memory
On Sun, Jan 5, 2014 at 1:44 PM, james wrote:
> I'm intrigued - how are the handles shared between children that are peers
> in the current scheme? Some handle transfer must already be in place.
That's up to the application. After calling dsm_create(), you call
dsm_segment_handle() to get the 32
On Sun, Jan 5, 2014 at 1:28 PM, knizhnik wrote:
> From my point of view it is not a big problem that it is not possible to
> place LWLock in DSM.
> I can allocate LWLocks in standard way - using RequestAddinLWLocks and use
> them for synchronization.
Sure, well, that works fine if you're being lo
On 05/01/2014 18:02, Robert Haas wrote:
On Sun, Jan 5, 2014 at 12:34 PM, james wrote:
>On 05/01/2014 16:50, Robert Haas wrote:
>
> But on Windows, segments are*automatically*
>destroyed*by the operating system* when the last process unmaps them,
>so it's not quite so clear to me how we can al
From my point of view it is not a big problem that it is not possible
to place LWLock in DSM.
I can allocate LWLocks in standard way - using RequestAddinLWLocks and
use them for synchronization.
Concerning support of huge pages - actually I do not think that it
should involve something more th
On Sun, Jan 5, 2014 at 12:34 PM, james wrote:
> On 05/01/2014 16:50, Robert Haas wrote:
>
> But on Windows, segments are *automatically*
> destroyed *by the operating system* when the last process unmaps them,
> so it's not quite so clear to me how we can allow it there. The main
> shared memory
On 05/01/2014 16:50, Robert Haas wrote:
But on Windows, segments are*automatically*
destroyed*by the operating system* when the last process unmaps them,
so it's not quite so clear to me how we can allow it there. The main
shared memory segment is no problem because the postmaster always has
On Sat, Jan 4, 2014 at 3:27 PM, knizhnik wrote:
> 1. I want IMCS to work with PostgreSQL versions not supporting DSM (dynamic
> shared memory), like 9.2, 9.3.1,...
Yeah. If it's loaded at postmaster start time, then it can work with
any version. On 9.4+, you could possibly make it work even if
On 01/04/2014 11:11 PM, Tom Lane wrote:
knizhnik writes:
On 01/04/2014 12:05 PM, David Fetter wrote:
Is there some way not to use shared memory for it?
No, IMCS ("In-Memory Columnar Store") is storing data in shared memory.
It would probably be better if it made use of the dynamic shared mem
knizhnik writes:
> On 01/04/2014 12:05 PM, David Fetter wrote:
>> Is there some way not to use shared memory for it?
> No, IMCS ("In-Memory Columnar Store") is storing data in shared memory.
It would probably be better if it made use of the dynamic shared memory
features that exist in HEAD.
On 01/04/2014 12:05 PM, David Fetter wrote:
I'm sorry I misunderstood about the extension you wrote.
Is there some way not to use shared memory for it?
No, IMCS ("In-Memory Columnar Store") is storing data in shared memory.
Certainly I could allocate shared memory myself, but due to portabilit
I'm sorry I misunderstood about the extension you wrote.
Is there some way not to use shared memory for it?
Cheers,
David.
On Sat, Jan 04, 2014 at 11:46:25AM +0400, knizhnik wrote:
> Hi David,
>
> Sorry, but I do not completely understand your suggestions:
>
> 1. IMCS really contains single pa
Hi David,
Sorry, but I do not completely understand your suggestions:
1. IMCS really contains single patch file sysv_shmem.patch.
Applying this patch is not mandatory for using IMCS: it just solves the
problem with support of > 256Gb of shared memory.
Right now PostgreSQL is not able to use mor
On Thu, Jan 02, 2014 at 08:48:24PM +0400, knizhnik wrote:
> I want to announce implementation of In-Memory Columnar Store
> extension for PostgreSQL.
> Vertical representation of data is stored in PostgreSQL shared memory.
Thanks for the hard work!
I noticed a couple of things about this that pro
38 matches
Mail list logo
