[issue9942] Allow memory sections to be OS MERGEABLE

Martin v. Löwis added the comment: In order to arrive at some resolution of this issue, I'm answering the original question ("Should Python enable a way for folks to inform the OS of MADV_MERGEABLE memory?"). The discussion has shown that the answer is "no"; there are no pages of memory where

[issue9942] Allow memory sections to be OS MERGEABLE

Changes by s7v7nislands : -- nosy: +s7v7nislands ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.pyt

[issue9942] Allow memory sections to be OS MERGEABLE

Dave Malcolm added the comment: One possible use for this: mark the "str" buffers of PyUnicodeObject instances when demarshalling docstrings from disk; in theory these ought not to change, and can be quite large: the bulk of the memory overhead is stored in a separate allocation from the obje

[issue9942] Allow memory sections to be OS MERGEABLE

Konstantin Svist added the comment: This issue sounds very interesting to me for a somewhat different reason. My problem is that I'm trying to run multiple processes on separate CPUs/cores with os.fork(). In short, the data set is the same (~2GB) and the separate processes do whatever they nee

[issue9942] Allow memory sections to be OS MERGEABLE

Kevin Hunter added the comment: > Why do you read it into a cStringIO? A cStringIO has the same interface > as a file, so you could simply operate on the file directly. In that particular case, because it isn't actually a file. That workflow was my attempt at simplification to illustrate a po

[issue9942] Allow memory sections to be OS MERGEABLE

Antoine Pitrou added the comment: > > Well, first, this would only work for large objects. [...] > > Why do you think you might have such duplication in your workload? > > Some of the projects with which I work involve multiple manipulations > of large datasets. Often, we use Python scripts as

[issue9942] Allow memory sections to be OS MERGEABLE

Kevin Hunter added the comment: > Well, first, this would only work for large objects. [...] > Why do you think you might have such duplication in your workload? Some of the projects with which I work involve multiple manipulations of large datasets. Often, we use Python scripts as "first and

[issue9942] Allow memory sections to be OS MERGEABLE

Antoine Pitrou added the comment: > Answering the question as best I can: I don't know how the reference > counter is implemented in CPython, but if it's just a field in a > struct, then madvise could be sent the memory location starting with > the byte immediately following the reference counte

[issue9942] Allow memory sections to be OS MERGEABLE

Georg Brandl added the comment: > My first thought is "Why is the reference counter stored with the object > itself?" Because if you move the reference counter out of the object, you a) add another indirection and b) depending on how you implement it require a certain amount of memory more p

[issue9942] Allow memory sections to be OS MERGEABLE

Kevin Hunter added the comment: My first thought is "Why is the reference counter stored with the object itself?" I imagine there are very good reasons, however, and this is not an area in which I have much mastery. Answering the question as best I can: I don't know how the reference counter

[issue9942] Allow memory sections to be OS MERGEABLE

Amaury Forgeot d'Arc added the comment: With CPython, even objects that don't change see their reference counter modified quite frequently, just by looking at them. What kind of memory would you mark this way? -- nosy: +amaury.forgeotdarc ___ Python

[issue9942] Allow memory sections to be OS MERGEABLE

New submission from Kevin Hunter : Should Python enable a way for folks to inform the OS of MADV_MERGEABLE memory? I can't speak for other OSs, but Linux added the ability for processes to inform the kernel that they have memory that will likely not change for a while in 2.6.32. This is done