Re: [Python-Dev] The untuned tunable parameter ARENA_SIZE
On Sat, 3 Jun 2017 21:46:09 -0500 Tim Peters wrote: > > A virtual address space span of a terabyte could hold 1M pools, so > would "only" need a 1M/8 = 128KB bit vector. That's minor compared to > a terabyte (one bit per megabyte). The virtual address space currently supported by x86-64 is 48 bits wide (spanning an address space of 2**48 bytes, that is 256 TB), so you would need a 2**(48-20) bits vector, i.e. 256 Mbits or 32 MB. Note Intel has plans to extend the virtual address space to 2**57 bytes: https://www.phoronix.com/scan.php?page=news_item&px=Intel-5-Level-Paging > The system calls can't be relied on to > return arena-aligned _or_ pool-aligned addresses. Unless using mmap() for pools with a size equal to or an exact divisor of the system's page size, that is ;-) Regards Antoine. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The untuned tunable parameter ARENA_SIZE
[Tim] >> A virtual address space span of a terabyte could hold 1M pools, so >> would "only" need a 1M/8 = 128KB bit vector. That's minor compared to >> a terabyte (one bit per megabyte). [Antoine] > The virtual address space currently supported by x86-64 is 48 bits > wide (spanning an address space of 2**48 bytes, that is 256 TB), so you > would need a 2**(48-20) bits vector, i.e. 256 Mbits or 32 MB. > > Note Intel has plans to extend the virtual address space to 2**57 bytes: > https://www.phoronix.com/scan.php?page=news_item&px=Intel-5-Level-Paging Fill in the blanks ;-) There's only a need for the bit vector to cover the range of addresses _actually returned_ by the system for arenas allocated so far (we're only trying to identify the memory obmalloc _does_ control) . I didn't spell that out, but it was implicit in glosses like "(or if the bit address is out of the vector's domain)". That is, it's up to the bit vector implementation to intelligently represent what's almost always going to be a relatively tiny slice of a theoretically massive address space. So my observation about a terabyte is to be read as applying to a case in which obmalloc has _actually allocated_ arenas spanning a terabyte of address space.(2**14 arenas of 64MB each, and if the system is kind enough to keep them contiguous). Since that's about 100 times more address space than any Python program I've run actually needed, it's a bare minimum for a thought experiment. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The untuned tunable parameter ARENA_SIZE
On Sun, 4 Jun 2017 09:46:10 -0500 Tim Peters wrote: > [Tim] > >> A virtual address space span of a terabyte could hold 1M pools, so > >> would "only" need a 1M/8 = 128KB bit vector. That's minor compared to > >> a terabyte (one bit per megabyte). > > [Antoine] > > The virtual address space currently supported by x86-64 is 48 bits > > wide (spanning an address space of 2**48 bytes, that is 256 TB), so you > > would need a 2**(48-20) bits vector, i.e. 256 Mbits or 32 MB. > > > > Note Intel has plans to extend the virtual address space to 2**57 bytes: > > https://www.phoronix.com/scan.php?page=news_item&px=Intel-5-Level-Paging > > Fill in the blanks ;-) There's only a need for the bit vector to > cover the range of addresses _actually returned_ by the system for > arenas allocated so far (we're only trying to identify the memory > obmalloc _does_ control) . I didn't spell that out, but it was > implicit in glosses like "(or if the bit address is out of the > vector's domain)". That is, it's up to the bit vector implementation > to intelligently represent what's almost always going to be a > relatively tiny slice of a theoretically massive address space. True. That works if the operating system doesn't go too wild in address space randomization, though ;-) Regards Antoine. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The untuned tunable parameter ARENA_SIZE
[Tim] >> ... That is, it's up to the bit vector implementation >> to intelligently represent what's almost always going to be a >> relatively tiny slice of a theoretically massive address space. [Antoine] > True. That works if the operating system doesn't go too wild in > address space randomization, though ;-) I don't know that it's a real problem, but if it is I'm sure you know of efficient ways to implement sparse sets (for example, picture Python's set implementation, but slimmed down to handle just C uint64 members). I was hoping to spur a discussion of much higher level issues. I bet Larry was too ;-) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The untuned tunable parameter ARENA_SIZE
[Larry Hastings ]
> ...
> Yet CPython's memory consumption continues to grow. By the time a current
> "trunk" build of CPython reaches the REPL prompt it's already allocated 16
> arenas.
I'd be surprised if that's true ;-) The first time `new_arena()` is
called, it allocates space for a vector of 16 (INITIAL_ARENA_OBJECTS)
`arena_object` structs. Those are tiny, and hold bookkeeping info for
the actual arenas, none of which are allocated at first.
`new_arena()` asks the system for memory for just one arena (and 15 of
its 16 initial arena_object structs remain unused, awaiting future
calls).
At the time, 16 wasn't picked because it was the minimum Python
needed, but because it was the _most_ the bulk of a variety of casual
Python scripts needed to complete.
It's certainly gotten worse on a 64-bit box. Here on a 64-bit Win10
under Python 3.6.1 upon reaching the interactive prompt in a DOS box
(set envar PYTHONMALLOCSTATS to get this kind of output whenever
`new_arena()` is called):
# arenas allocated total = 14
# arenas reclaimed =5
# arenas highwater mark=9
# arenas allocated current =9
9 arenas * 262144 bytes/arena =2,359,296
# bytes in allocated blocks=2,166,656
# bytes in available blocks= 140,104
0 unused pools * 4096 bytes=0
# bytes lost to pool headers = 27,648
# bytes lost to quantization = 24,888
# bytes lost to arena alignment=0
Total =2,359,296
So at most 9 arenas ("highwater mark") were ever simultaneously allocated..
Change the arena size to 64MB, and most programs would stop with the
first arena allocated ;-)
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 7 and braces { .... } on if
03.06.17 23:30, Barry Warsaw пише:
On Jun 03, 2017, at 07:25 PM, Serhiy Storchaka wrote:
But the latter example continuation lines are intended at the same level as
the following block of code. I propose to make exception for that case and
allow moving an open brace to the start of the next line.
if (type->tp_dictoffset != 0 && base->tp_dictoffset == 0 &&
type->tp_dictoffset == b_size &&
(size_t)t_size == b_size + sizeof(PyObject *))
{
return 0; /* "Forgive" adding a __dict__ only */
}
Agreed!
https://github.com/python/peps/issues/283
https://github.com/python/peps/pull/284
Thank you for opening a PR Barry! But there is some disputation. Barry
and Victor prefer moving a brace on a new line in all multiline
conditional cases. I think that it should be done only when the
condition continuation lines and the following block of the code have
the same indentation (as in the example above), and the following code
is enough readable:
if (PyErr_WarnFormat(PyExc_DeprecationWarning, 1,
"invalid escape sequence '\\%c'",
*first_invalid_escape) < 0) {
Py_DECREF(result);
return NULL;
}
What other core developers think about this?
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
