Re: right list for SIGABRT python binary question ?

[dieter]

Karsten Hilbert  writes:

> On Sat, Oct 21, 2017 at 09:31:54AM +0200, dieter wrote:
>> It points to a memory corruption.
> While running valgrind and friends is beyond my capabilitis I
> have run the problem through gdb to at least obtain a stack
> trace to see what gives:

The i386/x64 architecture supports memory access breakpoints
and GDB, too, has support for this. You know the address which
gets corrupted. Thus, the following apporach could have a chance
to succeed:

   Put a "memory write" breakpoint on the address which gets corrupted.
   this should stop the program each time this address is written;
   Check then the backtrace. As the address forms part of the
   address block prologue, it should only be accessed from
   Python's "malloc" (and "free") implementation. Any other access
   indicates bad behaviour.

   Should your program get stopped too often (because the memory
   block is reused too often), you must find a good point in your
   program to activate the memory access breakpoint in a delayed way.
   You could use the Python debugger for this: execute significant
   parts of your program in the Python debugger; switching to
   GDB, check whether the address is already corrupted - if
   so, restart and refine the checks above in the portion of your program
   which caused the corruption. If the part in your program
   is sufficiently small, you can activate the memory access breakpoint.
   This approach may also be feasible, should you use a CPU
   without memory access breakpoints.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Objects with __name__ attribute

[dieter]

"ast"  writes:
> I know two Python's objects which have an intrinsic name, classes and
> functions.
> ...
> Are there others objects with a __name__ attribute
> and what is it used for ?

Some Python objects naturally have a name: functions, classes, modules, ...;
others don't: tuples, lists, integers, ...

If a Python object natureally has a name, there is a good chance that
it is accessible in a standard way: "obj.__name__".

Sometimes, the name is a bit special. An example is the "filename"
(potentially) associated with a "file" like object. In those cases,
the name can be accessed in a slightly different way, e.g. "file.filename".


I recommend that you do not try to enumerate all cases where an object
has a name accessible via "obj.__name__". Instead, start with
your concrete task. Select the objects appropriate for your task.
Use the "buildin" functions "dir" and "help" on a sample object
to find out (in an interactive Python sesssion)
which attributes and methods it supports. If this does not give
enough information, consult the documentation.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Let's talk about debuggers!

[dieter]

Thomas Jollans  writes:
> I just wanted to know what tools everyone used for debugging Python
> applications - scripts / backend / desktop apps / notebooks / whatever.
> Apart from the usual dance with log files and strategically inserted
> print() calls, that is.
>
> Of course we all know and mildly dislike pdb.

I mostly develop backend components for internet applications
based on the web application framework "Zope". There,
the extension "Products.PDBDebugMode" is highly helpfull: it enters
the Python debugger ("pdb"), whenever an uncatched exception is raised
or an error is logged; "pdb" can then be used to examine the state
and check what caused the problem.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Performance of map vs starmap.

[Serhiy Storchaka]

30.10.17 12:10, Kirill Balunov пише:

Sometime ago I asked this question at SO [1], and among the responses
received was paragraph:

  - `zip` re-uses the returned `tuple` if it has a reference count of 1 when
the `__next__` call is made.
  - `map` build a new `tuple` that is passed to the mapped function every
time a `__next__` call is made.

Why can not `map` use the same approach as `zip`?

Also it turns out that a faster solution looks not reasonable, since it
requires additional calculations..

[1] https://stackoverflow.com/questions/46172018/perfomance-
of-map-vs-starmap


Sometime ago I had wrote a sample patch for using cached arg tuples in 
map() and several other functions [1]. It increased the speed in 
microbenchmarks up to 24-38%. But the code was too complex and fragile, 
I didn't want to commit it. Later, after numerous of attempts, this 
resulted in implementing by Victor Stinner the new private "fastcall" 
calling method which allowed to avoid creation a new tuple (and a dict 
for keyword arguments) in many cases. It was added just before releasing 
of 3.6 and was used in few places. In 3.7 it was optimized further. Now 
map() is faster than starmap()+zip().


Python 3.6:

$ ./python -m timeit -s 'from operator import eq; from itertools import 
starmap; seq1 = [1]*1; seq2 = [1]*1' 'list(map(eq, seq1, seq2))'

1000 loops, best of 3: 559 usec per loop

$ ./python -m timeit -s 'from operator import eq; from itertools import 
starmap; seq1 = [1]*1; seq2 = [1]*1' 'list(starmap(eq, zip(seq1, 
seq2)))'

1000 loops, best of 3: 399 usec per loop

Python 3.7:

$ ./python -m timeit -s 'from operator import eq; from itertools import 
starmap; seq1 = [1]*1; seq2 = [1]*1' 'list(map(eq, seq1, seq2))'

1000 loops, best of 5: 338 usec per loop

$ ./python -m timeit -s 'from operator import eq; from itertools import 
starmap; seq1 = [1]*1; seq2 = [1]*1' 'list(starmap(eq, zip(seq1, 
seq2)))'

1000 loops, best of 5: 359 usec per loop

[1] https://bugs.python.org/issue23507

--
https://mail.python.org/mailman/listinfo/python-list

Re: right list for SIGABRT python binary question ?

[Karsten Hilbert]

On Wed, Nov 01, 2017 at 09:21:46AM +0100, dieter wrote:

> >> It points to a memory corruption.
> 
> The i386/x64 architecture supports memory access breakpoints
> and GDB, too, has support for this. You know the address which
> gets corrupted. Thus, the following apporach could have a chance
> to succeed:
> 
>Put a "memory write" breakpoint on the address which gets corrupted.
>this should stop the program each time this address is written;
>Check then the backtrace. As the address forms part of the
>address block prologue, it should only be accessed from
>Python's "malloc" (and "free") implementation. Any other access
>indicates bad behaviour.

I understand. Thank you for the explanation. This may seem
like a dumb question: the actual address that gets corrupted
varies from run to run (it may be the same "place" in the
code but that place gets put at a different address each
run). Since I don't know the address of a given run, how do I
set a break point on that address ?  It seems to me I first
need to map the "place" to its address before the SIGABRT
happens. How do I find out out which "place" needs to be
mapped from the output I already have ?

(the "place" is the memory block you refer to)

Other than that I understand what you are getting at and am
willing to try.

Thanks,
Karsten

>Should your program get stopped too often (because the memory
>block is reused too often), you must find a good point in your
>program to activate the memory access breakpoint in a delayed way.
>You could use the Python debugger for this: execute significant
>parts of your program in the Python debugger; switching to
>GDB, check whether the address is already corrupted - if
>so, restart and refine the checks above in the portion of your program
>which caused the corruption. If the part in your program
>is sufficiently small, you can activate the memory access breakpoint.
>This approach may also be feasible, should you use a CPU
>without memory access breakpoints.


-- 
GPG key ID E4071346 @ eu.pool.sks-keyservers.net
E167 67FD A291 2BEA 73BD  4537 78B9 A9F9 E407 1346
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: matplot plot hangs

[Wolfgang Maier]

On 01.11.2017 00:40, Andrew Z wrote:

hello,
  learning python's plotting by using matplotlib with python35 on fedora 24
x86.

Installed matplotlib into user's directory.
tk, seemed to work -
http://www.tkdocs.com/tutorial/install.html#installlinux - the window shows
up just fine.
but when trying to run the simple plot (
https://matplotlib.org/examples/pylab_examples/simple_plot.html) the script
is hanging on;

plt.plot(t, s)

attempts to
matplotlib.interactive(True) didn't bring anything,



Hi Andrew,

From which environment are you trying to run the example? In the 
terminal, from within some IDE, inside a jupyter notebook?


Are you sure the script "is hanging on plt.plot(t, s)" and not after that?

Best,
Wolfgang

--
https://mail.python.org/mailman/listinfo/python-list

Re: right list for SIGABRT python binary question ?

[Karsten Hilbert]

On Wed, Nov 01, 2017 at 10:27:54AM +0100, Karsten Hilbert wrote:

> > >> It points to a memory corruption.
> > 
> > The i386/x64 architecture supports memory access breakpoints
> > and GDB, too, has support for this. You know the address which
> > gets corrupted. Thus, the following apporach could have a chance
> > to succeed:
> > 
> >Put a "memory write" breakpoint on the address which gets corrupted.
> >this should stop the program each time this address is written;
> >Check then the backtrace. As the address forms part of the
> >address block prologue, it should only be accessed from
> >Python's "malloc" (and "free") implementation. Any other access
> >indicates bad behaviour.
> 
> I understand. Thank you for the explanation. This may seem
> like a dumb question: the actual address that gets corrupted
> varies from run to run (it may be the same "place" in the
> code but that place gets put at a different address each
> run). Since I don't know the address of a given run, how do I
> set a break point on that address ?  It seems to me I first
> need to map the "place" to its address before the SIGABRT
> happens. How do I find out out which "place" needs to be
> mapped from the output I already have ?

Or rather: I need to find out which "place" a given address
refers to, check whether the changing addresses always belong
to the same "place" between runs and _then_ map a "place" to
its address and breakpoint that address on yet another run.

It might seem

gdb> info symbol 

should give me the "place".

Then

gdb> info address 

on another run. Or even just "watch https://mail.python.org/mailman/listinfo/python-list

Re: right list for SIGABRT python binary question ?

[Karsten Hilbert]

On Wed, Nov 01, 2017 at 11:14:08AM +0100, Karsten Hilbert wrote:

> Or rather: I need to find out which "place" a given address
> refers to, check whether the changing addresses always belong
> to the same "place" between runs and _then_ map a "place" to
> its address and breakpoint that address on yet another run.
> 
> It might seem
> 
>   gdb> info symbol 
> 
> should give me the "place".

Given this:

Debug memory block at address p=0x6aab7c: API ''
0 bytes originally requested
The 3 pad bytes at p-3 are not all FORBIDDENBYTE (0xfb):
at p-3: 0x33 *** OUCH
at p-2: 0x47 *** OUCH
at p-1: 0x00 *** OUCH
Because memory is corrupted at the start, the count of bytes 
requested
   may be bogus, and checking the trailing pad bytes may segfault.
The 4 pad bytes at tail=0x6aab7c are not all FORBIDDENBYTE (0xfb):
at tail+0: 0x00 *** OUCH
at tail+1: 0x00 *** OUCH
at tail+2: 0x00 *** OUCH
at tail+3: 0x00 *** OUCH
The block was made by call #0 to debug malloc/realloc.
Fatal Python error: bad ID: Allocated using API '', verified using API 
'o'

Program received signal SIGABRT, Aborted.
0xb7fd9ce9 in __kernel_vsyscall ()
(gdb) info symbol 0x6aab7c
_Py_ZeroStruct in section .data of /usr/bin/python2.7-dbg
(gdb)

my assumption would be that something clobbers 0x6aab7c,
which seems to be in (?) _Py_ZeroStruct in this run. I'll
re-run a few times to make sure the corruption "reliably"
hits _Py_ZeroStruct.

If so, I'll set a memory write breakpoint on _Py_ZeroStruct.

Am I on the right track ?

Thanks,
Karsten

BTW, the backtrace for this run was ...

(gdb) bt
#0  0xb7fd9ce9 in __kernel_vsyscall ()
#1  0xb7d70dd0 in __libc_signal_restore_set (set=0xbfffee90) at 
../sysdeps/unix/sysv/linux/nptl-signals.h:79
#2  __GI_raise (sig=6) at ../sysdeps/unix/sysv/linux/raise.c:48
#3  0xb7d72297 in __GI_abort () at abort.c:89
#4  0x0055fb74 in Py_FatalError (msg=0xb13c "bad ID: Allocated 
using API '\037', verified using API 'o'") at ../Python/pythonrun.c:1700
#5  0x00499adb in _PyObject_DebugCheckAddressApi (api=111 'o', 
p=0x6aab7c <_Py_ZeroStruct>) at ../Objects/obmalloc.c:1640
#6  0x004997a5 in _PyObject_DebugFreeApi (api=111 'o', p=0x6aab7c 
<_Py_ZeroStruct>) at ../Objects/obmalloc.c:1527
#7  0x0049964f in _PyObject_DebugFree (p=0x6aab7c <_Py_ZeroStruct>) at 
../Objects/obmalloc.c:1471
#8  0x00471043 in int_dealloc (v=0x6aab7c <_Py_ZeroStruct>) at 
../Objects/intobject.c:139

... so I could've known without "info symbol" :-)

#9  0x00497bee in _Py_Dealloc (op=False) at ../Objects/object.c:2262
#10 0x004885d7 in insertdict_by_entry (mp=0xb7fc5674, 
key='dont_write_bytecode', hash=591857026, ep=0x7c5c08, value=None) at 
../Objects/dictobject.c:519
#11 0x00488857 in insertdict (mp=0xb7fc5674, key='dont_write_bytecode', 
hash=591857026, value=None) at ../Objects/dictobject.c:556
#12 0x0048910f in dict_set_item_by_hash_or_entry (
op={
'setrecursionlimit': None,
'dont_write_bytecode': None,
'getfilesystemencoding': ,
'long_info': ,
'path_importer_cache': None,
'stdout': ,
'getprofile': ,
'__stdin__': ,
'version_info': ,
'exc_clear': , 'gettotalrefcount': 
, 'getrefcount': , 'byteorder': 'little', '_clear_type_cache': None, 'excepthook': 
, 'subversion': ('CPython', '', ''), 
'_multiarch': None, 'exc_type': None, 'ps1': None, '__excepthook__': , 'executable': '/usr/bin/python2.7-dbg', 'float_info': 
None, 'copyright': 'Copyright (c) 2001-2017 Python Software Foundation.\nAll 
Rights Reserved.\n\nCopyright (c) 2000 BeOpen.com.\nAll Rights 
Reserved.\n\nCopyright (c) 1995-2001 Corporation for Nation...(truncated), 
key='dont_write_bytecode', hash=591857026, ep=0x0, value=None
) at ../Objects/dictobject.c:795
#13 0x00489285 in PyDict_SetItem (
op={'setrecursionlimit': None, 'dont_write_bytecode': None, 
'getfilesystemencoding': , 
'long_info': , 'path_importer_cache': None, 'stdout': , 'getprofile': , '__stdin__': , 'version_info': , 
'exc_clear': , 'gettotalrefcount': , 'getrefcount': , 
'byteorder': 'little', '_clear_type_cache': None, 'excepthook': , 'subversion': ('CPython', '', ''), '_multiarch': None, 
'exc_type': None, 'ps1': None, '__excepthook__': , 'executable': '/usr/bin/python2.7-dbg', 'float_info': None, 
'copyright': 'Copyright (c) 2001-2017 Python Software Foundation.\nAll Rights 
Reserved.\n\nCopyright (c) 2000 BeOpen.com.\nAll Rights Reserved.\n\nCopyright 
(c) 1995-2001 Corporation for Nation...(truncated), key='dont_write_bytecode', 
value=None) at ../Objects/dic

Re: right list for SIGABRT python binary question ?

[Karsten Hilbert]

> my assumption would be that something clobbers 0x6aab7c,
> which seems to be in (?) _Py_ZeroStruct in this run. I'll
> re-run a few times to make sure the corruption "reliably"
> hits _Py_ZeroStruct.
> 
> If so, I'll set a memory write breakpoint on _Py_ZeroStruct.

Interestingly, on subsequent runs, it seems to hit the same
address, 0x6aab7c, belonging to the same symbol, _Py_ZeroStruct.

This is what happens:

(gdb) watch *0x6aab7c
Hardware watchpoint 1: *0x6aab7c
(gdb) run
Starting program: /usr/bin/python2.7-dbg ./bootstrap_gm_db_system.py 
--log-file=./bootstrap-latest.log --conf-file=bootstrap-latest.conf --
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".

Hardware watchpoint 1: *0x6aab7c

Old value = 0
New value = -1208182272
_Py_AddToAllObjects (op=False, force=0) at ../Objects/object.c:70
70  ../Objects/object.c: Datei oder Verzeichnis nicht gefunden.
(gdb)

which means I'll probably have to apply the delayed
breakpoint setting strategy, or else it is just some initial
relocation at startup. Let's see what "cont" brings. The next
hit after the Python script has run until just before it
usually aborts:

Hardware watchpoint 1: *0x6aab7c

Old value = -1208182272
New value = 0
_Py_ForgetReference (op=False) at ../Objects/object.c:2255
2255in ../Objects/object.c
(gdb)

The backtrace at this point says:

(gdb) bt
#0  _Py_ForgetReference (op=False) at ../Objects/object.c:2255
#1  0x00497be0 in _Py_Dealloc (op=False) at ../Objects/object.c:2261
#2  0x004885d7 in insertdict_by_entry (mp=0xb7fc5674, 
key='dont_write_bytecode', hash=591857026, ep=0x7c5c08, value=None) at 
../Objects/dictobject.c:519
#3  0x00488857 in insertdict (mp=0xb7fc5674, key='dont_write_bytecode', 
hash=591857026, value=None) at ../Objects/dictobject.c:556
#4  0x0048910f in dict_set_item_by_hash_or_entry (
op={'setrecursionlimit': None, 'dont_write_bytecode': None, 
'getfilesystemencoding': , 
'long_info': , 'path_importer_cache': None, 'stdout': , 'getprofile': , '__stdin__': , 'version_info': , 
'exc_clear': , 'gettotalrefcount': , 'getrefcount': , 
'byteorder': 'little', '_clear_type_cache': None, 'excepthook': , 'subversion': ('CPython', '', ''), '_multiarch': None, 
'exc_type': None, 'ps1': None, '__excepthook__': , 'executable': '/usr/bin/python2.7-dbg', 'float_info': None, 
'copyright': 'Copyright (c) 2001-2017 Python Software Foundation.\nAll Rights 
Reserved.\n\nCopyright (c) 2000 BeOpen.com.\nAll Rights Reserved.\n\nCopyright 
(c) 1995-2001 Corporation for Nation...(truncated), key='dont_write_bytecode', 
hash=591857026, ep=0x0, value=None) at ../Objects/dictobject.c:795
#5  0x00489285 in PyDict_SetItem (
op={'setrecursionlimit': None, 'dont_write_bytecode': None, 
'getfilesystemencoding': , 
'long_info': , 'path_importer_cache': None, 'stdout': , 'getprofile': , '__stdin__': , 'version_info': , 
'exc_clear': , 'gettotalrefcount': , 'getrefcount': , 
'byteorder': 'little', '_clear_type_cache': None, 'excepthook': , 'subversion': ('CPython', '', ''), '_multiarch': None, 
'exc_type': None, 'ps1': None, '__excepthook__': , 'executable': '/usr/bin/python2.7-dbg', 'float_info': None, 
'copyright': 'Copyright (c) 2001-2017 Python Software Foundation.\nAll Rights 
Reserved.\n\nCopyright (c) 2000 BeOpen.com.\nAll Rights Reserved.\n\nCopyright 
(c) 1995-2001 Corporation for Nation...(truncated), key='dont_write_bytecode', 
value=None) at ../Objects/dictobject.c:848
#6  0x0049281f in _PyModule_Clear (m=) at 
../Objects/moduleobject.c:139
#7  0x0054a3ec in PyImport_Cleanup () at ../Python/import.c:540
#8  0x0055c53c in Py_Finalize () at ../Python/pythonrun.c:458
#9  0x0055fe9c in Py_Exit (sts=1) at ../Python/pythonrun.c:1783
#10 0x0055e0fc in handle_system_exit () at ../Python/pythonrun.c:1151
#11 0x0055e152 in PyErr_PrintEx (set_sys_last_vars=1) at 
../Python/pythonrun.c:1161
#12 0x0055dd5b in PyErr_Print () at ../Python/pythonrun.c:1064
#13 0x0055d61f in PyRun_SimpleFileExFlags (fp=0x7ee010, 
filename=0xb7e6 "./bootstrap_gm_db_system.py", closeit=1, flags=0xb4f4) 
at ../Python/pythonrun.c:952
#14 0x0055cc4e in PyRun_AnyFileExFlags (fp=0x7ee010, 
filename=0xb7e6 "./bootstrap_gm_db_system.py", closeit=1, flags=0xb4f4) 
at ../Python/pythonrun.c:752
#15 0x00577cb0 in Py_Main (argc=5, argv=0xb684) at 
../Modules/main.c:645
#16 0x004259c8 in main (argc=5, argv=0xb684) at 
../Modules/python.c:20

And continuing hits the SIGABRT right away:

(gdb) cont
Continuing.
Debug memory block at address p=0x6aab7c: API ''
0 bytes originally requested
The 3 pad bytes at p-3 are not all FORBIDDENBY

Re: right list for SIGABRT python binary question ?

[Karsten Hilbert]

On Wed, Nov 01, 2017 at 12:40:56PM +0100, Karsten Hilbert wrote:

> Interestingly, on subsequent runs, it seems to hit the same
> address, 0x6aab7c, belonging to the same symbol, _Py_ZeroStruct.
> 
> This is what happens:
> 
>   (gdb) watch *0x6aab7c
>   Hardware watchpoint 1: *0x6aab7c
>   (gdb) run
>   Starting program: /usr/bin/python2.7-dbg ./bootstrap_gm_db_system.py 
> --log-file=./bootstrap-latest.log --conf-file=bootstrap-latest.conf --
>   [Thread debugging using libthread_db enabled]
>   Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".
> 
>   Hardware watchpoint 1: *0x6aab7c
> 
>   Old value = 0
>   New value = -1208182272
>   _Py_AddToAllObjects (op=False, force=0) at ../Objects/object.c:70
>   70  ../Objects/object.c: Datei oder Verzeichnis nicht gefunden.
>   (gdb)

I forgot the backtrace from this point:

(gdb) bt
#0  _Py_AddToAllObjects (op=False, force=0) at ../Objects/object.c:70
#1  0x0051e052 in _PyBuiltin_Init () at ../Python/bltinmodule.c:2708
#2  0x0055bebf in Py_InitializeEx (install_sigs=1) at 
../Python/pythonrun.c:233
#3  0x0055c4dd in Py_Initialize () at ../Python/pythonrun.c:381
#4  0x00577805 in Py_Main (argc=5, argv=0xb684) at 
../Modules/main.c:551
#5  0x004259c8 in main (argc=5, argv=0xb684) at 
../Modules/python.c:20

Karsten
-- 
GPG key ID E4071346 @ eu.pool.sks-keyservers.net
E167 67FD A291 2BEA 73BD  4537 78B9 A9F9 E407 1346
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: The syntax of replacement fields in format strings

[Ned Batchelder]

On 10/31/17 12:45 PM, John Smith wrote:

If we keep the current implementation as is, perhaps the documentation
should at least be altered ?


John, it looks like you are responding to a Python-Dev message, but on 
this list, somehow...


--Ned.
--
https://mail.python.org/mailman/listinfo/python-list

Re: matplot plot hangs

[Andrew Z]

Wolfgang,
 I tried to ran from ide with no rwsults, so now im trying from a terminal
in xwindow.
The .plot is the last line in the script and it does hang trying to execute
it.


On Nov 1, 2017 05:44, "Wolfgang Maier" <
wolfgang.ma...@biologie.uni-freiburg.de> wrote:

On 01.11.2017 00:40, Andrew Z wrote:

> hello,
>   learning python's plotting by using matplotlib with python35 on fedora 24
> x86.
>
> Installed matplotlib into user's directory.
> tk, seemed to work -
> http://www.tkdocs.com/tutorial/install.html#installlinux - the window
> shows
> up just fine.
> but when trying to run the simple plot (
> https://matplotlib.org/examples/pylab_examples/simple_plot.html) the
> script
> is hanging on;
>
> plt.plot(t, s)
>
> attempts to
> matplotlib.interactive(True) didn't bring anything,
>
>
Hi Andrew,

>From which environment are you trying to run the example? In the terminal,
from within some IDE, inside a jupyter notebook?

Are you sure the script "is hanging on plt.plot(t, s)" and not after that?

Best,
Wolfgang

-- 
https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: matplot plot hangs

[Vlastimil Brom]

2017-11-01 13:49 GMT+01:00 Andrew Z :
> Wolfgang,
>  I tried to ran from ide with no rwsults, so now im trying from a terminal
> in xwindow.
> The .plot is the last line in the script and it does hang trying to execute
> it.
>
>
> On Nov 1, 2017 05:44, "Wolfgang Maier" <
> wolfgang.ma...@biologie.uni-freiburg.de> wrote:
>
> On 01.11.2017 00:40, Andrew Z wrote:
>
>> hello,
>>   learning python's plotting by using matplotlib with python35 on fedora 24
>> x86.
>>
>> Installed matplotlib into user's directory.
>> tk, seemed to work -
>> http://www.tkdocs.com/tutorial/install.html#installlinux - the window
>> shows
>> up just fine.
>> but when trying to run the simple plot (
>> https://matplotlib.org/examples/pylab_examples/simple_plot.html) the
>> script
>> is hanging on;
>>
>> plt.plot(t, s)
>>
>> attempts to
>> matplotlib.interactive(True) didn't bring anything,
>>
>>
> Hi Andrew,
>
> From which environment are you trying to run the example? In the terminal,
> from within some IDE, inside a jupyter notebook?
>
> Are you sure the script "is hanging on plt.plot(t, s)" and not after that?
>
> Best,
> Wolfgang
>
> --
Hi,
sorry if it is too trivial, just to make sure, do you have a call to
"show()" the resulting plot in the code?

An elementary plotting code might be e.g.:

import matplotlib.pyplot as plt
plt.plot([1,2,4,1])
plt.show()

Does this work in your environment?

It was not quite clear, what do you plan with interactive drawing, or
whether you are using e.g. plt.interactive(True) already - this might
be a problem as there could be collisions or the plot window is closed
after the standalone script finishes.

hth,
 vbr
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: right list for SIGABRT python binary question ?

[Karsten Hilbert]

On Wed, Nov 01, 2017 at 11:58:53AM -0400, Dennis Lee Bieber wrote:

> >I understand. Thank you for the explanation. This may seem
> >like a dumb question: the actual address that gets corrupted
> >varies from run to run (it may be the same "place" in the
> 
>   Since the process virtual memory space should be the same on each run
> -- even heap allocations should be deterministic, UNLESS the program is
> doing things with external non-deterministic data (the time of day, random
> numbers not initialized with a repeatable seed, dynamic web pages, multiple
> threads that are non-deterministic due to I/O delays, etc.),

Mine doesn't to my knowledge hence I can expect "fixed"
addresses -- which is confirmed by what I found during
consecutive runs. Thanks for the affirmative explanation.

> the only other source for varying addresses would be
> OS-level shared libraries that are getting mapped at
> different locations due to changes in the order of other
> programs running.

IOW I should take care I haven't run any relevant amount of
Python code before debugging this problem, it seems ?

> The physical memory may vary due to paging fragmentation,
> but should still be the same virtual addresses.

I see.

Karsten
-- 
GPG key ID E4071346 @ eu.pool.sks-keyservers.net
E167 67FD A291 2BEA 73BD  4537 78B9 A9F9 E407 1346
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: right list for SIGABRT python binary question ?

[Grant Edwards]

On 2017-11-01, Dennis Lee Bieber  wrote:
> On Wed, 1 Nov 2017 10:27:54 +0100, Karsten Hilbert
> declaimed the following:
>
>
>>
>>I understand. Thank you for the explanation. This may seem
>>like a dumb question: the actual address that gets corrupted
>>varies from run to run (it may be the same "place" in the
>
>   Since the process virtual memory space should be the same on each run

Not with modern OS kernels:

https://en.wikipedia.org/wiki/Address_space_layout_randomization

-- 
Grant Edwards   grant.b.edwardsYow! ... My pants just went
  at   on a wild rampage through a
  gmail.comLong Island Bowling Alley!!

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Thread safety issue (I think) with defaultdict

[Israel Brewster]

Let me rephrase the question, see if I can simplify it. I need to be able to 
access a defaultdict from two different threads - one thread that responds to 
user requests which will populate the dictionary in response to a user request, 
and a second thread that will keep the dictionary updated as new data comes in. 
The value of the dictionary will be a timestamp, with the default value being 
datetime.min, provided by a lambda:

lambda: datetime.min

At the moment my code is behaving as though each thread has a *separate* 
defaultdict, even though debugging shows the same addresses - the background 
update thread never sees the data populated into the defaultdict by the main 
thread. I was thinking race conditions or the like might make it so one 
particular loop of the background thread occurs before the main thread, but 
even so subsequent loops should pick up on the changes made by the main thread.

How can I *properly* share a dictionary like object between two threads, with 
both threads seeing the updates made by the other?
---
Israel Brewster
Systems Analyst II
Ravn Alaska
5245 Airport Industrial Rd
Fairbanks, AK 99709
(907) 450-7293
---




> On Oct 31, 2017, at 9:38 AM, Israel Brewster  wrote:
> 
> A question that has arisen before (for example, here: 
> https://mail.python.org/pipermail/python-list/2010-January/565497.html 
> ) is 
> the question of "is defaultdict thread safe", with the answer generally being 
> a conditional "yes", with the condition being what is used as the default 
> value: apparently default values of python types, such as list, are thread 
> safe, whereas more complicated constructs, such as lambdas, make it not 
> thread safe. In my situation, I'm using a lambda, specifically:
> 
> lambda: datetime.min
> 
> So presumably *not* thread safe.
> 
> My goal is to have a dictionary of aircraft and when they were last "seen", 
> with datetime.min being effectively "never". When a data point comes in for a 
> given aircraft, the data point will be compared with the value in the 
> defaultdict for that aircraft, and if the timestamp on that data point is 
> newer than what is in the defaultdict, the defaultdict will get updated with 
> the value from the datapoint (not necessarily current timestamp, but rather 
> the value from the datapoint). Note that data points do not necessarily 
> arrive in chronological order (for various reasons not applicable here, it's 
> just the way it is), thus the need for the comparison.
> 
> When the program first starts up, two things happen:
> 
> 1) a thread is started that watches for incoming data points and updates the 
> dictionary as per above, and
> 2) the dictionary should get an initial population (in the main thread) from 
> hard storage.
> 
> The behavior I'm seeing, however, is that when step 2 happens (which 
> generally happens before the thread gets any updates), the dictionary gets 
> populated with 56 entries, as expected. However, none of those entries are 
> visible when the thread runs. It's as though the thread is getting a separate 
> copy of the dictionary, although debugging says that is not the case - 
> printing the variable from each location shows the same address for the 
> object.
> 
> So my questions are:
> 
> 1) Is this what it means to NOT be thread safe? I was thinking of race 
> conditions where individual values may get updated wrong, but this apparently 
> is overwriting the entire dictionary.
> 2) How can I fix this?
> 
> Note: I really don't care if the "initial" update happens after the thread 
> receives a data point or two, and therefore overwrites one or two values. I 
> just need the dictionary to be fully populated at some point early in 
> execution. In usage, the dictionary is used to see of an aircraft has been 
> seen "recently", so if the most recent datapoint gets overwritten with a 
> slightly older one from disk storage, that's fine - it's just if it's still 
> showing datetime.min because we haven't gotten in any datapoint since we 
> launched the program, even though we have "recent" data in disk storage thats 
> a problem. So I don't care about the obvious race condition between the two 
> operations, just that the end result is a populated dictionary. Note also 
> that as datapoint come in, they are being written to disk, so the disk 
> storage doesn't lag significantly anyway.
> 
> The framework of my code is below:
> 
> File: watcher.py
> 
> last_points = defaultdict(lambda:datetime.min)
> 
> # This function is launched as a thread using the threading module when the 
> first client connects
> def watch():
>   while true:
>   
>   pointtime= 
>   if last_points[] < pointtime:
>   
>   last_points[]=pointtime
>   #DEBUGGING
>

Re: Thread safety issue (I think) with defaultdict

[Israel Brewster]

On Nov 1, 2017, at 9:04 AM, Israel Brewster  wrote:
> 
> Let me rephrase the question, see if I can simplify it. I need to be able to 
> access a defaultdict from two different threads - one thread that responds to 
> user requests which will populate the dictionary in response to a user 
> request, and a second thread that will keep the dictionary updated as new 
> data comes in. The value of the dictionary will be a timestamp, with the 
> default value being datetime.min, provided by a lambda:
> 
> lambda: datetime.min
> 
> At the moment my code is behaving as though each thread has a *separate* 
> defaultdict, even though debugging shows the same addresses - the background 
> update thread never sees the data populated into the defaultdict by the main 
> thread. I was thinking race conditions or the like might make it so one 
> particular loop of the background thread occurs before the main thread, but 
> even so subsequent loops should pick up on the changes made by the main 
> thread.
> 
> How can I *properly* share a dictionary like object between two threads, with 
> both threads seeing the updates made by the other?

For what it's worth, if I insert a print statement in both threads (which I am 
calling "Get AC", since that is the function being called in the first thread, 
and "update", since that is the purpose of the second thread), I get the 
following output:

Length at get AC:  54 ID: 4524152200  Time: 2017-11-01 09:41:24.474788
Length At update:  1 ID: 4524152200  Time: 2017-11-01 09:41:24.784399
Length At update:  2 ID: 4524152200  Time: 2017-11-01 09:41:25.228853
Length At update:  3 ID: 4524152200  Time: 2017-11-01 09:41:25.530434
Length At update:  4 ID: 4524152200  Time: 2017-11-01 09:41:25.532073
Length At update:  5 ID: 4524152200  Time: 2017-11-01 09:41:25.682161
Length At update:  6 ID: 4524152200  Time: 2017-11-01 09:41:26.807127
...

So the object ID hasn't changed as I would expect it to if, in fact, we have 
created a separate object for the thread. And the first call that populates it 
with 54 items happens "well" before the first update call - a full .3 seconds, 
which I would think would be an eternity is code terms. So it doesn't even look 
like it's a race condition causing the issue.

It seems to me this *has* to be something to do with the use of threads, but 
I'm baffled as to what.

> ---
> Israel Brewster
> Systems Analyst II
> Ravn Alaska
> 5245 Airport Industrial Rd
> Fairbanks, AK 99709
> (907) 450-7293
> ---
> 
> 
> 
> 
>> On Oct 31, 2017, at 9:38 AM, Israel Brewster  wrote:
>> 
>> A question that has arisen before (for example, here: 
>> https://mail.python.org/pipermail/python-list/2010-January/565497.html 
>> ) is 
>> the question of "is defaultdict thread safe", with the answer generally 
>> being a conditional "yes", with the condition being what is used as the 
>> default value: apparently default values of python types, such as list, are 
>> thread safe, whereas more complicated constructs, such as lambdas, make it 
>> not thread safe. In my situation, I'm using a lambda, specifically:
>> 
>> lambda: datetime.min
>> 
>> So presumably *not* thread safe.
>> 
>> My goal is to have a dictionary of aircraft and when they were last "seen", 
>> with datetime.min being effectively "never". When a data point comes in for 
>> a given aircraft, the data point will be compared with the value in the 
>> defaultdict for that aircraft, and if the timestamp on that data point is 
>> newer than what is in the defaultdict, the defaultdict will get updated with 
>> the value from the datapoint (not necessarily current timestamp, but rather 
>> the value from the datapoint). Note that data points do not necessarily 
>> arrive in chronological order (for various reasons not applicable here, it's 
>> just the way it is), thus the need for the comparison.
>> 
>> When the program first starts up, two things happen:
>> 
>> 1) a thread is started that watches for incoming data points and updates the 
>> dictionary as per above, and
>> 2) the dictionary should get an initial population (in the main thread) from 
>> hard storage.
>> 
>> The behavior I'm seeing, however, is that when step 2 happens (which 
>> generally happens before the thread gets any updates), the dictionary gets 
>> populated with 56 entries, as expected. However, none of those entries are 
>> visible when the thread runs. It's as though the thread is getting a 
>> separate copy of the dictionary, although debugging says that is not the 
>> case - printing the variable from each location shows the same address for 
>> the object.
>> 
>> So my questions are:
>> 
>> 1) Is this what it means to NOT be thread safe? I was thinking of race 
>> conditions where individual values may get updated wrong, but this 
>> apparently is overwriting the entire dictionary.
>> 

Re: Thread safety issue (I think) with defaultdict

[Ian Kelly]

On Tue, Oct 31, 2017 at 11:38 AM, Israel Brewster  wrote:
> A question that has arisen before (for example, here: 
> https://mail.python.org/pipermail/python-list/2010-January/565497.html 
> ) is 
> the question of "is defaultdict thread safe", with the answer generally being 
> a conditional "yes", with the condition being what is used as the default 
> value: apparently default values of python types, such as list, are thread 
> safe,

I would not rely on this. It might be true for current versions of
CPython, but I don't think there's any general guarantee and you could
run into trouble with other implementations.

> whereas more complicated constructs, such as lambdas, make it not thread 
> safe. In my situation, I'm using a lambda, specifically:
>
> lambda: datetime.min
>
> So presumably *not* thread safe.
>
> My goal is to have a dictionary of aircraft and when they were last "seen", 
> with datetime.min being effectively "never". When a data point comes in for a 
> given aircraft, the data point will be compared with the value in the 
> defaultdict for that aircraft, and if the timestamp on that data point is 
> newer than what is in the defaultdict, the defaultdict will get updated with 
> the value from the datapoint (not necessarily current timestamp, but rather 
> the value from the datapoint). Note that data points do not necessarily 
> arrive in chronological order (for various reasons not applicable here, it's 
> just the way it is), thus the need for the comparison.

Since you're going to immediately replace the default value with an
actual value, it's not clear to me what the purpose of using a
defaultdict is here. You could use a regular dict and just check if
the key is present, perhaps with the additional argument to .get() to
return a default value.

Individual lookups and updates of ordinary dicts are atomic (at least
in CPython). A lookup followed by an update is not, and this would be
true for defaultdict as well.

> When the program first starts up, two things happen:
>
> 1) a thread is started that watches for incoming data points and updates the 
> dictionary as per above, and
> 2) the dictionary should get an initial population (in the main thread) from 
> hard storage.
>
> The behavior I'm seeing, however, is that when step 2 happens (which 
> generally happens before the thread gets any updates), the dictionary gets 
> populated with 56 entries, as expected. However, none of those entries are 
> visible when the thread runs. It's as though the thread is getting a separate 
> copy of the dictionary, although debugging says that is not the case - 
> printing the variable from each location shows the same address for the 
> object.
>
> So my questions are:
>
> 1) Is this what it means to NOT be thread safe? I was thinking of race 
> conditions where individual values may get updated wrong, but this apparently 
> is overwriting the entire dictionary.

No, a thread-safety issue would be something like this:

account[user] = account[user] + 1

where the value of account[user] could potentially change between the
time it is looked up and the time it is set again. That said it's not
obvious to me what your problem actually is.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Code Snippets

[Neil Cerutti]

On 2017-11-01, Stefan Ram  wrote:
>   I started to collect some code snippets:
>
>   Sleep one second
>
> __import__( "time" ).sleep( 1 )
>
>   What I'm supposed to do instead, I guess, is:
>
>   Sleep one second
>
> import time
> ...
> time.sleep( 1 )
>
>   Get current directory
>
> import os
> ...
> os.getcwd()
>
>   Get a random number
>
> import random
> ...
> random.random()
>
>   Now, the user has to cut the import, paste it to the top
>   of his code, then go back to the list of snippets, find
>   the same snippet again, copy the expression, go to his code,
>   then find the point where he wanted to insert the snippet again,
>   and finally insert the snippet. And still there now is a
>   risk of name collisions. So, it seems to me that __import__
>   is just so much better!

You can import wherever you like--only good style requires you to
put them at the top of your file.

Moreover, snippets could be a library, with each snippet a
function, with the import inside the function. That would keep
the module name out of your global namespace.

-- 
Neil Cerutti

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Code Snippets

[Ned Batchelder]

On 11/1/17 1:25 PM, Stefan Ram wrote:

   I started to collect some code snippets:

   Sleep one second

__import__( "time" ).sleep( 1 )

   Get current directory

__import__( "os" ).getcwd()

   Get a random number

__import__( "random" ).random()

   And so on. You get the idea.

   However, reportedly, all those snippets are anti-patterns
   because they use »__import__«.

   But what I like about them: You just paste them in where
   you want them to be, and your done.

   What I'm supposed to do instead, I guess, is:

   Sleep one second

import time
...
time.sleep( 1 )

   Get current directory

import os
...
os.getcwd()

   Get a random number

import random
...
random.random()

   Now, the user has to cut the import, paste it to the top
   of his code, then go back to the list of snippets, find
   the same snippet again, copy the expression, go to his code,
   then find the point where he wanted to insert the snippet again,
   and finally insert the snippet. And still there now is a
   risk of name collisions. So, it seems to me that __import__
   is just so much better!



You should not optimize for the shortest time to paste a line of code.  
You should take time and care writing your code, so that it reads best 
and runs best.  If you needed another os function, would you have two 
__import__("os") in your code? Ugh.


--Ned.
--
https://mail.python.org/mailman/listinfo/python-list

Re: Report on non-breaking spaces in posts

[Ned Batchelder]

On 10/31/17 1:23 PM, Stefan Ram wrote:

   Ok, here's a report on me seing non-breaking spaces in
   posts in this NG. I have written this report so that you
   can see that it's not my newsreader that is converting
   something, because there is no newsreader involved.


You've worded this as if 1) non-breaking spaces are a problem, 2) you've 
reported them before, and 3) someone has claimed that it is your 
newsreader at fault.  Have I missed a previous discussion?


--Ned.
--
https://mail.python.org/mailman/listinfo/python-list

Re: Thread safety issue (I think) with defaultdict

[Israel Brewster]

On Nov 1, 2017, at 9:58 AM, Ian Kelly  wrote:
> 
> On Tue, Oct 31, 2017 at 11:38 AM, Israel Brewster  
> wrote:
>> A question that has arisen before (for example, here: 
>> https://mail.python.org/pipermail/python-list/2010-January/565497.html 
>> ) is 
>> the question of "is defaultdict thread safe", with the answer generally 
>> being a conditional "yes", with the condition being what is used as the 
>> default value: apparently default values of python types, such as list, are 
>> thread safe,
> 
> I would not rely on this. It might be true for current versions of
> CPython, but I don't think there's any general guarantee and you could
> run into trouble with other implementations.

Right, completely agreed. Kinda feels "dirty" to rely on things like this to me.

> 
>> [...]
> 
> [...] You could use a regular dict and just check if
> the key is present, perhaps with the additional argument to .get() to
> return a default value.

True. Using defaultdict is simply saves having to stick the same default in 
every call to get(). DRY principal and all. That said, see below - I don't 
think the defaultdict is the issue.

> 
> Individual lookups and updates of ordinary dicts are atomic (at least
> in CPython). A lookup followed by an update is not, and this would be
> true for defaultdict as well.
> 
>> [...]
>> 1) Is this what it means to NOT be thread safe? I was thinking of race 
>> conditions where individual values may get updated wrong, but this 
>> apparently is overwriting the entire dictionary.
> 
> No, a thread-safety issue would be something like this:
> 
>account[user] = account[user] + 1
> 
> where the value of account[user] could potentially change between the
> time it is looked up and the time it is set again.

That's what I thought - changing values/different values from expected, not 
missing values.

All that said, I just had a bit of an epiphany: the main thread is actually a 
Flask app, running through UWSGI with multiple *processes*, and using the 
flask-uwsgi-websocket plugin, which further uses greenlets. So what I was 
thinking was simply a separate thread was, in reality, a completely separate 
*process*. I'm sure that makes a difference. So what's actually happening here 
is the following:

1) the main python process starts, which initializes the dictionary (since it 
is at a global level)
2) uwsgi launches off a bunch of child worker processes (10 to be exact, each 
of which is set up with 10 gevent threads)
3a) a client connects (web socket connection to be exact). This connection is 
handled by an arbitrary worker, and an arbitrary green thread within that 
worker, based on UWSGI algorithms.
3b) This connection triggers launching of a *true* thread (using the python 
threading library) which, presumably, is now a child thread of that arbitrary 
uwsgi worker. <== BAD THING, I would think
4) The client makes a request for the list, which is handled by a DIFFERENT 
(presumably) arbitrary worker process and green thread.

So the end result is that the thread that "updates" the dictionary, and the 
thread that initially *populates* the dictionary are actually running in 
different processes. In fact, any given request could be in yet another 
process, which would seem to indicate that all bets are off as to what data is 
seen.

Now that I've thought through what is really happening, I think I need to 
re-architect things a bit here. For one thing, the update thread should be 
launched from the main process, not an arbitrary UWSGI worker. I had launched 
it from the client connection because there is no point in having it running if 
there is no one connected, but I may need to launch it from the __init__.py 
file instead. For another thing, since this dictionary will need to be accessed 
from arbitrary worker processes, I'm thinking I may need to move it to some 
sort of external storage, such as a redis database. Oy, I made my life 
complicated :-)

> That said it's not
> obvious to me what your problem actually is.
> -- 
> https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

String changing size on failure?

[Ned Batchelder]

From David Beazley (https://twitter.com/dabeaz/status/925787482515533830):

>>> a = 'n'
>>> b = 'ñ'
>>> sys.getsizeof(a)
   50
>>> sys.getsizeof(b)
   74
>>> float(b)
   Traceback (most recent call last):
  File "", line 1, in 
   ValueError: could not convert string to float: 'ñ'
>>> sys.getsizeof(b)
   77

Huh?

--Ned.

--
https://mail.python.org/mailman/listinfo/python-list

Re: String changing size on failure?

[Skip Montanaro]

Leave it to DB to ask the tough questions other people won't. :-)

Skip


On Wed, Nov 1, 2017 at 2:26 PM, Ned Batchelder  wrote:
> From David Beazley (https://twitter.com/dabeaz/status/925787482515533830):
>
> >>> a = 'n'
> >>> b = 'ñ'
> >>> sys.getsizeof(a)
>50
> >>> sys.getsizeof(b)
>74
> >>> float(b)
>Traceback (most recent call last):
>   File "", line 1, in 
>ValueError: could not convert string to float: 'ñ'
> >>> sys.getsizeof(b)
>77
>
> Huh?
>
> --Ned.
>
> --
> https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: String changing size on failure?

[Chris Angelico]

On Thu, Nov 2, 2017 at 6:26 AM, Ned Batchelder  wrote:
> From David Beazley (https://twitter.com/dabeaz/status/925787482515533830):
>
> >>> a = 'n'
> >>> b = 'ñ'
> >>> sys.getsizeof(a)
>50
> >>> sys.getsizeof(b)
>74
> >>> float(b)
>Traceback (most recent call last):
>   File "", line 1, in 
>ValueError: could not convert string to float: 'ñ'
> >>> sys.getsizeof(b)
>77
>
> Huh?

There are optional parts to the Python string object that don't get
memory allocated for them until they're used. Apparently calling
float() on a string triggers one of those allocations (possibly the
UTF-8 representation?). You'd have to mess around with ctypes to see
exactly what's going on.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Code Snippets

[Wolfgang Maier]

On 01.11.2017 18:25, Stefan Ram wrote:

   I started to collect some code snippets:

   Sleep one second

__import__( "time" ).sleep( 1 )

   Get current directory

__import__( "os" ).getcwd()

   Get a random number

__import__( "random" ).random()

   And so on. You get the idea.

   However, reportedly, all those snippets are anti-patterns
   because they use »__import__«.

   But what I like about them: You just paste them in where
   you want them to be, and your done.

   What I'm supposed to do instead, I guess, is:

   Sleep one second

import time
...
time.sleep( 1 )

   Get current directory

import os
...
os.getcwd()

   Get a random number

import random
...
random.random()

   Now, the user has to cut the import, paste it to the top
   of his code, then go back to the list of snippets, find
   the same snippet again, copy the expression, go to his code,
   then find the point where he wanted to insert the snippet again,
   and finally insert the snippet. And still there now is a
   risk of name collisions. So, it seems to me that __import__
   is just so much better!



I'm not sure why you think this has to do with import vs __import__.
If you're worried bout having things on separate lines, you could write:

import os; os.getcwd()

,etc., which is actually saving a few characters :)



--
https://mail.python.org/mailman/listinfo/python-list

Re: String changing size on failure?

[MRAB]

On 2017-11-01 19:26, Ned Batchelder wrote:

  From David Beazley (https://twitter.com/dabeaz/status/925787482515533830):

  >>> a = 'n'
  >>> b = 'ñ'
  >>> sys.getsizeof(a)
 50
  >>> sys.getsizeof(b)
 74
  >>> float(b)
 Traceback (most recent call last):
    File "", line 1, in 
 ValueError: could not convert string to float: 'ñ'
  >>> sys.getsizeof(b)
 77

Huh?


It's all explained in PEP 393.

It's creating an additional representation (UTF-8 + zero-byte 
terminator) of the value and is caching that, so there'll then be the 
bytes for 'ñ' and the bytes for the UTF-8 (0xC3 0xB1 0x00).


When the string is ASCII, the bytes of the UTF-8 representation is 
identical to those or the original string, so it can share them.

--
https://mail.python.org/mailman/listinfo/python-list

Re: Code Snippets

[Chris Angelico]

On Thu, Nov 2, 2017 at 7:17 AM, Stefan Ram  wrote:
> Wolfgang Maier  writes:
>>If you're worried bout having things on separate lines, you could write:
>>import os; os.getcwd()
>>,etc., which is actually saving a few characters :)
>
>   Yes, but there still is the risk of the identifier »os«
>   already being used in the sorrounding code. While
>
> __import__( "os" ).getcwd()
>
>   does not seem to "leak" names into the enclosing scope.

If you're using the name "os" for something else, you need to be aware
of that anyway. Leaking names of core modules shouldn't normally be a
problem.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: String changing size on failure?

[Ned Batchelder]

On 11/1/17 4:17 PM, MRAB wrote:

On 2017-11-01 19:26, Ned Batchelder wrote:
  From David Beazley 
(https://twitter.com/dabeaz/status/925787482515533830):


  >>> a = 'n'
  >>> b = 'ñ'
  >>> sys.getsizeof(a)
 50
  >>> sys.getsizeof(b)
 74
  >>> float(b)
 Traceback (most recent call last):
    File "", line 1, in 
 ValueError: could not convert string to float: 'ñ'
  >>> sys.getsizeof(b)
 77

Huh?


It's all explained in PEP 393.

It's creating an additional representation (UTF-8 + zero-byte 
terminator) of the value and is caching that, so there'll then be the 
bytes for 'ñ' and the bytes for the UTF-8 (0xC3 0xB1 0x00).


When the string is ASCII, the bytes of the UTF-8 representation is 
identical to those or the original string, so it can share them.


That explains why b is larger than a to begin with, but it doesn't 
explain why float(b) is changing the size of b.


--Ned.
--
https://mail.python.org/mailman/listinfo/python-list

Re: String changing size on failure?

[Chris Angelico]

On Thu, Nov 2, 2017 at 7:34 AM, Ned Batchelder  wrote:
> On 11/1/17 4:17 PM, MRAB wrote:
>>
>> On 2017-11-01 19:26, Ned Batchelder wrote:
>>>
>>>   From David Beazley
>>> (https://twitter.com/dabeaz/status/925787482515533830):
>>>
>>>   >>> a = 'n'
>>>   >>> b = 'ñ'
>>>   >>> sys.getsizeof(a)
>>>  50
>>>   >>> sys.getsizeof(b)
>>>  74
>>>   >>> float(b)
>>>  Traceback (most recent call last):
>>> File "", line 1, in 
>>>  ValueError: could not convert string to float: 'ñ'
>>>   >>> sys.getsizeof(b)
>>>  77
>>>
>>> Huh?
>>>
>> It's all explained in PEP 393.
>>
>> It's creating an additional representation (UTF-8 + zero-byte terminator)
>> of the value and is caching that, so there'll then be the bytes for 'ñ' and
>> the bytes for the UTF-8 (0xC3 0xB1 0x00).
>>
>> When the string is ASCII, the bytes of the UTF-8 representation is
>> identical to those or the original string, so it can share them.
>
>
> That explains why b is larger than a to begin with, but it doesn't explain
> why float(b) is changing the size of b.

b doesn't initially even _have_ a UTF-8 representation. When float()
tries to parse the string, it asks for the UTF-8 form, and that form
gets saved into the string object in case it's needed later.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Code Snippets

[Irmen de Jong]

On 11/01/2017 06:25 PM, Stefan Ram wrote:

> import random
> ...
> random.random()
> 
>   Now, the user has to cut the import, paste it to the top
>   of his code, then go back to the list of snippets, find
>   the same snippet again, copy the expression, go to his code,
>   then find the point where he wanted to insert the snippet again,
>   and finally insert the snippet. And still there now is a
>   risk of name collisions. So, it seems to me that __import__
>   is just so much better!

..or suggest them to use an IDE instead? For instance in PyCharm you can type:
random.random()   (squiggle appears under random)

(it suggests: Import This name)  
(it suggests: from random) 
done, an import random has been added at the top of my module.


Irmen


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: String changing size on failure?

[Grant Edwards]

On 2017-11-01, Ned Batchelder  wrote:
> On 11/1/17 4:17 PM, MRAB wrote:
>> On 2017-11-01 19:26, Ned Batchelder wrote:
>>>   From David Beazley 
>>> (https://twitter.com/dabeaz/status/925787482515533830):
>>>
>>>   >>> a = 'n'
>>>   >>> b = 'ñ'
>>>   >>> sys.getsizeof(a)
>>>  50
>>>   >>> sys.getsizeof(b)
>>>  74
>>>   >>> float(b)
>>>  Traceback (most recent call last):
>>>     File "", line 1, in 
>>>  ValueError: could not convert string to float: 'ñ'
>>>   >>> sys.getsizeof(b)
>>>  77
>>>
>>> Huh?
>>>
>> It's all explained in PEP 393.
>>
>> It's creating an additional representation (UTF-8 + zero-byte 
>> terminator) of the value and is caching that, so there'll then be the 
>> bytes for 'ñ' and the bytes for the UTF-8 (0xC3 0xB1 0x00).
>>
>> When the string is ASCII, the bytes of the UTF-8 representation is 
>> identical to those or the original string, so it can share them.
>
> That explains why b is larger than a to begin with

No, that size difference is due to the additional bytes required for
the internal representation of the string.

> but it doesn't explain why float(b) is changing the size of b.

The additional UTF-8 representation isn't being created and cached
until the float() call is made.

-- 
Grant Edwards   grant.b.edwardsYow! ONE LIFE TO LIVE for
  at   ALL MY CHILDREN in ANOTHER
  gmail.comWORLD all THE DAYS OF
   OUR LIVES.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Code Snippets

[Ben Bacarisse]

r...@zedat.fu-berlin.de (Stefan Ram) writes:

> Wolfgang Maier  writes:
>>If you're worried bout having things on separate lines, you could write:
>>import os; os.getcwd()
>>,etc., which is actually saving a few characters :)
>
>   Yes, but there still is the risk of the identifier »os«
>   already being used in the sorrounding code. While
>
> __import__( "os" ).getcwd()
>
>   does not seem to "leak" names into the enclosing scope.

Also it's an expression which may be important in your "quick and dirty"
scripts.

-- 
Ben.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Code Snippets

[Chris Angelico]

On Thu, Nov 2, 2017 at 8:02 AM, Ben Bacarisse  wrote:
> r...@zedat.fu-berlin.de (Stefan Ram) writes:
>
>> Wolfgang Maier  writes:
>>>If you're worried bout having things on separate lines, you could write:
>>>import os; os.getcwd()
>>>,etc., which is actually saving a few characters :)
>>
>>   Yes, but there still is the risk of the identifier »os«
>>   already being used in the sorrounding code. While
>>
>> __import__( "os" ).getcwd()
>>
>>   does not seem to "leak" names into the enclosing scope.
>
> Also it's an expression which may be important in your "quick and dirty"
> scripts.

If your quick-and-dirties are needing these kinds of imports all the
time, the best solution might be to slap something into site.py that
"pre-imports" those into the builtins. Then you can just use
os.getcwd() without worrying about the import, and without calling a
dunder.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: String changing size on failure?

[David Beazley]

Nah. I discuss this behavior (caching of UTF-8 conversions in the C API)  in 
section 15.14 of the Python Cookbook.  The tweet was a tutorial, not a question 
;-).   Admittedly, an evil tutorial...

Cheers,
Dave

> On Nov 1, 2017, at 2:53 PM, Skip Montanaro  wrote:
> 
> Leave it to DB to ask the tough questions other people won't. :-)
> 
> Skip
> 
> 
> On Wed, Nov 1, 2017 at 2:26 PM, Ned Batchelder  wrote:
>> From David Beazley (https://twitter.com/dabeaz/status/925787482515533830):
>> 
> a = 'n'
> b = 'ñ'
> sys.getsizeof(a)
>>   50
> sys.getsizeof(b)
>>   74
> float(b)
>>   Traceback (most recent call last):
>>  File "", line 1, in 
>>   ValueError: could not convert string to float: 'ñ'
> sys.getsizeof(b)
>>   77
>> 
>> Huh?
>> 
>> --Ned.
>> 
>> --
>> https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

replacing `else` with `then` in `for` and `try`

[Alexey Muranov]

Hello,

what do you think about the idea of replacing "`else`" with "`then`" in 
the contexts of `for` and `try`?


It seems clear that it should be rather "then" than "else."  Compare 
also "try ... then ... finally" with "try ... else ... finally".


Currently, with "else", it is almost impossible to guess the meaning 
without looking into the documentation.


Off course, it should not be changed in Python 3, maybe in Python 4 or 
5, but in Python 3 `then` could be an alias of `else` in these contexts.


Alexey.

--
https://mail.python.org/mailman/listinfo/python-list

Re: replacing `else` with `then` in `for` and `try`

[Chris Angelico]

On Thu, Nov 2, 2017 at 8:12 AM, Alexey Muranov  wrote:
> Hello,
>
> what do you think about the idea of replacing "`else`" with "`then`" in the
> contexts of `for` and `try`?
>
> It seems clear that it should be rather "then" than "else."  Compare also
> "try ... then ... finally" with "try ... else ... finally".
>
> Currently, with "else", it is almost impossible to guess the meaning without
> looking into the documentation.
>
> Off course, it should not be changed in Python 3, maybe in Python 4 or 5,
> but in Python 3 `then` could be an alias of `else` in these contexts.
>

The cost of creating a new keyword is incredibly high. You'll need to
demonstrate much more than a marginal improvement.

With try/except/else, it's "do this, and if an exception happens, do
this, else do this". So else makes perfect sense. With the 'for' loop,
it's a bit more arguable, but I've never seen anything more than a
weak argument in favour of 'then', and since it'd be a completely new
keyword, there's very approximately 0% chance that this will be
changed.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: replacing `else` with `then` in `for` and `try`

[Ned Batchelder]

On 11/1/17 5:12 PM, Alexey Muranov wrote:

Hello,

what do you think about the idea of replacing "`else`" with "`then`" 
in the contexts of `for` and `try`?


It seems clear that it should be rather "then" than "else." Compare 
also "try ... then ... finally" with "try ... else ... finally".


Currently, with "else", it is almost impossible to guess the meaning 
without looking into the documentation.


Off course, it should not be changed in Python 3, maybe in Python 4 or 
5, but in Python 3 `then` could be an alias of `else` in these contexts.


Alexey.



Apart from the questions of backward compatibility etc (Python is 
unlikely to ever go through another shift like the 2/3 breakage), are 
you sure "then" is what you mean?  This won't print "end":


    for i in range(10):
    print(i)
    else:
    print(end)

--Ned.
--
https://mail.python.org/mailman/listinfo/python-list

Re: replacing `else` with `then` in `for` and `try`

[Chris Angelico]

On Thu, Nov 2, 2017 at 8:23 AM, Ned Batchelder  wrote:
> On 11/1/17 5:12 PM, Alexey Muranov wrote:
>>
>> Hello,
>>
>> what do you think about the idea of replacing "`else`" with "`then`" in
>> the contexts of `for` and `try`?
>>
>> It seems clear that it should be rather "then" than "else." Compare also
>> "try ... then ... finally" with "try ... else ... finally".
>>
>> Currently, with "else", it is almost impossible to guess the meaning
>> without looking into the documentation.
>>
>> Off course, it should not be changed in Python 3, maybe in Python 4 or 5,
>> but in Python 3 `then` could be an alias of `else` in these contexts.
>>
>> Alexey.
>>
>
> Apart from the questions of backward compatibility etc (Python is unlikely
> to ever go through another shift like the 2/3 breakage), are you sure "then"
> is what you mean?  This won't print "end":
>
> for i in range(10):
> print(i)
> else:
> print(end)

Well, it'll bomb with NameError when it tries to look up the *name*
end. But it will run that line of code - if you quote it, it will
work.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: matplot plot hangs

[Andrew Z]

nope. it doesnt:

I added print-s after each line and that produced:
[az@hp src]$ cat ./main1.py
import matplotlib.pyplot as plt
print("imported")
plt.plot([1,2,4,1])
print("plot is done")
plt.show()
print("show is done")

[az@hp src]$ python3.5 ./main1.py
imported
^C^Z
[1]+  Stopped python3.5 ./main1.py


On Wed, Nov 1, 2017 at 9:31 AM, Vlastimil Brom 
wrote:

> 2017-11-01 13:49 GMT+01:00 Andrew Z :
> > Wolfgang,
> >  I tried to ran from ide with no rwsults, so now im trying from a
> terminal
> > in xwindow.
> > The .plot is the last line in the script and it does hang trying to
> execute
> > it.
> >
> >
> > On Nov 1, 2017 05:44, "Wolfgang Maier" <
> > wolfgang.ma...@biologie.uni-freiburg.de> wrote:
> >
> > On 01.11.2017 00:40, Andrew Z wrote:
> >
> >> hello,
> >>   learning python's plotting by using matplotlib with python35 on
> fedora 24
> >> x86.
> >>
> >> Installed matplotlib into user's directory.
> >> tk, seemed to work -
> >> http://www.tkdocs.com/tutorial/install.html#installlinux - the window
> >> shows
> >> up just fine.
> >> but when trying to run the simple plot (
> >> https://matplotlib.org/examples/pylab_examples/simple_plot.html) the
> >> script
> >> is hanging on;
> >>
> >> plt.plot(t, s)
> >>
> >> attempts to
> >> matplotlib.interactive(True) didn't bring anything,
> >>
> >>
> > Hi Andrew,
> >
> > From which environment are you trying to run the example? In the
> terminal,
> > from within some IDE, inside a jupyter notebook?
> >
> > Are you sure the script "is hanging on plt.plot(t, s)" and not after
> that?
> >
> > Best,
> > Wolfgang
> >
> > --
> Hi,
> sorry if it is too trivial, just to make sure, do you have a call to
> "show()" the resulting plot in the code?
>
> An elementary plotting code might be e.g.:
>
> import matplotlib.pyplot as plt
> plt.plot([1,2,4,1])
> plt.show()
>
> Does this work in your environment?
>
> It was not quite clear, what do you plan with interactive drawing, or
> whether you are using e.g. plt.interactive(True) already - this might
> be a problem as there could be collisions or the plot window is closed
> after the standalone script finishes.
>
> hth,
>  vbr
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: replacing `else` with `then` in `for` and `try`

[Ned Batchelder]

On 11/1/17 5:29 PM, Chris Angelico wrote:

On Thu, Nov 2, 2017 at 8:23 AM, Ned Batchelder  wrote:

On 11/1/17 5:12 PM, Alexey Muranov wrote:

Hello,

what do you think about the idea of replacing "`else`" with "`then`" in
the contexts of `for` and `try`?

It seems clear that it should be rather "then" than "else." Compare also
"try ... then ... finally" with "try ... else ... finally".

Currently, with "else", it is almost impossible to guess the meaning
without looking into the documentation.

Off course, it should not be changed in Python 3, maybe in Python 4 or 5,
but in Python 3 `then` could be an alias of `else` in these contexts.

Alexey.


Apart from the questions of backward compatibility etc (Python is unlikely
to ever go through another shift like the 2/3 breakage), are you sure "then"
is what you mean?  This won't print "end":

 for i in range(10):
 print(i)
 else:
 print(end)

Well, it'll bomb with NameError when it tries to look up the *name*
end. But it will run that line of code - if you quote it, it will
work.

ChrisA


Naturally, I messed up on both a shallow and deep level! :) Or should I 
say derp level... :)


--Ned.
--
https://mail.python.org/mailman/listinfo/python-list

what exactly does type.__call__ do?

[Jason Maldonis]

Hi everyone,

I want to use a metaclass to override how class instantiation works. I've
done something analogous to using the Singleton metaclass from the Python3
Cookbook example.

However, I want to provide a classmethod that allows for "normal" class
instantiation that prevents this metaclass from being used.

To do that, I think I just make a @classmethod constructor function.
However, I can imagine a few different ways of writing this:

@classmethod
def normal_constructor(cls, *args, **kwargs):
return type.__call__(*args, **kwargs)

@classmethod
def normal_constructor(cls, *args, **kwargs):
return super(???).__call__(*args, **kwargs)  # I'm not sure what should
go in the super here  (I'm using python3)

@classmethod
def normal_constructor(cls, *args, **kwargs):
self = cls.__new__(cls)
self.__init__(*args, **kwargs)
return self

Is one of these correct? Or do they all do the same thing?

I was looking for documentation for what exactly `type.__call__` does so
that I can emulate it, but I wasn't able to find any docs explicitly
detailing what that method does. If someone knows where this info is that
would be great too.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: what exactly does type.__call__ do?

[Jason Maldonis]

Thanks for the reply. And I think I wasn't clear enough. I was wondering
what the metaclass `type`'s `type.__call__` does explicitly. I'm reasonably
comfortable writing metaclasses when I need them, and I understand how
`.__call__` works for non-metaclass objects.

In my first email I gave three possible ways to write a @classmethod that
returns an instance of that class. I'm not sure which one is the "most
pythonic" or "most correct", but I'm pretty sure I can get them all to
work. I'll paste them below again, and I'm wondering which one I should use
and why. Thanks!

@classmethod
def normal_constructor(cls, *args, **kwargs):
return type.__call__(*args, **kwargs)

@classmethod
def normal_constructor(cls, *args, **kwargs):
return super(???).__call__(*args, **kwargs)  # I'm not sure what should
go in the super here  (I'm using python3)

@classmethod
def normal_constructor(cls, *args, **kwargs):
self = cls.__new__(cls)
self.__init__(*args, **kwargs)
return self

On Wed, Nov 1, 2017 at 6:51 PM, Stefan Ram  wrote:

> Jason Maldonis  writes:
> >I was looking for documentation for what exactly `type.__call__` does so
> >that I can emulate it, but I wasn't able to find any docs explicitly
> >detailing what that method does. If someone knows where this info is that
> >would be great too.
>
>   Instances are callable if their class has a __call__ method.
>
>   When an instance is called, this __call__ method will be called.
>
>   main.py
>
> class example:
> def __call__( this ):
> print( 'called' )
> instance = example()
> instance()
>
>   transcript
>
> called
>
>   The general contract of this method is that it can do
>   whatever the author of the class deems appropriate.
>
>   What exectly this is has to be documented in the
>   documentation of each specific class.
>
>   In fact, the attribute »instance.__call__« has a »__call__«
>   attribute itself, thus:
>
>   main.py
>
> class example:
> def __call__( this ):
> print( 'called' )
> instance = example()
> instance()
> instance.__call__()
> instance.__call__.__call__()
>
>   transcript
>
> called
> called
> called
>
>   A type also is an instance, so
>
> int.__call__(whatever)
>
>   should do the same as
>
> int( whatever )
>
>   . In Python 2 they had
>
> operator.isCallable( obj )
>
>   , in Python 3 we can emulate this using
>
> hasattr( obj, '__call__' )
>
>   .
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Code Snippets

[Steve D'Aprano]

On Thu, 2 Nov 2017 04:25 am, Stefan Ram wrote:

>   I started to collect some code snippets:
[...]

> __import__( "random" ).random()
>
>   And so on. You get the idea.
>
>   However, reportedly, all those snippets are anti-patterns
>   because they use »__import__«.

Correct. Nearly all dunder functions and methods are reserved for use by the
interpreter.

It isn't an outright error to call __import__ directly, but you should avoid
it unless absolutely necessary.


[...]
>   What I'm supposed to do instead, I guess, is:
[...]

> import random
> ...
> random.random()
> 
>   Now, the user has to cut the import, paste it to the top
>   of his code, then go back to the list of snippets, find
>   the same snippet again, copy the expression, go to his code,
>   then find the point where he wanted to insert the snippet again,
>   and finally insert the snippet.

The ellipsis do nothing, why are they there? And surely you are capable of
copying two lines at a time.


import random
random.random()


requires only one copy operation.

There is no outright requirement to collect all the imports at the top of your
module. That's merely a very good convention to follow. If you don't mind
breaking the convention, you can simply paste the result where you want the
random number:

# code here
# more code
# paste here >>
import random
random.random()  # and edit as needed


This has saved you ten seconds of editing time while writing the code. It will
probably cost you ten minutes, when you come back to maintain the program in
six months and the imports are scattered all through the module, inside
functions and classes, but that's your decision to make.


>   And still there now is a 
>   risk of name collisions. So, it seems to me that __import__
>   is just so much better!

Only if you wish to write ugly, inefficient code.

The idea of code snippets is that they are intended as *templates*, not that
you repeat them over and over again. That would be the worst sort of copy and
paste programming, an anti-pattern.

If you need four random numbers, would you write this?


a = __import__( "random" ).random()
b = __import__( "random" ).random()
c = __import__( "random" ).random()
d = __import__( "random" ).random()


Or a list of them?


numbers = [__import__( "random" ).random() for i in range(1000)]


Do I need to explain how terrible that code is?

Code snippets are not an alternative to actually thinking about your code and
writing the best code you can.


-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Thread safety issue (I think) with defaultdict

[Steve D'Aprano]

On Thu, 2 Nov 2017 05:53 am, Israel Brewster wrote:

[...]
> So the end result is that the thread that "updates" the dictionary, and the
> thread that initially *populates* the dictionary are actually running in
> different processes.

If they are in different processes, that would explain why the second
(non)thread sees an empty dict even after the first thread has populated it:


# from your previous post
> Length at get AC:  54 ID: 4524152200  Time: 2017-11-01 09:41:24.474788
> Length At update:  1 ID: 4524152200  Time: 2017-11-01 09:41:24.784399
> Length At update:  2 ID: 4524152200  Time: 2017-11-01 09:41:25.228853


You cannot rely on IDs being unique across different processes. Its an
unfortunately coincidence(!) that they end up with the same ID.

Or possibly there's some sort of weird side-effect or bug in Flask that, when
it shares the dict between two processes (how?) it clears the dict.

Or... have you considered the simplest option, that your update thread clears
the dict when it is first called? Since you haven't shared your code with us,
I cannot rule out a simple logic error like this:

def launch_update_thread(dict):
dict.clear()
# code to start update thread


> In fact, any given request could be in yet another 
> process, which would seem to indicate that all bets are off as to what data
> is seen.
> 
> Now that I've thought through what is really happening, I think I need to
> re-architect things a bit here. 

Indeed. I've been wondering why you are using threads at all, since there
doesn't seem to be any benefit to initialising the dict and updating it in
different thread. Now I learn that your architecture is even more complex. I
guess some of that is unavailable, due to it being a web app, but still.


> For one thing, the update thread should be 
> launched from the main process, not an arbitrary UWSGI worker. I had
> launched it from the client connection because there is no point in having
> it running if there is no one connected, but I may need to launch it from
> the __init__.py file instead. For another thing, since this dictionary will
> need to be accessed from arbitrary worker processes, I'm thinking I may need 
> to move it to some sort of external storage, such as a redis database

That sounds awful. What if the arbitrary worker decides to remove a bunch of
planes from your simulation, or insert them? There should be one and only one
way to insert or remove planes from the simulation (I **really** hope it is a
simulation).

Surely the right solution is to have the worker process request whatever
information it needs, like "the next plane", and have the main process
provide the data. Having worker processes have the ability to reach deep into
the data structures used by the main program and mess with them seems like a
good way to have mind-boggling bugs.



> Oy, I made my life complicated :-)

"Some people, when confronted with a problem, think, 'I know, I'll use
threads. Nothew y htwo probave lems."

:-)



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: replacing `else` with `then` in `for` and `try`

[Steve D'Aprano]

On Thu, 2 Nov 2017 08:12 am, Alexey Muranov wrote:

> Hello,
> 
> what do you think about the idea of replacing "`else`" with "`then`" in
> the contexts of `for` and `try`?


Yes, this, exactly!!!

(For while and for loops, but not try -- see below.)

I have argued this for many years. The current choice of "else" is painfully
misleading, and it causes people (including myself) to wrongly guess that
the "else" block runs only if the for/while block doesn't run at all:


# This is wrong!
for x in sequence:
...
else:
print("sequence is empty")


The actually semantics of "else" is that the block is UNCONDITIONALLY run
after the for/while loop completes, unless you jump out of the loop using
return, raise or break. That makes it a "then" block, not "else".


> It seems clear that it should be rather "then" than "else."  Compare
> also "try ... then ... finally" with "try ... else ... finally".

I disagree about the try block though. The semantics of the try block are:

try:
   A
except:
   B
else:
   C
finally:
   D

(1) code block A is attempted;

(2) IF an exception occurs, jump to code block B;

(3) otherwise (else), no exception occurs, so jump to code block C;

(4) finally run code block D on your way out, regardless of which blocks are
executed and how you exit them.


So I think "else" is correct here. The else block only gets called if there is
no exception.


> Currently, with "else", it is almost impossible to guess the meaning
> without looking into the documentation.

It is worse than that: it is easy to guess the WRONG meaning, namely that the
else block runs when the for/while loop doesn't execute at all (the for-loop
sequence is empty, or the while-loop condition is initially false).


> Off course, it should not be changed in Python 3, maybe in Python 4 or
> 5, but in Python 3 `then` could be an alias of `else` in these contexts.

Unfortunately, this is almost certainly not going to happen. It would require
adding a new keyword, and unless Guido changes his mind, he doesn't think
this change is worthwhile.

If I were the BDFL, this would be the first change I make :-)



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Code Snippets

[Steve D'Aprano]

On Thu, 2 Nov 2017 05:57 am, Stefan Ram wrote:

> I also have heard that there was a module cache, so I
> was hoping that a second import of the same module might
> not be such an effort for the implementation.


There is: sys.modules.

Although `import spam` is cheap when spam is in the cache, its not free, and
`__import__("spam")` is even less cheap (more costly). Its a function call,
not a statement, so it requires a runtime name lookup and a function call on
top of the same process of checking the cache and importing the module.




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Code Snippets

[Steve D'Aprano]

On Thu, 2 Nov 2017 08:02 am, Ben Bacarisse wrote:

> r...@zedat.fu-berlin.de (Stefan Ram) writes:
> 
>> Wolfgang Maier  writes:
>>>If you're worried bout having things on separate lines, you could write:
>>>import os; os.getcwd()
>>>,etc., which is actually saving a few characters :)
>>
>>   Yes, but there still is the risk of the identifier »os«
>>   already being used in the sorrounding code. While
>>
>> __import__( "os" ).getcwd()
>>
>>   does not seem to "leak" names into the enclosing scope.
> 
> Also it's an expression which may be important in your "quick and dirty"
> scripts.

No script is so quick or so dirty to justify calling __import__ with a string
literal argument instead of import.

We've all written quick and dirty throw-away scripts where we don't care too
much about best practices. But this isn't so much less-than-best practices as
worst-practices: optimizing to save a few seconds during the initial editing
run, by using the copy-and-paste anti-pattern.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: replacing `else` with `then` in `for` and `try`

[Steve D'Aprano]

On Thu, 2 Nov 2017 08:21 am, Chris Angelico wrote:

> With the 'for' loop,
> it's a bit more arguable, but I've never seen anything more than a
> weak argument in favour of 'then'

Thhpptpt!

"else" is an completely inappropriate term that doesn't describe the semantics
of the statement even a little bit. The argument that it means "else no
break" is feeble because break is not the only way to exit the loop and
therefore skip executing the else clause.

It is not even necessarily the most common: I wouldn't be surprised if there
were more returns out of the middle of a loop than breaks, although I
wouldn't necessarily predict it either.

If we spoke in ordinary English using "else" the way Python uses it for
looping, we would say:

"Shampoo your hair twice, ELSE apply conditioner and leave for five minutes
before rinsing."

"Boil the pasta until it is soft, ELSE drain it and mix in the sauce."




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: replacing `else` with `then` in `for` and `try`

[Steve D'Aprano]

On Thu, 2 Nov 2017 08:23 am, Ned Batchelder wrote:

> Apart from the questions of backward compatibility etc (Python is
> unlikely to ever go through another shift like the 2/3 breakage), are
> you sure "then" is what you mean?  This won't print "end":
> 
>  for i in range(10):
>  print(i)
>  else:
>  print(end)


You are neither the first nor the last person to have mistakenly understood
the "else" clause to run only if the main loop does not.

It is a common error. I've done it. And this demonstrates exactly why the
choice of keyword is so poor. If somebody of your experience can misread it,
I don't feel so bad about how long it too me to understand it when I was a
newbie.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: replacing `else` with `then` in `for` and `try`

[bartc]

On 02/11/2017 01:19, Steve D'Aprano wrote:

On Thu, 2 Nov 2017 08:21 am, Chris Angelico wrote:


With the 'for' loop,
it's a bit more arguable, but I've never seen anything more than a
weak argument in favour of 'then'


Thhpptpt!

"else" is an completely inappropriate term that doesn't describe the semantics
of the statement even a little bit. The argument that it means "else no
break" is feeble because break is not the only way to exit the loop and
therefore skip executing the else clause.

It is not even necessarily the most common: I wouldn't be surprised if there
were more returns out of the middle of a loop than breaks, although I
wouldn't necessarily predict it either.

If we spoke in ordinary English using "else" the way Python uses it for
looping, we would say:

"Shampoo your hair twice, ELSE apply conditioner and leave for five minutes
before rinsing."

"Boil the pasta until it is soft, ELSE drain it and mix in the sauce."


1 Start boiling the pasta
2 Wait one minute
3 If it's not soft (some of us prefer al dente), repeat from step 2,
  else drain it
--
https://mail.python.org/mailman/listinfo/python-list

Re: replacing `else` with `then` in `for` and `try`

[bartc]

On 02/11/2017 01:06, Steve D'Aprano wrote:

On Thu, 2 Nov 2017 08:12 am, Alexey Muranov wrote:


Hello,

what do you think about the idea of replacing "`else`" with "`then`" in
the contexts of `for` and `try`?



Yes, this, exactly!!!

(For while and for loops, but not try -- see below.)

I have argued this for many years. The current choice of "else" is painfully
misleading, and it causes people (including myself) to wrongly guess that
the "else" block runs only if the for/while block doesn't run at all:


# This is wrong!
for x in sequence:
 ...
else:
 print("sequence is empty")


The actually semantics of "else" is that the block is UNCONDITIONALLY run
after the for/while loop completes,


/If/ it completes, and /when/ it completes. Otherwise why bother with 
using 'else'? Just have the code immediately following the loop.


And there are some circumstances where the 'else' part is never executed.

But if people prefer a different keyword, then why not? I think 'then' 
can be used, without impacting its use as an identifier, because it will 
always be followed by ":". Of course you would need to allow both "else" 
and "then" for backwards compatibility.


--
bartc
--
https://mail.python.org/mailman/listinfo/python-list

Re: replacing `else` with `then` in `for` and `try`

[Skip Montanaro]

I don't know. The word "then" doesn't connote different ways of exiting a
loop to me ("else" doesn't really either, I will grant you that, but it's
what we have). Here's how I would read things:

   - *while* some condition holds, execute the loop, possibly breaking out,
   *then* do some finishing work
   - *for* each element in some sequence, execute the loop, possibly
   breaking out, *then* do some finishing work

In neither case does it seem to me that you execute the finishing work only
if you break out of the loop, but not if the loop terminates when the while
condition becomes false or the for loop's sequence is exhausted. You might
consider that while/then or for/then actually reads too much like English,
fooling you into interpreting it as English, opening you up to ambiguity.
You might argue that "else" doesn't either, but it has the strangely nice
property of actually being a bit clumsier to read, forcing the reader to
learn and apply the precise rules of the programming language instead of
infer the more ambiguous rules of English. Either way, we are down to two
imperfect solutions, and have a case of tomato, tomahto, I think.

If I was starting with a clean sheet of paper, I might put the raise and
except keywords to work, and add a LoopExit exception:

while some condition holds:
blah blah blah
if some other condition rears its ugly head:
raise LoopExit
blah blah blah
except LoopExit:
execute exceptional code

English and other natural languages aren't precise enough to serve as
programming languages. Neither are programming languages fluid enough that
we can always expect them to read naturally. There will always be cases
like this where there is no perfect solution. Other examples I can think of
are the if/else expression added to the language relatively recently (never
really reads well to me, though I agree it can be handy), or all the
proposals for switch/case/computed goto statements which litter the Python
PEP cemetery. The desire to add such a statement has been very strong at
times (there is a powerful desire from a performance perspective to have
something akin to C's switch statement), but nothing ever worked well
enough to be accepted.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: replacing `else` with `then` in `for` and `try`

[Ben Bacarisse]

Steve D'Aprano  writes:

> On Thu, 2 Nov 2017 08:12 am, Alexey Muranov wrote:
>
>> what do you think about the idea of replacing "`else`" with "`then`" in
>> the contexts of `for` and `try`?
>
> Yes, this, exactly!!!
>
> (For while and for loops, but not try -- see below.)
>
> I have argued this for many years. The current choice of "else" is painfully
> misleading, and it causes people (including myself) to wrongly guess that
> the "else" block runs only if the for/while block doesn't run at all:
>
>
> # This is wrong!
> for x in sequence:
> ...
> else:
> print("sequence is empty")
>
>
> The actually semantics of "else" is that the block is UNCONDITIONALLY run
> after the for/while loop completes, unless you jump out of the loop using
> return, raise or break. That makes it a "then" block, not "else".
>
>
>> It seems clear that it should be rather "then" than "else."  Compare
>> also "try ... then ... finally" with "try ... else ... finally".
>
> I disagree about the try block though. The semantics of the try block are:
>
> try:
>A
> except:
>B
> else:
>C
> finally:
>D
>
> (1) code block A is attempted;
>
> (2) IF an exception occurs, jump to code block B;
>
> (3) otherwise (else), no exception occurs, so jump to code block C;
>
> (4) finally run code block D on your way out, regardless of which blocks are
> executed and how you exit them.
>
>
> So I think "else" is correct here. The else block only gets called if there is
> no exception.
>
>
>> Currently, with "else", it is almost impossible to guess the meaning
>> without looking into the documentation.
>
> It is worse than that: it is easy to guess the WRONG meaning, namely that the
> else block runs when the for/while loop doesn't execute at all (the for-loop
> sequence is empty, or the while-loop condition is initially false).
>
>
>> Off course, it should not be changed in Python 3, maybe in Python 4 or
>> 5, but in Python 3 `then` could be an alias of `else` in these contexts.
>
> Unfortunately, this is almost certainly not going to happen. It would require
> adding a new keyword, and unless Guido changes his mind, he doesn't think
> this change is worthwhile.

Re-using finally would not need a new keyword and might be close enough
in meaning.


-- 
Ben.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: replacing `else` with `then` in `for` and `try`

[Chris Angelico]

On Thu, Nov 2, 2017 at 12:19 PM, Steve D'Aprano
 wrote:
> On Thu, 2 Nov 2017 08:21 am, Chris Angelico wrote:
>
>> With the 'for' loop,
>> it's a bit more arguable, but I've never seen anything more than a
>> weak argument in favour of 'then'
>
> Thhpptpt!
>
> "else" is an completely inappropriate term that doesn't describe the semantics
> of the statement even a little bit. The argument that it means "else no
> break" is feeble because break is not the only way to exit the loop and
> therefore skip executing the else clause.
>
> It is not even necessarily the most common: I wouldn't be surprised if there
> were more returns out of the middle of a loop than breaks, although I
> wouldn't necessarily predict it either.

I'd say that's plausible, partly because it's easier than messing
around with the else clause. Sometimes, it's easier to just make
another function so you can use 'return' as flow control.

> If we spoke in ordinary English using "else" the way Python uses it for
> looping, we would say:
>
> "Shampoo your hair twice, ELSE apply conditioner and leave for five minutes
> before rinsing."

You wouldn't use 'else' with a simple iteration loop like that -
there's no difference between 'else' and simply having more code after
the loop, unless you have a break.

for _ in range(2):
hair.shampoo()
condition()
sleep(300)

> "Boil the pasta until it is soft, ELSE drain it and mix in the sauce."

Except that the else keyword doesn't mean that. Here's the nearest I
can come up with as an equivalent:

while pot.water_level:
pasta.boil()
if pasta.is_soft(): break
else:
print("HELP! You boiled the pot dry!")

The else clause happens if you don't break. If you break, you skip the
rest of the loop - including the else. You could use "then" for this,
and it would make sense, but not enough to create a new keyword, and
DEFINITELY not enough to justify changing it now.

Yes, there are arguments against calling this feature "else". But I
stand by my assertion that there are none strong enough to justify a
new keyword.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: what exactly does type.__call__ do?

[Jason Maldonis]

Ok no worries then! Thanks for the tips. I might wait until tomorrow then
until someone comes along who deals with metaclasses and alternate class
constructors.

In case you're curious, I'm doing two things that are relevant here, and
I'll link the python3 cookbook examples that are super useful (I love that
book):
http://chimera.labs.oreilly.com/books/123000393/ch09.
html#_discussion_155
and
http://chimera.labs.oreilly.com/books/123000393/ch08.html#_solution_134

They explain things very well. Metaclasses are so useful when you hit a use
case that they fit into. I don't often deal with multiple class
constructors, so I'm a bit new to that territory and I'm trying to delve
into the details of how python classes are constructed (namely, whether
`type.__call__` does more than just call `cls.__new__` and `self.__init__`).

Thanks for the help too.

On Wed, Nov 1, 2017 at 8:49 PM, Stefan Ram  wrote:

> r...@zedat.fu-berlin.de (Stefan Ram) writes:
> >|PyObject *meth = lookup_method(self, &PyId___call__);
> ...
> >|Call self as a function.
>
>   Oh, I guess this might be what makes
>
> int(x)
>
>   call
>
> int.__call__(x)
>
>   . And all other instances of this metaclass.
>
>   Let's try to overwrite it to call 'charles' instead.
>
>   main.py
>
> class mymetaclass( type ):
> def __call__( this, *args ):
> this.charles( *args )
>
> class myclass( metaclass=mymetaclass ):
> def charles():
> print( "Charles" )
>
> myclass()
>
>   transcript
>
> Charles
>
>   Ok, while I managed to make »myclass« print »Charles«
>   using a »__call__« function of it's metaclass, I still
>   don't know what I am doing. This is the first time I
>   actually deal with the topic of metaclasses. I have not
>   yet read a text book chapter about them, so I'm still
>   in the dark like the Chinese in the Chinese room.
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: replacing `else` with `then` in `for` and `try`

[Steve D'Aprano]

On Thu, 2 Nov 2017 12:50 pm, Ben Bacarisse wrote:

> Steve D'Aprano  writes:
> 
>> On Thu, 2 Nov 2017 08:12 am, Alexey Muranov wrote:
>>
>>> what do you think about the idea of replacing "`else`" with "`then`" in
>>> the contexts of `for` and `try`?
[...]
> Re-using finally would not need a new keyword and might be close enough
> in meaning.

Reusing finally would be *completely* wrong.

The semantics of `finally` is that it should be executed no matter[1] how you
exit the previous block. E.g. if we write:


try:
return 1
finally:
print("exiting")


then "exiting" is printed. Replace the return with a raise, and the same
applies. Reusing `finally` in for and while loops would imply the similar
behaviour:


for i in range(100):
return i
finally:
print("exiting")


should print "exiting", when in fact it does not. Likewise if you replace the
return with a break.





[1] Within the bounds of normal processing. There are ways to halt Python
without executing any subsequent code, namely os._exit, and os.abort dumps
core to exit immediately.


-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: what exactly does type.__call__ do?

[Steve D'Aprano]

On Thu, 2 Nov 2017 10:13 am, Jason Maldonis wrote:

> Hi everyone,
> 
> I want to use a metaclass to override how class instantiation works. I've
> done something analogous to using the Singleton metaclass from the Python3
> Cookbook example.

In my opinion, nine times out of ten, using a metaclass for something like
that is overkill.

(And that's putting aside the fact that 999 out of a thousand, using a
Singleton is the wrong solution, no matter what the question is.)


> However, I want to provide a classmethod that allows for "normal" class
> instantiation that prevents this metaclass from being used.

To me, that strongly suggests that a metaclass is the wrong solution.


> To do that, I think I just make a @classmethod constructor function.
> However, I can imagine a few different ways of writing this:
> 
> @classmethod
> def normal_constructor(cls, *args, **kwargs):
> return type.__call__(*args, **kwargs)

Untested, but I think that should be:

return type.__call__(cls, *args, **kwargs)


> @classmethod
> def normal_constructor(cls, *args, **kwargs):
> return super(???).__call__(*args, **kwargs)  # I'm not sure what should
> go in the super here  (I'm using python3)
> 
> @classmethod
> def normal_constructor(cls, *args, **kwargs):
> self = cls.__new__(cls)
> self.__init__(*args, **kwargs)
> return self
> 
> Is one of these correct? Or do they all do the same thing?

None of them look "correct", they all look "weird and scary" :-)

If I had to pick one of these three -- and I hope that I would not -- I'd pick
the first one.


> I was looking for documentation for what exactly `type.__call__` does so
> that I can emulate it, 

And then if type.__call__ changes, your emulation will be wrong.


> but I wasn't able to find any docs explicitly 
> detailing what that method does. If someone knows where this info is that
> would be great too.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list