Re: Can't find latest version of 3.4.x on download page

2017-10-18 Thread Terry Reedy 

On 10/18/2017 2:09 AM, Christopher Reimer wrote:

Greetings,

I'm setting up different test environments for tox. I can't find Windows 
installer for the latest version of Python 3.4 on the download page.

Versions 3.4.5 to 3.4.7 only have the source files available.


Correct.  These are security releases, mainly for servers and people who 
would compile themselves.



Version 3.4.4 is the last version with Windows installers.


Correct.

--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list

Re: Can't find latest version of 3.4.x on download page

2017-10-18 Thread Ben Finney 
Christopher Reimer  writes:

> I'm setting up different test environments for tox. I can't find
> Windows installer for the latest version of Python 3.4 on the download
> page.

Python 3.4 is provided in source form only; the Python core developers
no longer generate installers for it.

Anyone who wants to run Python 3 older than 3.5, needs to find a way
themselves to build and install it; that's the trade-off for wanting an
old version :-)

> Testing on Python 3.4 might be moot for my program. It choked on a
> starred expression in a unit test. The starred expression got changed
> in Python 3.5 and my unit tests pass in Python 3.5/3.6. I don't think
> I can live without that feature.

That sounds reasonable. If you can declare Python 3.4 is not supported
by your app, your support job becomes easier.

-- 
 \“Without cultural sanction, most or all of our religious |
  `\  beliefs and rituals would fall into the domain of mental |
_o__) disturbance.” —John F. Schumaker |
Ben Finney

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: why del is not a function or method?

2017-10-18 Thread Robin Becker 

On 16/10/2017 16:37, Xue Feng via Python-list wrote:

Hi,
    I wonder why 'del' is not a function or method. Most operations can be used 
as follows

     len(team)
or
     team.append("tom")
But, I think
     del team[2]
is somewhat uncommon. Why does not it take a syntax we are famillar with?

It can look like a function


x = 3
del(x)
x

Traceback (most recent call last):
  File "", line 1, in 
NameError: name 'x' is not defined





--
Robin Becker

--
https://mail.python.org/mailman/listinfo/python-list

Re: PEP 249 Compliant error handling

2017-10-18 Thread Abdur-Rahmaan Janhangeer 
all corruption systematically ignored but data piece logged in for analysis

Abdur-Rahmaan Janhangeer,
Mauritius
abdurrahmaanjanhangeer.wordpress.com

On 17 Oct 2017 21:43, "Israel Brewster"  wrote:

> I have written and maintain a PEP 249 compliant (hopefully) DB API for the
> 4D database, and I've run into a situation where corrupted string data from
> the database can cause the module to error out. Specifically, when decoding
> the string, I get a "UnicodeDecodeError: 'utf-16-le' codec can't decode
> bytes in position 86-87: illegal UTF-16 surrogate" error. This makes sense,
> given that the string data got corrupted somehow, but the question is "what
> is the proper way to deal with this in the module?" Should I just throw an
> error on bad data? Or would it be better to set the errors parameter to
> something like "replace"? The former feels a bit more "proper" to me
> (there's an error here, so we throw an error), but leaves the end user dead
> in the water, with no way to retrieve *any* of the data (from that row at
> least, and perhaps any rows after it as well). The latter option sort of
> feels like sweeping the problem under the rug, but does at least leave an
> error character in the string to l
>  et them know there was an error, and will allow retrieval of any good
> data.
>
> Of course, if this was in my own code I could decide on a case-by-case
> basis what the proper action is, but since this a module that has to work
> in any situation, it's a bit more complicated.
> ---
> Israel Brewster
> Systems Analyst II
> Ravn Alaska
> 5245 Airport Industrial Rd
> Fairbanks, AK 99709
> (907) 450-7293
> ---
>
>
>
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: how to read in the newsreader

2017-10-18 Thread Thomas Jollans 
On 2017-10-17 20:38, Pete Forman wrote:
> Thomas Jollans  writes:
> 
>> On 16/10/17 20:02, Pete Forman wrote:
>>> Thomas Jollans  writes:
>>>
 On 2017-10-16 08:48, Pete Forman wrote:
> Andrew Z  writes:
>
>> hmm. i did do that.  maybe just a delay.
>> I'll see how it will go tomorrow then. Thank you gents.
>>
>> On Mon, Oct 16, 2017 at 12:30 AM, Chris Angelico  
>> wrote:
>>
>>> On Mon, Oct 16, 2017 at 3:19 PM, Andrew Z  wrote:
 Michael, that's what i use too - gmail. But i get the digest only
 and can't really reply that way. i was hoping to get the
 mail.python.org list
>>>
>>> Turn off digests then. Easy!
>
> If you do stick with a digest then check your newsreader for a feature
> to expand it. Then you can read and reply as if you were getting
> individual posts.
>
 That exists? How does it work?
>>>
>>> The Gnus newsreader in Emacs allows you to type C-d on a digest to run
>>> gnus-summary-enter-digest-group. That then behaves the same as if you
>>> opened any other summary such as a regular Usenet group.
>>>
>>
>> Does it set the References header correctly when replying?
> 
> Sorry, I am not in a position to test. The only digest I subscribe to is
> comp.risks. The only messsage id in that is a single one for the whole
> digest. Each article only has date, subject and from headers. You would
> need to look inside a Python digest to see if it carries more headers
> for the articles. If they are not present then they cannot be used when
> composing a reply.
> 

I see. I've never subscribed to anything as a digest so I don't really
know what they look like, but judging from
https://mail.python.org/pipermail/python-list/2017-October/727421.html
(section of the digest quoted at the bottom) it *could* work.


-- 
Thomas Jollans
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: right list for SIGABRT python binary question ?

2017-10-18 Thread Karsten Hilbert 
OK, here's the first bit which might be part of the puzzle of
why my Python script SIGABRT's.

When run under "normal" py2.7 it runs all the way through but
upon shutdown (*after* sys.exit(0)) faulthandler shows a
problem (and returns 134 which made me think of SIGABRT):

*** Error in `python': free(): invalid pointer: 0x00770b14 ***
=== Backtrace: =
/lib/i386-linux-gnu/libc.so.6(+0x6738a)[0xb7ccc38a]
/lib/i386-linux-gnu/libc.so.6(+0x6dfc7)[0xb7cd2fc7]
/lib/i386-linux-gnu/libc.so.6(+0x6e806)[0xb7cd3806]
python(PyObject_Free+0xfb)[0x44affb]
python(+0x110b51)[0x534b51]
python(+0x110b51)[0x534b51]
python(+0x110b51)[0x534b51]
python(+0xf3ea7)[0x517ea7]
python(PyDict_SetItem+0x201)[0x4d4f81]
python(_PyModule_Clear+0x150)[0x539630]
python(PyImport_Cleanup+0x61d)[0x53927d]
python(Py_Finalize+0x9d)[0x53635d]
python(Py_Exit+0x14)[0x55cb24]
python(+0x136102)[0x55a102]
python(PyErr_PrintEx+0x35)[0x559975]
python(+0x3dd41)[0x461d41]
python(Py_Main+0x753)[0x4d2c83]
python(main+0x1b)[0x4d251b]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf6)[0xb7c7d286]
python(+0xae3f3)[0x4d23f3]
=== Memory map: 
00424000-00763000 r-xp  08:01 6285092/usr/bin/python2.7
00763000-00764000 r--p 0033e000 08:01 6285092/usr/bin/python2.7
00764000-007c4000 rw-p 0033f000 08:01 6285092/usr/bin/python2.7
007c4000-007d9000 rw-p  00:00 0 
026f1000-02921000 rw-p  00:00 0  [heap]
b680-b6821000 rw-p  00:00 0 
b6821000-b690 ---p  00:00 0 
b6995000-b69b r-xp  08:01 8151085
/lib/i386-linux-gnu/libgcc_s.so.1
b69b-b69b1000 r--p 0001a000 08:01 8151085
/lib/i386-linux-gnu/libgcc_s.so.1
b69b1000-b69b2000 rw-p 0001b000 08:01 8151085
/lib/i386-linux-gnu/libgcc_s.so.1
b69f5000-b6b35000 rw-p  00:00 0 
b6b35000-b6b4 r-xp  08:01 8151337
/lib/i386-linux-gnu/libnss_files-2.24.so
b6b4-b6b41000 r--p a000 08:01 8151337
/lib/i386-linux-gnu/libnss_files-2.24.so
b6b41000-b6b42000 rw-p b000 08:01 8151337
/lib/i386-linux-gnu/libnss_files-2.24.so
b6b42000-b6b48000 rw-p  00:00 0 
b6b48000-b6b53000 r-xp  08:01 8151339
/lib/i386-linux-gnu/libnss_nis-2.24.so
b6b53000-b6b54000 r--p a000 08:01 8151339
/lib/i386-linux-gnu/libnss_nis-2.24.so
b6b54000-b6b55000 rw-p b000 08:01 8151339
/lib/i386-linux-gnu/libnss_nis-2.24.so
b6b64000-b6b6b000 r--p  08:01 6286752
/usr/share/locale/de/LC_MESSAGES/libpq5-10.mo
b6b6b000-b6b72000 r--s  08:01 6903281
/usr/lib/i386-linux-gnu/gconv/gconv-modules.cache
b6b72000-b6b98000 r--p  08:01 6288754
/usr/share/locale/de/LC_MESSAGES/libc.mo
b6b98000-b6c98000 rw-p  00:00 0 
b6c9e000-b6cb4000 r-xp  08:01 8151334
/lib/i386-linux-gnu/libnsl-2.24.so
b6cb4000-b6cb5000 r--p 00016000 08:01 8151334
/lib/i386-linux-gnu/libnsl-2.24.so
b6cb5000-b6cb6000 rw-p 00017000 08:01 8151334
/lib/i386-linux-gnu/libnsl-2.24.so
b6cb6000-b6cb8000 rw-p  00:00 0 
b6cb8000-b6cc r-xp  08:01 8151335
/lib/i386-linux-gnu/libnss_compat-2.24.so
b6cc-b6cc1000 r--p 7000 08:01 8151335
/lib/i386-linux-gnu/libnss_compat-2.24.so
b6cc1000-b6cc2000 rw-p 8000 08:01 8151335
/lib/i386-linux-gnu/libnss_compat-2.24.so
b6cc2000-b6d02000 rw-p  00:00 0 
b6d02000-b6d09000 r-xp  08:01 6899491
/usr/lib/i386-linux-gnu/libffi.so.6.0.4
b6d09000-b6d0a000 r--p 6000 08:01 6899491
/usr/lib/i386-linux-gnu/libffi.so.6.0.4
b6d0a000-b6d0b000 rw-p 7000 08:01 6899491
/usr/lib/i386-linux-gnu/libffi.so.6.0.4
b6d0b000-b6d96000 r-xp  08:01 6899509
/usr/lib/i386-linux-gnu/libgmp.so.10.3.2
b6d96000-b6d97000 r--p 0008a000 08:01 6899509
/usr/lib/i386-linux-gnu/libgmp.so.10.3.2
b6d97000-b6d98000 rw-p 0008b000 08:01 6899509
/usr/lib/i386-linux-gnu/libgmp.so.10.3.2
b6d98000-b6dcc000 r-xp  08:01 6899139
/usr/lib/i386-linux-gnu/libhogweed.so.4.3
b6dcc000-b6dcd000 r--p 00033000 08:01 6899139
/usr/lib/i386-linux-gnu/libhogweed.so.4.3
b6dcd000-b6dce000 rw-p 00034000 08:01 6899139
/usr/lib/i386-linux-gnu/libhogweed.so.4.3
b6dce000-b6e08000 r-xp  08:01 6897991
/usr/lib/i386-linux-gnu/libnettle.so.6.3
b6e08000-b6e09000 ---p 0003a000 08:01 6897991
/usr/lib/i386-linux-gnu/libnettle.so.6.3
b6e09000-b6e0a000 r--p 0003a000 08:01 6897991
/usr/lib/i386-linux-gnu/libnettle.so.6.3
b6e0a000-b6e0b000 rw-p 0003b000 08:01 6897991
/u

Re: multiprocessing shows no benefit

2017-10-18 Thread Jason 
I've read the docs several times, but I still have questions.
I've even used multiprocessing before, but not map() from it.

I am not sure if map() will let me use a common object (via a manager) and if 
so, how to set that up.





-- 
https://mail.python.org/mailman/listinfo/python-list

Re: multiprocessing shows no benefit

2017-10-18 Thread Thomas Nyberg 
On 10/18/2017 05:10 PM, Jason wrote:
> I've read the docs several times, but I still have questions.
> I've even used multiprocessing before, but not map() from it.
> 
> I am not sure if map() will let me use a common object (via a manager) and if 
> so, how to set that up.
> 

As I said earlier, you should really post the code in question. Fake any
data you need so that the problem is computationally the same. Post some
file `script.py` so that running `python script.py` works without errors
(other than e.g. possible import errors from numpy and stuff like that).
This will make it much more likely that someone will be able to help you.

Cheers,
Thomas
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: multiprocessing shows no benefit

2017-10-18 Thread Jason 
#When I change line19 to True to use the multiprocessing stuff it all slows 
down. 

from multiprocessing import Process, Manager, Pool, cpu_count
from timeit import default_timer as timer

def f(a,b):
return dict_words[a]-b

def f_unpack(args):
return f(*args)

def init():
global dict_words, dict_keys
d = {str(k):k for k in range(10)}
for k in d:
dict_words[k] = d[k]
dict_keys.append(k)

if __name__ == '__main__':
print 'CPUs:', cpu_count() # 4 on my machine
if False: # make this a zero to use PODs
'''
CPUs: 4
pool.map: 47.1494848728 100 21209.1394571
map: 49.2051260471 100 20323.0858314
'''
manager = Manager()
dict_words = manager.dict()
dict_keys = manager.list()
else:
'''
CPUs: 4
pool.map: 3.2546248436 100 307255.074872
map: 1.19043779373 100 840027.093617
'''
dict_words = {}
dict_keys = []

init()

pool = Pool(cpu_count())
z=5
funcs = {'pool.map:': pool.map, 'map': map}
for name in funcs:
start = timer()
x = funcs[name](f_unpack, [(k,z) for k in dict_keys])
duration = float(timer() -start)
print funcs[name], duration, len(x), len(x) / duration
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: multiprocessing shows no benefit

2017-10-18 Thread Ian Kelly 
On Wed, Oct 18, 2017 at 9:46 AM, Jason  wrote:
> #When I change line19 to True to use the multiprocessing stuff it all slows 
> down.
>
> from multiprocessing import Process, Manager, Pool, cpu_count
> from timeit import default_timer as timer
>
> def f(a,b):
> return dict_words[a]-b

Since the computation is so simple my suspicion is that the run time
is dominated by IPC, in other words the cost of sending objects back
and forth outweighs the gains you get from parallelization.

What happens if you remove dict_words from the Manager and just pass
dict_words[a] across instead of just a? Also, I'm not sure why
dict_keys is a managed list to begin with since it only appears to be
handled by the main process.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: PEP 249 Compliant error handling

2017-10-18 Thread Israel Brewster 


> On Oct 18, 2017, at 1:46 AM, Abdur-Rahmaan Janhangeer  
> wrote:
> 
> all corruption systematically ignored but data piece logged in for analysis

Thanks. Can you expound a bit on what you mean by "data piece logged in" in 
this context? I'm not aware of any logging specifications in the PEP 249, and 
would think that would be more end-user configured rather than module level.

---
Israel Brewster
Systems Analyst II
Ravn Alaska
5245 Airport Industrial Rd
Fairbanks, AK 99709
(907) 450-7293
---

> 
> Abdur-Rahmaan Janhangeer,
> Mauritius
> abdurrahmaanjanhangeer.wordpress.com 
> 
> 
> On 17 Oct 2017 21:43, "Israel Brewster"  > wrote:
> I have written and maintain a PEP 249 compliant (hopefully) DB API for the 4D 
> database, and I've run into a situation where corrupted string data from the 
> database can cause the module to error out. Specifically, when decoding the 
> string, I get a "UnicodeDecodeError: 'utf-16-le' codec can't decode bytes in 
> position 86-87: illegal UTF-16 surrogate" error. This makes sense, given that 
> the string data got corrupted somehow, but the question is "what is the 
> proper way to deal with this in the module?" Should I just throw an error on 
> bad data? Or would it be better to set the errors parameter to something like 
> "replace"? The former feels a bit more "proper" to me (there's an error here, 
> so we throw an error), but leaves the end user dead in the water, with no way 
> to retrieve *any* of the data (from that row at least, and perhaps any rows 
> after it as well). The latter option sort of feels like sweeping the problem 
> under the rug, but does at least leave an error character in the string to
  l
>  et them know there was an error, and will allow retrieval of any good data.
> 
> Of course, if this was in my own code I could decide on a case-by-case basis 
> what the proper action is, but since this a module that has to work in any 
> situation, it's a bit more complicated.
> ---
> Israel Brewster
> Systems Analyst II
> Ravn Alaska
> 5245 Airport Industrial Rd
> Fairbanks, AK 99709
> (907) 450-7293 
> ---
> 
> 
> 
> 
> --
> https://mail.python.org/mailman/listinfo/python-list 
> 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: multiprocessing shows no benefit

2017-10-18 Thread Jason 
On Wednesday, October 18, 2017 at 12:14:30 PM UTC-4, Ian wrote:
> On Wed, Oct 18, 2017 at 9:46 AM, Jason  wrote:
> > #When I change line19 to True to use the multiprocessing stuff it all slows 
> > down.
> >
> > from multiprocessing import Process, Manager, Pool, cpu_count
> > from timeit import default_timer as timer
> >
> > def f(a,b):
> > return dict_words[a]-b
> 
> Since the computation is so simple my suspicion is that the run time
> is dominated by IPC, in other words the cost of sending objects back
> and forth outweighs the gains you get from parallelization.
> 
> What happens if you remove dict_words from the Manager and just pass
> dict_words[a] across instead of just a? Also, I'm not sure why
> dict_keys is a managed list to begin with since it only appears to be
> handled by the main process.

You can try that code by changing line 17 :-) It's 10 times faster.

Well given the many objects to be iterated over, I was hoping pool.map() would 
distribute them across the processors. So that each processor gets 
len(dict_words)/cpu_count() items to process. The actual computation is much 
longer than a single subtraction, currently I can process about 16k items per 
second single core.  My target is to get those 16k items processed in .25s.



-- 
https://mail.python.org/mailman/listinfo/python-list

Re: multiprocessing shows no benefit

2017-10-18 Thread Ian Kelly 
On Wed, Oct 18, 2017 at 10:13 AM, Ian Kelly  wrote:
> On Wed, Oct 18, 2017 at 9:46 AM, Jason  wrote:
>> #When I change line19 to True to use the multiprocessing stuff it all slows 
>> down.
>>
>> from multiprocessing import Process, Manager, Pool, cpu_count
>> from timeit import default_timer as timer
>>
>> def f(a,b):
>> return dict_words[a]-b
>
> Since the computation is so simple my suspicion is that the run time
> is dominated by IPC, in other words the cost of sending objects back
> and forth outweighs the gains you get from parallelization.
>
> What happens if you remove dict_words from the Manager and just pass
> dict_words[a] across instead of just a? Also, I'm not sure why
> dict_keys is a managed list to begin with since it only appears to be
> handled by the main process.

Timings from my system:

# Original code without using Manager
$ python test.py
CPUs: 12
 0.0757319927216 10 1320445.9094
> 0.143120765686 10 698710.627495

# Original code with Manager
$ python test.py
CPUs: 12
 5.5354039669 10 18065.5288391
> 4.3253660202 10 23119.4307101

# Modified code without Manager and avoiding sharing the dict
$ python test.py
CPUs: 12
 0.0657241344452 10 1521511.09854
> 0.0966320037842 10 1034853.83811
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: multiprocessing shows no benefit

2017-10-18 Thread Ian Kelly 
On Wed, Oct 18, 2017 at 10:21 AM, Jason  wrote:
> On Wednesday, October 18, 2017 at 12:14:30 PM UTC-4, Ian wrote:
>> On Wed, Oct 18, 2017 at 9:46 AM, Jason  wrote:
>> > #When I change line19 to True to use the multiprocessing stuff it all 
>> > slows down.
>> >
>> > from multiprocessing import Process, Manager, Pool, cpu_count
>> > from timeit import default_timer as timer
>> >
>> > def f(a,b):
>> > return dict_words[a]-b
>>
>> Since the computation is so simple my suspicion is that the run time
>> is dominated by IPC, in other words the cost of sending objects back
>> and forth outweighs the gains you get from parallelization.
>>
>> What happens if you remove dict_words from the Manager and just pass
>> dict_words[a] across instead of just a? Also, I'm not sure why
>> dict_keys is a managed list to begin with since it only appears to be
>> handled by the main process.
>
> You can try that code by changing line 17 :-) It's 10 times faster.
>
> Well given the many objects to be iterated over, I was hoping pool.map() 
> would distribute them across the processors. So that each processor gets 
> len(dict_words)/cpu_count() items to process. The actual computation is much 
> longer than a single subtraction, currently I can process about 16k items per 
> second single core.  My target is to get those 16k items processed in .25s.

Well, in any case I would try keeping the communication to a minimum.
Instead of sending 1 item at a time, break the data set up into
batches of 4k and send each child task a batch. Avoid using managers
or other shared state.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: PEP 249 Compliant error handling

2017-10-18 Thread Israel Brewster 
On Oct 17, 2017, at 12:02 PM, MRAB  wrote:
> 
> On 2017-10-17 20:25, Israel Brewster wrote:
>> 
>>> On Oct 17, 2017, at 10:35 AM, MRAB >> > wrote:
>>> 
>>> On 2017-10-17 18:26, Israel Brewster wrote:
 I have written and maintain a PEP 249 compliant (hopefully) DB API for the 
 4D database, and I've run into a situation where corrupted string data 
 from the database can cause the module to error out. Specifically, when 
 decoding the string, I get a "UnicodeDecodeError: 'utf-16-le' codec can't 
 decode bytes in position 86-87: illegal UTF-16 surrogate" error. This 
 makes sense, given that the string data got corrupted somehow, but the 
 question is "what is the proper way to deal with this in the module?" 
 Should I just throw an error on bad data? Or would it be better to set the 
 errors parameter to something like "replace"? The former feels a bit more 
 "proper" to me (there's an error here, so we throw an error), but leaves 
 the end user dead in the water, with no way to retrieve *any* of the data 
 (from that row at least, and perhaps any rows after it as well). The 
 latter option sort of feels like sweeping the problem under the rug, but 
 does at least leave an error character in the s
>>> tring to
>>> l
  et them know there was an error, and will allow retrieval of any good 
 data.
 Of course, if this was in my own code I could decide on a case-by-case 
 basis what the proper action is, but since this a module that has to work 
 in any situation, it's a bit more complicated.
>>> If a particular text field is corrupted, then raising UnicodeDecodeError 
>>> when trying to get the contents of that field as a Unicode string seems 
>>> reasonable to me.
>>> 
>>> Is there a way to get the contents as a bytestring, or to get the contents 
>>> with a different errors parameter, so that the user has the means to fix it 
>>> (if it's fixable)?
>> 
>> That's certainly a possibility, if that behavior conforms to the DB API 
>> "standards". My concern in this front is that in my experience working with 
>> other PEP 249 modules (specifically psycopg2), I'm pretty sure that columns 
>> designated as type VARCHAR or TEXT are returned as strings (unicode in 
>> python 2, although that may have been a setting I used), not bytes. The 
>> other complication here is that the 4D database doesn't use the UTF-8 
>> encoding typically found, but rather UTF-16LE, and I don't know how well 
>> this is documented. So not only is the bytes representation completely 
>> unintelligible for human consumption, I'm not sure the average end-user 
>> would know what decoding to use.
>> 
>> In the end though, the main thing in my mind is to maintain "standards" 
>> compatibility - I don't want to be returning bytes if all other DB API 
>> modules return strings, or visa-versa for that matter. There may be some 
>> flexibility there, but as much as possible I want to conform to the 
>> majority/standard/whatever
>> 
> The average end-user might not know which encoding is being used, but 
> providing a way to read the underlying bytes will give a more experienced 
> user the means to investigate and possibly fix it: get the bytes, figure out 
> what the string should be, update the field with the correctly decoded string 
> using normal DB instructions.

I agree, and if I was just writing some random module I'd probably go with it, 
or perhaps with the suggestion offered by Karsten Hilbert. However, neither 
answer addresses my actual question, which is "how does the STANDARD (PEP 249 
in this case) say to handle this, or, baring that (since the standard probably 
doesn't explicitly say), how do the MAJORITY of PEP 249 compliant modules 
handle this?" Not what is the *best* way to handle it, but rather what is the 
normal, expected behavior for a Python DB API module when presented with bad 
data? That is, how does psycopg2 behave? pyodbc? pymssql (I think)? Etc. Or is 
that portion of the behavior completely arbitrary and different for every 
module?

It may well be that one of the suggestions *IS* the normal, expected, behavior, 
but it sounds more like you are suggesting how you think would be best to 
handle it, which is appreciated but not actually what I'm asking :-) Sorry if I 
am being difficult.

> -- 
> https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: multiprocessing shows no benefit

2017-10-18 Thread Jason 
I refactored the map call to break dict_keys into cpu_count() chunks, (so each 
f() call gets to run continuously over n/cpu_count() items) virtually the same 
results. pool map is much slower (4x)  than regular map, and I don't know why. 


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: PEP 249 Compliant error handling

2017-10-18 Thread Thomas Jollans 
On 17/10/17 19:26, Israel Brewster wrote:
> I have written and maintain a PEP 249 compliant (hopefully) DB API for the 4D 
> database, and I've run into a situation where corrupted string data from the 
> database can cause the module to error out. Specifically, when decoding the 
> string, I get a "UnicodeDecodeError: 'utf-16-le' codec can't decode bytes in 
> position 86-87: illegal UTF-16 surrogate" error. This makes sense, given that 
> the string data got corrupted somehow, but the question is "what is the 
> proper way to deal with this in the module?" Should I just throw an error on 
> bad data? Or would it be better to set the errors parameter to something like 
> "replace"? The former feels a bit more "proper" to me (there's an error here, 
> so we throw an error), but leaves the end user dead in the water, with no way 
> to retrieve *any* of the data (from that row at least, and perhaps any rows 
> after it as well). The latter option sort of feels like sweeping the problem 
> under the rug, but does at least leave an error character in the string to l
>  et them know there was an error, and will allow retrieval of any good data.
> 
> Of course, if this was in my own code I could decide on a case-by-case basis 
> what the proper action is, but since this a module that has to work in any 
> situation, it's a bit more complicated.


The sqlite3 module falls back to returning bytes if there's a decoding
error. I don't know what the other modules do. It should be easy enough
for you to test this, though!

Python 3.5.3 (default, Jan 19 2017, 14:11:04)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.1.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import sqlite3

In [2]: db = sqlite3.connect("malformed-test1.sqlite3")

In [3]: db.execute("CREATE TABLE test (txt TEXT)")
Out[3]: 

In [4]: db.execute("INSERT INTO test VALUES(?)", ("utf-8: é",))
Out[4]: 

In [5]: db.execute("INSERT INTO test VALUES(?)", ("latin1:
é".encode('latin1'),))
Out[5]: 

In [6]: db.execute("SELECT * FROM test").fetchall()
Out[6]: [('utf-8: é',), (b'latin1: \xe9',)]

In [7]: db.text_factory = bytes # sqlite3 extension to the API

In [8]: db.execute("SELECT * FROM test").fetchall()
Out[8]: [(b'utf-8: \xc3\xa9',), (b'latin1: \xe9',)]


For what it's worth, this is also what os.listdir() does when it
encounters filenames in the wrong encoding on operating systems where
this is possible (e.g. Linux, but not Windows)

If the encoding could be anything, I think you should give the user some
kind of choice between using bytes, raising errors, and escaping.

In the particular case of UTF-16 (especially if the encoding is always
UTF-16), the best solution is almost certainly to use
errors='surrogatepass' in both en- and decoding. I believe this is
fairly common practice when full interoperability with software that
predates UTF-16 (and previously used UCS-2) is required. This should
solve all your problems as long as you don't get strings with an odd
number of bytes.

See: https://en.wikipedia.org/wiki/UTF-16#U.2BD800_to_U.2BDFFF

-- Thomas

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Searching For Old Posts On Python

2017-10-18 Thread Ian Kelly 
On Mon, Oct 16, 2017 at 10:42 PM, Cai Gengyang  wrote:
> https://duckduckgo.com/html/?q=%22Cai%20Gengyang%22%20python  This
> seems to be the only method that works, using DuckDuckGo

Or any search engine, really.
https://www.google.com/search?q=site%3Apython.org+cai+gengyang
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: PEP 249 Compliant error handling

2017-10-18 Thread Karsten Hilbert 
On Wed, Oct 18, 2017 at 08:32:48AM -0800, Israel Brewster wrote:

> actual question, which is "how does the STANDARD (PEP 249 in
> this case) say to handle this, or, baring that (since the
> standard probably doesn't explicitly say), how do the
> MAJORITY of PEP 249 compliant modules handle this?" Not what
> is the *best* way to handle it, but rather what is the
> normal, expected behavior for a Python DB API module when
> presented with bad data? That is, how does psycopg2 behave?
> pyodbc? pymssql (I think)? Etc. Or is that portion of the
> behavior completely arbitrary and different for every module?

For what it is worth, psycopg2 does not give you bad data to
the best of my knowledge. In fact, given PostgreSQL's quite
tight checking of text data to be "good" psycopg2 hardly has
a chance to give you bad data. Most times the database itself
will detect the corruption and not even hand the data to
psycopg2.

IMHO a driver should not hand over to the client any bad data
unless explicitely told to do so, which latter case does not
seem to be covered by the DB-API specs, regardless of what
the majority of drivers might do these days.

2 cent...

Karsten
-- 
GPG key ID E4071346 @ eu.pool.sks-keyservers.net
E167 67FD A291 2BEA 73BD  4537 78B9 A9F9 E407 1346
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: how to read in the newsreader

2017-10-18 Thread Cholo Lennon 

On 16/10/17 01:00, Michael Torrie wrote:

On 10/15/2017 08:50 PM, Andrew Z wrote:

Gents,
  how do i get this group in a newsreader? The digest i'm getting is not
workable for me - i can't reply , can only read the replies from the
members of the group. Or. maybe, it shouldn't be a news reader
please advise..

P.S. Oh the comp.lang.python is a nightmare because of spam...


Regardless of what usenet reader you use, com.lang.python will have the
same spam everywhere.  So maybe reading via usenet isn't that useful anyway.


I use Mozilla Thunderbird as a newsreader. I am subscribed to 
'comp.lang.python' through the free NNTP server 'news.aioe.org'. My spam 
filter for python only has 6 address!




I just subscribe to the mailing list and then have a rule in my email
filter to stick all list messages in their own folder (or gmail label).
I'm using gmail, so I just use gmail's filtering settings.  Then I view
email using IMAP and a conventional email reader that supports real
threading of messages.  Works great.  A lot of spam is filtered out this
way, also.  In fact very little spam gets through to my folder between
python.org's filtering and gmail's spam detection.

I'm subscribed to maybe a dozen email lists, and I filter all of them
into their own folders.



In my case I am subscribed (with Thunderbird) to more than 30 newsgroups 
(in 3 news servers). I have to say that spam is not a problem.


Regards


--
Cholo Lennon
Bs.As.
ARG
--
https://mail.python.org/mailman/listinfo/python-list

Re: PEP 249 Compliant error handling

2017-10-18 Thread Cameron Simpson 

On 17Oct2017 21:38, Karsten Hilbert  wrote:

That's certainly a possibility, if that behavior conforms to the DB API 
"standards". My concern in this front is that in my experience working with 
other PEP 249 modules (specifically psycopg2), I'm pretty sure that columns designated as 
type VARCHAR or TEXT are returned as strings (unicode in python 2, although that may have 
been a setting I used), not bytes. The other complication here is that the 4D database 
doesn't use the UTF-8 encoding typically found, but rather UTF-16LE, and I don't know how 
well this is documented. So not only is the bytes representation completely 
unintelligible for human consumption, I'm not sure the average end-user would know what 
decoding to use.

In the end though, the main thing in my mind is to maintain "standards" 
compatibility - I don't want to be returning bytes if all other DB API modules return 
strings, or visa-versa for that matter. There may be some flexibility there, but as much 
as possible I want to conform to the majority/standard/whatever



The thing here is that you don't want to return data AS IF it was correct 
despite it having been
"corrected" by some driver logic.


I just want to say that I think this is correct and extremely important.


What might be interesting to users is to set an attribute on the cursor, say,
  cursor.faulty_data = unicode(faulty_data, errors='replace')
or some such in order to improve error messages to the end user.


Or perhaps more conveniently for the end user, possibly an option supplied at 
db connect time, though I'd entirely understand wanting a cursor option so that 
one can pick and choose in a fine grained fashion.


Cheers,
Cameron Simpson  (formerly c...@zip.com.au)
--
https://mail.python.org/mailman/listinfo/python-list

How to debug an unfired tkinter event?

2017-10-18 Thread jfong 
In last few days, I tried to experiment with the scrolling table implemented in 
canvas, started from this example: 
http://code.activestate.com/recipes/580793-tkinter-table-with-scrollbars/. 
Everything works fine until I moved the scrolling_area instance (which the 
canvas is in) from column=0 to column=1.

The canvas has a binding:
self.canvas.bind('', self._on_canvas_configure)

After this movement, the callback was only triggered when dragging the root 
widget to resize canvas vertically, but not horizontally.

Event seems has OS involved(it's MS Windows XP in my case). How to debug this 
kind of problem, under pdb?

Best Regards,
Jach Fong
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: right list for SIGABRT python binary question ?

2017-10-18 Thread Thomas Jollans 
On 2017-10-18 13:43, Karsten Hilbert wrote:
> OK, here's the first bit which might be part of the puzzle of
> why my Python script SIGABRT's.
> 
> When run under "normal" py2.7 it runs all the way through but
> upon shutdown (*after* sys.exit(0)) faulthandler shows a
> problem (and returns 134 which made me think of SIGABRT):
> 
>   *** Error in `python': free(): invalid pointer: 0x00770b14 ***
>   === Backtrace: =
>   /lib/i386-linux-gnu/libc.so.6(+0x6738a)[0xb7ccc38a]
>   /lib/i386-linux-gnu/libc.so.6(+0x6dfc7)[0xb7cd2fc7]
>   /lib/i386-linux-gnu/libc.so.6(+0x6e806)[0xb7cd3806]
>   python(PyObject_Free+0xfb)[0x44affb]
>   python(+0x110b51)[0x534b51]
>   python(+0x110b51)[0x534b51]
>   python(+0x110b51)[0x534b51]
>   python(+0xf3ea7)[0x517ea7]
>   python(PyDict_SetItem+0x201)[0x4d4f81]
>   python(_PyModule_Clear+0x150)[0x539630]
>   python(PyImport_Cleanup+0x61d)[0x53927d]
>   python(Py_Finalize+0x9d)[0x53635d]
>   python(Py_Exit+0x14)[0x55cb24]
>   python(+0x136102)[0x55a102]
>   python(PyErr_PrintEx+0x35)[0x559975]
>   python(+0x3dd41)[0x461d41]
>   python(Py_Main+0x753)[0x4d2c83]
>   python(main+0x1b)[0x4d251b]
>   /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf6)[0xb7c7d286]
>   python(+0xae3f3)[0x4d23f3]
>   === Memory map: 
>   00424000-00763000 r-xp  08:01 6285092/usr/bin/python2.7
>   00763000-00764000 r--p 0033e000 08:01 6285092/usr/bin/python2.7
>   00764000-007c4000 rw-p 0033f000 08:01 6285092/usr/bin/python2.7
>   007c4000-007d9000 rw-p  00:00 0 
>   026f1000-02921000 rw-p  00:00 0  [heap]
>   b680-b6821000 rw-p  00:00 0 
>   b6821000-b690 ---p  00:00 0 
>   b6995000-b69b r-xp  08:01 8151085
> /lib/i386-linux-gnu/libgcc_s.so.1
>   b69b-b69b1000 r--p 0001a000 08:01 8151085
> /lib/i386-linux-gnu/libgcc_s.so.1
>   b69b1000-b69b2000 rw-p 0001b000 08:01 8151085
> /lib/i386-linux-gnu/libgcc_s.so.1
>   b69f5000-b6b35000 rw-p  00:00 0 
>   b6b35000-b6b4 r-xp  08:01 8151337
> /lib/i386-linux-gnu/libnss_files-2.24.so
>   b6b4-b6b41000 r--p a000 08:01 8151337
> /lib/i386-linux-gnu/libnss_files-2.24.so
>   b6b41000-b6b42000 rw-p b000 08:01 8151337
> /lib/i386-linux-gnu/libnss_files-2.24.so
>   b6b42000-b6b48000 rw-p  00:00 0 
>   b6b48000-b6b53000 r-xp  08:01 8151339
> /lib/i386-linux-gnu/libnss_nis-2.24.so
>   b6b53000-b6b54000 r--p a000 08:01 8151339
> /lib/i386-linux-gnu/libnss_nis-2.24.so
>   b6b54000-b6b55000 rw-p b000 08:01 8151339
> /lib/i386-linux-gnu/libnss_nis-2.24.so
>   b6b64000-b6b6b000 r--p  08:01 6286752
> /usr/share/locale/de/LC_MESSAGES/libpq5-10.mo
>   b6b6b000-b6b72000 r--s  08:01 6903281
> /usr/lib/i386-linux-gnu/gconv/gconv-modules.cache
>   b6b72000-b6b98000 r--p  08:01 6288754
> /usr/share/locale/de/LC_MESSAGES/libc.mo
>   b6b98000-b6c98000 rw-p  00:00 0 
>   b6c9e000-b6cb4000 r-xp  08:01 8151334
> /lib/i386-linux-gnu/libnsl-2.24.so
>   b6cb4000-b6cb5000 r--p 00016000 08:01 8151334
> /lib/i386-linux-gnu/libnsl-2.24.so
>   b6cb5000-b6cb6000 rw-p 00017000 08:01 8151334
> /lib/i386-linux-gnu/libnsl-2.24.so
>   b6cb6000-b6cb8000 rw-p  00:00 0 
>   b6cb8000-b6cc r-xp  08:01 8151335
> /lib/i386-linux-gnu/libnss_compat-2.24.so
>   b6cc-b6cc1000 r--p 7000 08:01 8151335
> /lib/i386-linux-gnu/libnss_compat-2.24.so
>   b6cc1000-b6cc2000 rw-p 8000 08:01 8151335
> /lib/i386-linux-gnu/libnss_compat-2.24.so
>   b6cc2000-b6d02000 rw-p  00:00 0 
>   b6d02000-b6d09000 r-xp  08:01 6899491
> /usr/lib/i386-linux-gnu/libffi.so.6.0.4
>   b6d09000-b6d0a000 r--p 6000 08:01 6899491
> /usr/lib/i386-linux-gnu/libffi.so.6.0.4
>   b6d0a000-b6d0b000 rw-p 7000 08:01 6899491
> /usr/lib/i386-linux-gnu/libffi.so.6.0.4
>   b6d0b000-b6d96000 r-xp  08:01 6899509
> /usr/lib/i386-linux-gnu/libgmp.so.10.3.2
>   b6d96000-b6d97000 r--p 0008a000 08:01 6899509
> /usr/lib/i386-linux-gnu/libgmp.so.10.3.2
>   b6d97000-b6d98000 rw-p 0008b000 08:01 6899509
> /usr/lib/i386-linux-gnu/libgmp.so.10.3.2
>   b6d98000-b6dcc000 r-xp  08:01 6899139
> /usr/lib/i386-linux-gnu/libhogweed.so.4.3
>   b6dcc000-b6dcd000 r--p 00033000 08:01 6899139
> /usr/lib/i386-linux-gnu/libhogweed.so.4.3
>   b6dcd000-b6dce000 rw-p 00034000 08:01 6899139
> /usr/lib/i386-linux-gnu/libhogweed.so.4.3
>   b6dce000-b6e08000 r-xp  08:01 6897991
> /usr/lib/i386-linux-gnu/libnettle.so.6.3
>   b6e08000-b6e09000 ---p 0003a000 08:01 6897991
> /usr/lib/i386-linux-gnu/libnettle.so.6.3
>   b6e09000-b6e0a000 r--p 0003a000 08:

Re: Python Interview Questions

2017-10-18 Thread lingmaaki 
Hope this will help you

http://net-informations.com/python/iq/default.htm

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: right list for SIGABRT python binary question ?

2017-10-18 Thread dieter 
Karsten Hilbert  writes:
> ...
> When run under "normal" py2.7 it runs all the way through but
> upon shutdown (*after* sys.exit(0)) faulthandler shows a
> problem (and returns 134 which made me think of SIGABRT):
>
>   *** Error in `python': free(): invalid pointer: 0x00770b14 ***

This indicates some form of memory corruption, usually caused by
some C extension. Unfortunately, memory corruption is rarely noticed
locally; therefore, the localization is typically complex.

Maybe, you are lucky and the "mxdatetime" is to blame.

-- 
https://mail.python.org/mailman/listinfo/python-list