Re: Embedded python: How to debug code in an isolated way

[Barry Scott]

> On 24 Aug 2020, at 12:52, Eko palypse  wrote:
> 
> Thank you very much for your interest in my little problem.
> 
>> When the app calls into python does the event loop of the gui block?
> 
> Yes, the cpp app calls a callback function from the embedded python
> interpreter synchronously.

> 
>> When you print does that trigger the event loop to run in a nested call
> back into the cpp?
> 
> I can't say for sure but I don't think so. At least my script or the
> embedded interpreter doesn't do that.
> 
> 
>> Maybe you need to run python in its own thread and Marshall in and out
> of the gui main thread with the event loop?
> 
> I've checked by using GetCurrentThreadId
> 
> that the registered callback function and debugger run in the same thread
> but it is different to the UI thread.

> Do you think I have to run the debugger in its own thread? I'll give it a
> try anyways. :-)

No I doubt that can will work. What I was thinking is that your life was made 
more
complex by multi threads and GUI event loops.

I think that you should not attempt to use a debugger at all.

Add code to put the information you need to find the bug into
a log file.

You might want to build a feature into the GUI that will show the log file to 
you
if it not easy to get at a log file on disk.

As an aside I have never used a python debugger to fix python code.
I have always relied on logging key information to allow me to fix bugs.
I have used gdb to debug C/C++ and python.

All the non-trivial projects I work on have multi stream logging builtin.
I tend to design the logging with the aim of debugging and maintaining
code from the outset these days. When a new bug turns up I just switch on
the appropriate stream of logs to help me fix it, add new logs as necessary.
 
Barry

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: "filterwarnings"

[Peter Otten]

Stefan Ram wrote:

>   I'm not sure I understand "filterwarnings" correctly.
>   It does not suppress the warning within the "with" statement below.
>   Outside of the "with" statement, it does work, but according
>   to sources it also should work within the "with" statement.
> 
>   Console transcript:
> 
> |Python 3.9
> |Type "help", "copyright", "credits" or "license" for more information.
> |>>> import warnings
> |>>> with warnings.catch_warnings():
> |... warnings.filterwarnings("ignore", category=SyntaxWarning)
> |... print( 1 is 1 )
> |...
> |:3: SyntaxWarning: "is" with a literal. Did you mean "=="?
> |True
> |>>> warnings.filterwarnings("ignore", category=SyntaxWarning)
> |>>> print( 1 is 1 )
> |True
> 
>   How could I make "filterwarnings" work inside the "with" statement?
>   TIA!

It does work already:

>>> with warnings.catch_warnings():
... warnings.filterwarnings("ignore", category=UserWarning)
... warnings.warn("yadda")
... 
>>> warnings.warn("yadda")
:1: UserWarning: yadda

The problem is that you picked the wrong category. As SyntaxWarnings are 
issued during compilation you need to trigger a compilation to see the 
effect:

>>> with warnings.catch_warnings():
... warnings.filterwarnings("ignore", category=SyntaxWarning)
... exec("1 is 1")
... 
>>> exec("1 is 1")
:1: SyntaxWarning: "is" with a literal. Did you mean "=="?

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Output showing "None" in Terminal

[Terry Reedy]

On 8/24/2020 4:41 PM, Calvin Spealman wrote:


"None" is the default return value of all functions in Python. But, the
interpreter is supposed to suppress it as a displayed result.


In batch mode, expressions are not echoed.  In interactive move, 
expression values other than None are echoed.  The string 'None' is not 
suppressed.


Type "help", "copyright", "credits" or "license()" for more information.
>>> None  # Not echoed.
>>> str(None)  # Not echoed in batch mode.
'None'
>>> repr(None)  # Ditto.
'None'
>>> print(None)  # Always printed to valid sys.stdout.
None

--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list

stat_result.st_ino from os.stat isn't constant on a network drive.

[Andrew Nelson]

OS: windows 7
Python: 3.8

I am trying to track which files have been modified in a directory on a
network mount. When I call the os.stat function for a given file in
that directory the stat_result.st_ino changes every time I call it:

```
>>> os.stat('U:\data\current\myfile')
os.stat_result(st_mode=33206, st_ino=18446735965107825360, st_dev=33674366,
st_nlink=1, st_uid=0, st_gid=0, st_size=433874, st_atime=1598333128,
st_mtime=1598330233, st_ctime=1598329773)

>>> os.stat('U:\data\current\myfile')
os.stat_result(st_mode=33206, st_ino=18446735965183676432, st_dev=33674366,
st_nlink=1, st_uid=0, st_gid=0, st_size=433874, st_atime=1598333128,
st_mtime=1598330233, st_ctime=1598329773)
```
It's the same file, why on earth does the st_ino field change everytime?
st_ino is supposed to "uniquely identify the file for a given value of
st_dev."
If I do the same thing on a local file instead of on a network mount st_ino
is constant and doesn't change.

Is this a Python bug, or is it an issue with the way Windows deals with
network mounts?

Andrew.
-- 
https://mail.python.org/mailman/listinfo/python-list

About float/double type number in range.

[ADITYA]

   Dear Sir/Ma’am

   I am requesting you to satisfy me about float number in Range function,
   because in the argument of range we can take integer but not double or
   float whenever double as well as float are integer in nature but when we
   use double/float in, it gives error that- “'float' object cannot be
   interpreted as an integer.” If we want to increment the number by half or
   quarter what can I do.

   For ex- Range(1,3,0.5) I want it gives Output as [1 1.5 2 2.5 3)

    

   I am requesting to change the nature of increment number nature in above
   example so that we can increase the number with half or quarter any point
   value.

    

   Your Sincerely

   Aditya Gautam

   Saharsa (Bihar)

   India

   Postal Code- 852201

    

    

    

    

    

   Sent from [1]Mail for Windows 10

    

   [2][IMG] Virus-free. [3]www.avg.com

References

   Visible links
   1. https://go.microsoft.com/fwlink/?LinkId=550986
   2. 
http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient
   3. 
http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: About float/double type number in range.

[Joel Goldstick]

On Wed, Aug 26, 2020 at 3:43 AM ADITYA  wrote:
>
>Dear Sir/Ma’am
>
>I am requesting you to satisfy me about float number in Range function,
>because in the argument of range we can take integer but not double or
>float whenever double as well as float are integer in nature but when we
>use double/float in, it gives error that- “'float' object cannot be
>interpreted as an integer.” If we want to increment the number by half or
>quarter what can I do.
>
>For ex- Range(1,3,0.5) I want it gives Output as [1 1.5 2 2.5 3)
>

Take a look at this:
>>> l = [i/2 for i in range(2,7)]
>>> l
[1.0, 1.5, 2.0, 2.5, 3.0]
>>>
The first line is a list comprehension which is a convenient way to
produce a list by performing an operation on each element in range.

In your code, you set the end of the range to 3 which will not include
3.  It stops at 2.
>
>
>I am requesting to change the nature of increment number nature in above
>example so that we can increase the number with half or quarter any point
>value.
>
   if you want to increase by 1/4, use a range that is 4 times normal
integers. and divide each element by 4
>
>Your Sincerely
>
>Aditya Gautam
>
>Saharsa (Bihar)
>
>India
>
>Postal Code- 852201
>
>
>
>
>
>
>
>
>
>
>
>Sent from [1]Mail for Windows 10
>
>
>
>[2][IMG] Virus-free. [3]www.avg.com
>
> References
>
>Visible links
>1. https://go.microsoft.com/fwlink/?LinkId=550986
>2. 
> http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient
>3. 
> http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient
> --
> https://mail.python.org/mailman/listinfo/python-list



-- 
Joel Goldstick
http://joelgoldstick.com/blog
http://cc-baseballstats.info/stats/birthdays
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: stat_result.st_ino from os.stat isn't constant on a network drive.

[Eryk Sun]

On 8/25/20, Andrew Nelson  wrote:
>
> st_ino is supposed to "uniquely identify the file for a given value of
> st_dev."

That assurance only applies to POSIX systems, not Windows.

st_ino is the file ID in Windows. Some filesystems do not support
reliable file IDs, if any at all. I would only rely on the st_ino
value if the filesystem flags include FILE_SUPPORTS_OPEN_BY_FILE_ID.
You can get the filesystem flags via GetVolumeInformationByHandleW,
using a handle for any file in the filesystem.

For example, FAT filesystems use a semi-stable file ID, which is based
on a file's byte offset in the parent directory. This offset can can
change if the filesystem is defragmented, so FAT file IDs are
unreliable. This is probably why FAT filesystems do not support
opening by file ID. Another example is the WebDAV filesystem, which
doesn't support file IDs at all, so the value is 0 for all files.

st_dev is the volume serial number (VSN) in Windows. Like the file ID,
filesystems are not required to support a VSN, in which case it may be
0 (e.g. it's 0 for WebDAV). Even if the VSN is supported, Windows does
not guarantee that the value is unique. It's a random 32-bit number,
so it's unlikely that a non-zero value isn't unique, but there's no
hard guarantee.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: About float/double type number in range.

[dn via Python-list]

On 26/08/2020 19:58, Joel Goldstick wrote:

On Wed, Aug 26, 2020 at 3:43 AM ADITYA  wrote:


Dear Sir/Ma’am

I am requesting you to satisfy me about float number in Range function,
because in the argument of range we can take integer but not double or
float whenever double as well as float are integer in nature but when we
use double/float in, it gives error that- “'float' object cannot be
interpreted as an integer.” If we want to increment the number by half or
quarter what can I do.

For ex- Range(1,3,0.5) I want it gives Output as [1 1.5 2 2.5 3)



Take a look at this:

l = [i/2 for i in range(2,7)]
l

[1.0, 1.5, 2.0, 2.5, 3.0]



This is a neat solution! To make it more user-friendly, the above needs 
to be wrapped in a convenience function that the user can call using 
arguments like "(1,3,0.5)" and not have to think about the 'multiplies' 
and 'divides'.


The problem with constructing such a list is that it is 'expensive', ie 
if the range is large, more storage-space will be required. In the same 
way that range() moved to a storage-averse model in Python3, herewith a 
pair of generator solutions - it might be quite fun to compare the 
performances (PSL's time library).


My first thought was that the count() from the itertools library enables 
one to step using a floating-point number. So, herewith two 
generator-based solutions:-



<<< Code NB Python v3.8 >>>
def fp_range( start:float, stop:float, step:float=1.0 )->float:
"""Generate a range of floating-point numbers."""
if stop <= start:
raise OverflowError( "RangeError: start must be less than stop" )
x = start
while x < stop:
yield x
x += step

try:
for fp_number in fp_range( 1, 3, 0.5 ):
print( fp_number )
except OverflowError as e:
print( e )

print( "*"*12 )

try:
for fp_number in fp_range( 1, 3, 0.25 ):
print( fp_number )
except OverflowError as e:
print( e )

print( "+"*12 )
import itertools

def fp_iter( start:float, stop:float, step:float=1.0 )->float:
"""Generate a range of floating-point numbers using 
itertools.count()."""

if stop <= start:
raise OverflowError( "RangeError: start must be less than stop" )
for x in itertools.count( start, step ):
if x >= stop:
break
yield x


for fp_number in fp_iter( 1, 3, 0.5 ):
print( fp_number )

print( "*"*12 )

for fp_number in fp_iter( 1, 3, 0.25 ):
print( fp_number )

print( "*"*12 )

try:
for fp_number in fp_iter( 3, 1, 0.5 ):
print( fp_number )
except OverflowError as e:
print( e )


<<< Console.out >>>

1
1.5
2.0
2.5

1
1.25
1.5
1.75
2.0
2.25
2.5
2.75

1
1.5
2.0
2.5

1
1.25
1.5
1.75
2.0
2.25
2.5
2.75

RangeError: start must be less than stop

--
Regards =dn
--
https://mail.python.org/mailman/listinfo/python-list

Re: About float/double type number in range.

[Peter J. Holzer]

On 2020-08-26 22:40:36 +1200, dn via Python-list wrote:
> On 26/08/2020 19:58, Joel Goldstick wrote:
> > On Wed, Aug 26, 2020 at 3:43 AM ADITYA  wrote:
> > > Dear Sir/Ma’am
> > > 
> > > I am requesting you to satisfy me about float number in Range
> > > function, because in the argument of range we can take integer
> > > but not double or float whenever double as well as float are
> > > integer in nature but when we use double/float in, it gives
> > > error that- “'float' object cannot be interpreted as an
> > > integer.” If we want to increment the number by half or
> > > quarter what can I do.
> > > 
> > > For ex- Range(1,3,0.5) I want it gives Output as [1 1.5 2 2.5 3)
> > > 
> > 
> > Take a look at this:
> > > > > l = [i/2 for i in range(2,7)]
> > > > > l
> > [1.0, 1.5, 2.0, 2.5, 3.0]
> 
> 
> This is a neat solution! To make it more user-friendly, the above needs to
> be wrapped in a convenience function that the user can call using arguments
> like "(1,3,0.5)" and not have to think about the 'multiplies' and 'divides'.
> 
[...]
> <<< Code NB Python v3.8 >>>
> def fp_range( start:float, stop:float, step:float=1.0 )->float:
> """Generate a range of floating-point numbers."""
> if stop <= start:
> raise OverflowError( "RangeError: start must be less than stop" )
> x = start
> while x < stop:
> yield x
> x += step

This is almost the same solution as I came up with, but note that this
is not quote the same as what Joel proposed. Repeated addition is not
the same as multiplication with floating point numbers. 

A generator based on Joel's idea would look like this:

def fp_range(start, end, step):
v = start
n = int(math.ceil((end - start) / step))
for i in range(n):
yield start + i * step

(I omitted the OverflowError: range() doesn't raise one, either)

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: About float/double type number in range.

[Richard Damon]

On 8/25/20 10:39 PM, ADITYA wrote:
>Dear Sir/Ma’am
>
>I am requesting you to satisfy me about float number in Range function,
>because in the argument of range we can take integer but not double or
>float whenever double as well as float are integer in nature but when we
>use double/float in, it gives error that- “'float' object cannot be
>interpreted as an integer.” If we want to increment the number by half or
>quarter what can I do.
>
>For ex- Range(1,3,0.5) I want it gives Output as [1 1.5 2 2.5 3)
>
>I am requesting to change the nature of increment number nature in above
>example so that we can increase the number with half or quarter any point
>value.
>
>Your Sincerely
>
>Aditya Gautam
>Saharsa (Bihar)
>India
>Postal Code- 852201

As was indicated, Range(1, 3, 0.5) if legal would generate ([1, 1.5, 2,
2.5] because range excludes the top value. This case happens to work in
the sense that if Python allowed it, this is the result that would come
out. On the other hand, the very similar Range(1, 3, 0.4) would be need
very detailed knowledge to predict, it could return [1, 1.4, 1.8, 2.2,
2.6] as expected or [1, 1.4, 1.8, 2.2, 2.6, 3.0]. The issue is that
there is no such exact number in binary floating point as 0.4, (it is
like trying to write out exactly 1/3), so the actual value used for 0.4
will be either slightly higher (where you get the expected value) or
slightly lower (where you would get the top 'excluded' value listed).
This sort of unpredictability is part of the difficulty dealing with
floating point.

As was pointed out, you can scale the range, or build your own generator
to get what you want.

-- 
Richard Damon

-- 
https://mail.python.org/mailman/listinfo/python-list

How do I do this in Python 3 (string.join())?

[Chris Green]

I have the following line in Python 2:-

msgstr = string.join(popmsg[1], "\n")  # popmsg[1] is a list containing 
the lines of the message

... so I changed it to:-

s = "\n"
msgstr = s.join(popmsg[1])  # popmsg[1] is a list containing the lines 
of the message

However this still doesn't work because popmsg[1] isn't a list of
strings, I get the error:-

TypeError: sequence item 0: expected str instance, bytes found

So how do I do this?  I can see clumsy ways by a loop working through
the list in popmsg[1] but surely there must be a way that's as neat
and elegant as the Python 2 way was?

-- 
Chris Green
·
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How do I do this in Python 3 (string.join())?

[2QdxY4RzWzUUiLuE]

On 2020-08-26 at 14:22:10 +0100,
Chris Green  wrote:

> I have the following line in Python 2:-
> 
> msgstr = string.join(popmsg[1], "\n")  # popmsg[1] is a list 
> containing the lines of the message
> 
> ... so I changed it to:-
> 
> s = "\n"
> msgstr = s.join(popmsg[1])  # popmsg[1] is a list containing the 
> lines of the message
> 
> However this still doesn't work because popmsg[1] isn't a list of
> strings, I get the error:-
> 
> TypeError: sequence item 0: expected str instance, bytes found
> 
> So how do I do this?  I can see clumsy ways by a loop working through
> the list in popmsg[1] but surely there must be a way that's as neat
> and elegant as the Python 2 way was?

Join bytes objects with a byte object:

b"\n".join(popmsg[1])
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How do I do this in Python 3 (string.join())?

[D'Arcy Cain]

On 2020-08-26 09:22, Chris Green wrote:
> I have the following line in Python 2:-
> 
> msgstr = string.join(popmsg[1], "\n")  # popmsg[1] is a list 
> containing the lines of the message
> 
> ... so I changed it to:-
> 
> s = "\n"
> msgstr = s.join(popmsg[1])  # popmsg[1] is a list containing the 
> lines of the message
> 
> However this still doesn't work because popmsg[1] isn't a list of
> strings, I get the error:-
> 
> TypeError: sequence item 0: expected str instance, bytes found
> 
> So how do I do this?  I can see clumsy ways by a loop working through
> the list in popmsg[1] but surely there must be a way that's as neat
> and elegant as the Python 2 way was?

Well, the simple fix is to set s to b"\n" but that may not solve all of your
problems.  The issue is that popmsg[1] is a list of bytes.  You probably
want a list of strings.  I would look further back and think about getting a
list of strings in the first place.  Without knowing how popmsg was created
we can't tell you how to do that.

Of course, if a bytes object is what you want then the above will work.  You
can also convert to string after the join.

Cheers.

-- 
D'Arcy J.M. Cain
Vybe Networks Inc.
A unit of Excelsior Solutions Corporation - Propelling Business Forward
http://www.VybeNetworks.com/
IM:da...@vybenetworks.com VoIP: sip:da...@vybenetworks.com



signature.asc
Description: OpenPGP digital signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How do I do this in Python 3 (string.join())?

[MRAB]

On 2020-08-26 14:22, Chris Green wrote:

I have the following line in Python 2:-

 msgstr = string.join(popmsg[1], "\n")  # popmsg[1] is a list 
containing the lines of the message

... so I changed it to:-

 s = "\n"
 msgstr = s.join(popmsg[1])  # popmsg[1] is a list containing the lines 
of the message

However this still doesn't work because popmsg[1] isn't a list of
strings, I get the error:-

 TypeError: sequence item 0: expected str instance, bytes found

So how do I do this?  I can see clumsy ways by a loop working through
the list in popmsg[1] but surely there must be a way that's as neat
and elegant as the Python 2 way was?


In Python 3, bytestring literals require the 'b' prefix:

msgstr = b"\n".join(popmsg[1])
--
https://mail.python.org/mailman/listinfo/python-list

Re: How do I do this in Python 3 (string.join())?

[Chris Green]

2qdxy4rzwzuui...@potatochowder.com wrote:
> On 2020-08-26 at 14:22:10 +0100,
> Chris Green  wrote:
> 
> > I have the following line in Python 2:-
> > 
> > msgstr = string.join(popmsg[1], "\n") # popmsg[1] is a list containing 
> the lines of the message 
> > 
> > ... so I changed it to:-
> > 
> > s = "\n"
> > msgstr = s.join(popmsg[1])  # popmsg[1] is a list containing the 
> > lines of the message
> > 
> > However this still doesn't work because popmsg[1] isn't a list of
> > strings, I get the error:-
> > 
> > TypeError: sequence item 0: expected str instance, bytes found
> > 
> > So how do I do this?  I can see clumsy ways by a loop working through
> > the list in popmsg[1] but surely there must be a way that's as neat
> > and elegant as the Python 2 way was?
> 
> Join bytes objects with a byte object:
> 
> b"\n".join(popmsg[1])

Aaahhh!  Thank you (and the other reply).

-- 
Chris Green
·
-- 
https://mail.python.org/mailman/listinfo/python-list

Another 2 to 3 mail encoding problem

[Chris Green]

I'm unearthing a few issues here trying to convert my mail filter and
delivery programs from 2 to 3!  

I have a simple[ish] local mbox mail delivery module as follows:-


import mailbox
import logging
import logging.handlers
import os
import time
#
#
# Class derived from mailbox.mbox so we can override _pre_message_hook()
# to do nothing instead of appending a blank line
#
class mymbox(mailbox.mbox):
def _pre_message_hook(self, f):
"""Don't write the blank line before the 'From '"""
pass
#
#
# log a message
#
def initLog(name):
log = logging.getLogger(name)
log.setLevel(logging.DEBUG)
f = logging.handlers.RotatingFileHandler("/home/chris/tmp/mail.log", 
'a', 100, 4)
f.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - 
%(message)s')
f.setFormatter(formatter)
log.addHandler(f)
return log
#
#
# Deliver a message to a local mbox
#
def deliverMboxMsg(dest, msg, log):
#
#
# Create the destination mbox instance
#
mbx = mymbox(dest, factory=None)

log.info("From: " + msg.get("From", "unknown"))
log.info("Destination is: " + dest)
#
#
# Lock the mbox while we append to it
#
for tries in range(3):
try:
mbx.lock()
#
#
# Append the incoming message to the appropriate mbox
#
mbx.add(msg)
#
#
# now set the modified time later than the access time (which 
is 'now') so
# that mutt will see new mail in the mbox.
#
os.utime(dest, ((time.time()), (time.time() + 5)))
mbx.unlock()
break

except mailbox.ExternalClashError:
log.info("Destination locked, try " + str(tries))
time.sleep(1)

else: # get here if we ran out of tries
log.warn("Failed to lock destination after 3 attempts, giving up")

return


It has run faultlessly for many years under Python 2.  I've now
changed the calling program to Python 3 and while it handles most
E-Mail OK I have just got the following error:-

Traceback (most recent call last):
  File "/home/chris/.mutt/bin/filter.py", line 102, in 
mailLib.deliverMboxMsg(dest, msg, log)
  File "/home/chris/.mutt/bin/mailLib.py", line 52, in deliverMboxMsg
mbx.add(msg)
  File "/usr/lib/python3.8/mailbox.py", line 603, in add
self._toc[self._next_key] = self._append_message(message)
  File "/usr/lib/python3.8/mailbox.py", line 758, in _append_message
offsets = self._install_message(message)
  File "/usr/lib/python3.8/mailbox.py", line 830, in _install_message
self._dump_message(message, self._file, self._mangle_from_)
  File "/usr/lib/python3.8/mailbox.py", line 215, in _dump_message
gen.flatten(message)
  File "/usr/lib/python3.8/email/generator.py", line 116, in flatten
self._write(msg)
  File "/usr/lib/python3.8/email/generator.py", line 181, in _write
self._dispatch(msg)
  File "/usr/lib/python3.8/email/generator.py", line 214, in _dispatch
meth(msg)
  File "/usr/lib/python3.8/email/generator.py", line 432, in _handle_text
super(BytesGenerator,self)._handle_text(msg)
  File "/usr/lib/python3.8/email/generator.py", line 249, in _handle_text
self._write_lines(payload)
  File "/usr/lib/python3.8/email/generator.py", line 155, in _write_lines
self.write(line)
  File "/usr/lib/python3.8/email/generator.py", line 406, in write
self._fp.write(s.encode('ascii', 'surrogateescape'))
UnicodeEncodeError: 'ascii' codec can't encode character '\ufeff' in 
position 4: ordinal not in range(128)

So what do I need to do to the message I'm adding with mbx.add(msg) to
fix this?  (I assume that's what I need to do).

-- 
Chris Green
·
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Another 2 to 3 mail encoding problem

[Alexa Oña]

Don’t send me more emails

Obtener Outlook para iOS

De: Python-list  
en nombre de Chris Green 
Enviado: Wednesday, August 26, 2020 5:10:35 PM
Para: python-list@python.org 
Asunto: Another 2 to 3 mail encoding problem

I'm unearthing a few issues here trying to convert my mail filter and
delivery programs from 2 to 3!

I have a simple[ish] local mbox mail delivery module as follows:-


import mailbox
import logging
import logging.handlers
import os
import time
#
#
# Class derived from mailbox.mbox so we can override _pre_message_hook()
# to do nothing instead of appending a blank line
#
class mymbox(mailbox.mbox):
def _pre_message_hook(self, f):
"""Don't write the blank line before the 'From '"""
pass
#
#
# log a message
#
def initLog(name):
log = logging.getLogger(name)
log.setLevel(logging.DEBUG)
f = logging.handlers.RotatingFileHandler("/home/chris/tmp/mail.log", 
'a', 100, 4)
f.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - 
%(message)s')
f.setFormatter(formatter)
log.addHandler(f)
return log
#
#
# Deliver a message to a local mbox
#
def deliverMboxMsg(dest, msg, log):
#
#
# Create the destination mbox instance
#
mbx = mymbox(dest, factory=None)

log.info("From: " + msg.get("From", "unknown"))
log.info("Destination is: " + dest)
#
#
# Lock the mbox while we append to it
#
for tries in range(3):
try:
mbx.lock()
#
#
# Append the incoming message to the appropriate mbox
#
mbx.add(msg)
#
#
# now set the modified time later than the access time (which 
is 'now') so
# that mutt will see new mail in the mbox.
#
os.utime(dest, ((time.time()), (time.time() + 5)))
mbx.unlock()
break

except mailbox.ExternalClashError:
log.info("Destination locked, try " + str(tries))
time.sleep(1)

else: # get here if we ran out of tries
log.warn("Failed to lock destination after 3 attempts, giving up")

return


It has run faultlessly for many years under Python 2.  I've now
changed the calling program to Python 3 and while it handles most
E-Mail OK I have just got the following error:-

Traceback (most recent call last):
  File "/home/chris/.mutt/bin/filter.py", line 102, in 
mailLib.deliverMboxMsg(dest, msg, log)
  File "/home/chris/.mutt/bin/mailLib.py", line 52, in deliverMboxMsg
mbx.add(msg)
  File "/usr/lib/python3.8/mailbox.py", line 603, in add
self._toc[self._next_key] = self._append_message(message)
  File "/usr/lib/python3.8/mailbox.py", line 758, in _append_message
offsets = self._install_message(message)
  File "/usr/lib/python3.8/mailbox.py", line 830, in _install_message
self._dump_message(message, self._file, self._mangle_from_)
  File "/usr/lib/python3.8/mailbox.py", line 215, in _dump_message
gen.flatten(message)
  File "/usr/lib/python3.8/email/generator.py", line 116, in flatten
self._write(msg)
  File "/usr/lib/python3.8/email/generator.py", line 181, in _write
self._dispatch(msg)
  File "/usr/lib/python3.8/email/generator.py", line 214, in _dispatch
meth(msg)
  File "/usr/lib/python3.8/email/generator.py", line 432, in _handle_text
super(BytesGenerator,self)._handle_text(msg)
  File "/usr/lib/python3.8/email/generator.py", line 249, in _handle_text
self._write_lines(payload)
  File "/usr/lib/python3.8/email/generator.py", line 155, in _write_lines
self.write(line)
  File "/usr/lib/python3.8/email/generator.py", line 406, in write
self._fp.write(s.encode('ascii', 'surrogateescape'))
UnicodeEncodeError: 'ascii' codec can't encode character '\ufeff' in 
position 4: ordinal not in range(128)

So what do I need to do to the message I'm adding with mbx.add(msg) to
fix this?  (I assume that's what I need to do).

--
Chris Green
·
--
https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Another 2 to 3 mail encoding problem

[Chris Green]

To add a little to this, the problem is definitely when I receive a
message with UTF8 (or at least non-ascci) characters in it.  My code
is basically very simple, the main program reads an E-Mail message
received from .forward on its standard input and makes it into an mbox
message as follows:-

msg = mailbox.mboxMessage(sys.stdin.read())

it then does various tests (but doesn't change msg at all) and at the
end delivers the message to my local mbox with:-

mbx.add(msg)

where mbx is an instance of mailbox.mbox.


So, how is one supposed to handle this, should I encode the incoming
message somewhere?

-- 
Chris Green
·
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Another 2 to 3 mail encoding problem

[Python]

Alexa Oña wrote:

Don’t send me more emails

Obtener Outlook para iOS


You are the one spamming the mailing list with unrelated posts.

STOP.


--
https://mail.python.org/mailman/listinfo/python-list

Re: Another 2 to 3 mail encoding problem

[Peter J. Holzer]

On 2020-08-26 16:10:35 +0100, Chris Green wrote:
> I'm unearthing a few issues here trying to convert my mail filter and
> delivery programs from 2 to 3!  
> 
> I have a simple[ish] local mbox mail delivery module as follows:-
> 
[...]
> class mymbox(mailbox.mbox):
> def _pre_message_hook(self, f):
> """Don't write the blank line before the 'From '"""
> pass
[...]
> def deliverMboxMsg(dest, msg, log):
[...]
> mbx = mymbox(dest, factory=None)
[...]
> mbx.add(msg)
[...]
> 
> 
> It has run faultlessly for many years under Python 2.  I've now
> changed the calling program to Python 3 and while it handles most
> E-Mail OK I have just got the following error:-
> 
> Traceback (most recent call last):
>   File "/home/chris/.mutt/bin/filter.py", line 102, in 
> mailLib.deliverMboxMsg(dest, msg, log)
>   File "/home/chris/.mutt/bin/mailLib.py", line 52, in deliverMboxMsg
> mbx.add(msg)
>   File "/usr/lib/python3.8/mailbox.py", line 603, in add
> self._toc[self._next_key] = self._append_message(message)
>   File "/usr/lib/python3.8/mailbox.py", line 758, in _append_message
> offsets = self._install_message(message)
>   File "/usr/lib/python3.8/mailbox.py", line 830, in _install_message
> self._dump_message(message, self._file, self._mangle_from_)
>   File "/usr/lib/python3.8/mailbox.py", line 215, in _dump_message
> gen.flatten(message)
>   File "/usr/lib/python3.8/email/generator.py", line 116, in flatten
> self._write(msg)
>   File "/usr/lib/python3.8/email/generator.py", line 181, in _write
> self._dispatch(msg)
>   File "/usr/lib/python3.8/email/generator.py", line 214, in _dispatch
> meth(msg)
>   File "/usr/lib/python3.8/email/generator.py", line 432, in _handle_text
> super(BytesGenerator,self)._handle_text(msg)
>   File "/usr/lib/python3.8/email/generator.py", line 249, in _handle_text
> self._write_lines(payload)
>   File "/usr/lib/python3.8/email/generator.py", line 155, in _write_lines
> self.write(line)
>   File "/usr/lib/python3.8/email/generator.py", line 406, in write
> self._fp.write(s.encode('ascii', 'surrogateescape'))
> UnicodeEncodeError: 'ascii' codec can't encode character '\ufeff' in 
> position 4: ordinal not in range(128)

The problem is that the message contains a '\ufeff' character (byte
order mark) where email/generator.py expects only ASCII characters.

I see two possible reasons for this:

 * The mbox writing code assumes that all messages with non-ascii
   characters are QP or base64 encoded, and some higher layer uses 8bit
   instead.

 * A mime-part is declared as charset=us-ascii but contains really
   Unicode characters.

Both reasons are weird.

The first would be an unreasonable assumption (8bit encoding has been
common since the mid-1990s), but even if the code made that assumption,
one would expect that other code from the same library honors it.

The second shouldn't be possible: If a message is mis-declared (that
happens) one would expect that the error happens during parsing, not
when trying to serialize the already parsed message. 

But then you haven't shown where msg comes from. How do you parse the
message to get "msg"?

Can you construct a minimal test message which triggers the bug?

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Another 2 to 3 mail encoding problem

[Michael Torrie]

On 8/26/20 9:27 AM, Alexa Oña wrote:
> Don’t send me more emails
>
> https://mail.python.org/mailman/listinfo/python-list
^
Please unsubscribe from the mailing list. Click on the link above.

Thank you.



-- 
https://mail.python.org/mailman/listinfo/python-list

Re: About float/double type number in range.

[Terry Reedy]

On 8/26/2020 6:40 AM, dn via Python-list wrote:

def fp_range( start:float, stop:float, step:float=1.0 )->float:
     """Generate a range of floating-point numbers."""
     if stop <= start:
     raise OverflowError( "RangeError: start must be less than stop" )
     x = start
     while x < stop:
     yield x
     x += step


This only works exactly as expected for fraction steps of k/2**n give 
exactly.  Your only tests used .5 and .25.


print(list(fp_range(0.0, 2.01, .1)))
produces
[0.0, 0.1, 0.2, 0.30004, 0.4, 0.5, 0.6, 0.7, 
0.7999, 0.8999, 0., 
1.0999, 1.2, 1.3, 1.4001, 1.5002, 
1.6003, 1.7004, 1.8005, 
1.9006, 2.0004]


For exact fractional steps, one should use fractions or multiply and 
divide by the denominator, as shown by Joel, after stepping by the 
numerator.



--
Terry Jan Reedy


--
https://mail.python.org/mailman/listinfo/python-list

Unsubscrip (Re: Another 2 to 3 mail encoding problem)

[Terry Reedy]

On 8/26/2020 11:27 AM, Alexa Oña wrote:

Don’t send me more emails



--
https://mail.python.org/mailman/listinfo/python-list


Unsubscribe yourself by going to the indicated url.


--
Terry Jan Reedy


--
https://mail.python.org/mailman/listinfo/python-list

Re: Another 2 to 3 mail encoding problem

[Terry Reedy]

On 8/26/2020 11:10 AM, Chris Green wrote:


I have a simple[ish] local mbox mail delivery module as follows:-

...

It has run faultlessly for many years under Python 2.  I've now
changed the calling program to Python 3 and while it handles most
E-Mail OK I have just got the following error:-

 Traceback (most recent call last):
   File "/home/chris/.mutt/bin/filter.py", line 102, in 
 mailLib.deliverMboxMsg(dest, msg, log)

...

   File "/usr/lib/python3.8/email/generator.py", line 406, in write
 self._fp.write(s.encode('ascii', 'surrogateescape'))
 UnicodeEncodeError: 'ascii' codec can't encode character '\ufeff' in 
position 4: ordinal not in range(128)


'\ufeff' is the Unicode byte-order mark.  It should not be present in an 
ascii-only 3.x string and would not normally be present in general 
unicode except in messages like this that talk about it.  Read about it, 
for instance, at

https://en.wikipedia.org/wiki/Byte_order_mark

I would catch the error and print part or all of string s to see what is 
going on with this particular message.  Does it have other non-ascii chars?



--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list

Why __hash__() does not return an UUID4?

[Marco Sulla]

As title. The reasons that came in my mind are:

1. speed
2. security
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why __hash__() does not return an UUID4?

[Richard Damon]

On 8/26/20 4:10 PM, Marco Sulla wrote:
> As title. The reasons that came in my mind are:
>
> 1. speed
> 2. security

My guess is that the typical use of the value for __hash__() doesn't
need that level of guarantee of uniqueness, so requiring it would be
unnecessarily expensive.

Since the typical use of hash will be followed by a real equality test
if the hashes match, we don't need the level of a UUID

-- 
Richard Damon

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why __hash__() does not return an UUID4?

[2QdxY4RzWzUUiLuE]

On 2020-08-26 at 22:10:26 +0200,
Marco Sulla  wrote:

> As title ...

Assuming that the title appears prominently with the content of your
email.  For the rest of us:

Why __hash__() does not return an UUID4?

> ... The reasons that came in my mind are:
> 
> 1. speed
> 2. security

Correctness?

Why would __hash__() return a random number?  An important property of
hashes is that if x == y, then hash(x) == hash(y).  If hash(x) and
hash(y) were random numbers, then how would this property be maintained?

Or do UUID4 mean something else to you than a random number?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why __hash__() does not return an UUID4?

[Richard Damon]

On 8/26/20 5:13 PM, 2qdxy4rzwzuui...@potatochowder.com wrote:
> On 2020-08-26 at 22:10:26 +0200,
> Marco Sulla  wrote:
>
>> As title ...
> Assuming that the title appears prominently with the content of your
> email.  For the rest of us:
>
> Why __hash__() does not return an UUID4?
>
>> ... The reasons that came in my mind are:
>>
>> 1. speed
>> 2. security
> Correctness?
>
> Why would __hash__() return a random number?  An important property of
> hashes is that if x == y, then hash(x) == hash(y).  If hash(x) and
> hash(y) were random numbers, then how would this property be maintained?
>
> Or do UUID4 mean something else to you than a random number?

Looking up which UUID type 4 is, yes it is a random number, so could
only be applicable for objects that currently return id(), which could
instead make id() be a UUID4 (and save it in the object, increasing its
side)

-- 
Richard Damon

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why __hash__() does not return an UUID4?

[Chris Angelico]

On Thu, Aug 27, 2020 at 7:52 AM Richard Damon  wrote:
>
> On 8/26/20 5:13 PM, 2qdxy4rzwzuui...@potatochowder.com wrote:
> > On 2020-08-26 at 22:10:26 +0200,
> > Marco Sulla  wrote:
> >
> >> As title ...
> > Assuming that the title appears prominently with the content of your
> > email.  For the rest of us:
> >
> > Why __hash__() does not return an UUID4?
> >
> >> ... The reasons that came in my mind are:
> >>
> >> 1. speed
> >> 2. security
> > Correctness?
> >
> > Why would __hash__() return a random number?  An important property of
> > hashes is that if x == y, then hash(x) == hash(y).  If hash(x) and
> > hash(y) were random numbers, then how would this property be maintained?
> >
> > Or do UUID4 mean something else to you than a random number?
>
> Looking up which UUID type 4 is, yes it is a random number, so could
> only be applicable for objects that currently return id(), which could
> instead make id() be a UUID4 (and save it in the object, increasing its
> side)
>

I can't see how this is any benefit over any other definition, but if
someone wants to propose that id() return a UUID4, then that would be
entirely separate from anything regarding hash. As long as there's an
absolute guarantee that the same ID will never be used for two objects
at once, it's fine to generate them any way you like.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: About float/double type number in range.

[Rob Cliffe via Python-list]

On 26/08/2020 12:02, Peter J. Holzer wrote:

On 2020-08-26 22:40:36 +1200, dn via Python-list wrote:

On 26/08/2020 19:58, Joel Goldstick wrote:


[...]

<<< Code NB Python v3.8 >>>
def fp_range( start:float, stop:float, step:float=1.0 )->float:
 """Generate a range of floating-point numbers."""
 if stop <= start:
 raise OverflowError( "RangeError: start must be less than stop" )
 x = start
 while x < stop:
 yield x
 x += step

This is almost the same solution as I came up with, but note that this
is not quote the same as what Joel proposed. Repeated addition is not
the same as multiplication with floating point numbers.
To expand on this: it is because floating point numbers cannot represent 
all real numbers exactly.
Repeated addition of a number represented inexactly will drift away from 
the mathematical value.
Example: fp_range(0, 70, 0.07) as written above will yield 1001 numbers 
ending with 69.66 (i.e. approximately, but not quite, 70).
Whereas Peter's version (below) will correctly yield 1000 numbers ending 
with 69.93.
(Even then some numbers are not represented exactly, e.g. the 
last-but-one is 69.860001.)


  


A generator based on Joel's idea would look like this:

def fp_range(start, end, step):
 v = start
 n = int(math.ceil((end - start) / step))
 for i in range(n):
 yield start + i * step



--
https://mail.python.org/mailman/listinfo/python-list

Re: Why __hash__() does not return an UUID4?

[Cameron Simpson]

On 26Aug2020 22:10, Marco Sulla  wrote:
>As title. The reasons that came in my mind are:
>1. speed
>2. security

Various reasons come to mind:
- it is overkill for what __hash__ has to do (computational and feature 
  overkill)
- requires importing the uuid module (overkill again)
- speed
- the hash must be the same each time it is called for a given object, 
  so you'd need to waste space saving a UUID

The requirements for a hash are that:
- it be stable for a given object
- for use in dicts, objects of equal value (via ==) have the same hash

That second is actually a problem for UUID4. Suppose we have 2 
independent objects with the same value - how are you to ensure they 
have the same UUID?

Think through how a hash is used in a hash table - the hash is used to 
distribute values into buckets in a statisticly even fashion, so that 
all the values which are "equal" land in the _same_ bucket. That 
requires coordination of the hash values. Where that matters, you 
compute the hash from the value. UUID4s are not like that.

Cheers,
Cameron Simpson 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How do I do this in Python 3 (string.join())?

[Cameron Simpson]

On 26Aug2020 15:09, Chris Green  wrote:
>2qdxy4rzwzuui...@potatochowder.com wrote:
>> Join bytes objects with a byte object:
>>
>> b"\n".join(popmsg[1])
>
>Aaahhh!  Thank you (and the other reply).

But note: joining bytes like strings is uncommon, and may indicate that 
you should be working in strings to start with. Eg you may want to 
convert popmsg from bytes to str and do a str.join anyway. It depends on 
exactly what you're dealing with: are you doing text work, or are you 
doing "binary data" work?

I know many network protocols are "bytes-as-text, but that is 
accomplished by implying an encoding of the text, eg as ASCII, where 
characters all fit in single bytes/octets.

Cheers,
Cameron Simpson 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: ABC with abstractmethod: kwargs on Base, explicit names on implementation

[Samuel Marks]

The main thing I want is type safety. I want Python to complain if the
callee uses the wrong argument types, and to provide suggestions on
what's needed and info about it.

Without a base class I can just have docstrings and type annotations
to achieve that.

What can I use that will require all implementers to have a minimum of
the same properties and arguments, but also allow them to add new
properties and arguments?

Preferably I would like this all to happen before compile/interpret
time. This also opens it up to AST driven stub generation; as can be
found in various IDEs (and also a little project I wrote that `import
ast`).

What are my options here? - It doesn't seem like the metaclass or
decorator approaches will help here…

Samuel Marks
Charity  | consultancy
 | open-source  |
LinkedIn 


Samuel Marks wrote at 2020-8-24 18:24 +1000:
>After a discussion on #python on Freenode, I'm here.
>
>The gist of it is:
>> Falc - Signature of method 'Pharm.foo()' does not match signature of base 
>> method in class 'Base'
>
>What's the right way of specialising the children and leaving the Base
>pretty much empty?
>
>Specifically I want:
>• All implementers to be forced to implement Base's method
>• Base to be able to do very basic things with kwargs, namely log it
>(to a file like stdout or a db)
>• Support for [at least] Python 3.6–3.9 (I don't think `Protocol` has
>been backported?)
>• Implementors to have access to the proper name, rather than having
>to unpack kwargs
>
>Should I be using a Base class? - Metaclass? - Something else
>entirely? - I know giving `b` a default value would resolve the
>[PyCharm] linter error… but I want it to be a required argument.
>
>Full example:
>
>from abc import ABC, abstractmethod
>
>
>class Base(ABC):
>@abstractmethod
>def foo(self, a, **kwargs):
> ...
>class Pharm(Base):
>def foo(self, a, b):
> ...

Python make a distinction between positional and keyword
arguments. A positional argument is identified by its
position in the parameter list; a keyword argument is
identified by its name.

`**` introduces arbitrary keyword arguments. In a call, all
those arguments must be passed as "name=value".
In your case above, `b` is not a keyword argument and thus
is not matched by `**kwargs`. The error you observe is justified.

You can try:

class Base(ABC):
@abstractmethod
def foo(self, a, *args, **kwargs):
...

class Pharm(Base):
def foo(self, a, b, *args, **kwargs):
...


Note that the base method signature allows arbitrary positional
and keyword arguments. As a consequence, derived methods must do
the same.

If this is not what you want, you might want to explore
the use of a decorator or a meta class rather than a base class.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Fwd: [BUG] missing ')' causes syntax error on next line

[Greg Ewing]

On 23/07/20 12:23 pm, dn wrote:

You be sure to have a nice day, now!


How do you feel about you be sure to have a nice day, now?

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list