Re: What's the meaning the "backlog" in the socket.listen(backlog) is?

[Karen Shaeffer via Python-list]

> On Feb 16, 2021, at 8:10 PM, Jason Friedman  wrote:
> 
>> 
>> I set listen(2) and expect to see "error" when more clients than "the
>> maximum number of queued connections" trying to connect the server. But, no
>> error!! Even 4 clients can run normally without problem.
>> 
>> Am I misunderstanding the meaning of this argument?
>> 
> 
> https://docs.python.org/3/library/socket.html#socket.socket.listen:
> Enable a server to accept connections. If backlog is specified ... it
> specifies the number of unaccepted connections that the system will allow
> before refusing new connections.
> 
> So I read that to say that the listen method will accept as many connection
> as it can handle, and will put into a backlog those it cannot. Your
> argument of 2 tells the method to allow up to 2 not-yet-accepted connection
> requests.
> -- 

Hi Jason,
The backlog parameter specifies the number of connection requests that the 
kernel network stack has waiting for the application layer to process. So, it’s 
at a lower level than the listen system call, which is the application layer 
API into the kernel network stack that simply establishes the socket. Listen 
doesn’t actually touch any data. Once the accept system call has touched it, 
then the network stack no longer counts it in the backlog. Backlog is a network 
stack queue associated with the application socket. The backlog is managed by 
the network driver software. Moving bits down there is extremely efficient. And 
the driver wouldn’t process and pass packets up the stack, if the backlog was 
maxed out. That would be very inefficient — rather the driver would drop the 
packets at the network driver ring buffer in such case. That’s at the API into 
the network stack looking up from the network driver.

Here is a good description from the perspective of the bottom of the tcp ip 
stack API into the network card driver, discussing the processing of packets 
depending on state of the backlog for the socket.

http://veithen.io/2014/01/01/how-tcp-backlog-works-in-linux.html 



Humbly,
Karen

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What's the meaning the "backlog" in the socket.listen(backlog) is?

[Karen Shaeffer via Python-list]

> On Feb 17, 2021, at 12:25 AM, Karen Shaeffer via Python-list 
>  wrote:
> 
> 
> 
>> On Feb 16, 2021, at 8:10 PM, Jason Friedman  wrote:
>> 
>>> 
>>> I set listen(2) and expect to see "error" when more clients than "the
>>> maximum number of queued connections" trying to connect the server. But, no
>>> error!! Even 4 clients can run normally without problem.
>>> 
>>> Am I misunderstanding the meaning of this argument?
>>> 
>> 
>> https://docs.python.org/3/library/socket.html#socket.socket.listen:
>> Enable a server to accept connections. If backlog is specified ... it
>> specifies the number of unaccepted connections that the system will allow
>> before refusing new connections.
>> 
>> So I read that to say that the listen method will accept as many connection
>> as it can handle, and will put into a backlog those it cannot. Your
>> argument of 2 tells the method to allow up to 2 not-yet-accepted connection
>> requests.
>> -- 
> 
> Hi Jason,
> The backlog parameter specifies the number of connection requests that the 
> kernel network stack has waiting for the application layer to process. So, 
> it’s at a lower level than the listen system call, which is the application 
> layer API into the kernel network stack that simply establishes the socket. 
> Listen doesn’t actually touch any data. Once the accept system call has 
> touched it, then the network stack no longer counts it in the backlog. 
> Backlog is a network stack queue associated with the application socket. The 
> backlog is managed by the network driver software. Moving bits down there is 
> extremely efficient. And the driver wouldn’t process and pass packets up the 
> stack, if the backlog was maxed out. That would be very inefficient — rather 
> the driver would drop the packets at the network driver ring buffer in such 
> case. That’s at the API into the network stack looking up from the network 
> driver.
> 
> Here is a good description from the perspective of the bottom of the tcp ip 
> stack API into the network card driver, discussing the processing of packets 
> depending on state of the backlog for the socket.
> 
> http://veithen.io/2014/01/01/how-tcp-backlog-works-in-linux.html 
> 
> 
The tcp ip stack software is the archetypical example of copy-on-write code. 
The packet comes over the wire and is copied onto the network driver ring 
buffer. Then, if the socket backlog is less than full, the network driver 
software copies the packet from the network driver ring buffer onto the very 
bottom layer of the ip stack. The data is never moved (copied) again, as it 
flows up the layers of the tcp ip stack, only pointers are used by each layer 
of the stack to maintain access to the data. Finally, the application will copy 
the data from the kernel memory over to application user space memory with a 
socket read call. It’s very beautiful code.

Karen.

-- 
https://mail.python.org/mailman/listinfo/python-list

[RELEASE] Python 3.9.2rc1 and 3.8.8rc1 are now available for testing

[Łukasz Langa]

I’m happy to announce two release candidates today: Python 3.9.2rc1, and Python 
3.8.8rc1. Get them from:

https://www.python.org/downloads/release/python-392rc1/ 


https://www.python.org/downloads/release/python-388rc1/ 


Unless critical issues are discovered, both release candidates will become 
their respective final versions on Monday, March 1st.
Following that, the last full regular maintenance release of Python 3.8 is 
planned for May 3rd 2021, after which it will shift to source releases only for 
security bug fixes only. Maintenance releases for the 3.9 series will continue 
at regular bi-monthly intervals, with 3.9.3 planned for early May 2021.

Notable security content in today’s releases

bpo-42967 : Fix web cache poisoning 
vulnerability by defaulting the query args separator to &, and allowing the 
user to choose a custom separator.

bpo-42938 : Avoid static buffers when 
computing the repr of ctypes.c_double and ctypes.c_longdouble values.

What’s new?

The Python 3.9 series contains many new features and optimizations over 3.8. 
See the “What’s New in Python 3.9 
” document for more information 
about features included in the 3.9 series. We also have a detailed change log 
for 3.9.2rc1 
 
specifically.

Detailed information about all changes made in version 3.8.8rc1 specifically 
can be found in its change log 
.

We hope you enjoy those new releases!

Thanks to all of the many volunteers who help make Python Development and these 
releases possible! Please consider supporting our efforts by volunteering 
yourself or through organization contributions to the Python Software 
Foundation.

Your friendly release team,
Ned Deily @nad 
Steve Dower @steve.dower 
Łukasz Langa @ambv 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 2.7 and 3.9

[Thomas Jollans]

On 16/02/2021 22:16, Ethan Furman wrote:
Kevin, please reply to the list (preferably Reply-to-list, or 
Reply-all), that way others can chime in with help.


On 2/16/21 12:55 PM, Kevin M. Wilson wrote:

Windows 7 OS, and typically run in conjunction with testing SSD', as 
for stand alone scripts.
Those require: python BogusFile.py. I too was expecting users to 
type: python my_Script.py!


On Windows, you should use the py.exe launcher:

py -2 python2_script.py

to run an old script, and

py -3 python3_script.py

or

py python3_script.py

to launch a new script. AKAIK, the launcher is always installed with 
recent versions of Python on Windows.



You could set up the PATH such that 'python' is python 2.7, and 'py' 
calls python 3.x. Have a look at the docs to figure about what other 
options py.exe gives you (such as shebang lines)


Docs: 
https://docs.python.org/3/using/windows.html#python-launcher-for-windows


PEP: https://www.python.org/dev/peps/pep-0397


-- Thomas



--
https://mail.python.org/mailman/listinfo/python-list

Re: Python 2.7 and 3.9

[MRAB]

On 2021-02-17 14:09, Thomas Jollans wrote:

On 16/02/2021 22:16, Ethan Furman wrote:
Kevin, please reply to the list (preferably Reply-to-list, or 
Reply-all), that way others can chime in with help.


On 2/16/21 12:55 PM, Kevin M. Wilson wrote:

Windows 7 OS, and typically run in conjunction with testing SSD', as 
for stand alone scripts.
Those require: python BogusFile.py. I too was expecting users to 
type: python my_Script.py!


On Windows, you should use the py.exe launcher:

py -2 python2_script.py

to run an old script, and

py -3 python3_script.py

or

py python3_script.py

to launch a new script. AKAIK, the launcher is always installed with
recent versions of Python on Windows.

The scripts simply need to start with a shebang line that specifies the 
Python version. Then it's just:


py python2_script.py

py python3_script.py



You could set up the PATH such that 'python' is python 2.7, and 'py'
calls python 3.x. Have a look at the docs to figure about what other
options py.exe gives you (such as shebang lines)

Docs:
https://docs.python.org/3/using/windows.html#python-launcher-for-windows

PEP: https://www.python.org/dev/peps/pep-0397


--
https://mail.python.org/mailman/listinfo/python-list

I need some help interpreting this error

[Chris Green]

I'm running this using Python 3.7 on a Linux system.

Most of the time (i.e. for a couple of days now) the program has been
satifactorily delivering mail messages, hundreds of them.  However one
mail message has provoked the following error:-

chris@cheddar$ tail mail.err
Traceback (most recent call last):
  File "/home/chris/.mutt/bin/filter.py", line 95, in 
if sbstrip in msghdr["subject"]:
TypeError: argument of type 'Header' is not iterable


But msghdr["subject"] is surely just a string isn't it?  Why is it 
complaining about something of type 'Header'?

As I said the program has been running happily for a couple of days 
with no errors, I guess it must be something strange in a mail that
has broken things - but what?

Full program listed below:-

#!/usr/bin/python3
#
#
# licenseApache v2 (http://www.apache.org/licenses/LICENSE-2.0) 
# author Chris Green - ch...@isbd.co.uk
# 
#
#
# Mail filtering script
#
import mailbox
import os
import sys
import time
import mailLib
import shlex
#
#
# Redirect any exceptions to a file
#
sys.stderr = open("/home/chris/tmp/mail.err", 'a')
#
#
# Some constants (i.e. configuration)
#
home = "/home/chris"
logfile = home + "/tmp/mail.log"
filtfile = home + "/.mutt/filter"
mldir = home + "/mail/"
indir = mldir + "In/"
#
#
# Set to log to mail.log in ~/tmp with name 'filter' and the
envelope/from
#
log = mailLib.initLog("filter")
#
#
# Initialise destination mailbox name to its default "In/default"
#
dest = indir + "default"
#
#
# Read the message from standard input and make a message object from it
#
msg = mailbox.MaildirMessage(sys.stdin.buffer.read())
#
#
# Extract various headers and the envelope/from
#
msghdr =  {}
msghdr["to"] = msg.get("To", "unknown").lower()
msghdr["subject"] = msg.get("Subject", "unknown")
msghdr["list-id"] = msg.get("List-Id", "unknown").lower()
msghdr["list-post"] = msg.get("List-Post", "unknown").lower()
msghdr["x-mailing-list"] = msg.get("X-Mailing-List", "unknown").lower()
msghdr["unknown"] = "unknown"
#
#
# See if there's a match in our filter file
#
f = open(filtfile, 'r')
for ln in f:# for each line in filter
if ln[0] == '#':# ignore comments
continue
#
#
# split the line into fields, shlex.split() does quoted strings, add a field
# to create a dummy fourth field if there isn't one in the filter file
#
fld = shlex.split(ln)
fld.append("")
#
#
# copy the fields into better named variables
#
nm = fld[0] # name/alias
destdir = fld[1] + "/"  # the destination directory
header = fld[2] # the header to find the match in
address = fld[3].lower()# list address to match
sbstrip = '[' + fld[4] + ']'# string to strip out of subject
#
#
# Does the address in the header match this entry
#
if (address in msghdr[header]):
#
#
# set the destination directory
#
dest = mldir + destdir + nm
#
#
# Strip out list name (4th field) from subject if it's there
#
if sbstrip in msghdr["subject"]:
msg.replace_header("Subject", msghdr["subject"].replace(sbstrip, 
''))
#
#
# we've found a match so assume we won't get another
#
break
#
#
# deliver the message
#
mailLib.deliverMdMsg(dest, msg, log)

-- 
Chris Green
·
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: I need some help interpreting this error

[2QdxY4RzWzUUiLuE]

On 2021-02-17 at 15:40:27 +,
Chris Green  wrote:

> I'm running this using Python 3.7 on a Linux system.
> 
> Most of the time (i.e. for a couple of days now) the program has been
> satifactorily delivering mail messages, hundreds of them.  However one
> mail message has provoked the following error:-
> 
> chris@cheddar$ tail mail.err
> Traceback (most recent call last):
>   File "/home/chris/.mutt/bin/filter.py", line 95, in 
> if sbstrip in msghdr["subject"]:
> TypeError: argument of type 'Header' is not iterable
> 
> 
> But msghdr["subject"] is surely just a string isn't it?  Why is it 
> complaining about something of type 'Header'?

Isn't it?  ;-)

First step:  Print msghdr["subject"] and its type to know for sure.  The
worst case is that you'll verify your assumption.

IIRC, the subject header is actually optional.  Maybe someone sent a
message without a subject?  Is msghdr["subject"] None?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: I need some help interpreting this error

[Chris Green]

2qdxy4rzwzuui...@potatochowder.com wrote:
> On 2021-02-17 at 15:40:27 +,
> Chris Green  wrote:
> 
> > I'm running this using Python 3.7 on a Linux system.
> > 
> > Most of the time (i.e. for a couple of days now) the program has been
> > satifactorily delivering mail messages, hundreds of them.  However one
> > mail message has provoked the following error:-
> > 
> > chris@cheddar$ tail mail.err
> > Traceback (most recent call last):
> >   File "/home/chris/.mutt/bin/filter.py", line 95, in 
> > if sbstrip in msghdr["subject"]:
> > TypeError: argument of type 'Header' is not iterable
> > 
> > 
> > But msghdr["subject"] is surely just a string isn't it?  Why is it 
> > complaining about something of type 'Header'?
> 
> Isn't it?  ;-)
> 
> First step:  Print msghdr["subject"] and its type to know for sure.  The
> worst case is that you'll verify your assumption.
> 
The documentation says "Headers are represented by customized
subclasses of str", so it's a sub-class of str.  


> IIRC, the subject header is actually optional.  Maybe someone sent a
> message without a subject?  Is msghdr["subject"] None?

If you look at the code (and the documentation) if there's no subject
header I'll get the string "unknown", I've also tried sending myself
an E-Mail with no header and not provoked the error.


-- 
Chris Green
·
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: I need some help interpreting this error

[2QdxY4RzWzUUiLuE]

On 2021-02-17 at 16:42:03 +,
Chris Green  wrote:

> 2qdxy4rzwzuui...@potatochowder.com wrote:
> > On 2021-02-17 at 15:40:27 +,
> > Chris Green  wrote:
> > 
> > > I'm running this using Python 3.7 on a Linux system.
> > > 
> > > Most of the time (i.e. for a couple of days now) the program has been
> > > satifactorily delivering mail messages, hundreds of them.  However one
> > > mail message has provoked the following error:-
> > > 
> > > chris@cheddar$ tail mail.err
> > > Traceback (most recent call last):
> > >   File "/home/chris/.mutt/bin/filter.py", line 95, in 
> > > if sbstrip in msghdr["subject"]:
> > > TypeError: argument of type 'Header' is not iterable
> > > 
> > > 
> > > But msghdr["subject"] is surely just a string isn't it?  Why is it 
> > > complaining about something of type 'Header'?
> > 
> > Isn't it?  ;-)
> > 
> > First step:  Print msghdr["subject"] and its type to know for sure.  The
> > worst case is that you'll verify your assumption.
> > 
> The documentation says "Headers are represented by customized
> subclasses of str", so it's a sub-class of str.  

So we still don't know what the content of msghdr["subject"] is at the
time the error occurs.  I don't mean to sound harsh, but that the
documentation and the code are correct, and that they match, remain
assumptions.  Sometimes, seeing an actual value tells you what went
wrong (e.g., "oh, that's the sender's address, not the receiver's
address," "oh, that's my 'time' class, not the one from the standard
library").

The traceback tells you that msghdr["subject"] is of type Header.  Is
Header a sub-class of str?

Again, the worst case of looking at the value (whether in a log or in a
debugger) is that you verify your assumption.

> > IIRC, the subject header is actually optional.  Maybe someone sent a
> > message without a subject?  Is msghdr["subject"] None?
> 
> If you look at the code (and the documentation) if there's no subject
> header I'll get the string "unknown", I've also tried sending myself
> an E-Mail with no header and not provoked the error.

That's good news about subject-less emails not generating an error, and
separate (possibly related) good news about your code handling an email
without a subject header.  ;-)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: I need some help interpreting this error

[Chris Green]

Stefan Ram  wrote:
> Chris Green  writes:
> >But msghdr["subject"] is surely just a string isn't it?  Why is it 
> >complaining about something of type 'Header'?
> 
>   What would you do to debug-print the type of an object?
> 
I don't know, what would I do?  :-)

Without knowing what provokes the problem I could be waiting for days
or weeks even before I see the error again.  As I'd need to print the
type for every message I'd get some big logs or I'd need to add a try:
except to trap the specific error.

-- 
Chris Green
·
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: I need some help interpreting this error

[Chris Green]

2qdxy4rzwzuui...@potatochowder.com wrote:
> On 2021-02-17 at 16:42:03 +,
> Chris Green  wrote:
> 
> > 2qdxy4rzwzuui...@potatochowder.com wrote:
> > > On 2021-02-17 at 15:40:27 +,
> > > Chris Green  wrote:
> > > 
> > > > I'm running this using Python 3.7 on a Linux system.
> > > > 
> > > > Most of the time (i.e. for a couple of days now) the program has been
> > > > satifactorily delivering mail messages, hundreds of them.  However one
> > > > mail message has provoked the following error:-
> > > > 
> > > > chris@cheddar$ tail mail.err
> > > > Traceback (most recent call last):
> > > >   File "/home/chris/.mutt/bin/filter.py", line 95, in 
> > > > if sbstrip in msghdr["subject"]:
> > > > TypeError: argument of type 'Header' is not iterable
> > > > 
> > > > 
> > > > But msghdr["subject"] is surely just a string isn't it?  Why is it 
> > > > complaining about something of type 'Header'?
> > > 
> > > Isn't it?  ;-)
> > > 
> > > First step:  Print msghdr["subject"] and its type to know for sure.  The
> > > worst case is that you'll verify your assumption.
> > > 
> > The documentation says "Headers are represented by customized
> > subclasses of str", so it's a sub-class of str.  
> 
> So we still don't know what the content of msghdr["subject"] is at the
> time the error occurs.  I don't mean to sound harsh, but that the
> documentation and the code are correct, and that they match, remain
> assumptions.  Sometimes, seeing an actual value tells you what went
> wrong (e.g., "oh, that's the sender's address, not the receiver's
> address," "oh, that's my 'time' class, not the one from the standard
> library").
> 
> The traceback tells you that msghdr["subject"] is of type Header.  Is
> Header a sub-class of str?
> 
That's exactly what puzzled me! The line that gets the value is:-

msghdr["subject"] = msg.get("Subject", "unknown")

What I need to know is how that can return a value of type Header, and
not a str.

> Again, the worst case of looking at the value (whether in a log or in a
> debugger) is that you verify your assumption.
> 
> > > IIRC, the subject header is actually optional.  Maybe someone sent a
> > > message without a subject?  Is msghdr["subject"] None?
> > 
> > If you look at the code (and the documentation) if there's no subject
> > header I'll get the string "unknown", I've also tried sending myself
> > an E-Mail with no header and not provoked the error.
> 
> That's good news about subject-less emails not generating an error, and
> separate (possibly related) good news about your code handling an email
> without a subject header.  ;-)

I think the only sane approach at the moment may be to add a try:
except: and output some diagnostic information.  Though there may
still be an issue when trying to output the "what is it" object to the
error log of course.

-- 
Chris Green
·
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: I need some help interpreting this error

[Stestagg]

I don't particularly like to encourage this shotgun help request because,
as previous commenter suggests, debugging this yourself is best.

Sometimes debugging is super hard, and especially so when uncommon
situations occur, but it's always far easier to debug things when you have
visibility into the system under test.

However, in this case, the email code is super complex, and this scenario
also looks very uncommon, but not unique: (
https://github.com/Sydius/mbox-to-txt/issues/2), so you successfully
nerd-sniped me :).

My *guess*, from reading the python standard library source code is that
you came across an email with some content in the subject line that is
considered a "surrogate", roughly, some badly encoded unicode or binary
data in it.

When this happens, the code in some situations (depending on the policy...)
may return a header.Header() instance, rather than a
headerregistry.UniqueUnstructuredHeader
(which would have had a headerregistry.BaseHeader (mro: str) dynamically
attached).

header.Header() does not inherit from str, and thus would throw the
traceback you observed.

Your suggestion of a try: catch: may make sense, alternately, you could
wrap the result in a call to str():

if sbstrip in str(msghdr["subject"]):

which should attempt to encode the binary into some form of string object
for comparison (I haven't checked exactly what would happen, beyond: it
tries).

It should be possible to create a test mbox with some funky bytes in the
subject, and try to reproduce it that way.

Steve


On Wed, Feb 17, 2021 at 5:07 PM Chris Green  wrote:

> Stefan Ram  wrote:
> > Chris Green  writes:
> > >But msghdr["subject"] is surely just a string isn't it?  Why is it
> > >complaining about something of type 'Header'?
> >
> >   What would you do to debug-print the type of an object?
> >
> I don't know, what would I do?  :-)
>
> Without knowing what provokes the problem I could be waiting for days
> or weeks even before I see the error again.  As I'd need to print the
> type for every message I'd get some big logs or I'd need to add a try:
> except to trap the specific error.
>
> --
> Chris Green
> ·
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: I need some help interpreting this error

[Chris Green]

Stefan Ram  wrote:
> Chris Green  writes:
> >chris@cheddar$ tail mail.err
> >Traceback (most recent call last):
> >  File "/home/chris/.mutt/bin/filter.py", line 95, in 
> >if sbstrip in msghdr["subject"]:
> >TypeError: argument of type 'Header' is not iterable
> >But msghdr["subject"] is surely just a string isn't it?  Why is it 
> >complaining about something of type 'Header'?
> 
>   I presume that the error message has been edited (abbreviated).
> 
>   In "if sbstrip in msghdr["subject"]:", there is no argument.
> 
>   But the error message says "argument of ...".
> 
>   When something that is not iterable is presented to a for loop,
>   the error message does not mention "argument":
> 
I have output everything that appears, I've not changed it at all.
It's the whole content of the file ~/tmp/mail.err as it's the only
error that has occurred for the last day or so.  The error log is
created by the line:-

sys.stderr = open("/home/chris/tmp/mail.err", 'a')

So that's everything that was output to stderr.

I think you are puzzled in the same way that I was, the error message
doesn't make a lot of sense.  

> |>>> for i in 0:
> |...print(i)
> |...
> |Traceback (most recent call last):
> |  File "", line 1, in 
> |TypeError: 'int' object is not iterable
> 
>   .
> 
> 

-- 
Chris Green
·
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: I need some help interpreting this error

[Stestagg]

Some sources, in case they help:

Message.get() calls policy.header_fetch_parse (
https://github.com/python/cpython/blob/cd80f430daa7dfe7feeb431ed34f88db5f64aa30/Lib/email/message.py#L471
)
Compat32.header_fetch_parse calls self._sanitize_header (
https://github.com/python/cpython/blob/cd80f430daa7dfe7feeb431ed34f88db5f64aa30/Lib/email/_policybase.py#L311
)
_sanitize_header calls _has_surrogates (
https://github.com/python/cpython/blob/cd80f430daa7dfe7feeb431ed34f88db5f64aa30/Lib/email/_policybase.py#L287
)
_has_surrogates check:
https://github.com/python/cpython/blob/cd80f430daa7dfe7feeb431ed34f88db5f64aa30/Lib/email/utils.py#L51



On Wed, Feb 17, 2021 at 5:42 PM Stestagg  wrote:

> I don't particularly like to encourage this shotgun help request because,
> as previous commenter suggests, debugging this yourself is best.
>
> Sometimes debugging is super hard, and especially so when uncommon
> situations occur, but it's always far easier to debug things when you have
> visibility into the system under test.
>
> However, in this case, the email code is super complex, and this scenario
> also looks very uncommon, but not unique: (
> https://github.com/Sydius/mbox-to-txt/issues/2), so you successfully
> nerd-sniped me :).
>
> My *guess*, from reading the python standard library source code is that
> you came across an email with some content in the subject line that is
> considered a "surrogate", roughly, some badly encoded unicode or binary
> data in it.
>
> When this happens, the code in some situations (depending on the
> policy...) may return a header.Header() instance, rather than a
> headerregistry.UniqueUnstructuredHeader (which would have had a
> headerregistry.BaseHeader (mro: str) dynamically attached).
>
> header.Header() does not inherit from str, and thus would throw the
> traceback you observed.
>
> Your suggestion of a try: catch: may make sense, alternately, you could
> wrap the result in a call to str():
>
> if sbstrip in str(msghdr["subject"]):
>
> which should attempt to encode the binary into some form of string object
> for comparison (I haven't checked exactly what would happen, beyond: it
> tries).
>
> It should be possible to create a test mbox with some funky bytes in the
> subject, and try to reproduce it that way.
>
> Steve
>
>
> On Wed, Feb 17, 2021 at 5:07 PM Chris Green  wrote:
>
>> Stefan Ram  wrote:
>> > Chris Green  writes:
>> > >But msghdr["subject"] is surely just a string isn't it?  Why is it
>> > >complaining about something of type 'Header'?
>> >
>> >   What would you do to debug-print the type of an object?
>> >
>> I don't know, what would I do?  :-)
>>
>> Without knowing what provokes the problem I could be waiting for days
>> or weeks even before I see the error again.  As I'd need to print the
>> type for every message I'd get some big logs or I'd need to add a try:
>> except to trap the specific error.
>>
>> --
>> Chris Green
>> ·
>> --
>> https://mail.python.org/mailman/listinfo/python-list
>>
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: I need some help interpreting this error

[Chris Green]

Stestagg  wrote:
> I don't particularly like to encourage this shotgun help request because,
> as previous commenter suggests, debugging this yourself is best.
> 
> Sometimes debugging is super hard, and especially so when uncommon
> situations occur, but it's always far easier to debug things when you have
> visibility into the system under test.
> 
> However, in this case, the email code is super complex, and this scenario
> also looks very uncommon, but not unique: (
> https://github.com/Sydius/mbox-to-txt/issues/2), so you successfully
> nerd-sniped me :).
> 
> My *guess*, from reading the python standard library source code is that
> you came across an email with some content in the subject line that is
> considered a "surrogate", roughly, some badly encoded unicode or binary
> data in it.
> 
> When this happens, the code in some situations (depending on the policy...)
> may return a header.Header() instance, rather than a
> headerregistry.UniqueUnstructuredHeader
> (which would have had a headerregistry.BaseHeader (mro: str) dynamically
> attached).
> 
> header.Header() does not inherit from str, and thus would throw the
> traceback you observed.
> 
Ah, thank you, a possible explanation.


> Your suggestion of a try: catch: may make sense, alternately, you could
> wrap the result in a call to str():
> 
> if sbstrip in str(msghdr["subject"]):
> 
> which should attempt to encode the binary into some form of string object
> for comparison (I haven't checked exactly what would happen, beyond: it
> tries).
> 
Yes, I did consider the str() approach but it feels a bit like making
the problem go away without actually fixing it.

However since the code in question is only 'cosmetic' (removing
unwanted [list name] from the subject, it's not all *that* important
to handle it properly.  I just need to stop the error from killing my
program.


> It should be possible to create a test mbox with some funky bytes in the
> subject, and try to reproduce it that way.
> 
That's a point, with the clues you have given me I can try some 'bad'
subject text and see if I can reproduce the error.

Thanks again.

-- 
Chris Green
·
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: I need some help interpreting this error

[2QdxY4RzWzUUiLuE]

On 2021-02-17 at 16:52:55 +,
Chris Green  wrote:

> Stefan Ram  wrote:
> > Chris Green  writes:
> > >But msghdr["subject"] is surely just a string isn't it?  Why is it 
> > >complaining about something of type 'Header'?
> > 
> >   What would you do to debug-print the type of an object?
> > 
> I don't know, what would I do?  :-)
> 
> Without knowing what provokes the problem I could be waiting for days
> or weeks even before I see the error again.  As I'd need to print the
> type for every message I'd get some big logs or I'd need to add a try:
> except to trap the specific error.

An intermittent problem!  They're our favorite!  ;-)

Seriously, though, do you have a message that provokes the problem?  Is
there a log at all?  Can you re-run that message through the code, in a
debugger or otherwise?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 2.7 and 3.9

[J. Pic]

The best would be to upgrade the scripts, did you try them with 2to3 ? It
should do most of the work, if not all.

https://docs.python.org/3/library/2to3.html
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: I need some help interpreting this error

[2QdxY4RzWzUUiLuE]

On 2021-02-17 at 17:36:48 +,
Chris Green  wrote:

> Stefan Ram  wrote:
> > Chris Green  writes:
> > >chris@cheddar$ tail mail.err
> > >Traceback (most recent call last):
> > >  File "/home/chris/.mutt/bin/filter.py", line 95, in 
> > >if sbstrip in msghdr["subject"]:
> > >TypeError: argument of type 'Header' is not iterable
> > >But msghdr["subject"] is surely just a string isn't it?  Why is it 
> > >complaining about something of type 'Header'?
> > 
> >   I presume that the error message has been edited (abbreviated).
> > 
> >   In "if sbstrip in msghdr["subject"]:", there is no argument.
> > 
> >   But the error message says "argument of ...".
> > 
> >   When something that is not iterable is presented to a for loop,
> >   the error message does not mention "argument":
> > 
> I have output everything that appears, I've not changed it at all.
> It's the whole content of the file ~/tmp/mail.err as it's the only
> error that has occurred for the last day or so.  The error log is
> created by the line:-
> 
> sys.stderr = open("/home/chris/tmp/mail.err", 'a')
> 
> So that's everything that was output to stderr.
> 
> I think you are puzzled in the same way that I was, the error message
> doesn't make a lot of sense.  

At some point, the internal code for the "in" operator is likely
iterating over the elements of msghdr["subject"].

That error message doesn't make a lot of sense if msghdr["subject"] is a
sub-class of str.  It makes more sense if msghdr["subject"] is something
else.

IMO, you need more information in the log, a try/except block to prevent
the code from crashing, and, yes, perhaps some patience to wait for it
to happen again.

Sometimes, developing software isn't all fortune, fame, and glory.  ;-)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: New Python implementation

[Alan Gauld via Python-list]

On 16/02/2021 22:23, boB Stepp wrote:

>> And that's just one example, the language is now full of meta goodness
>> that makes it incomprehensible to beginners. 
> 
> Hmm.  I'm not sure I can agree, Alan.  My son took to Python 3 like a duck to
> water.  

That's interesting. I knew you were teaching him but not
how he got on. Good for him.

My comments stem from the feedback emails I get from my tutorial.
People get messages like "foo is not an iterable" or references
to range objects and such and it confuses them.

I never got those kinds of messages for the v.1 tutorial...

> Yes, there is a ton of meta-goodness in Python, but most of it can be ignored
> to be able to *just use it*.  

But the problem I see is that you can't ignore it because
it bubbles to the surface in error messages. And if you
don't know what it means you don't know that you can ignore it!
That's where a teacher/mentor comes in, they can say "Oh that's OK
just ignore it!" but if you are on your own you think
"Oh no! What do I do now?"

But as I say, I can't find anything better so I'll stick
with it for now.

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: New Python implementation

[Alan Gauld via Python-list]

On 16/02/2021 21:22, Tarjei Bærland via Python-list wrote:

> To me, it depends on what you want out of including programming in
> mathematics education.

That's a really important subclass distinction.
If programming is seen as an adjunct to math then the aims
can be simplified considerably since you are only interested
in pure computation. Things like networking, interfacing
to peripherals and the like can be ignored.
Likewise you probably don't care about creating large projects wit
multiple files etc.

With those constraints there are probably better languages
than python. You mention Logo and I had fun with that back
in the 80s and early 90s. But ultimately it wasn't suitable
for the kind of real-world programming I needed.

That's why I chose python for my tutorial. Its not only
teachable at a basic level but it is actually usable on
larger, real-world type projects once you've learned it.
You never need to throw away your skills.

> If the aim is to quickly get the students to "be creative" with
> programming by themselves, my feeling is that Python is too unwieldy for
> a beginner.

Don't underestimate the interactive prompt. It gives instant
feedback and is moire usable for beginners than most Lisp-style
REPLs.

Logo works too of course. But almost anything you can do in
Logo you can do almost as easily in Python. Be it list
handling or turtle graphics.

> minutes. They get the feeling of designing their own computational
> procedure much quicker than in Python. (Yes, of course, the introductory
> excercises in Python are a bit more advanced than "add two numbers".)

But they don't need to be. My tutor starts off with precisely that...

> I am honestly not sure what quite what my feelings are regarding this,
> perhaps Python is the best of all possible options.

Its a good compromise. Its got faults (see my other post!) but
I haven't found anything clearly better.

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: I need some help interpreting this error

[Terry Reedy]

On 2/17/2021 10:40 AM, Chris Green wrote:

I'm running this using Python 3.7 on a Linux system.

Most of the time (i.e. for a couple of days now) the program has been
satifactorily delivering mail messages, hundreds of them.  However one
mail message has provoked the following error:-

 chris@cheddar$ tail mail.err
 Traceback (most recent call last):
   File "/home/chris/.mutt/bin/filter.py", line 95, in 
 if sbstrip in msghdr["subject"]:
 TypeError: argument of type 'Header' is not iterable


But msghdr["subject"] is surely just a string isn't it?


Obviously not.


 Why is it
complaining about something of type 'Header'?


Because you tried to iterate it.  Header is defined in email.header 
(find 'class Header').  It has a __str__ to turn one into a string.  I 
have never read the email doc so I have no idea if 'subject' being a 
Header is a bug.


Grepping email package for case-sensitive word 'Header' also returns 3 
comment saying that something might be a Header, so stringify it.  I 
have the impression that these might have been added after the original 
code, perhaps in response to error reports.  In any case, you can do the 
same.



As I said the program has been running happily for a couple of days
with no errors, I guess it must be something strange in a mail that
has broken things - but what?


To determine that, look at (after logging or saving) the raw email and 
maybe the resulting Message (using msg.as_string).



# Read the message from standard input and make a message object from it
#
msg = mailbox.MaildirMessage(sys.stdin.buffer.read())


raw = sys.stdin.buffer.read()  # So can save
msg = mailbox.MaildirMessage(raw)


msghdr["subject"] = msg.get("Subject", "unknown")

...

 if sbstrip in msghdr["subject"]:


Replace with

  sub = msghdr["subject"]
  if 'Header' in str(sub.__class__):
# Or import email.message.class and use isinstance
   # stringify or skip or ???
   else:

 msg.replace_header("Subject", msghdr["subject"].replace(sbstrip, 


--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list