Re: virtualenv and make DESTDIR=

[Barry Scott]

> On 5 Mar 2022, at 19:56, Hartmut Goebel  wrote:
> 
> Am 05.03.22 um 17:34 schrieb Barry Scott:
>> Have the RPM install all the pythone code and dependencies and also install 
>> a short script that
>> sets up PYTHONPATH, LD_LIBRARY_PATH, etc and execs the python3 .py.
> The scripts are already created by entry-points. So basically this means to 
> reinvent the wheel. Or did I miss something?
> 
I see what are getting at - I did miss the something.
It is the setup.py that will create your entry point!

For my own understanding I turned one of my PyPI projects into an RPM.

The project source is on https://github.com/barry-scott/CLI-tools.git 


First I turned the this into a tarball:

$ git archive --format=tar --prefix=cli-tools-3.1.0/ master >cli-tools-3.1.0.tar
$ gzip cli-tools-3.1.0.tar

Then I created a RPM spec file:

Summary:cli-tools example packaging for python
Name:   cli-tools
Version:3.1.0
Release:1

BuildArch:  noarch
Source: %{name}-%{version}.tar.gz
License:APL2.0
Prefix: /opt/barry-example

BuildRequires:  python3
BuildRequires:  python3-setuptools

Requires:   python3

%description
cli-tools example packaging for python

%prep
# unpack Source tarball
%setup

%build
cd Source
python3 setup_ssh_wait.py build

%install
cd Source
python3 setup_ssh_wait.py install --prefix=%{prefix} --root=%{buildroot}

# need to add the site-packages folder to the sys.path as
# setup.py assumes it is installing into the systems python library
cat 

Execute in a multiprocessing child dynamic code loaded by the parent process

[Martin Di Paola]

Hi everyone. I implemented time ago a small plugin engine to load code
dynamically.

So far it worked well but a few days ago an user told me that he wasn't
able to run in parallel a piece of code in MacOS.

He was using multiprocessing.Process to run the code and in MacOS, the
default start method for such process is using "spawn". My understanding
is that Python spawns an independent Python server (the child) which
receives what to execute (the target function) from the parent process.

In pseudo code this would be like:

modules = loader() # load the plugins (Python modules at the end)
objs = init(modules) # initialize the plugins

# One of the plugins wants to execute part of its code in parallel
# In MacOS this fails
ch = multiprocessing.Process(target=objs[0].sayhi)
ch.start()

The code fails with "ModuleNotFoundError: No module named 'foo'" (where
'foo' is the name of the loaded plugin).

This is because the parent program sends to the serve (the child) what
needs to execute (objs[0].sayhi) using pickle as the serialization
mechanism.

Because Python does not really serialize code but only enough
information to reload it, the serialization of "objs[0].sayhi" just
points to its module, "foo".

Module which it cannot be imported by the child process.

So the question is, what would be the alternatives and workarounds?

I came with a hack: use a trampoline() function to load the plugins
in the child before executing the target function.

In pseudo code it is:

modules = loader() # load the plugins (Python modules at the end)
objs = init(modules) # initialize the plugins

def trampoline(target_str):
   loader() # load the plugins now that we are in the child process

   # deserialize the target and call it
   target = reduction.loads(target_str)
   target()

# Serialize the real target function, but call in the child
# trampoline(). Because it can be accessed by the child it will
# not fail
target_str = reduction.dumps(objs[0].sayhi)
ch = multiprocessing.Process(target=trampoline, args=(target_str,))
ch.start()

The hack works but is this the correct way to do it?

The following gist has the minimal example code that triggers the issue
and its workaround:
https://gist.github.com/eldipa/d9b02875a13537e72fbce4cdb8e3f282

Thanks!
Martin.
--
https://mail.python.org/mailman/listinfo/python-list

Re: Execute in a multiprocessing child dynamic code loaded by the parent process

[Chris Angelico]

On Sun, 6 Mar 2022 at 23:43, Martin Di Paola  wrote:
>
> Hi everyone. I implemented time ago a small plugin engine to load code
> dynamically.
>
> So far it worked well but a few days ago an user told me that he wasn't
> able to run in parallel a piece of code in MacOS.
>
> He was using multiprocessing.Process to run the code and in MacOS, the
> default start method for such process is using "spawn". My understanding
> is that Python spawns an independent Python server (the child) which
> receives what to execute (the target function) from the parent process.

> Because Python does not really serialize code but only enough
> information to reload it, the serialization of "objs[0].sayhi" just
> points to its module, "foo".
>

Hmm. This is a route that has some tricky hazards on it. Generally, in
Python code, we can assume that a module is itself, no matter what; it
won't be a perfect clone of itself, it will actually be the same
module.

If you want to support multiprocessing, I would recommend
disconnecting yourself from the concept of loaded modules, and instead
identify the target by its module name.

> I came with a hack: use a trampoline() function to load the plugins
> in the child before executing the target function.
>
> In pseudo code it is:
>
> modules = loader() # load the plugins (Python modules at the end)
> objs = init(modules) # initialize the plugins
>
> def trampoline(target_str):
> loader() # load the plugins now that we are in the child process
>
> # deserialize the target and call it
> target = reduction.loads(target_str)
> target()
>
> # Serialize the real target function, but call in the child
> # trampoline(). Because it can be accessed by the child it will
> # not fail
> target_str = reduction.dumps(objs[0].sayhi)
> ch = multiprocessing.Process(target=trampoline, args=(target_str,))
> ch.start()
>
> The hack works but is this the correct way to do it?
>

The way you've described it, it's a hack. Allow me to slightly redescribe it.

modules = loader()
objs = init(modules)

def invoke(mod, func):
# I'm assuming that the loader is smart enough to not load
# a module that's already loaded. Alternatively, load just the
# module you need, if that's a possibility.
loader()
target = getattr(modules[mod], func)
target()

ch = multiprocessing.Process(target=invoke, args=("some_module", "sayhi"))
ch.start()


Written like this, it achieves the same goal, but looks a lot less
hacky, and as such, I would say that yes, this absolutely IS a correct
way to do it. (I won't say "the" correct way, as there are other valid
ways, but there's certainly nothing wrong with this idea.)

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Issues of pip install gdal and fiona

[Shaozhong SHI]

I downloaded .whl files for fiona and gdal to go with Python3.6.5.

However, I am having trouble with red error messages.

Though Gdal is now working, there is a warning message - Missing global ~
gdal: DRIVER_NAME declaration   gdal_array,py

Can anyone advise on how to resolve the issues?

Regards,

David
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Cpython: when to incref before insertdict

[Marco Sulla]

On Sun, 6 Mar 2022 at 03:20, Inada Naoki  wrote:
> In general, when reference is borrowed from a caller, the reference is
> available during the API.
> But merge_dict borrows reference of key/value from other dict, not caller.
> [...]
> Again, insertdict takes the reference. So _PyDict_FromKeys() **does**
> INCREF before calling insertdict, when key/value is borrowed
> reference.
> https://github.com/python/cpython/blob/6927632492cbad86a250aa006c1847e03b03e70b/Objects/dictobject.c#L2287-L2290
> https://github.com/python/cpython/blob/6927632492cbad86a250aa006c1847e03b03e70b/Objects/dictobject.c#L2309-L2311
>
> On the other hand, slow path uses PyIter_Next() which returns strong
> reference. So no need to INCREF it.

Thank you Inada, these points make me things clear now.
(PS: dictobject will change a lot in 3.11... sigh :D)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Behavior of the for-else construct

[Peter J. Holzer]

On 2022-03-05 14:25:35 -0500, Dennis Lee Bieber wrote:
> On Sat, 5 Mar 2022 12:39:36 -0600, "Michael F. Stemper"
>  declaimed the following:
> >... especially Pascal, which was probably bigger in Germany and Austria
> >in the 1980s than was C.
> 
>   Pascal also defined alternate representations (per Jensen&Wirth) for
> some of those (and I don't recall ever seeing a system that actually had an
> up-arrow character -- and selecting one in character map doesn't help, my
> client doesn't render it).
> 
> directalternate
> ? ^ or @  
> [ (.
> ] .)
> { (*
> } *)

(* *) for comments was actually pretty commonly used - maybe because it
stands out more than { }. I don't know if I've ever seen (. .) instead
of [ ].

C also has alternative rerpresentations for characters not in the common
subset of ISO-646 and EBCDIC. However, the trigraphs are extremely ugly
(e.g ??< ??> instead of { }). I have seen them used (on an IBM/390
system with an EBCDIC variant without curly braces) and it's really no
fun to read that.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: virtualenv and make DESTDIR=

[Peter J. Holzer]

On 2022-03-05 17:59:48 +0100, Marco Sulla wrote:
> On Sat, 5 Mar 2022 at 17:36, Barry Scott  wrote:
> > Note: you usually cannot use pip when building an RPM with mock as the 
> > network is disabled inside the build for
> > security reasons.
> 
> Can't he previously download the packages and run pip on the local packages?

That shouldn't be necessary. Any packages needed by that new package
should be dependencies of that package, so they must also already exist
as RPMs. The rpm build process will automatically install them if they
aren't already.

(That may not be true for build dependencies, though.)

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Timezone for datetime.date objects

[Morten W. Petersen]

On Wed, Mar 2, 2022 at 6:20 PM Peter J. Holzer  wrote:

> On 2022-02-28 23:28:23 +0100, Morten W. Petersen wrote:
> > Well, let's say I specify the datetime 2022-02-22 02:02 (AM). I think
> > everyone could agree that it also means 2022-02-22 02:02:00:00, to
> > 2022-02-22 02:02:59:59.
>
> I disagree. The datetime 2022-02-22 02:02 specifies a point in time, not
> a time period. It means 2022-02-22 02:02:00.0.
>
> In reality, a point in time may be fuzzy. "The train departs at 02:02"
> definitely doesn't mean that the train will depart exactly at 02:02:00,
> but it also doesn't mean that it will depart between 02:02 and 02:03.
> Rather it's a smooth probability distribution starting a bit before
> 02:02:00 (a train should never leave early, but sometimes clocks are
> wrong or somebody doesn't pay attention) a peak shortly after 02:02:00
> and a very long tail.
>

Well, the precision we have is probably nowhere as close as it will be
possible
to specify in the future, so it depends on context.

But I agree on the train departing time, it should never be before the
scheduled time, and yes, probably peaks right after the departure time.

If I say I'll meet someone at 2 o'clock, that means 14:00, coming 5 minutes
early or late is acceptable.  So saying that implies a range in the given
context.

If I have a bill with a due date, it means I have to pay it on that date or
earlier
to do it correctly. Or even a couple of days later, if the due date is a
holiday
and it's not possible to make a bank transfer that day.

But with crypto currencies it is. :)

If I say I'll do something on the 15th of March, it would be reasonable to
think
that means between 06:00 and 21:00 on that day, but having it done at any
time that day would be correct.

So I think the correct interpretation depends on context, and if I say
14:02,
in a computer and programming context, that means 14:02:00 to 14:02:59,
because the seconds aren't specified, and 14:01 would not be a match, and
14:03 would also not be a match.

If I had specified in terms of seconds into that day

>>> (14*60*60)+(2*60)
50520

the implied and correct range would have been 50520.0 to 50520.9.



>
> > And I think the same applies for a date.
>
> Depends on the context, but without s specific context (like business
> days) I would agree. A day *is* a time period with a beginning and an
> end.
>

Yeah, more or less what I'm saying above.

Regards,

Morten


-- 
I am https://leavingnorway.info
Videos at https://www.youtube.com/user/TheBlogologue
Twittering at http://twitter.com/blogologue
Blogging at http://blogologue.com
Playing music at https://soundcloud.com/morten-w-petersen
Also playing music and podcasting here:
http://www.mixcloud.com/morten-w-petersen/
On Google+ here https://plus.google.com/107781930037068750156
On Instagram at https://instagram.com/morphexx/
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Getting Syslog working on OSX Monterey

[Peter J. Holzer]

On 2022-03-05 16:25:38 +, Barry Scott wrote:
> On 4 Mar 2022, at 21:23, Peter J. Holzer  wrote:
> > If you are saying that SysLogHandler should use a system specific
> > default (e.g. "/dev/log" on Linux) instead of UDP port 514 everywhere, I
> > agree 99 % (the remaining 1 % is my contrarian alter ego arguing that
> > that's really the distribution maintainer's job since a Linux
> > distribution might use some other socket).
> > 
> > If you are saying it should use the libc syslog routines, I disagree for
> > at least two reasons: a) they are OS specific, b) you can't configure the
> > destination. So that would remove useful functionality.
> > 
> > In any case it seems strange to me that you want to rewrite it just to
> > avoid passing a single parameter to the constructor (or - more likely -
> > adding a single line to a config file).
> 
> What I am used to is a setup where programs use syslog() to log and
> a dameon like rsyslog is responsible for routing the logs to local files
> and/or remote systems.

Yes, that's a pretty common setup.

> On a modern linux I think the route is syslog() -> journald. And you
> can have syslog() -> journald -> rsyslogd.

Yes. Journald accepts syslog (RFC 5424) protocol on /dev/log and can
optionally forward messages to another syslog daemon (but only via Unix
sockets, AFAICS, not via UDP or TCP). rsyslog is then configured to
accept those forwarded messages (and may also be configured to accept
messages on port 514, but this isn't the case by default on Debian or
Ubuntu).


> Using the syslog() function means that any platform/distro details are
> hidden from the user of syslog() and as is the case of macOS it
> "just works".

That doesn't seem to be case. Philip Bloom reported in
,
that syslog in Python 3.6 broke in OS X Monterey. So even using
syslog(3) doesn't seem to be safe across upgrades (no, I don't know
how Apple managed to break this - as I wrote, I don't use OS X).


> (I assume, not checked, that the write to the socket does not work
> because Apple is not implementing the syslog protocol from the RFC).
> 
> > 
> >> What you do not see used in the SyslogHandler() is the import syslog
> >> and hence its nor using openlog() etc from syslog API.
> > 
> > For good reasons. The C syslog API is missing important functionality.
> 
> What are you think about being missing? Just curious.

As I already wrote:

* System independence. The libc syslog(3) function (and the Python
  syslog package using it) exists only on Unix-like OSs. Not Windows.
  Probably not some embedded platforms. SyslogHandler works on all
  platforms with networking.
* Configurable targets. The SyslogHandler allows you to specify where to
  send the messages. syslog(3) does not. So with syslog(3) you always
  need a locally running daemon. In containers or embedded systems this
  might add extra unnecessary complexity.

> As an aside if I had the need to log into a system log mechanism I'd be
> looking to use the jounald API so that I can use structured logging on linux
> systems.

That might be useful. Journald already adds a bit of extra information
to messages received over the syslog, but using the API directly would
probably give the user more control.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Behavior of the for-else construct

[Grant Edwards]

On 2022-03-05, Avi Gross via Python-list  wrote:

> I am not sure how we end up conversing about PASCAL on a Python
> forum.
> [...]
> I paid no attention to where PASCAL was being used other than I did
> much of my grad school work in PASCAL [...]

It's "Pascal". It's not an acronym. It's a guy's name:

https://en.wikipedia.org/wiki/Blaise_Pascal

-- 
Grant
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Behavior of the for-else construct

[Peter J. Holzer]

On 2022-03-06 09:29:19 -0800, Grant Edwards wrote:
> On 2022-03-05, Avi Gross via Python-list  wrote:
> > I am not sure how we end up conversing about PASCAL on a Python
> > forum.
> > [...]
> > I paid no attention to where PASCAL was being used other than I did
> > much of my grad school work in PASCAL [...]
> 
> It's "Pascal". It's not an acronym. It's a guy's name:
> 
> https://en.wikipedia.org/wiki/Blaise_Pascal

And similarly, it's "Perl", not "PERL" (also misspelled in this thread).

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Execute in a multiprocessing child dynamic code loaded by the parent process

[Dieter Maurer]

Martin Di Paola wrote at 2022-3-6 12:42 +:
>Hi everyone. I implemented time ago a small plugin engine to load code
>dynamically.
>
>So far it worked well but a few days ago an user told me that he wasn't
>able to run in parallel a piece of code in MacOS.
>
>He was using multiprocessing.Process to run the code and in MacOS, the
>default start method for such process is using "spawn". My understanding
>is that Python spawns an independent Python server (the child) which
>receives what to execute (the target function) from the parent process.
>
>In pseudo code this would be like:
>
>modules = loader() # load the plugins (Python modules at the end)
>objs = init(modules) # initialize the plugins
>
># One of the plugins wants to execute part of its code in parallel
># In MacOS this fails
>ch = multiprocessing.Process(target=objs[0].sayhi)
>ch.start()
>
>The code fails with "ModuleNotFoundError: No module named 'foo'" (where
>'foo' is the name of the loaded plugin).
>
>This is because the parent program sends to the serve (the child) what
>needs to execute (objs[0].sayhi) using pickle as the serialization
>mechanism.
>
>Because Python does not really serialize code but only enough
>information to reload it, the serialization of "objs[0].sayhi" just
>points to its module, "foo".
>
>Module which it cannot be imported by the child process.
>
>So the question is, what would be the alternatives and workarounds?

Try to use `fork` as "start method" (instead of "spawn").
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Getting Syslog working on OSX Monterey

[Peter J. Holzer]

On 2022-03-06 18:28:59 +0100, Peter J. Holzer wrote:
> On 2022-03-05 16:25:38 +, Barry Scott wrote:
> > Using the syslog() function means that any platform/distro details are
> > hidden from the user of syslog() and as is the case of macOS it
> > "just works".
> 
> That doesn't seem to be case. Philip Bloom reported in
> ,
> that syslog in Python 3.6 broke in OS X Monterey. So even using
> syslog(3) doesn't seem to be safe across upgrades (no, I don't know
> how Apple managed to break this - as I wrote, I don't use OS X).
> 
> 
> > (I assume, not checked, that the write to the socket does not work
> > because Apple is not implementing the syslog protocol from the RFC).

I noticed that RFC 5424 describes a very different protocol from RFC
3164 and was curious which one SysLogHandler implements.

Surprise: Neither.

It just sends the PRI part and the MSG part. While that neatly sidesteps
the question of which of the two incompatible header formats to use,
it's not a a valid syslog message in either format.

Obviously Linux accepts this format (the timestamp and the hostname are
kind of redundant anyway when you're using a unix socket).

I strongly suspect that the OS X syslog daemon got stricter and now
insists on properly formatted messages.

hp


-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Execute in a multiprocessing child dynamic code loaded by the parent process

[Martin Di Paola]

The way you've described it, it's a hack. Allow me to slightly redescribe it.

modules = loader()
objs = init(modules)

def invoke(mod, func):
   # I'm assuming that the loader is smart enough to not load
   # a module that's already loaded. Alternatively, load just the
   # module you need, if that's a possibility.
   loader()
   target = getattr(modules[mod], func)
   target()

ch = multiprocessing.Process(target=invoke, args=("some_module", "sayhi"))
ch.start()



Yeup, that would be my first choice but the catch is that "sayhi" may
not be a function of the given module. It could be a static method of
some class or any other callable.

And doing the lookup by hand sounds complex.

The thing is that the use of multiprocessing is not something required by me
(by my plugin-engine), it was a decision of the developer of a particular
plugin so I don't have any control on that.

Using multiprocessing.reduction was a practical decision: if the user
wants to call something non-pickleable, it is not my fault, it is
multiprocessing's fault.

It *would* be my fault if multiprocessing.Process fails only because I'm
loading the code dynamically.


[...] I won't say "the" correct way, as there are other valid
ways, but there's certainly nothing wrong with this idea.


Do you have some in mind? Or may be a project that I could read?

Thanks!
Martin
--
https://mail.python.org/mailman/listinfo/python-list

Re: Execute in a multiprocessing child dynamic code loaded by the parent process

[Martin Di Paola]

Try to use `fork` as "start method" (instead of "spawn").


Yes but no. Indeed with `fork` there is no need to pickle anything. In
particular the child process will be a copy of the parent so it will
have all the modules loaded, including the dynamic ones. Perfect.

The problem is that `fork` is the default only in Linux. It works in
MacOS but it may lead to crashes if the parent process is multithreaded
(and the my is!) and `fork` does not work in Windows.
--
https://mail.python.org/mailman/listinfo/python-list

Re: Execute in a multiprocessing child dynamic code loaded by the parent process

[Chris Angelico]

On Mon, 7 Mar 2022 at 07:37, Martin Di Paola  wrote:
>
>
>
> >
> >The way you've described it, it's a hack. Allow me to slightly redescribe it.
> >
> >modules = loader()
> >objs = init(modules)
> >
> >def invoke(mod, func):
> ># I'm assuming that the loader is smart enough to not load
> ># a module that's already loaded. Alternatively, load just the
> ># module you need, if that's a possibility.
> >loader()
> >target = getattr(modules[mod], func)
> >target()
> >
> >ch = multiprocessing.Process(target=invoke, args=("some_module", "sayhi"))
> >ch.start()
> >
>
> Yeup, that would be my first choice but the catch is that "sayhi" may
> not be a function of the given module. It could be a static method of
> some class or any other callable.

Ah, fair. Are you able to define it by a "path", where each step in
the path is a getattr() call?

The trouble is, arbitrary callables might not be available in a
reconstructed version of the module.

> Using multiprocessing.reduction was a practical decision: if the user
> wants to call something non-pickleable, it is not my fault, it is
> multiprocessing's fault.
>
> It *would* be my fault if multiprocessing.Process fails only because I'm
> loading the code dynamically.

Fair. I guess, then, that the best thing to do is to preload the
modules, then unpickle. So, basically what you already have, but with
more caveats.

> Do you have some in mind? Or may be a project that I could read?

Not handy, but there are always many different ways to do things. For
instance, instead of saying "spawn a subprocess and call this
function", you could invert it, and have the function register itself
as the target. Then it's just "spawn a subprocess and load this
module", and that calls the registered invocation. It all depends on
what the rest of your project is doing. Mainly, though, I'm just not
ruling out the possibility of other options :)

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: virtualenv and make DESTDIR=

[Barry]

> On 6 Mar 2022, at 16:53, Peter J. Holzer  wrote:
> 
> On 2022-03-05 17:59:48 +0100, Marco Sulla wrote:
>>> On Sat, 5 Mar 2022 at 17:36, Barry Scott  wrote:
>>> Note: you usually cannot use pip when building an RPM with mock as the 
>>> network is disabled inside the build for
>>> security reasons.
>> 
>> Can't he previously download the packages and run pip on the local packages?
> 
> That shouldn't be necessary. Any packages needed by that new package
> should be dependencies of that package, so they must also already exist
> as RPMs. The rpm build process will automatically install them if they
> aren't already.
> 
> (That may not be true for build dependencies, though.)

In the spec file use BuildRequires: to pull in an rpm package that is needed to 
do the build.
For example python3-setuptools.

Barry

> 
>hp
> 
> -- 
>   _  | Peter J. Holzer| Story must make more sense than reality.
> |_|_) ||
> | |   | h...@hjp.at |-- Charles Stross, "Creative writing
> __/   | http://www.hjp.at/ |   challenge!"
> -- 
> https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Getting Syslog working on OSX Monterey

[Barry]

> On 6 Mar 2022, at 19:38, Peter J. Holzer  wrote:
> 
> On 2022-03-06 18:28:59 +0100, Peter J. Holzer wrote:
>>> On 2022-03-05 16:25:38 +, Barry Scott wrote:
>>> Using the syslog() function means that any platform/distro details are
>>> hidden from the user of syslog() and as is the case of macOS it
>>> "just works".
>> 
>> That doesn't seem to be case. Philip Bloom reported in
>> ,
>> that syslog in Python 3.6 broke in OS X Monterey. So even using
>> syslog(3) doesn't seem to be safe across upgrades (no, I don't know
>> how Apple managed to break this - as I wrote, I don't use OS X).
>> 
>> 
>>> (I assume, not checked, that the write to the socket does not work
>>> because Apple is not implementing the syslog protocol from the RFC).
> 
> I noticed that RFC 5424 describes a very different protocol from RFC
> 3164 and was curious which one SysLogHandler implements.
> 
> Surprise: Neither.

So the logging code works by accident…

> 
> It just sends the PRI part and the MSG part. While that neatly sidesteps
> the question of which of the two incompatible header formats to use,
> it's not a a valid syslog message in either format.
> 
> Obviously Linux accepts this format (the timestamp and the hostname are
> kind of redundant anyway when you're using a unix socket).
> 
> I strongly suspect that the OS X syslog daemon got stricter and now
> insists on properly formatted messages.

Interesting. If I can find a spare hour I can play with this.

Barry

> 
>hp
> 
> 
> -- 
>   _  | Peter J. Holzer| Story must make more sense than reality.
> |_|_) ||
> | |   | h...@hjp.at |-- Charles Stross, "Creative writing
> __/   | http://www.hjp.at/ |   challenge!"
> -- 
> https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Behavior of the for-else construct

[Avi Gross via Python-list]

>>>
Pascal versus PASCAL versus pascal (not versus paschal) and
Perl versus PERL versus perl (not versus pearl)

seems to be a topic.
<<<

The nitpickers here are irrelevant. I happen to know how things are formally 
spelled and if I was publishing a book, might carefully proofread it.

I sometimes use upper case for Emphasis and do not give a dAMN. But feel free 
to correct my spelling in case you suspect someone here truly misunderstood a 
message because it was written with the wrong case.

I do learn, but when it is something I already know and am not particularly 
interested in proofreading, the result tends to be that I simply get annoyed 
and the one pointing it out is taken as wandering off topic even more than I do.

Python is named after a snake right? So is it upper case or lower case or mixed 
case? No, wait, it is named after Monty Python. But what made them choose that 
name and why would anyone then name a programming language after them? I can 
see some rudimentary reasoning in naming Ada and maybe for Pascal, albeit I can 
think of many others who deserve it as much or more that nobody has chosen to 
honor. Did Turing catch on, I mean as a programming language rather than a 
machine? 

I don't care if someone says PYTHON or Python or python when informally 
discussing things. They all mean the same thing to me. On the other hand, names 
like "R" and "S" and "C" are rarely presented as lower-case while I sometimes 
see c++ or c# for some reason. Obviously the right version is upper case, 
albeit the plus sign and sharp sign have no upper case version.

FORTRAN supposedly stands for FORmula TRANslator or something. So why does it 
have to be all uppercase? Was COBOL named for COmmon Business Oriented 
Language. Should it be CoBOL?

I have lots of awareness that Perl is generally written in mixed case, except 
when it isn't. I have seen people justify the name by saying it was short for 
Practical Extraction and Report Language but also for   Pathologically 
Eclectic Rubbish Lister! If I felt it was important, I would write it whatever 
way is formally required. 

What I think often happens, from a human psychology perspective, is we see Perl 
in middle of a sentence as a spelling mistake while PERL is obviously something 
like an acronym. Pascal may be named for a French person I admire but may still 
look like a nonsense word to those who do not know or care. Not a great excuse, 
of course.

But if this was being graded in these ways, and I see lots of such nitpicking 
in other ways when an example focusing on something does not carefully do lots 
of other things, I probably would focus on the many other things I don't get 
around to doing in my life because I waste the time here. As a voluntary 
effort, I can opt out.

The argument that a programming language is named after a person and thus must 
be spelled some way is not impressing me. Yes, to be strictly correct, it has a 
proper spelling. But following that reasoning, why does anyone give an email 
address of john.sm...@gmail.com or jane...@yahoo.com instead of ...?



-Original Message-
From: Peter J. Holzer 
To: python-list@python.org
Sent: Sun, Mar 6, 2022 12:48 pm
Subject: Re: Behavior of the for-else construct

On 2022-03-06 09:29:19 -0800, Grant Edwards wrote:

> On 2022-03-05, Avi Gross via Python-list  wrote:
> > I am not sure how we end up conversing about PASCAL on a Python
> > forum.
> > [...]
> > I paid no attention to where PASCAL was being used other than I did
> > much of my grad school work in PASCAL [...]
> 
> It's "Pascal". It's not an acronym. It's a guy's name:
> 
>     https://en.wikipedia.org/wiki/Blaise_Pascal


And similarly, it's "Perl", not "PERL" (also misspelled in this thread).

        hp

-- 
   _  | Peter J. Holzer    | Story must make more sense than reality.
|_|_) |                    |
| |   | h...@hjp.at         |    -- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |       challenge!"


-- 
https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Behavior of the for-else construct

[Chris Angelico]

On Mon, 7 Mar 2022 at 09:51, Avi Gross via Python-list
 wrote:
>
> >>>
> Pascal versus PASCAL versus pascal (not versus paschal) and
> Perl versus PERL versus perl (not versus pearl)
>
> seems to be a topic.
> <<<
>
> The nitpickers here are irrelevant. I happen to know how things are formally 
> spelled and if I was publishing a book, might carefully proofread it.

So what you're saying is that you don't really care unless it's in a
book. Good to know. I make errors now and then, but I consider them to
be errors, and I would like to reduce their number.

> Python is named after a snake right? So is it upper case or lower case or 
> mixed case? No, wait, it is named after Monty Python. But what made them 
> choose that name and why would anyone then name a programming language after 
> them? I can see some rudimentary reasoning in naming Ada and maybe for 
> Pascal, albeit I can think of many others who deserve it as much or more that 
> nobody has chosen to honor. Did Turing catch on, I mean as a programming 
> language rather than a machine?
>

If you want strange naming conventions, apparently there's "Apache Pig
Latin", "Ballerina", "Hopscotch", "Mercury", and even this bizarre
thing called "ECMAScript", can't possibly imagine why anyone would use
a thing like that, right?

Things get named, usually by their creators. Deliberately misnaming
something is insulting it.

> I don't care if someone says PYTHON or Python or python when informally 
> discussing things. They all mean the same thing to me. On the other hand, 
> names like "R" and "S" and "C" are rarely presented as lower-case while I 
> sometimes see c++ or c# for some reason. Obviously the right version is upper 
> case, albeit the plus sign and sharp sign have no upper case version.
>
> FORTRAN supposedly stands for FORmula TRANslator or something. So why does it 
> have to be all uppercase? Was COBOL named for COmmon Business Oriented 
> Language. Should it be CoBOL?
>

Because that's what their creators decided. Or are you in a more
authoritative position?

I think you should have been named AVI144, not Avi Gross. We shouldn't
bother to try to call you by your name, it's much more reasonable to
call you something else.

> But if this was being graded in these ways, and I see lots of such nitpicking 
> in other ways when an example focusing on something does not carefully do 
> lots of other things, I probably would focus on the many other things I don't 
> get around to doing in my life because I waste the time here. As a voluntary 
> effort, I can opt out.
>
> The argument that a programming language is named after a person and thus 
> must be spelled some way is not impressing me. Yes, to be strictly correct, 
> it has a proper spelling. But following that reasoning, why does anyone give 
> an email address of john.sm...@gmail.com or jane...@yahoo.com instead of ...?
>

Email addresses consist of a mailbox part, interpreted entirely by the
server, and a domain, which is defined as case insensitive and
conventionally written in lowercase. There's no difference between
john.sm...@gmail.com and john.sm...@gmail.com, but it's up to Google's
servers to decide whether john.smith is the same as JOHN.SMITH (and,
in the case of Google, they've decided that it's the same as johnsmith
and johns.mith too).

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: mac app from a python script?

[Dan Stromberg]

On Sun, Jan 23, 2022 at 9:59 AM Dan Stromberg  wrote:

>
> Hi folks.
>
> I have a Python 3 script (built on top of gi.respository.Gtk) that runs on
> Linux and macOS 11.5.  It's at
> https://stromberg.dnsalias.org/~strombrg/hcm/ if you're curious.
>
> It works the way I want on Linux, but on macOS I seem to have to start it
> from the command line, like:
> hcm --gui
> ...because I don't know how to create a macOS "app" that goes under
> /Applications.
>
> I don't really care about having a single executable on macOS, and I don't
> really care about creating a .dmg or .pkg file. I'd be perfectly happy just
> running "make install" and putting a #!'d script under /Applications with
> appropriate metadata - but if it's easier to do a single executable, .dmg
> or .pkg, I'd be fine with that.
>

It turns out that using something called "appify" does this well.

I found it on a gist I believe.

I liked it so much that I adopted it and put it at
https://stromberg.dnsalias.org/~strombrg/mactools/ , with a few style
changes and small bug fixes.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Behavior of the for-else construct

[Dennis Lee Bieber]

On Sun, 6 Mar 2022 17:39:51 +0100, "Peter J. Holzer" 
declaimed the following:

>
>(* *) for comments was actually pretty commonly used - maybe because it
>stands out more than { }. I don't know if I've ever seen (. .) instead
>of [ ].
>
Or some terminals provided [ ] but not { }  

Modula-2 appears to have fixed on (* *) for comments, and only [ ] for
indexing.

Consider the potential mayhem going from a language where { } are
comment delimiters to one where they are block delimiters 


>C also has alternative rerpresentations for characters not in the common
>subset of ISO-646 and EBCDIC. However, the trigraphs are extremely ugly
>(e.g ??< ??> instead of { }). I have seen them used (on an IBM/390
>system with an EBCDIC variant without curly braces) and it's really no
>fun to read that.
>
My college mainframe used EBCDIC, but the available languages did not
include C or Pascal. We had APL, FORTRAN-IV (in full separate compilation
form, and FLAG [FORTRAN Load and Go] which was a "all in one file, compile
& run" used by first year students), COBOL (74?), BASIC, SNOBOL,
Meta-Symbol and AP (both assemblers, though Meta-Symbol could, provided the
proper definition file, generate absolute binary code for pretty much any
processor), and something called SL-1 (Simulation Language-1, which
produced FORTRAN output for discrete event models).

UCSD Pascal, and PDP-11 assembly were run on a pair of LSI-11 systems.
Assembly used for the operating system principles course.

I didn't encounter "real" C until getting a TRS-80 (first as integer
LC, then Pro-MC), along with Supersoft LISP (on cassette tape!). (I had
books for C and Ada before encountering compilers for them)


-- 
Wulfraed Dennis Lee Bieber AF6VN
wlfr...@ix.netcom.comhttp://wlfraed.microdiversity.freeddns.org/
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Execute in a multiprocessing child dynamic code loaded by the parent process

[Greg Ewing]

On 7/03/22 9:36 am, Martin Di Paola wrote:

It *would* be my fault if multiprocessing.Process fails only because I'm
loading the code dynamically.


I'm not so sure about that. The author of the plugin knows they're
writing code that will be dynamically loaded, and can therefore
expect the kind of problem they're having. It could be argued that
it's their responsibility to ensure that all the needed code is loaded
into the subprocess.

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

C API PyObject_CallFunctionObjArgs returns incorrect result

[Jen Kris via Python-list]

I am using the C API in Python 3.8 with the nltk library, and I have a problem 
with the return from a library call implemented with 
PyObject_CallFunctionObjArgs.  

This is the relevant Python code:

import nltk
from nltk.corpus import gutenberg
fileids = gutenberg.fileids()
sentences = gutenberg.sents(fileids[0])
sentence = sentences[0]
sentence = " ".join(sentence)
pt = nltk.word_tokenize(sentence)

I run this at the Python command prompt to show how it works:
>>> sentence = " ".join(sentence)
>>> pt = nltk.word_tokenize(sentence)
>>> print(pt)
['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ']']
>>> type(pt)


This is the relevant part of the C API code:

PyObject* str_sentence = PyObject_Str(pSentence);  
// nltk.word_tokenize(sentence)  
PyObject* pNltk_WTok = PyObject_GetAttrString(pModule_mstr, "word_tokenize");
PyObject* pWTok = PyObject_CallFunctionObjArgs(pNltk_WTok, str_sentence, 0);

(where pModule_mstr is the nltk library). 

That should produce a list with a length of 7 that looks like it does on the 
command line version shown above:

['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ']']

But instead the C API produces a list with a length of 24, and the REPR looks 
like this:

'[\'[\', "\'", \'[\', "\'", \',\', "\'Emma", "\'", \',\', "\'by", "\'", \',\', 
"\'Jane", "\'", \',\', "\'Austen", "\'", \',\', "\'1816", "\'", \',\', "\'", 
\']\', "\'", \']\']'

I also tried this with PyObject_CallMethodObjArgs and PyObject_Call without 
success. 

Thanks for any help on this. 

Jen

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: C API PyObject_CallFunctionObjArgs returns incorrect result

[MRAB]

On 2022-03-07 00:32, Jen Kris via Python-list wrote:

I am using the C API in Python 3.8 with the nltk library, and I have a problem 
with the return from a library call implemented with 
PyObject_CallFunctionObjArgs.

This is the relevant Python code:

import nltk
from nltk.corpus import gutenberg
fileids = gutenberg.fileids()
sentences = gutenberg.sents(fileids[0])
sentence = sentences[0]
sentence = " ".join(sentence)
pt = nltk.word_tokenize(sentence)

I run this at the Python command prompt to show how it works:

sentence = " ".join(sentence)
pt = nltk.word_tokenize(sentence)
print(pt)

['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ']']

type(pt)



This is the relevant part of the C API code:

PyObject* str_sentence = PyObject_Str(pSentence);
// nltk.word_tokenize(sentence)
PyObject* pNltk_WTok = PyObject_GetAttrString(pModule_mstr, "word_tokenize");
PyObject* pWTok = PyObject_CallFunctionObjArgs(pNltk_WTok, str_sentence, 0);

(where pModule_mstr is the nltk library).

That should produce a list with a length of 7 that looks like it does on the 
command line version shown above:

['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ']']

But instead the C API produces a list with a length of 24, and the REPR looks 
like this:

'[\'[\', "\'", \'[\', "\'", \',\', "\'Emma", "\'", \',\', "\'by", "\'", \',\', "\'Jane", "\'", \',\', "\'Austen", "\'", 
\',\', "\'1816", "\'", \',\', "\'", \']\', "\'", \']\']'

I also tried this with PyObject_CallMethodObjArgs and PyObject_Call without 
success.

Thanks for any help on this.


What is pSentence? Is it what you think it is?
To me it looks like it's either the list:

['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ']']

or that list as a string:

"['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ']']"

and that what you're tokenising.
--
https://mail.python.org/mailman/listinfo/python-list

Re: Execute in a multiprocessing child dynamic code loaded by the parent process

[Martin Di Paola]

I'm not so sure about that. The author of the plugin knows they're
writing code that will be dynamically loaded, and can therefore
expect the kind of problem they're having. It could be argued that
it's their responsibility to ensure that all the needed code is loaded
into the subprocess.


Yes but I try to always make my libs/programs as much as usable as
possible. "Ergonomic" would be the word.

In the case of the plugin-engine I'm trying to hide any side-effect or
unexpected behaviour of the engine so the developer of the plugin
does not have take that into account.

I agree that if the developer uses multiprocessing he/she needs to know
its implications. But if I can "smooth" any rough corner, I will try to
do it.

For example, the main project (developed by me) uses threads for
concurrency. It would be simpler to load the plugins and instantiate
them *once* and ask the plugins developers to take care of any
race condition (RC) within their implementation.

Because the plugins were instantiated *once*, it is almost guaranteed
that the plugins will suffer from race conditions and they will require
some sort of locking.

This is quite risky: you may forget to protect something and you will
end up with a RC and/or you may put the lock in the wrong place and the
whole thing will not work concurrently.

My decision back then was to instantiate each plugin N+1 times: once in
the main thread and then once per worker thread.

With this, no single plugin instance will be shared so there is no risk
of RC and no need for locking. (Yes, I know, the developer just needs to
use a module variable or a class attribute and it will get a RC and
these are shared but it is definitely not the default scenario).

If sharing is required I provide an object that minimizes the locking
needed.

It was much complex for me at the design and at the implementation level
but I think that it is safer and requires less from the plugin
developer.

Reference: https://byexamples.github.io/byexample/contrib/concurrency-model
--
https://mail.python.org/mailman/listinfo/python-list

Re: Execute in a multiprocessing child dynamic code loaded by the parent process

[Martin Di Paola]

Yeup, that would be my first choice but the catch is that "sayhi" may
not be a function of the given module. It could be a static method of
some class or any other callable.


Ah, fair. Are you able to define it by a "path", where each step in
the path is a getattr() call?


Yes but I think that unpickle (pickle.loads()) does that plus
importing any module needed
in the path which it is handy because I can preload the plugins
(modules) before the unpickle but the path may contain others
more-standard modules as well.

Something like "myplugin.re.match". unpickle should import 're' module
automatically will it is loading the function "match".


Fair. I guess, then, that the best thing to do is to preload the
modules, then unpickle. So, basically what you already have, but with
more caveats.


Yes, this will not be transparent for the user, just trying to minimize
the changes needed.

And it will require some documentation for those caveats. And tests.

Thanks for the brainstorming!
Martin.
--
https://mail.python.org/mailman/listinfo/python-list

Re: Behavior of the for-else construct

[Grant Edwards]

On 2022-03-06, Avi Gross via Python-list  wrote:

> Python is named after a snake right?

No. It's named after a comedy troupe.

--
Grant

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Behavior of the for-else construct

[Peter J. Holzer]

On 2022-03-06 18:34:39 -0800, Grant Edwards wrote:
> On 2022-03-06, Avi Gross via Python-list  wrote:
> > Python is named after a snake right?
> 
> No. It's named after a comedy troupe.

He actually wrote that two sentences later.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list