Re: poetry script fails to find module
r...@zedat.fu-berlin.de (Stefan Ram) writes: > "Loris Bennett" writes: >>However, this raises the question of why it worked in the first place >>in the poetry shell. > > It might have had a different or extended sys.path. In the poetry shell sys.path has this additional path /home/loris/gitlab/stoat The module not being found was /home/gitlab/stoat/stoat/hpc/main.py But if I run [~/gitlab/stoat] $ python stoat/main.py hpc user --help wouldn't import hpc.main still fail? Or is it because I am calling stoat/main.py and so Python looks for 'hpc' relative to the second 'stoat' directory? Confused, loris -- This signature is currently under construction. -- https://mail.python.org/mailman/listinfo/python-list
Re: ANN: Dogelog Player 1.0.0
Sofar we have resisted in adding a Next or Abort button to any of our Dogelog Player HTML pages. Behind the lack of these buttons is a method of exploring sequence meta-predicates and what they could do for user interfaces. Here is a little new article about the topic: Enumerating Solutions with Dogelog Payer https://twitter.com/dogelogch/status/1552802997247303680 Enumerating Solutions with Dogelog Payer https://www.facebook.com/groups/dogelog -- https://mail.python.org/mailman/listinfo/python-list
Re: poetry script fails to find module
Hi Stefan, r...@zedat.fu-berlin.de (Stefan Ram) writes: > "Loris Bennett" writes: >>Why is the module 'hpc' not found by the poetry script? > > I have tried to execute the following sequence of shell > commands to understand your problem. Here they all worked > without error messages. Warning: Some of these commands might > alter your directories or data, so only execute them if you > are aware of their meaning/consequences and agree with them! I do know what the commands do. However, just to be on the safe side, but mainly in honour of a classic dad-joke, I changed 'stoat' to 'weasel'. > mkdir stoat > mkdir stoat/hpc > echo import hpc.main >stoat/main.py > echo >stoat/hpc/main.py > python3 stoat/main.py > cd stoat > python3 main.py I guess with the production version, the package stoat is visible in sys.path and thus the subpackages have to be referred to via the main package, e.g. import stoat.hpc.main However, in the development environment, if I run python stoat/main.py hpc user --help then is stoat/hpc/main.py being found via import hpc.main because Python looks in stoat as the parent directory of stoat/main.py rather than the current working directory? That doesn't seem likely to me, but I am already confused. Cheers, Loris -- This signature is currently under construction. -- https://mail.python.org/mailman/listinfo/python-list
Re: poetry script fails to find module
r...@zedat.fu-berlin.de (Stefan Ram) writes: > "Loris Bennett" writes: >>However, in the development environment, if I run >> python stoat/main.py hpc user --help >>then is >> stoat/hpc/main.py >>being found via >> import hpc.main >>because Python looks in >> stoat >>as the parent directory of >> stoat/main.py >>rather than the current working directory? > > When you run "stoat/main.py", the directory of that file > "main.py" is automatically added at the front of sys.path. > (This also would happen after "cd stoat; python main.py".) OK, that explains it. I initially found that a bit odd, but thinking about it I see that "the directory containing the file being run" is a more sensible reference point than the current working directory, which is totally arbitrary. > Then, when "import hpc.main" is being executed, the system > will search for "hpc/main.py" in every entry of sys.path > and will use the first entry wherein it finds "hpc/main.py". > So it will use the directory of the file "stoat/main.py", > i.e., the directory "stoat". It finds "stoat/hpc/main.py". > > You can call "python" with "-v", and it will show some lines > with information about the imports executed, including the > directories used (albeit hidden in a lot of other lines). That's useful, although in the production case I would have to tweak the script generated by poetry. Cheers, Loris -- This signature is currently under construction. -- https://mail.python.org/mailman/listinfo/python-list
How to generate a .pyi file for a C Extension using stubgen
I tried to follow the instructions here: https://mypy.readthedocs.io/en/stable/stubgen.html but the instructions about creating a stub for a C Extension are a little mysterious. I tried to use it on the .so file without luck. -- https://mail.python.org/mailman/listinfo/python-list
Re: Simple TCP proxy
OK, that's useful to know. Thanks. :) -Morten On Fri, Jul 29, 2022 at 3:43 AM Andrew MacIntyre wrote: > On 29/07/2022 8:08 am, Chris Angelico wrote: > > It takes a bit of time to start ten thousand threads, but after that, > > the system is completely idle again until I notify them all and they > > shut down. > > > > (Interestingly, it takes four times as long to start 20,000 threads, > > suggesting that something in thread spawning has O(n²) cost. Still, > > even that leaves the system completely idle once it's done spawning > > them.) > > Another cost of threads can be memory allocated as thread stack space, > the default size of which varies by OS (see e.g. > > https://ariadne.space/2021/06/25/understanding-thread-stack-sizes-and-how-alpine-is-different/ > ). > > threading.stack_size() can be used to check and perhaps adjust the > allocation size. > > -- > - > Andrew I MacIntyre "These thoughts are mine alone..." > E-mail: andy...@pcug.org.au(pref) | Snail: PO Box 370 > andy...@bullseye.apana.org.au (alt) |Belconnen ACT 2616 > Web:http://www.andymac.org/ |Australia > -- > https://mail.python.org/mailman/listinfo/python-list > -- I am https://leavingnorway.info Videos at https://www.youtube.com/user/TheBlogologue Twittering at http://twitter.com/blogologue Blogging at http://blogologue.com Playing music at https://soundcloud.com/morten-w-petersen Also playing music and podcasting here: http://www.mixcloud.com/morten-w-petersen/ On Google+ here https://plus.google.com/107781930037068750156 On Instagram at https://instagram.com/morphexx/ -- https://mail.python.org/mailman/listinfo/python-list
Re: Simple TCP proxy
OK.
Well, I've worked with web hosting in the past, and proxies like squid were
used to lessen the load on dynamic backends. There was also a website
opensourcearticles.com that we had with Firefox, Thunderbird articles etc.
that got quite a bit of traffic.
IIRC, that website was mostly static with some dynamic bits and heavily
cached by squid.
Most websites don't get a lot of traffic though, and don't have a big
budget for "website system administration". So maybe that's where I'm
partly going with this, just making a proxy that can be put in front and
deal with a lot of common situations, in a reasonably good way.
If I run into problems with threads that can't be managed, then a switch to
something like the queue_manager function which has data and then functions
that manage the data and connections is an option.
-Morten
On Fri, Jul 29, 2022 at 12:11 AM Chris Angelico wrote:
> On Fri, 29 Jul 2022 at 07:24, Morten W. Petersen
> wrote:
> >
> > Forwarding to the list as well.
> >
> > -- Forwarded message -
> > From: Morten W. Petersen
> > Date: Thu, Jul 28, 2022 at 11:22 PM
> > Subject: Re: Simple TCP proxy
> > To: Chris Angelico
> >
> >
> > Well, an increase from 0.1 seconds to 0.2 seconds on "polling" in each
> > thread whether or not the connection should become active doesn't seem
> like
> > a big deal.
>
> Maybe, but polling *at all* is the problem here. It shouldn't be
> hammering the other server. You'll quickly find that there are limits
> that simply shouldn't exist, because every connection is trying to
> check to see if it's active now. This is *completely unnecessary*.
> I'll reiterate the advice given earlier in this thread (of
> conversation): Look into the tools available for thread (of execution)
> synchronization, such as mutexes (in Python, threading.Lock) and
> events (in Python, threading.Condition). A poll interval enforces a
> delay before the thread notices that it's active, AND causes inactive
> threads to consume CPU, neither of which is a good thing.
>
> > And there's also some point where it is pointless to accept more
> > connections, and where maybe remedies like accepting known good IPs,
> > blocking IPs / IP blocks with more than 3 connections etc. should be
> > considered.
>
> Firewalling is its own science. Blocking IPs with too many
> simultaneous connections should be decided administratively, not
> because your proxy can't handle enough connections.
>
> > I think I'll be getting closer than most applications to an eventual
> > ceiling for what Python can handle of threads, and that's interesting and
> > could be beneficial for Python as well.
>
> Here's a quick demo of the cost of threads when they're all blocked on
> something.
>
> >>> import threading
> >>> finish = threading.Condition()
> >>> def thrd(cond):
> ... with cond: cond.wait()
> ...
> >>> threading.active_count() # Main thread only
> 1
> >>> import time
> >>> def spawn(n):
> ... start = time.monotonic()
> ... for _ in range(n):
> ... t = threading.Thread(target=thrd, args=(finish,))
> ... t.start()
> ... print("Spawned", n, "threads in", time.monotonic() - start,
> "seconds")
> ...
> >>> spawn(1)
> Spawned 1 threads in 7.548425202025101 seconds
> >>> threading.active_count()
> 10001
> >>> with finish: finish.notify_all()
> ...
> >>> threading.active_count()
> 1
>
> It takes a bit of time to start ten thousand threads, but after that,
> the system is completely idle again until I notify them all and they
> shut down.
>
> (Interestingly, it takes four times as long to start 20,000 threads,
> suggesting that something in thread spawning has O(n²) cost. Still,
> even that leaves the system completely idle once it's done spawning
> them.)
>
> If your proxy can handle 20,000 threads, I would be astonished. And
> this isn't even close to a thread limit.
>
> Obviously the cost is different if the threads are all doing things,
> but if you have thousands of active socket connections, you'll start
> finding that there are limitations in quite a few places, depending on
> how much traffic is going through them. Ultimately, yes, you will find
> that threads restrict you and asynchronous I/O is the only option; but
> you can take threads a fairly long way before they are the limiting
> factor.
>
> ChrisA
> --
> https://mail.python.org/mailman/listinfo/python-list
>
--
I am https://leavingnorway.info
Videos at https://www.youtube.com/user/TheBlogologue
Twittering at http://twitter.com/blogologue
Blogging at http://blogologue.com
Playing music at https://soundcloud.com/morten-w-petersen
Also playing music and podcasting here:
http://www.mixcloud.com/morten-w-petersen/
On Google+ here https://plus.google.com/107781930037068750156
On Instagram at https://instagram.com/morphexx/
--
https://mail.python.org/mailman/listinfo/python-list
Re: Simple TCP proxy
On Sat, 30 Jul 2022 at 04:54, Morten W. Petersen wrote: > > OK. > > Well, I've worked with web hosting in the past, and proxies like squid were > used to lessen the load on dynamic backends. There was also a website > opensourcearticles.com that we had with Firefox, Thunderbird articles etc. > that got quite a bit of traffic. > > IIRC, that website was mostly static with some dynamic bits and heavily > cached by squid. Yep, and squid almost certainly won't have a thread for every incoming connection, spinning and waiting for the back end server. But squid does a LOT more than simply queue connections - it'll be inspecting headers and retaining a cache of static content, so it's not really comparable. > Most websites don't get a lot of traffic though, and don't have a big budget > for "website system administration". So maybe that's where I'm partly going > with this, just making a proxy that can be put in front and deal with a lot > of common situations, in a reasonably good way. > > If I run into problems with threads that can't be managed, then a switch to > something like the queue_manager function which has data and then functions > that manage the data and connections is an option. > I'll be quite frank with you: this is not production-quality code. It should not be deployed by anyone who doesn't have a big budget for "website system administration *training*". This code is good as a tool for YOU to learn how these things work; it shouldn't be a tool for anyone who actually has server load issues. I'm sorry if that sounds harsh, but the fact is, you can do a lot better by using this to learn more about networking than you'll ever do by trying to pitch it to any specific company. That said though: it's still good to know what your (theoretical) use-case is. That'll tell you what kinds of connection spam to throw at your proxy (lots of idle sockets? lots of HTTP requests? billions of half open TCP connections?) to see what it can cope with. Keep on playing with this code. There's a lot you can gain from it, still. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Simple TCP proxy
OK, sounds like sunshine is getting the best of you. It's working with a pretty heavy load, I see ways of solving potential problems that haven't become a problem yet, and I'm enjoying it. Maybe you should tone down the coaching until someone asks for it. Regards, Morten On Fri, Jul 29, 2022 at 10:46 PM Chris Angelico wrote: > On Sat, 30 Jul 2022 at 04:54, Morten W. Petersen > wrote: > > > > OK. > > > > Well, I've worked with web hosting in the past, and proxies like squid > were used to lessen the load on dynamic backends. There was also a website > opensourcearticles.com that we had with Firefox, Thunderbird articles > etc. that got quite a bit of traffic. > > > > IIRC, that website was mostly static with some dynamic bits and heavily > cached by squid. > > Yep, and squid almost certainly won't have a thread for every incoming > connection, spinning and waiting for the back end server. But squid > does a LOT more than simply queue connections - it'll be inspecting > headers and retaining a cache of static content, so it's not really > comparable. > > > Most websites don't get a lot of traffic though, and don't have a big > budget for "website system administration". So maybe that's where I'm > partly going with this, just making a proxy that can be put in front and > deal with a lot of common situations, in a reasonably good way. > > > > If I run into problems with threads that can't be managed, then a switch > to something like the queue_manager function which has data and then > functions that manage the data and connections is an option. > > > > I'll be quite frank with you: this is not production-quality code. It > should not be deployed by anyone who doesn't have a big budget for > "website system administration *training*". This code is good as a > tool for YOU to learn how these things work; it shouldn't be a tool > for anyone who actually has server load issues. > > I'm sorry if that sounds harsh, but the fact is, you can do a lot > better by using this to learn more about networking than you'll ever > do by trying to pitch it to any specific company. > > That said though: it's still good to know what your (theoretical) > use-case is. That'll tell you what kinds of connection spam to throw > at your proxy (lots of idle sockets? lots of HTTP requests? billions > of half open TCP connections?) to see what it can cope with. > > Keep on playing with this code. There's a lot you can gain from it, still. > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > -- I am https://leavingnorway.info Videos at https://www.youtube.com/user/TheBlogologue Twittering at http://twitter.com/blogologue Blogging at http://blogologue.com Playing music at https://soundcloud.com/morten-w-petersen Also playing music and podcasting here: http://www.mixcloud.com/morten-w-petersen/ On Google+ here https://plus.google.com/107781930037068750156 On Instagram at https://instagram.com/morphexx/ -- https://mail.python.org/mailman/listinfo/python-list
Re: How to generate a .pyi file for a C Extension using stubgen
> On 29 Jul 2022, at 19:33, Marco Sulla wrote: > > I tried to follow the instructions here: > > https://mypy.readthedocs.io/en/stable/stubgen.html > > but the instructions about creating a stub for a C Extension are a little > mysterious. I tried to use it on the .so file without luck. It says that stubgen works on .py files not .so files. You will need to write the .pyi for your .so manually. The docs could do with splitting the need for .pyi for .so away from the stubgen description. Barry > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
mailbox misbehavior with non-ASCII
The following code produces a nonsense result with the input
described below:
import mailbox
box = mailbox.Maildir("/home/peter/Temp/temp",create=False)
x = box.values()[0]
h = x.get("X-DSPAM-Factors")
print(type(h))
#
The output is the desired "str" when the message file contains this:
To: recipi...@example.com
Message-ID: <123>
Date: Sun, 24 Jul 2022 15:31:19 +
Subject: Blah blah
From: f...@from.com
X-DSPAM-Factors: a'b
xxx
... but if the apostrophe in "a'b" is replaced with a
RIGHT SINGLE QUOTATION MARK, the returned h is of type
"email.header.Header", and seems to contain inscrutable garbage.
I realize that one should not put non-ASCII characters in
message headers, but of course I didn't put it there, it
just showed up, pretty much beyond my control. And I realize
that when software is given input that breaks the rules, one
cannot expect optimal results, but I'd think an exception
would be the right answer.
Is this worth a bug report?
--
To email me, substitute nowhere->runbox, invalid->com.
--
https://mail.python.org/mailman/listinfo/python-list
Re: mailbox misbehavior with non-ASCII
On 7/29/22 16:24, Peter Pearson wrote: > ... but if the apostrophe in "a'b" is replaced with a > RIGHT SINGLE QUOTATION MARK, the returned h is of type > "email.header.Header", and seems to contain inscrutable garbage. > > I'd think an exception would be the right answer. > > Is this worth a bug report? I would say yes. -- ~Ethan~ -- https://mail.python.org/mailman/listinfo/python-list
Re: mailbox misbehavior with non-ASCII
On 2022-07-29 at 23:24:57 +,
Peter Pearson wrote:
> The following code produces a nonsense result with the input
> described below:
>
> import mailbox
> box = mailbox.Maildir("/home/peter/Temp/temp",create=False)
> x = box.values()[0]
> h = x.get("X-DSPAM-Factors")
> print(type(h))
> #
>
> The output is the desired "str" when the message file contains this:
>
> To: recipi...@example.com
> Message-ID: <123>
> Date: Sun, 24 Jul 2022 15:31:19 +
> Subject: Blah blah
> From: f...@from.com
> X-DSPAM-Factors: a'b
>
> xxx
>
> ... but if the apostrophe in "a'b" is replaced with a
> RIGHT SINGLE QUOTATION MARK, the returned h is of type
> "email.header.Header", and seems to contain inscrutable garbage.
>
> I realize that one should not put non-ASCII characters in
> message headers, but of course I didn't put it there, it
> just showed up, pretty much beyond my control. And I realize
> that when software is given input that breaks the rules, one
> cannot expect optimal results, but I'd think an exception
> would be the right answer.
Be strict in what you send, but generous is what you receive.
I agree that email headers are supposed to be ASCII (RFC 822, 2822, and
now 5322) all say that, but always throwing an exception seems a little
harsh, and arguably (I'm not arguing for or against) breaks backwards
compatibility. At least let the exception contain, in its own
attribute, the inscrutable garbage after the space after the colon and
before next CR/LF pair.
> Is this worth a bug report?
If nothing else, the documentation could specify or disclaim the
existing behavior.
--
https://mail.python.org/mailman/listinfo/python-list
