Re: ESR "Waning of Python" post

[Marko Rauhamaa]

Paul Rubin :

> Marko Rauhamaa  writes:
>>> Right, if I need near realtime behaviour and must live
>>> with [C]Python's garbage collector.
>> Or any other GC ever invented.
>
> There are realtime ones, like the Azul GC for Java, that have bounded
> delay in the milliseconds or lower. The total overhead is higher
> though.

I'd be interested in a definitive, non-anecdotal analysis on the topic.
Do you happen to have a link?

One reference I found stated there was no upper bound for heap use:

  A second cost of concurrent garbage collection is unpredictable heap
  growth. The program can allocate arbitrary amounts of memory while the
  GC is running.

  https://making.pusher.com/golangs-real-time-gc-in-theory-and-prac
  tice/>

If that worst-case behavior were tolerated, it would be trivial to
implement real-time GC: just let the objects pile up and never reclaim.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Is it dangeous when using custom metaclass?

[jfong]

Gregory Ewing at 2018/10/16 UTC+8 PM 2:01:01 wrote
> jf...@ms4.hinet.net wrote:
> > class Structure(metaclass=StructureMeta): ...
> > 
> > class PolyHeader(Structure): ...
> > 
> > As my understanding, the metaclass's __init__ was called when a class was
> > created. In the above example, both the Structure and PolyHeader called it.
> > My question is: because the PolyHeader inherited Structure, is it reasonable
> > for PolyHeader to call this __init__ again? Will it cause any possible
> > trouble?
> 
> It's reasonable for both to call it, because they're distinct
> instances of StructureMeta, each of which need to be initialised.

The PolyHeader is already initialized by inheritance. Is there any way to 
bypass this __init__?

> Whether it will cause a problem depends on what StructureMeta's
> __init__ is supposed to do. Presumably you want a given structure
> class to start allocating its offsets where its base class left
> off, in which case you may need to do something like this:
> 
> class StructureMeta(type):
>  def __init__(self, clsname, bases, clsdict):
>  if bases:
> offset = bases[0].offset # assuming there isn't more than one base
>  else:
> offset = 0
>  ...
> 
> (BTW, why do you use setattr() to set the offset attribute
> instead of just doing self.offset = offset?)

No particular reason, just follows the same code pattern ahead of it where the 
attribute names are from a list:-)

--Jach
-- 
https://mail.python.org/mailman/listinfo/python-list

Questions about weakref

[Frank Millman]

Hi all

I have some questions about using weakrefs.

My first question is whether weakrefs are the correct tool for my situation. 
My use-case is not mentioned in the docs, so maybe it is not intended to be 
used this way.


I have a lot of objects active in my app. Some of them (A) are fairly 
long-lived. Others (B) are short-lived, but hold a reference to one of the A 
objects. For the lifetime of the existence of B, it needs to be notified of 
any change of state of A. Therefore when B is created, it adds itself to a 
list maintained by A. It adds a tuple consisting of itself and a function 
(C). Whenever A detects a change of state, it iterates through the list and 
calls the function C, passing B as an argument.


I have quite a bit of additional housekeeping to perform to ensure that, 
when B goes out of scope, it removes itself from the list in A. I was hoping 
to use weakrefs to make this automatic. So the first question is, is this a 
reasonable use of weakrefs?


If yes, my second question is how to guard against A calling methods using 
dead B objects, as A has no control over when B might be gc'd.


My thinking is to use a WeakKeyDictionary, with the B object as the key and 
the C function as the data.


When A detects a change of state, instead of iterating over the keys, it 
should call keyrefs() to get the weakrefs and iterate over that. For each 
weakref it should 'call' it to get the actual key, and check for None to 
ensure that it is still alive. Does that sound safe?


Incidentally, the docs are slightly misleading. It says that the method 
keyrefs returns 'an iterable of the weak references to the keys'. I was not 
sure if that would expose me to the 'change size while iterating' error, so 
I checked the source. The source says 'return list(self.data)', and the 
docstring clearly states 'Return a list of weak references to the keys'. I 
think the docs should say the same. Should I raise an issue for this?


Frank Millman


--
https://mail.python.org/mailman/listinfo/python-list

Re: Questions about weakref

[Thomas Jollans]

On 2018-10-16 11:10, Frank Millman wrote:
> Hi all
> 
> I have some questions about using weakrefs.
> 
> My first question is whether weakrefs are the correct tool for my
> situation. My use-case is not mentioned in the docs, so maybe it is not
> intended to be used this way.
> 
> I have a lot of objects active in my app. Some of them (A) are fairly
> long-lived. Others (B) are short-lived, but hold a reference to one of
> the A objects. For the lifetime of the existence of B, it needs to be
> notified of any change of state of A. Therefore when B is created, it
> adds itself to a list maintained by A. It adds a tuple consisting of
> itself and a function (C). Whenever A detects a change of state, it
> iterates through the list and calls the function C, passing B as an
> argument.
> 
> I have quite a bit of additional housekeeping to perform to ensure that,
> when B goes out of scope, it removes itself from the list in A. I was
> hoping to use weakrefs to make this automatic. So the first question is,
> is this a reasonable use of weakrefs?
> 
> If yes, my second question is how to guard against A calling methods
> using dead B objects, as A has no control over when B might be gc'd.
> 
> My thinking is to use a WeakKeyDictionary, with the B object as the key
> and the C function as the data.

This sounds reasonable, though what you probably really want is a
signal/slot library that does all of this for you, like PySignal.

I believe all the pure-Python signal/slot libraries use weakrefs internally.

-- Thomas
-- 
https://mail.python.org/mailman/listinfo/python-list

Package creation documentation?

[Spencer Graves]

  Where can I find a reasonable tutorial on how to create a Python 
package?



  I've not created a Python package before, and I want to. Sadly, 
I'm having trouble finding reasonable documentation on how to do so.



  According to the Python 3 Glossary, "a package is a Python module 
with an __path__ attribute."[1]



  I found "packaging.python.org", which recommends "Packaging 
Python Projects"[2] and "An Overview of Packaging for Python".[3] I 
failed to find "__path__" in either.



  I've started a project on GitHub for this, which includes what I 
have so far toward building a Python package I want along with RStudio 
Rmarkdown Documents summarizing what I've tried so far.[4]



  What would you suggest I do to understand what I should do to 
create a "__path__ attribute" for this, and what I should do after that?



  I have decades of coding experience, but only a small portion of 
that was with Python, and that was roughly 7 years ago.



  Thanks,
  Spencer Graves


[1] https://docs.python.org/3/glossary.html#term-package


[2] https://packaging.python.org/tutorials/packaging-projects/


[3] https://packaging.python.org/overview/


[4] https://github.com/sbgraves237/radioMonitor

--
https://mail.python.org/mailman/listinfo/python-list

Re: Package creation documentation?

[Léo El Amri via Python-list]

Given your coding experience also you may want to look at
https://docs.python.org/3/reference/import.html#packages,
which is the technical detail of what a package is (And "how" it's
implemented).
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Package creation documentation?

[Thomas Jollans]

On 2018-10-16 17:15, Spencer Graves wrote:
>   Where can I find a reasonable tutorial on how to create a Python
> package?
> 
> 
>   I've not created a Python package before, and I want to. Sadly,
> I'm having trouble finding reasonable documentation on how to do so.
> 
> 
>   According to the Python 3 Glossary, "a package is a Python module
> with an __path__ attribute."[1]

Never mind that. Pay attention to the next line:

See also regular package and namespace package.

A regular package is just a directory with an __init__.py file in it
(and normally some submodules).

Generally you should only create a package if your codebase has multiple
somewhat independent but related bits, or is to large or complex to be
maintainable as a single module (.py file).

Now, if you want to package (verb) your module (noun) or package (noun)
into a package (different noun) that others can install, you want to
read the setuptools documentation.

https://setuptools.readthedocs.io/en/latest/

The overloading of the term "package" is a bit unfortunate, but most of
the time we don't notice since most packages contain packages, and most
packages come in packages. Some packages just contain modules, and some
packages are not packaged, though.

I would like to apologize for the previous paragraph.

-- Thomas
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Package creation documentation?

[Léo El Amri via Python-list]

Hello Spencer,

On 16/10/2018 17:15, Spencer Graves wrote:
>   Where can I find a reasonable tutorial on how to create a Python
> package?

IMO, the best documentation about this is the tutorial:
https://docs.python.org/3/tutorial/modules.html#packages

>   According to the Python 3 Glossary, "a package is a Python module
> with an __path__ attribute."[1]

What you are looking at are the technical details of what a package is.
Incidentally, if you follow the tutorial, everything will get in-place.

>   I found "packaging.python.org", which recommends "Packaging Python
> Projects"[2] and "An Overview of Packaging for Python".[3]

packaging.python.org is centered on "How to install and distribute
Python packages (Or modules)"

- Léo
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: ESR "Waning of Python" post

[jfine2358]

On Tuesday, October 16, 2018 at 8:00:26 AM UTC+1, Marko Rauhamaa wrote:
>https://making.pusher.com/golangs-real-time-gc-in-theory-and-practice/>

I'm all in favour of collecting useful URLs. Here's some more suggestions:

https://stackoverflow.com/questions/4491260/explanation-of-azuls-pauseless-garbage-collector
https://pdfs.semanticscholar.org/9770/fc9baf0f2b6c7521f00958973657bf03337d.pdf
https://www.researchgate.net/publication/220800769_Tax-and-spend_Democratic_scheduling_for_real-time_garbage_collection
http://digg.com/2018/private-garbage-collection-propublica
http://flyingfrogblog.blogspot.com/

Aside: One of the above is not about software garbage collection. Can you guess 
which one?

-- 
Jonathan




-- 
https://mail.python.org/mailman/listinfo/python-list

Wikipedia on Python

[Spencer Graves]

  Thanks to Léo El Amri and Thomas Jollans for their quick and 
helpful replies to my question about "Package creation documentation".



  Beyond that, I'd like to encourage people on this list to review 
the Wikipedia article on "Python (programming language)",[1] especially 
the claim that "a package is a Python module with an __path__ 
attribute", which I added on 2018-09-24 to help me understand the 
distinction.



  That Wikipedia article has averaged over 6,000 views per day over 
the past 3 years.  Therefore, any improvements will benefit lots of people.



  If you have suggestions for how the article might be improved, 
you can post them to the "Talk" page associated with that article or 
send them to me.  If you are "autoconfirmed" with the Wikimedia system, 
you can make the changes yourself.



  Thanks,
  Spencer Graves


[1] https://en.wikipedia.org/wiki/Python_(programming_language)


On 2018-10-16 11:14, Léo El Amri wrote:

Hello Spencer,

On 16/10/2018 17:15, Spencer Graves wrote:

   Where can I find a reasonable tutorial on how to create a Python
package?

IMO, the best documentation about this is the tutorial:
https://docs.python.org/3/tutorial/modules.html#packages


   According to the Python 3 Glossary, "a package is a Python module
with an __path__ attribute."[1]

What you are looking at are the technical details of what a package is.
Incidentally, if you follow the tutorial, everything will get in-place.


   I found "packaging.python.org", which recommends "Packaging Python
Projects"[2] and "An Overview of Packaging for Python".[3]

packaging.python.org is centered on "How to install and distribute
Python packages (Or modules)"

- Léo



--
https://mail.python.org/mailman/listinfo/python-list

Re: Wikipedia on Python

[Chris Angelico]

On Wed, Oct 17, 2018 at 5:05 AM Spencer Graves
 wrote:
>Beyond that, I'd like to encourage people on this list to review
> the Wikipedia article on "Python (programming language)",[1] especially
> the claim that "a package is a Python module with an __path__
> attribute", which I added on 2018-09-24 to help me understand the
> distinction.
>

You're welcome to put whatever you like into Wikipedia, but
personally, I don't think that particular piece of terminology is all
that helpful to the typical reader. Technical distinctions aren't
important to someone who's trying to find out what Python's all about,
or why s/he should learn the language.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: Wikipedia on Python

[Ryan Johnson]

Not really having read the whole story here, just wanna say that Wikipedia 
articles already suffer from a tug-of-war between professionals and students. 
Don’t worsen the problem. Include lay explanation and go into the technical 
details in a natural progression. Don’t force the reader to make large 
perspective jumps or conceptual jumps or require them to approach the article 
from the perspective of the designer from one sentence to the next. Pick your 
audience first, then write. Wikipedia requires a bridge between student and 
professional. Write it like it’s a synopsis that goes into detail, not like a 
reference manual for the already-initiated reader. If you like to write like 
it’s a reference, then there are list metapages on Wikipedia that tend to be 
agreeable to this.

peace

Sent from Mail for Windows 10

From: Spencer Graves
Sent: Tuesday, October 16, 2018 1:06 PM
To: Léo El Amri; python-list@python.org
Subject: Wikipedia on Python

   Thanks to Léo El Amri and Thomas Jollans for their quick and 
helpful replies to my question about "Package creation documentation".


   Beyond that, I'd like to encourage people on this list to review 
the Wikipedia article on "Python (programming language)",[1] especially 
the claim that "a package is a Python module with an __path__ 
attribute", which I added on 2018-09-24 to help me understand the 
distinction.


   That Wikipedia article has averaged over 6,000 views per day over 
the past 3 years.  Therefore, any improvements will benefit lots of people.


   If you have suggestions for how the article might be improved, 
you can post them to the "Talk" page associated with that article or 
send them to me.  If you are "autoconfirmed" with the Wikimedia system, 
you can make the changes yourself.


   Thanks,
   Spencer Graves


[1] https://en.wikipedia.org/wiki/Python_(programming_language)


On 2018-10-16 11:14, Léo El Amri wrote:
> Hello Spencer,
>
> On 16/10/2018 17:15, Spencer Graves wrote:
>>Where can I find a reasonable tutorial on how to create a Python
>> package?
> IMO, the best documentation about this is the tutorial:
> https://docs.python.org/3/tutorial/modules.html#packages
>
>>According to the Python 3 Glossary, "a package is a Python module
>> with an __path__ attribute."[1]
> What you are looking at are the technical details of what a package is.
> Incidentally, if you follow the tutorial, everything will get in-place.
>
>>I found "packaging.python.org", which recommends "Packaging Python
>> Projects"[2] and "An Overview of Packaging for Python".[3]
> packaging.python.org is centered on "How to install and distribute
> Python packages (Or modules)"
>
> - Léo
>

-- 
https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

RE: ESR "Waning of Python" post

[Ryan Johnson]

Have seen this waning of python thread so many times. Hoping it would have 
waned by now. Lol.

Sent from Mail for Windows 10

From: jfine2...@gmail.com
Sent: Tuesday, October 16, 2018 12:42 PM
To: python-list@python.org
Subject: Re: ESR "Waning of Python" post

On Tuesday, October 16, 2018 at 8:00:26 AM UTC+1, Marko Rauhamaa wrote:
>https://making.pusher.com/golangs-real-time-gc-in-theory-and-practice/>

I'm all in favour of collecting useful URLs. Here's some more suggestions:

https://stackoverflow.com/questions/4491260/explanation-of-azuls-pauseless-garbage-collector
https://pdfs.semanticscholar.org/9770/fc9baf0f2b6c7521f00958973657bf03337d.pdf
https://www.researchgate.net/publication/220800769_Tax-and-spend_Democratic_scheduling_for_real-time_garbage_collection
http://digg.com/2018/private-garbage-collection-propublica
http://flyingfrogblog.blogspot.com/

Aside: One of the above is not about software garbage collection. Can you guess 
which one?

-- 
Jonathan




-- 
https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Wikipedia on Python

[Chris Warrick]

On Tue, 16 Oct 2018 at 20:11, Chris Angelico  wrote:
>
> On Wed, Oct 17, 2018 at 5:05 AM Spencer Graves
>  wrote:
> >Beyond that, I'd like to encourage people on this list to review
> > the Wikipedia article on "Python (programming language)",[1] especially
> > the claim that "a package is a Python module with an __path__
> > attribute", which I added on 2018-09-24 to help me understand the
> > distinction.
> >
>
> You're welcome to put whatever you like into Wikipedia, but
> personally, I don't think that particular piece of terminology is all
> that helpful to the typical reader. Technical distinctions aren't
> important to someone who's trying to find out what Python's all about,
> or why s/he should learn the language.

Seconded. This is not useful at all on Wikipedia.
I took the liberty to remove this paragraph, because I don’t think
anyone would find it useful; in fact, it would only confuse people.
Here’s a diff for anyone interested in the original content:
https://en.wikipedia.org/w/index.php?title=Python_(programming_language)&diff=prev&oldid=861064627

-- 
Chris Warrick 
PGP: 5EAAEA16
-- 
https://mail.python.org/mailman/listinfo/python-list

check whether a process is still running on remote hosts

[tina_zy_qian--- via Python-list]

I newly learned Python, and I need to wrap up a script to do following.

Env:
1: One launcher Linux VM (from where to run the Python script)
2. 100+ Linux VM

requirement:
In general, run a remote_script on remote 100 VMs and get the log files 
generated to remote hosts back to launcher.

steps
1. cp (pexpect.spawn('scp  ')) supporting files including the remote_script to 
remotehost:/remote_folder
2. run remote_script on remote host  ( pexpect.spawn('ssh %s@%s "%s"' % 
(user,host,cmd))
3. wait and check until the remote_script was run on remote host. (to check "ps 
-ef" result on remote hosts)
4. collect data from remote hosts -- reverse to step 1. 


I briefly implemented other steps expect step 3. Two options are below. 

option 1: run another script "ps -ef|grep remote_script and output result to a 
local file, then collected the files to launcher.

option 2: run pexect.spawn('ssh' 'ps -e') to get the result directly to 
laucher console.
But how I can get only the output for "ps -ef" command only?

I must use the (users, passwords, and hosts) way to do ssh and scp because I 
may not be allowed to use ssh key on some hosts. Any suggestions or sample code 
for step 3 or the whole script are appreciated. Thanks. 


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: ESR "Waning of Python" post

[Marko Rauhamaa]

Paul Rubin :

> But it's possible to do parallel GC with bounded latency. Perry
> Cheng's 2001 PhD thesis says how to do it and is fairly readable:
>
> http://reports-archive.adm.cs.cmu.edu/anon/2001/CMU-CS-01-174.pdf

Thanks. On a quick glance, it is difficult to judge what the worst-case
time and space behavior are as the thesis mixes theory and practice and
leans heavily on practice. The thesis says in its introduction:

   A real-time collector comprises two important features: pauses are
   bounded by some reasonably small value and the mutator can make
   sufficient progress between pauses. Different collectors meet these
   conditions with varying degrees of success and their viability
   depends on application needs. It is important to note that a
   collector must also complete collection within a reasonable time. A
   "real-time" collector which mereloy stops collections whenever it
   runs out of time would be hard real-time but useless if it never
   finishes a collection. In such cases, memory is soon exhausted. As
   with other real-time applications, the most important distinction
   among real-time collectors is the strength of the guarantee.

> If you hang out with users of Lisp, Haskell, Ocaml, Java, Ruby, etc.,
> they (like Python users) have all kinds of complaints about their
> languages, but GC pauses aren't a frequent topic of those complaints.

I don't suffer from it, either.

> Most applications don't actually care about sub-millisecond realtime.
> They just want pauses to be small or infrequent enough to not interfere
> with interactively using a program.  If there's a millisecond pause
> every few seconds of operation and an 0.2 second pause a few times an
> hour, that's usually fine.

Emacs occasionally hangs for about a minute to perform garbage
collection.

Similarly, Firefox occasionally becomes unresponsive for a long time,
and I'm guessing it's due to GC.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: check whether a process is still running on remote hosts

[Larry Martell]

On Tue, Oct 16, 2018 at 5:15 PM tina_zy_qian--- via Python-list
 wrote:
>
> I newly learned Python, and I need to wrap up a script to do following.
>
> Env:
> 1: One launcher Linux VM (from where to run the Python script)
> 2. 100+ Linux VM
>
> requirement:
> In general, run a remote_script on remote 100 VMs and get the log files 
> generated to remote hosts back to launcher.
>
> steps
> 1. cp (pexpect.spawn('scp  ')) supporting files including the remote_script 
> to remotehost:/remote_folder
> 2. run remote_script on remote host  ( pexpect.spawn('ssh %s@%s "%s"' % 
> (user,host,cmd))
> 3. wait and check until the remote_script was run on remote host. (to check 
> "ps -ef" result on remote hosts)
> 4. collect data from remote hosts -- reverse to step 1.
>
>
> I briefly implemented other steps expect step 3. Two options are below.
>
> option 1: run another script "ps -ef|grep remote_script and output result to 
> a local file, then collected the files to launcher.
>
> option 2: run pexect.spawn('ssh' 'ps -e') to get the result directly to 
> laucher console.
> But how I can get only the output for "ps -ef" command only?
>
> I must use the (users, passwords, and hosts) way to do ssh and scp because I 
> may not be allowed to use ssh key on some hosts. Any suggestions or sample 
> code for step 3 or the whole script are appreciated. Thanks.

This looks amazingly similar to a pre interview programming assignment
was given once.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: check whether a process is still running on remote hosts

[tina_zy_qian--- via Python-list]

On Tuesday, October 16, 2018 at 2:22:29 PM UTC-7, larry@gmail.com wrote:
> On Tue, Oct 16, 2018 at 5:15 PM tina_zy_qian--- via Python-list
>  wrote:
> >
> > I newly learned Python, and I need to wrap up a script to do following.
> >
> > Env:
> > 1: One launcher Linux VM (from where to run the Python script)
> > 2. 100+ Linux VM
> >
> > requirement:
> > In general, run a remote_script on remote 100 VMs and get the log files 
> > generated to remote hosts back to launcher.
> >
> > steps
> > 1. cp (pexpect.spawn('scp  ')) supporting files including the remote_script 
> > to remotehost:/remote_folder
> > 2. run remote_script on remote host  ( pexpect.spawn('ssh %s@%s "%s"' % 
> > (user,host,cmd))
> > 3. wait and check until the remote_script was run on remote host. (to check 
> > "ps -ef" result on remote hosts)
> > 4. collect data from remote hosts -- reverse to step 1.
> >
> >
> > I briefly implemented other steps expect step 3. Two options are below.
> >
> > option 1: run another script "ps -ef|grep remote_script and output result 
> > to a local file, then collected the files to launcher.
> >
> > option 2: run pexect.spawn('ssh' 'ps -e') to get the result directly to 
> > laucher console.
> > But how I can get only the output for "ps -ef" command only?
> >
> > I must use the (users, passwords, and hosts) way to do ssh and scp because 
> > I may not be allowed to use ssh key on some hosts. Any suggestions or 
> > sample code for step 3 or the whole script are appreciated. Thanks.
> 
> This looks amazingly similar to a pre interview programming assignment
> was given once.

Nice to know that. This is my daily work related though. Not interview question 
from me or to me. 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: check whether a process is still running on remote hosts

[Chris Angelico]

On Wed, Oct 17, 2018 at 10:11 AM tina_zy_qian--- via Python-list
 wrote:
>
> On Tuesday, October 16, 2018 at 2:22:29 PM UTC-7, larry@gmail.com wrote:
> > This looks amazingly similar to a pre interview programming assignment
> > was given once.
>
> Nice to know that. This is my daily work related though. Not interview 
> question from me or to me.

I suggest that you interview Larry for a programming job. Give him
this challenge. Take his code, use it.

http://dilbert.com/strip/2015-08-01

More seriously, though: You will find your task a LOT easier if (1)
you use public key logins rather than passwords, and (2) you break the
problem down into individual parts, each of which can be assessed
separately, and then write each one on its own.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: check whether a process is still running on remote hosts

[Cameron Simpson]

On 16Oct2018 14:14, tina_zy_q...@yahoo.com  wrote:

I newly learned Python, and I need to wrap up a script to do following.

Env:
1: One launcher Linux VM (from where to run the Python script)
2. 100+ Linux VM

requirement:
In general, run a remote_script on remote 100 VMs and get the log files 
generated to remote hosts back to launcher.

steps
1. cp (pexpect.spawn('scp  ')) supporting files including the remote_script to 
remotehost:/remote_folder
2. run remote_script on remote host  ( pexpect.spawn('ssh %s@%s "%s"' % 
(user,host,cmd))
3. wait and check until the remote_script was run on remote host. (to check "ps 
-ef" result on remote hosts)
4. collect data from remote hosts -- reverse to step 1.


I briefly implemented other steps expect step 3. Two options are below.

option 1: run another script "ps -ef|grep remote_script and output result to a 
local file, then collected the files to launcher.

option 2: run pexect.spawn('ssh' 'ps -e') to get the result directly to 
laucher console.
But how I can get only the output for "ps -ef" command only?

I must use the (users, passwords, and hosts) way to do ssh and scp 
because I may not be allowed to use ssh key on some hosts. Any 
suggestions or sample code for step 3 or the whole script are 
appreciated. Thanks.


Have you looked at ansible? It is a Python/ssh library/tool for remote 
admin. It is almost designed for this kind of thing. Run local script on 
remote hosts in parallel and wait for them?  Tick.


Ansible can prompt (once) for a remote ssh password and apply it to all 
the remote ssh invocations.


I _strongly_ recommend you do this with keys though. Passwords are 
something of a security nightmare.


You can be sure that the remote script has run if the ssh which invoked 
it returned with a zero exit status. If the ssh itself aborts, or the 
script fails, you get a nonzero status back from ssh.


Personally I would be (a) trying to do this without passwords - install 
keys and (b) using threads to spawn all the remote sshes, have each 
thread wait for its own ssh, and wait for all the threads.


Or reaching for ansible, which has this kind of logic built in.

Cheers,
Cameron Simpson 
--
https://mail.python.org/mailman/listinfo/python-list

Re: Is it dangeous when using custom metaclass?

[dieter]

jf...@ms4.hinet.net writes:
> Gregory Ewing at 2018/10/16 UTC+8 PM 2:01:01 wrote
>> jf...@ms4.hinet.net wrote:
>> > class Structure(metaclass=StructureMeta): ...
>> > 
>> > class PolyHeader(Structure): ...
>> > 
>> > As my understanding, the metaclass's __init__ was called when a class was
>> > created. In the above example, both the Structure and PolyHeader called it.
>> > My question is: because the PolyHeader inherited Structure, is it 
>> > reasonable
>> > for PolyHeader to call this __init__ again? Will it cause any possible
>> > trouble?
>> 
>> It's reasonable for both to call it, because they're distinct
>> instances of StructureMeta, each of which need to be initialised.
>
> The PolyHeader is already initialized by inheritance. Is there any way to 
> bypass this __init__?

If there were any, you should find it described in metaclass related 
documentation.

Likely, you can ensure that followup calls of "__init__" effectively
behave as "no-op"s: let the first call place a marker in the initialized
object and check in later calls whether it is already there.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: ESR "Waning of Python" post

[Marko Rauhamaa]

Paul Rubin :

> Marko Rauhamaa  writes:
>> Emacs occasionally hangs for about a minute to perform garbage
>> collection.
>
> I've never experienced that, especially with more recent versions that I
> think do a little bit of heap tidying in the background.  Even in the
> era of much slower computers I never saw an Emacs GC pause of more than
> a second or two unless something had run amuck and exhausted memory.
> It's always near imperceptible in my experience now.  Is your system
> swapping or something?

I can't be positive about swapping. I don't remember hearing thrashing.
However, I do admit running emacs for months on end and occasionally
with huge buffers so the resident size can be a couple of gigabytes.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: ESR "Waning of Python" post

[Brian Oney via Python-list]

On October 17, 2018 7:56:51 AM GMT+02:00, Marko Rauhamaa  
wrote:
>I can't be positive about swapping. I don't remember hearing thrashing.
>However, I do admit running emacs for months on end and occasionally
>with huge buffers so the resident size can be a couple of gigabytes.
>
That's a pretty good stress test for any program, especially one with so much 
human interaction.
-- 
https://mail.python.org/mailman/listinfo/python-list