Question about Multi-processing

[Anthony Flury via Python-list]

I decided to spend this morning to get my head around multi-processing, 
and decided to try to experiment initially in the command line 
interpreter, using the Python 3 documentation 



I fired up Python 3.6 and ran this :

   Python 3.6.6 (default, Sep 12 2018, 18:26:19)
   [GCC 8.0.1 20180414 (experimental) [trunk revision 259383]] on linux
   Type "help", "copyright", "credits" or "license" for more information.
>>> from multiprocessing import Pool
>>> def f(x):
   ... return x*2 + 1
   ...
>>> with Pool(10) as p:
   ...  print(p.map(f, range(1,100)))
   ...
   [3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29, 31, 33, 35, 37,
   39, 41, 43, 45, 47, 49, 51, 53, 55, 57, 59, 61, 63, 65, 67, 69, 71,
   73, 75, 77, 79, 81, 83, 85, 87, 89, 91, 93, 95, 97, 99, 101, 103,
   105, 107, 109, 111, 113, 115, 117, 119, 121, 123, 125, 127, 129,
   131, 133, 135, 137, 139, 141, 143, 145, 147, 149, 151, 153, 155,
   157, 159, 161, 163, 165, 167, 169, 171, 173, 175, 177, 179, 181,
   183, 185, 187, 189, 191, 193, 195, 197, 199]

As you can see - it clearly worked as expected - queue a happy dance.

I continued to read the documentation and came to this quote :

   /Functionality within this package requires that the
   //|__main__|//module be importable by the children. This is covered
   in //Programming guidelines
   
//however
   it is worth pointing out here. This means that some examples, such
   as the //|multiprocessing.pool.Pool|
   
//examples
   will not work in the interactive interpreter./

The Documentation then gives a clear example very similar to mine :

   >>> from multiprocessing import Pool >>> p = Pool(5) >>> def f(x): ...
   return x*x ... >>> p.map(f, [1,2,3]) Process PoolWorker-1: Process
   PoolWorker-2: Process PoolWorker-3: Traceback (most recent call
   last): AttributeError: 'module' object has no attribute 'f'
   AttributeError: 'module' object has no attribute 'f' AttributeError:
   'module' object has no attribute 'f'

But when I run this exact example in the command line interpreter it 
works fine :


>>> p = Pool(5)
>>> def f(x):
   ... return x*x
   ...
>>> p.map(f,[1,2,3])
   [3, 5, 7]

Is the documentation wrong ? is something weird going on ?

--

Anthony Flury
*Email* : anthony.fl...@btinternet.com 
*Twitter* : @TonyFlury 
--
https://mail.python.org/mailman/listinfo/python-list

Re: Observations on the List - "Be More Kind"

[Anthony Flury via Python-list]

Bruce,

I completely agree; it is worth everyone remember that we are all here 
(even the moderators) as volunteers; we are here because either we want 
to ask questions, or we want to learn, or we want to help.


We do need to remember that the moderators is here to be nasty or 
because of a power trip - they are here to help, and they are volunteers.


Part of the really important thing about Python is the community and 
this list is a critical part of that community.; lets keep it that way 
please :-)


--
Anthony Flury
*Email* : anthony.fl...@btinternet.com 
*Twitter* : @TonyFlury 

On 05/10/18 11:22, Bruce Coram wrote:
I will declare at the outset, I am a lurker.  I don't know enough 
about Python to give advice that I could 100% guarantee would be helpful.


There have been two recent threads that summarise for me where the 
Python Mailing List has lost its way (and this started before Trump 
arrived as a new role model for how to treat your fellow man):


"Re: This thread is closed [an actual new thread]"

"Re: So apparently I've been banned from this list"

The level of vitriol and personal attacks on the moderators was 
profoundly disappointing, but not totally out of character for those 
who made the attacks.  There is no doubt that these people know 
software and Python and this certainly earns my respect,  but perhaps 
they need to retain a sense of perspective.  There are 7 billion 
people in the world.  There are plenty more people at least as good as 
you and many better, but they don't have the compelling urge to 
demonstrate their genius.  They get on with their work in a quiet 
professional manner.


Some humility in acknowledging that you stand on the shoulders of 
giants would not go amiss.  It might also reflect that you understand 
the good fortune that dealt you such a good hand in life.


You aren't always right, and you don't always have to insist on being 
right.  I found Steve D'Aprano always had to have the last word and 
had to prove he was right.  I found some of his posts to be 
intemperate in tone.


Why is there a need to score points with caustic remarks or throwaway 
comments?  Perhaps the person who posed his question should have read 
the documents, perhaps he should have searched the archives.  Tell 
them so politely and firmly.  If you cannot manage that then why say 
anything?  Not everyone who posts a poorly prepared question is idle 
and deserves a response that is less than polite.  Prepare a 
boilerplate standard reply that is polite for those questions that are 
easily resolved by the poster.


Perhaps the person who posts something you regard as nonsense is 
ignorant and lacks the knowledge they think they possess.  Instead of 
wasting your time with scholastic debate, put the time to good use 
improving your education in subjects you don't excel at.  I can 
guarantee the depth of your ignorance will be profound - there will be 
much for you to learn.  The effort some of you put in to the endless 
debates suggests that you have plenty of time on your hands - don't 
waste it.  It will be gone soon enough.


Don't waste time on the trolls, some of whom undoubtedly enjoy the 
ability to provoke a response.  Develop a greater sense of self 
awareness to enable you to recognise that you are being played. The 
intemperate tone of some of the exchanges damages the reputation of 
the List.


Life is hard enough without us adding to it.  Try silence as a response.

Listen to Frank Turner's latest album: "Be More Kind".   That is not a 
plug to buy the album, but the title seems apposite - and the music is 
good.


Regards

Bruce Coram


--
https://mail.python.org/mailman/listinfo/python-list

Re: Encounter issues to install Python

[Anthony Flury via Python-list]

Olivier,

Welcome to the list - before we can help you, we need some more 
information :


 * What Operating system are you using - Windows/Mac/Linux/Raspberry
   Pi/Android for something else ?
 * What command or installer did you use to try to install Python.
 * What issues did you have during installation - if any ?
 * What interface are you trying to access, and how are you doing that ?
 * Do you get error messages?

Unless you tell us what the problem is we can't possibly help.

On 08/10/18 20:21, Olivier Oussou via Python-list wrote:

Hi!I downloaded and installed python 3.6.4 (32-bit) on my computer but I have 
problems and can not access the python interface.
I need your technical assistance to solve this matter.

Best regard!

Olivier OUSSOUMedical entomologist, Benin


--
Anthony Flury
*Email* : anthony.fl...@btinternet.com 
*Twitter* : @TonyFlury 
--
https://mail.python.org/mailman/listinfo/python-list

Side by side comparison - CPython, nuitka, PyPy

[Anthony Flury via Python-list]

I thought I would look at a side by side comparison of CPython, nuitka 
and PyPy


*The functionality under test**
*

I have a library (called primelib) which implements a Sieve of 
Erathoneses in pure Python - it was orginally written as part of my 
project Euler attempts


Not only does it build a sieve to test primality, it also builds an 
iterable list of primes, and has functionality to calculate the prime 
factors (and exponents) and also calculate all divisors of a given 
integer (up to the size of the sieve).


To test the primelib there is a simple harness which :

 * Builds a sieve for integers from 2 to 104729 (104729 is the 10,000th
   prime number)
 * Using a pre-built list from primes.utm.edu -
 o For every integer from 2 to 104729 the prime sieve and pre-built
   list agree on the primality or non-primality
 o confirm that the list of ALL primes identified by the sieve is
   the same as the pre-built list.
 o For every integer from 2 to 104729, get primelib to generate the
   prime factors and exponents - and comfirm that they multiply up
   to the expected integer
 o For every integer from 2 to 104729 get primelib to generate the
   divisors on the integer, and confirm that each divisor does
   divide cleanly into the integer

The Sieve is rebuilt between each test, there is no caching of data 
between test cases, so the test harness forces a lot of recalculations.


I have yet to convert primelib to be Python 3 compatible.

Exactly the same test harness was run in all 3 cases :

 * Under CPython 2.7.15, the execution of the test harness took around
   75 seconds to execute over 5 runs - fastest 73, slowest 78.
 * Under Nuitka 0.6, the execution of the test harness after compiler
   took around 85 seconds over 5 runes, fastest 84, slowest 86.
 * Under PyPy, the execution of the test harness took 4.9 seconds on
   average over 5 runs, fastest 4.79, slowest 5.2

I was very impressed at the execution time improvement under PyPy, and a 
little surprised about the lack of improvement under Nuitka.


I know Nuitka is a work in progress, but given that Nuitka compiles 
Python to C code I would have expected some level of gain, especially in 
a maths heavy implementation.


This comparison is provided for information only, and is not intended as 
any form of formal benchmark. I don't claim that primelib is as 
efficient as it could be - although every effort was made to try to make 
it as fast as I could.


--
Anthony Flury
*Email* : anthony.fl...@btinternet.com 
*Twitter* : @TonyFlury 
--
https://mail.python.org/mailman/listinfo/python-list

Re: Encounter issues to install Python

[Anthony Flury via Python-list]

What error are you getting ?

'Can not install the software' isn't enough information for me to assist 
in detail


In the mean time - things to check :

 * Does your user have permission to install to the directory you are
   trying to install to ?
 * Does the disk have enough disk space to install Python ?
 * have you actually got a 32 bit windows installation - many modern
   PCs are actually 64 bit now.

On 06/01/2019 15:20, Olivier Oussou wrote:

Hi dear Anthony,
I am using Windows systeme. I have download the set up uf python 3.6.4 
(32-bit) and I can not install the software on my computer.
I need your technical assistance to solve this matter and I will be 
glad if you do so.

Best regard!
Olivier
Medical entomologist, Benin



Le samedi 13 octobre 2018 à 19:24:28 UTC+2, Anthony Flury 
 a écrit :



Olivier,

Welcome to the list - before we can help you, we need some more 
information :


  * What Operating system are you using - Windows/Mac/Linux/Raspberry
Pi/Android for something else ?
  * What command or installer did you use to try to install Python.
  * What issues did you have during installation - if any ?
  * What interface are you trying to access, and how are you doing that ?
  * Do you get error messages?

Unless you tell us what the problem is we can't possibly help.

On 08/10/18 20:21, Olivier Oussou via Python-list wrote:

Hi!I downloaded and installed python 3.6.4 (32-bit) on my computer but I have 
problems and can not access the python interface.
I need your technical assistance to solve this matter.

Best regard!

Olivier OUSSOUMedical entomologist, Benin


--
Anthony Flury
*Email* : anthony.fl...@btinternet.com 


*Twitter* : @TonyFlury 

--
Anthony Flury
*Email* : anthony.fl...@btinternet.com 
*Twitter* : @TonyFlury 
--
https://mail.python.org/mailman/listinfo/python-list

Using PyArg_ParseTuple to with optional fields.

[Anthony Flury via Python-list]

I am trying to write an extension module with a function (actually an 
__init__ method, but I am not sure that matters) where the function can 
be called as either :


    my_func()

or

    my_func( a, b, c, d) - where a,b,c,d are all doubles.

I would prefer not to allow the function to be called with some 
arguments specified - it is either all four - or none.


I have tried the following format strings to PyArg_ParseTuple :

 * "" - but this a mandatory 4 doubles - doesn't allow for the no
   argument scenario
 * "|" - allows no arguments - but also allows only one, or just
   two, or three - i.e. not all or nothing
 * |() - allows no argument, but passing 4 arguments expects a
   4-tuple - i.e. my_func( (a,b,c,d) ) which is unnecessarily complex

Can I do what I want with a single call to PyArg_ParseTuple  ?

--
Anthony Flury
*Email* : anthony.fl...@btinternet.com 
*Twitter* : @TonyFlury 
--
https://mail.python.org/mailman/listinfo/python-list

Extension Module for Python 3.6 +

[Anthony Flury via Python-list]

In my brave and noble quest to get to grips with the CAPI - I am trying 
to write a C extension module which provides a new class


The exact details are not important, but what is important is that 
instances of my new class are imutable, and therefore from time to time, 
my extension module needs to build a new instance of my extension class.


The documentation is pretty sparse here, and suggests using PyObject_New 
& PyObject_Init but gives no examples.


An internet search suggests using PyObject_CallObject in some way - but 
the examples here are call backs to Python functions.


What is the canonical way to create New Instances of the Extension Type 
from within the Extension Module - even if someone can point me to a 
working example that would be great.


Another option would be to call the extension class new and init 
functions directly - is there a reason that is not allowed ?


PS - I think I also need to use Py_BuildValue to 'encapsulate' the 
initial arguments for the new instance - or could I just build a 'blank' 
instance and then directly set the fields as neccessary ?


--
Anthony Flury
*Email* : anthony.fl...@btinternet.com 
*Twitter* : @TonyFlury 
--
https://mail.python.org/mailman/listinfo/python-list

Interesting performance question

[Anthony Flury via Python-list]

I have just noticed an oddity :

Using python 3.6 building a tuple like this :

my_tuple = tuple([x*x for x in range(1,1000)])

is about 1/3 quicker than

    my_tuple = tuple(x*x for x in range(1,1000))

Measurements :

   $  python3 -m timeit 'my_tuple = tuple([x*x for x in range(1,1000)])'
   1 loops, best of 3: 33.5 usec per loop

   $  python3 -m timeit 'my_tuple = tuple(x*x for x in range(1,1000))'
   1 loops, best of 3: 44.1 usec per loop

My first assumption was that on the first example (with the list 
comprehension) the tuple constructor is able allocate the right amount 
of memory immediately, and therefore doesn't have the cost overhead of 
having to resize compared to the version having to resize every so often 
as new items are fetched from the generator.


One question though - wouldn't the list generator have a similar 
resizing issue, since the list comprehension wont predict the final size 
of the list ahead of time.


Is there a good explanation of the performance difference - I don't 
think that the memory resizing overhead is the whole answer.


--

Tony Flury

--
https://mail.python.org/mailman/listinfo/python-list