exit ThreadPoolExecutor immediately

[Atul Johri]

I am looking for a way to stop a ThreadPoolExecutor immediately under the
assumption that I don't care about what's currently running or pending.

```
limit = 2
executor = ThreadPoolExecutor(10)
posts = itertools.islice(mygen(executor=executor, **kwargs), 0, limit)
for post in posts:
  print(post)
executor.shutdown(wait=False)
```

Basically I have a generator, mygen, which is using the executor to submit
many tasks in parallel and yield the result one at a time. I would like to
be able to limit the generator and have the executor stop processing
immediately.

I was considering clearing the _work_queue or iterating over it and running
future.cancel() on each future but both seem to be quite hacky and didn't
work in my initial try.

https://github.com/python/cpython/blob/master/Lib/concurrent/futures/thread.py#L83

Any ideas?

- Al
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Access to the caller's globals, not your own

[Rob Gaddi]

Steven D'Aprano wrote:

> Suppose I have a library function that reads a global configuration setting:
>
> # Toy example
> SPAMIFY = True
> def make_spam(n):
> if SPAMIFY:
> return "spam"*n
> else:
> return "ham"*n
>
>
> Don't tell me to make SPAMIFY a parameter of the function. I know that. 
> That's 
> what I would normally do, but *occasionally* it is still useful to have a 
> global configuration setting, and those are the cases I'm talking about.
>
>
> Now the caller can say:
>
> import library
> library.SPAMIFY = False
> result = library.make_spam(99)
>
>
> but that would be Wrong and Bad and Evil, because you're affecting *other* 
> code, not your code, which relies on SPAMIFY being True. Doing this is 
> (maybe) 
> okay when testing the library, but not for production use.
>
> What I really want is for make_spam() to be able to look at the caller's 
> globals, so that each caller can create their own per-module configuration 
> setting:
>
> import library
> SPAMIFY = False  # only affects this module, no other modules
> result = library.make_spam(99)
>
>
> But... how can make_spam() look for the caller's globals rather than its own?
>
> I'd rather not do this:
>
>
> def make_spam(n, namespace):
> SPAMIFY = namespace['SPAMIFY']
> ...
>
> which forces the user to do this:
>
> result = library.make_spam(99, globals())
>
>
> which defeats the purpose of making it a global setting. (If I wanted to 
> *require* the caller to pass a parameter, I would just force them to pass in 
> the flag itself. The whole point is to not require that.)
>
> I'd be okay with making the namespace optional:
>
>
> def make_spam(n, namespace=None):
> if namespace is None:
> namespace = ... # magic to get the caller's globals
> SPAMIFY = namespace['SPAMIFY']
> ...
>
>
> but what magic do I need? globals() is no good, because it returns the 
> library's global namespace, not the caller's.
>
>
> Any solution ought to work for CPython, IronPython and Jython, at a minimum.
>

class Library:
  SPAMIFY = False
  def make_spam(m):
...

import library
Library = library.Library()

result = Library.make_spam(99)

-- 
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order.  See above to fix.
-- 
https://mail.python.org/mailman/listinfo/python-list

Why does this list swap fail?

[380162267qq]

L=[2,1]
L[0],L[L[0]-1]=L[L[0]-1],L[0]

The L doesn't change. Can someone provide me the detail procedure of this 
expression?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why does this list swap fail?

[Jussi Piitulainen]

38016226...@gmail.com writes:

> L=[2,1]
> L[0],L[L[0]-1]=L[L[0]-1],L[0]
>
> The L doesn't change. Can someone provide me the detail procedure of
> this expression?

The right-hand side evaluates to (1,2), but then the assignments to the
targets on the left-hand side are processed in order from left to right.
This happens:

L[0] = 1
L[L0] - 1] = 2

https://docs.python.org/3/reference/simple_stmts.html#assignment-statements
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why does this list swap fail?

[Alain Ketterlin]

38016226...@gmail.com writes:

> L=[2,1]
> L[0],L[L[0]-1]=L[L[0]-1],L[0]
>
> The L doesn't change. Can someone provide me the detail procedure of
> this expression?

From:

https://docs.python.org/3/reference/simple_stmts.html#grammar-token-assignment_stmt

| Although the definition of assignment implies that overlaps between the
| left-hand side and the right-hand side are ‘simultaneous’ (for example
| a, b = b, a swaps two variables), overlaps within the collection of
| assigned-to variables occur left-to-right, sometimes resulting in
| confusion. For instance, the following program prints [0, 2]:
|
| x = [0, 1]
| i = 0
| i, x[i] = 1, 2 # i is updated, then x[i] is updated
| print(x)

In your case:

- L[0] get 1 (from L[L[0]-1])
- L[L[0]-1] (from lhs) is now L[0], which gets 2 (from L[0] in rhs)

Since both operations happen in sequence (not simultaneously), you just
overwrite the first element twice.

Congratulations, a very good corner case (you can't call it a bug, since
it conforms to the definition of the language).

-- Alain.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why does this list swap fail?

[Rob Gaddi]

38016226...@gmail.com wrote:

> L=[2,1]
> L[0],L[L[0]-1]=L[L[0]-1],L[0]
>
> The L doesn't change. Can someone provide me the detail procedure of this 
> expression?

Better question: Why does that list swap exist?  If you're just trying
to put the language through its paces that's one thing, but I certainly
hope that's not actually going to be production code.  Even if it were
right it's unreadable.

-- 
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order.  See above to fix.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Access to the caller's globals, not your own

[ojacobson]

On Monday, November 14, 2016 at 12:21:00 AM UTC-5, Steven D'Aprano wrote:

> Don't tell me to make SPAMIFY a parameter of the function. I know that. 
> That's 
> what I would normally do, but *occasionally* it is still useful to have a 
> global configuration setting, and those are the cases I'm talking about.

This is the motivation behind Racket's parameters system 
. A _parameter_ has the 
following properties:

* It can be read or written whenever it is in scope. By convention, parameters 
are global variables, but they can also be created in more limited lexical 
scopes.

* A parameter can be set. Once set, all subsequent reads in any scope will 
produce the last set value.

So far, so good: these are exactly like module-scoped globals in Python. Where 
it gets interesting is the parameterize function 
:
 it introduces a dynamically-scoped set of new bindings for all included 
parameters, then evaluates a body (a procedure) with those bindings, then 
removes the bindings and restores the parameters to their values prior to 
parameterize. Any changes to those parameters within the body is also reverted: 
it's applied to the new binding introduced by parameterize, rather than to the 
"parent" binding.

Parameterizing a call means that changes to the "global variables" implemented 
as parameters have predictable scope and can be reliably restored to their 
prior values, meaning they're a fairly safe way to implement "config" globals.

Parameters are implemented using thread-local variables that point to stacks of 
bindings, where parameterize introduces a new stack frame for each listed 
parameter.

It's a neat system.

-o
-- 
https://mail.python.org/mailman/listinfo/python-list

Fwd: Problems with Anaconda and Pyzo

[Dan Pineau]

Hello,
I'm a french student and I'm running Python 3.5.2 on windows.

When I installed Anaconda3 4.2.0 version, Pyzo couldn't find a module named
conda.
Then I registered it as my default Python 3.5.2 version, but I couldn't use
Pyzo anymore (I sent you a screenshot). So I tried to fix this problem by
using the option "Repair" in the Python 3.5.2 setup, but this prevent me
from using Anaconda.

The problem might be simple, but I don't know what to do.
Could I have some help please ?

Have a good day.
-- 
https://mail.python.org/mailman/listinfo/python-list

Lexer/Parser question: TPG

[Johannes Bauer]

Hi group,

this is not really a Python question, but I use Python to lex/parse some
input. In particular, I use the amazing TPG (http://cdsoft.fr/tpg/).
However, I'm now stuck at a point and am sure I'm not doing something
correctly -- since there's a bunch of really smart people here, I hope
to get some insights. Here we go:

I've created a minimal example in which I'm trying to parse some tokens
(strings and ints in the minimal example). Strings are delimited by
braces (). Therefore

(Foo) -> "Foo"

Braces inside braces are taken literally when balanced. If not balanced,
it's a parsing error.

(Foo (Bar)) -> "Foo (Bar)"

Braces may be escaped:

(Foo \)Bar) -> "Foo )Bar"

In my first (naive) attempt, I ignored the escaping and went with lexing
and then these rules:

token string_token '[^()]*';

[...]

String/s -> start_string   $ s = ""
   (
   string_token/e  $ s += e
   | String/e  $ s += "(" + e + ")"
   )*
   end_string
   ;

While this worked a little bit (with some erroneous parsing,
admittedly), at least it *somewhat* worked. In my second attempt, I
tried to do it properly. I omitted the tokenization and instead used
inline terminals (which have precendence in TPG):

String/s -> start_string   $ s = ""
   (
   '\\.'/e $ s += "ESCAPED[" + e + "]"
   | '[^\\()]+'/e  $ s += e
   | String/e  $ s += "(" + e + ")"
   )*
   end_string
   ;

(the "ESCAPED" part is just for demonstration to get the idea).

While the latter parser parses all strings perfectly, it now isn't able
to parse anything else anymore (including integer values!). Instead, it
appears to match the inline terminal '[^\\()]+' to my integer and then
dies (when trying, for example, to parse "12345"):

[  1][ 3]START.Expression.Value: (1,1) _tok_2 12345 != integer
[  2][ 3]START.Expression.String: (1,1) _tok_2 12345 != start_string
Traceback (most recent call last):
  File "example.py", line 56, in 
print(Parser()(example))
  File "example/tpg.py", line 942, in __call__
return self.parse('START', input, *args, **kws)
  File "example/tpg.py", line 1125, in parse
return Parser.parse(self, axiom, input, *args, **kws)
  File "example/tpg.py", line 959, in parse
value = getattr(self, axiom)(*args, **kws)
  File "", line 3, in START
  File "", line 14, in Expression
UnboundLocalError: local variable 'e' referenced before assignment

"_tok_2" seems to correspond to one of the inline terminal symbols, the
only one that fits would be '[^\\()]+'. But why would that *ever* match?
I thought it'd only match once a "start_string" was encountered (which
it isn't).

Since I'm the parsing noob, I don't think TPG (which is FREAKING
AMAZING, seriously!) is at fault but rather my understanding of TPG. Can
someone help me with this?

I've uploaded a complete working example to play around with here:

http://wikisend.com/download/642120/example.tar.gz
(if it's not working, please tell me and I'll look for some place else).

Thank you so much for your help,
Best regards,
Johannes

-- 
>> Wo hattest Du das Beben nochmal GENAU vorhergesagt?
> Zumindest nicht öffentlich!
Ah, der neueste und bis heute genialste Streich unsere großen
Kosmologen: Die Geheim-Vorhersage.
 - Karl Kaos über Rüdiger Thomas in dsa 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: help on "from deen import *" vs. "import deen"

[Ned Batchelder]

On Sunday, November 13, 2016 at 9:39:12 PM UTC-5, jf...@ms4.hinet.net wrote:
> Running the following codes (deen.py) under Win32 python 3.4.4 terminal:
> 
> tbli = [0x66, 0x27, 0xD0]
> tblm = [0 for x in range(3)]
> def gpa(data):
> td = data ^ tblm[2]
> return td

The function gpa references the global tblm. Globals in Python are
global to the module (.py file) they appear in, and each module has
its own set of globals.  So the when the gpa function accesses tblm,
it's always going to be accessing the tblm in the deen module,
because gpa is defined in the deen module.


> 
> I can get a correct answer this way:
> >>> import deen
> >>> deen.tblm = deen.tbli
> >>> deen.gpa(0x7d)
> 173  # 0xad (= 0x7d ^ 0xd0) is decimal 173

Here you are updating deen.tblm, so when you call gpa, it sees the
new value you assigned to deen.tblm.

> 
> But I get a wrong answer this way:
> >>> from deen import *
> >>> tblm = tbli
> >>> gpa(0x7d)
> 125  # it's 0x7d, the tblm[2] is 0
> 
> Why? why! why:-(

Here you are assigning a value to your own tblm, not deen.tblm,
so gpa does not see the new value.

HTH,

--Ned.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: help on "from deen import *" vs. "import deen"

[jfong]

Ned Batchelder at 2016/11/15 6:33:54AM wrote:
> > But I get a wrong answer this way:
> > >>> from deen import *
> > >>> tblm = tbli
> > >>> gpa(0x7d)
> > 125  # it's 0x7d, the tblm[2] is 0
> > 
> > Why? why! why:-(
> 
> Here you are assigning a value to your own tblm, not deen.tblm,
> so gpa does not see the new value.
> 
Hi Ned, below is the capture of my terminal screen:

D:\Temp>python
Python 3.4.4 (v3.4.4:737efcadf5a6, Dec 20 2015, 19:28:18) [MSC v.1600 32 bit (In
tel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> dir()
['__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__']

>>> from deen import *
>>> dir()
['__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__',
 'gpa', 'tbli', 'tblm']
>>> tblm
[0, 0, 0]
>>> tblm=tbli
>>> tblm
[102, 39, 208]
>>> dir()
['__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__',
 'gpa', 'tbli', 'tblm']
>>>

Do you mean the last 'tblm' is my own? How it can be? The 'tblm' already exist 
before the assignment and this assignment shouldn't create a new one, right? 
and how do I assign the 'deen.tblm' under this circumstance?

--Jach

-- 
https://mail.python.org/mailman/listinfo/python-list

Who owns the memory in ctypes?

[Cem Karan]

Hi all, I'm hoping that this will be an easy question.

I have a pile of C code that I wrote that I want to interface to via the ctypes 
module (https://docs.python.org/3/library/ctypes.html).  The C code uses the 
Boehm-Demers-Weiser garbage collector (http://www.hboehm.info/gc/) for all of 
its memory management.  What I want to know is, who owns allocated memory?  
That is, if my C code allocates memory via GC_MALLOC() (the standard call for 
allocating memory in the garbage collector), and I access some object via 
ctypes in python, will the python garbage collector assume that it owns it and 
attempt to dispose of it when it goes out of scope?

Ideally, the memory is owned by the side that created it, with the other side 
simply referencing it, but I want to be sure before I invest a lot of time 
interfacing the two sides together.

Thanks,
Cem Karan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: help on "from deen import *" vs. "import deen"

[MRAB]

On 2016-11-15 01:36, jf...@ms4.hinet.net wrote:

Ned Batchelder at 2016/11/15 6:33:54AM wrote:

> But I get a wrong answer this way:
> >>> from deen import *
> >>> tblm = tbli
> >>> gpa(0x7d)
> 125  # it's 0x7d, the tblm[2] is 0
>
> Why? why! why:-(

Here you are assigning a value to your own tblm, not deen.tblm,
so gpa does not see the new value.


Hi Ned, below is the capture of my terminal screen:

D:\Temp>python
Python 3.4.4 (v3.4.4:737efcadf5a6, Dec 20 2015, 19:28:18) [MSC v.1600 32 bit (In
tel)] on win32
Type "help", "copyright", "credits" or "license" for more information.

dir()

['__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__']


from deen import *
dir()

['__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__',
 'gpa', 'tbli', 'tblm']

tblm

[0, 0, 0]

tblm=tbli
tblm

[102, 39, 208]

dir()

['__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__',
 'gpa', 'tbli', 'tblm']




Do you mean the last 'tblm' is my own? How it can be? The 'tblm' already exist 
before the assignment and this assignment shouldn't create a new one, right? 
and how do I assign the 'deen.tblm' under this circumstance?

--Jach

In the module 'deen' there's the name 'tblm' and the name 'tbli', among 
others:


[module deen]

tblm ---> [0, 0, 0]

tbli ---> [102, 39, 208]

When you say "from deen import *" you're copying names and their 
references from the module's namespace to your local namespace:


[module deen]   [locally]

tblm > [0, 0, 0] <- tblm

tbli > [102, 39, 208] < tbli

When you say "tblm = ..." you're binding the local name to a different 
object, so when you say "tblm = tbli" you get:


[module deen]   [locally]

tblm > [0, 0, 0]:-- tblm
V
tbli > [102, 39, 208] < tbli

--
https://mail.python.org/mailman/listinfo/python-list

Re: Access to the caller's globals, not your own

[Dan Sommers]

On Mon, 14 Nov 2016 16:20:49 +1100, Steven D'Aprano wrote:

> import library
> SPAMIFY = False  # only affects this module, no other modules
> result = library.make_spam(99)

I must be missing something, because it seems too obvious:

import library
# only affects this module, no other modules
library = library.make_library(spamify=False)
# ...
result = library.make_spam(99)

And then in library.py:

class Library:
def__init__(self, spamify=False):
self.spamify = spamify
def make_spam(self, p):
return ((p, "spammified")
if self.spamify else
(p, "hammified"))

def make_library(spamify=False):
return Library(spamify)

How do you want the following code to work:

import library
SPAMIFY=False

def make_false_spam():
return library.make_spam(99)

def make_true_spam():
global SPAMIFY
SPAMIFY=True
return library.make_spam(99)

I don't have to tell you how many things can go wrong with code like
that.  ;-)

Yes, it's a straw man.  No, I don't think we have all the details of
your use case.  Yes, I'm willing to have missed something subtle (or not
so subtle).

Dan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Who owns the memory in ctypes?

[eryk sun]

On Tue, Nov 15, 2016 at 3:15 AM, Cem Karan  wrote:
> if my C code allocates memory via GC_MALLOC() (the standard call for 
> allocating memory
> in the garbage collector), and I access some object via ctypes in python, 
> will the python
> garbage collector assume that it owns it and attempt to dispose of it when it 
> goes out of
> scope?

ctypes objects own only the memory that they allocate. Inspect the
_b_needsfree_ attribute to determine whether a ctypes object owns the
referenced memory.

For example:

This array object owns a 12-byte buffer for the array elements:

>>> arr = (ctypes.c_uint * 3)(0, 1, 2)
>>> arr._b_needsfree_
1

This pointer object owns an 8-byte buffer for the 64-bit target address:

>>> p = ctypes.POINTER(ctypes.c_uint * 3)(arr)
>>> p._b_needsfree_
1

The following new array object created by dereferencing the pointer
does not own the 12-byte buffer:

>>> ref = p[0]
>>> ref[:]
[0, 1, 2]
>>> ref._b_needsfree_
0

However, it does have a reference chain back to the original array:

>>> ref._b_base_ is p
True
>>> p._objects['1'] is arr
True

On the other hand, if you use the from_address() class method, the
resulting object is a dangling reference to memory that it doesn't own
and for which there's no supporting reference chain to the owning
object.

For example:

>>> arr = (ctypes.c_uint * 2**24)()
>>> arr[-1] = 42
>>> ref = type(arr).from_address(ctypes.addressof(arr))
>>> ref[-1]
42

>>> ref._b_base_ is None
True
>>> ref._objects is None
True

2**24 bytes is a big allocation that uses mmap (Unix) instead of the
heap. Thus accessing ref[-1] causes a segfault after arr is
deallocated:

>>> del arr
>>> ref[-1]
Segmentation fault (core dumped)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: help on "from deen import *" vs. "import deen"

[jfong]

MRAB at 2016/11/15 11:31:41AM wrote:
> When you say "from deen import *" you're copying names and their 
> references from the module's namespace to your local namespace:
> 
>  [module deen]   [locally]
> 
>  tblm > [0, 0, 0] <- tblm
> 
>  tbli > [102, 39, 208] < tbli

hmm...I always thought that the picture of "from deen import *" is

[locally]

 [0, 0, 0] <- tblm

 [102, 39, 208] < tbli

But obviously it's not. The compiled code of function gpa is still reference to 
a hidden deen.tblm 'global' object which I can't access anymore. A bad news:-(

--Jach

-- 
https://mail.python.org/mailman/listinfo/python-list