extending PATH on Windows?

2016-02-16 Thread Ulli Horlacher
I need to extend the PATH environment variable on Windows.

So far, I use:

   system('setx PATH "%PATH%;'+bindir+'"')

The problem: In a new process (cmd.exe) PATH contains a lot of double
elements. As far as I have understood, Windows builds the PATH
environment variable from a system component and a user component. With
the setx command from above I have copied the system PATH into the user
PATH component.

Is there a better way to extend the PATH environment variable for the user?
It must be persistent, not only for the current process.


-- 
Ullrich Horlacher  Server und Virtualisierung
Rechenzentrum IZUS/TIK E-Mail: horlac...@tik.uni-stuttgart.de
Universitaet Stuttgart Tel:++49-711-68565868
Allmandring 30aFax:++49-711-682357
70550 Stuttgart (Germany)  WWW:http://www.tik.uni-stuttgart.de/
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Make a unique filesystem path, without creating the file

2016-02-16 Thread Oscar Benjamin
On 16 Feb 2016 05:57, "Ben Finney"  wrote:
>
> Cameron Simpson  writes:
>
> > I've been watching this for a few days, and am struggling to
> > understand your use case.
>
> Yes, you're not alone. This surprises me, which is why I'm persisting.
>
> > Can you elaborate with a concrete example and its purpose which would
> > work with a mktemp-ish official function?
>
> An example::
>
> import io
> import tempfile
> names = tempfile._get_candidate_names()
>
> def test_frobnicates_configured_spungfile():
> """ ‘foo’ should frobnicate the configured spungfile. """
>
> fake_file_path = os.path.join(tempfile.gettempdir(), names.next())
> fake_file = io.BytesIO("Lorem ipsum, dolor sit
amet".encode("utf-8"))
>
> patch_builtins_open(
> when_accessing_path=fake_file_path,
> provide_file=fake_file)
>
> system_under_test.config.spungfile_path = fake_file_path
> system_under_test.foo()
> assert_correctly_frobnicated(fake_file)

If you're going to patch open to return a fake file when asked to open
fake_file_path why do you care whether there is a real file of that name?

--
Oscar
-- 
https://mail.python.org/mailman/listinfo/python-list


Will file be closed automatically in a "for ... in open..." statement?

2016-02-16 Thread jfong
I know

with open('foo.txt') as f:
...do something...

will close the file automatically when the "with" block ends. 

I also saw codes in a book:

for line in open('foo.txt'):
...do something...

but it didn't mention if the file will be closed automatically or not when the 
"for" block ends. Is there any document talking about this? and how to know if 
a file is in "open" or not?

--Jach Fong

-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Will file be closed automatically in a "for ... in open..." statement?

2016-02-16 Thread Chris Angelico
On Tue, Feb 16, 2016 at 7:39 PM,   wrote:
> I know
>
> with open('foo.txt') as f:
> ...do something...
>
> will close the file automatically when the "with" block ends.
>
> I also saw codes in a book:
>
> for line in open('foo.txt'):
> ...do something...
>
> but it didn't mention if the file will be closed automatically or not when 
> the "for" block ends. Is there any document talking about this? and how to 
> know if a file is in "open" or not?
>

The file will be closed when the open file object is disposed of. That
will happen at some point after there are no more references to it.
You're guaranteed that it stays around for the entire duration of the
'for' loop (the loop keeps track of the thing it's iterating over),
but exactly when after that is not guaranteed. In current versions of
CPython, the garbage collector counts references, so the file will be
closed immediately; but other Python interpreters, and future versions
of CPython, may not behave the same way. So the file will *probably*
be closed *reasonably* promptly, but unlike the "with" case, you have
no guarantee that it'll be immediate.

For small scripts, it probably won't even matter, though. You're
unlikely to run out of file handles, and the only time it would matter
is if you're opening, closing, and then reopening the file - for
example:

fn = input("Name a file to frobnosticate: ")
with open(fn) as f:
data = []
for line in f:
data = frobnosticate(data, line)
with open(fn, "w") as f:
f.writelines(data)

For this to work reliably, the file MUST be closed for reading before
it's opened for writing. The context managers are important. But this
is pretty unusual.

Of course, since it's so little trouble to use the 'with' block, it's
generally worth just using it everywhere. Why run the risk? :)

ChrisA
who often forgets to use 'with' anyway
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Will file be closed automatically in a "for ... in open..." statement?

2016-02-16 Thread Cameron Simpson

On 16Feb2016 00:39, jf...@ms4.hinet.net  wrote:

I know

   with open('foo.txt') as f:
   ...do something...

will close the file automatically when the "with" block ends.


Yes, because open is a context manager - they're great for reliably tidying up 
in the face of exceptions or "direct" departure from the block, such as a 
"return" statement.



I also saw codes in a book:

   for line in open('foo.txt'):
...do something...

but it didn't mention if the file will be closed automatically or not when the 
"for" block ends. Is there any document talking about this? and how to know if 
a file is in "open" or not?


This does not reliably close the file.

In CPython (the common implementation, and likely what you are using), objects 
are reference counted and when the interpreter notices their counter go to 
zero, the object's __del__ method is called before releasing the object's 
memory.


For an open file, __del__ _does_ call close if the file is open. However, only 
reference counting Pythons will call __del__ promptly - other systems rely on 
garbage collectors to detect unused objects.


In the for loop above, for interpreter obtains an iterator from the open file 
which returns lines of text. That iterator has a reference to the open file, 
and the for loop has a reference to the iterator. Therefore the file remains 
references while the loop runs. AT the end of the loop the iterator is 
discarded, reducing its references to zero. That in turn triggers releasing the 
open file, dropping its references to zero.


In CPython, that in turn will fire the open file's __del__, which will close 
the file. In other Pythons, not necessarily that promptly.


Also, there are plenty of ways to phrase this where the file reference doesn't 
go to zero. FOr example:


 f = open(...)
 for line in f:
   ...

You can see that it is easy to forget to close the file here (especially if you 
have an exception or exit the function precipitously).


Try to use the "with open(...) as f:" formulation when possible. It is much 
better.


Cheers,
Cameron Simpson 
--
https://mail.python.org/mailman/listinfo/python-list


Re: Will file be closed automatically in a "for ... in open..." statement?

2016-02-16 Thread Terry Reedy

On 2/16/2016 3:39 AM, jf...@ms4.hinet.net wrote:

I know

 with open('foo.txt') as f:
 ...do something...

will close the file automatically when the "with" block ends.

I also saw codes in a book:

 for line in open('foo.txt'):
...do something...


Some books were originally written before 'with context_manager' was 
added, or a little later, before it became the normal thing to do.


--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list


Re: extending PATH on Windows?

2016-02-16 Thread Thorsten Kampe
* Ulli Horlacher (Tue, 16 Feb 2016 08:30:59 + (UTC))
> I need to extend the PATH environment variable on Windows.
> 
> So far, I use:
> 
>system('setx PATH "%PATH%;'+bindir+'"')
> 
> The problem: In a new process (cmd.exe) PATH contains a lot of double
> elements. As far as I have understood, Windows builds the PATH
> environment variable from a system component and a user component. With
> the setx command from above I have copied the system PATH into the user
> PATH component.
> 
> Is there a better way to extend the PATH environment variable for the user?
> It must be persistent, not only for the current process.

`os.system` should be `subprocess.call` on modern systems (actually 
`subprocess.run` if you have Python 3.5). Since `setx` doesn't seem 
to have unique and add options, you basically have two options:

1. Add the path component yourself into HKEY_CURRENT_USER and make 
sure it's not there already (pure Python).

2. a) use a shell that offers that capability with `set`: 
https://jpsoft.com/help/set.htm (TCC/LE is free)

   b) use a dedicated environment variable editor: 
http://www.rapidee.com/en/command-line

Thorsten

-- 
https://mail.python.org/mailman/listinfo/python-list


Re: extending PATH on Windows?

2016-02-16 Thread Ulli Horlacher
Thorsten Kampe  wrote:
> * Ulli Horlacher (Tue, 16 Feb 2016 08:30:59 + (UTC))
> > I need to extend the PATH environment variable on Windows.
> 
> 1. Add the path component yourself into HKEY_CURRENT_USER and make 
> sure it's not there already (pure Python).

Preferred!
What is HKEY_CURRENT_USER? Another environment variable?

-- 
Ullrich Horlacher  Server und Virtualisierung
Rechenzentrum IZUS/TIK E-Mail: horlac...@tik.uni-stuttgart.de
Universitaet Stuttgart Tel:++49-711-68565868
Allmandring 30aFax:++49-711-682357
70550 Stuttgart (Germany)  WWW:http://www.tik.uni-stuttgart.de/
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Unable to insert data into MongoDB.

2016-02-16 Thread Arjun Srivatsa
Thanks a lot. Will implement that. Although I am able to do using just 2 
scripts as well. 

On Monday, February 15, 2016 at 5:34:08 PM UTC+1, Peter Otten wrote:
> Arjun Srivatsa wrote:
> 
> > Hi Peter.
> > 
> > Thank you for the reply.
> > 
> > This is the read_server code:
> > 
> > import socket
> > from pymongo import MongoClient
> > #import datetime
> > import sys
> > 
> > # Connection to server (PLC) on port 27017
> > host = "10.52.124.135"
> > port = 27017
> > 
> > s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
> > s.connect((host, port))
> > sys.stdout.write(s.recv(1024))
> > 
> > 
> > And the write_db code:
> > 
> > from pymongo import MongoClient
> > import datetime
> > import socket
> > import sys
> > 
> > client = MongoClient('mongodb://localhost:27017/')
> > db = client.test_database
> > 
> > mongodoc = { "data": 'data', "date" : datetime.datetime.utcnow() }
> > values = db.values
> > values_id = values.insert_one(mongodoc).inserted_id
> > 
> > 
> > So, both these scripts work independently. While, read_server shows the
> > output of the actual data from PLC, write_db inserts the sample data into
> > the MongoDB.
> > 
> > I am not sure as to how to combine these both and get the desired output.
> 
> What I mean is once you have working scripts
> 
> connect_to_mongodb()
> while True:
> record = make_fake_data()
> insert_record_into_mongodb(record)
> 
> and
> 
> connect_to_server()
> while True:
> record = read_record_from_server()
> print(record)
> 
> you can combine the code in a third script to
> 
> connect_to_server()
> connect_to_mongodb()
> while True:
> record = read_record_from_server()
> insert_record_into_mongodb(record)
> 
> and be fairly sure that the combination works, too.

-- 
https://mail.python.org/mailman/listinfo/python-list


Multiple Assignment a = b = c

2016-02-16 Thread srinivas devaki
Hi,

a = b = c

as an assignment doesn't return anything, i ruled out a = b = c as
chained assignment, like a = (b = c)
SO i thought, a = b = c is resolved as
a, b = [c, c]


at-least i fixed in my mind that every assignment like operation in
python is done with references and then the references are binded to
the named variables.
like globals()['a'] = result()

but today i learned that this is not the case with great pain(7 hours
of debugging.)

class Mytest(object):
def __init__(self, a):
self.a = a
def __getitem__(self, k):
print('__getitem__', k)
return self.a[k]
def __setitem__(self, k, v):
print('__setitem__', k, v)
self.a[k] = v

roots = Mytest([0, 1, 2, 3, 4, 5, 6, 7, 8])
a = 4
roots[4] = 6
a = roots[a] = roots[roots[a]]


the above program's output is
__setitem__ 4 6
__getitem__ 4
__getitem__ 6
__setitem__ 6 6


But the output that i expected is
__setitem__ 4 6
__getitem__ 4
__getitem__ 6
__setitem__ 4 6

SO isn't it counter intuitive from all other python operations.
like how we teach on how python performs a swap operation???

I just want to get a better idea around this.

-- 
Regards
Srinivas Devaki
Junior (3rd yr) student at Indian School of Mines,(IIT Dhanbad)
Computer Science and Engineering Department
ph: +91 9491 383 249
telegram_id: @eightnoteight
-- 
https://mail.python.org/mailman/listinfo/python-list


Installation error, compiling from source on Oracle Linux

2016-02-16 Thread John D. Gwinner
I'm installing an app that requires Carbon and some other Python 2.7 features.

The version of Oracle Linux we're using comes with 2.6.

I've read that it is not a good idea to directly update the O/S as it "may 
break things" so I'm doing make altinstall.

I've downloaded Python-2.7.11
Downloaded zlib-1.2.8
Done
./configure --prefix=/root/Python-2.7.8 --with-libs=/usr/local/lib 
--disable-ipv6

However, I get an error while compiling.

make altinstall
gcc -pthread -c -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall 
-Wstrict-prototypes  -I. -IInclude -I./Include   -DPy_BUILD_CORE -o 
Modules/python.o ./Modules/python.c
In file included from Include/Python.h:58,
 from ./Modules/python.c:3:
Include/pyport.h:256:13: error: #error "This platform's pyconfig.h needs to 
define PY_FORMAT_LONG_LONG"
make: *** [Modules/python.o] Error 1

I CAN compile without zlib, but then pip gives an error.

python2.7 get-pip.py
Traceback (most recent call last):
  File "get-pip.py", line 19017, in 
main()
  File "get-pip.py", line 194, in main
bootstrap(tmpdir=tmpdir)
  File "get-pip.py", line 82, in bootstrap
import pip
zipimport.ZipImportError: can't decompress data; zlib not available

I'd use findRPM but that seems to be 2.7.8, not 2.7.11, and it seems 
reasonable, if I'm building this, to build the most recent version.

Any ideas? I have web searched this; I found a bug that was closed in 2014, and 
I just got all new source *right now*.

I'm doing a very vanilla install on Oracle Linux Server release 6.7

Thank you,

== John ==

-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Multiple Assignment a = b = c

2016-02-16 Thread Sven R. Kunze

Hi Srinivas,

On 16.02.2016 13:46, srinivas devaki wrote:

Hi,

a = b = c

as an assignment doesn't return anything, i ruled out a = b = c as
chained assignment, like a = (b = c)
SO i thought, a = b = c is resolved as
a, b = [c, c]


at-least i fixed in my mind that every assignment like operation in
python is done with references and then the references are binded to
the named variables.
like globals()['a'] = result()

but today i learned that this is not the case with great pain(7 hours
of debugging.)

class Mytest(object):
 def __init__(self, a):
 self.a = a
 def __getitem__(self, k):
 print('__getitem__', k)
 return self.a[k]
 def __setitem__(self, k, v):
 print('__setitem__', k, v)
 self.a[k] = v

roots = Mytest([0, 1, 2, 3, 4, 5, 6, 7, 8])
a = 4
roots[4] = 6
a = roots[a] = roots[roots[a]]


the above program's output is
__setitem__ 4 6
__getitem__ 4
__getitem__ 6
__setitem__ 6 6


But the output that i expected is
__setitem__ 4 6
__getitem__ 4
__getitem__ 6
__setitem__ 4 6

SO isn't it counter intuitive from all other python operations.
like how we teach on how python performs a swap operation???

I just want to get a better idea around this.


I think the tuple assignment you showed basically nails it.

First, the rhs is evaluated.
Second, the lhs is evaluated from left to right.

Completely wrong?

Best,
Sven
--
https://mail.python.org/mailman/listinfo/python-list


Re: Multiple Assignment a = b = c

2016-02-16 Thread Sven R. Kunze

On 16.02.2016 14:05, Sven R. Kunze wrote:

Hi Srinivas,

I think the tuple assignment you showed basically nails it.

First, the rhs is evaluated.
Second, the lhs is evaluated from left to right.

Completely wrong?

Best,
Sven


As you mentioned swapping. The following two statements do the same (as 
you suggested at the beginning).


a,b=b,a=4,5
(a,b),(b,a)=(4,5),(4,5)

Best,
Sven
--
https://mail.python.org/mailman/listinfo/python-list


Re: extending PATH on Windows?

2016-02-16 Thread eryk sun
On Tue, Feb 16, 2016 at 2:30 AM, Ulli Horlacher
 wrote:
>
> So far, I use:
>
>system('setx PATH "%PATH%;'+bindir+'"')
>
> The problem: In a new process (cmd.exe) PATH contains a lot of double
> elements. As far as I have understood, Windows builds the PATH
> environment variable from a system component and a user component. With
> the setx command from above I have copied the system PATH into the user
> PATH component.

setx broadcasts a WM_SETTINGCHANGE [1] message that notifies Explorer
to reload its environment from the registry, so the user doesn't have
to start a new session. It also decides whether to use REG_SZ or
REG_EXPAND_SZ depending on the presence of mutliple "%" characters in
the string.

[1]: https://msdn.microsoft.com/en-us/library/ms725497

But as you note it's no good for extending an existing value,
especially not for PATH or a value that references other "%variables%"
that you want to remain unexpanded. To do this right, you have to at
least use winreg to query the user's PATH value from the registry. But
then you may as well replace setx completely. Here's a little
something to get you started.

import os
import sys
import types
import ctypes

user32 = ctypes.WinDLL('user32', use_last_error=True)

try:
import winreg
except ImportError:
import _winreg as winreg

def extend_path(new_paths, persist=True):
if isinstance(new_paths, getattr(types, 'StringTypes', str)):
new_paths = [new_paths]
new_paths = [os.path.abspath(p) for p in new_paths]
paths = [p for p in os.environ.get('PATH', '').split(os.pathsep) if p]
for p in new_paths:
if p not in paths:
paths.append(p)
os.environ['PATH'] = os.pathsep.join(paths)
if persist:
_persist_path(new_paths)

def _persist_path(new_paths):
if sys.version_info[0] == 2:
temp_paths = []
for p in new_paths:
if isinstance(p, unicode):
temp_paths.append(p)
else:
temp_paths.append(p.decode('mbcs'))
new_paths = temp_paths
with winreg.OpenKey(winreg.HKEY_CURRENT_USER,
'Environment', 0,
winreg.KEY_QUERY_VALUE |
winreg.KEY_SET_VALUE) as hkey:
try:
user_path, dtype = winreg.QueryValueEx(hkey, 'PATH')
except WindowsError as e:
ERROR_FILE_NOT_FOUND = 0x0002
if e.winerror != ERROR_FILE_NOT_FOUND:
raise
paths = []
else:
if dtype in (winreg.REG_SZ, winreg.REG_EXPAND_SZ):
paths = [p for p in user_path.split(os.pathsep) if p]
else:
paths = []
for p in new_paths:
if p not in paths:
paths.append(p)
pathstr = os.pathsep.join(paths)
if pathstr.count('%') < 2:
dtype = winreg.REG_SZ
else:
dtype = winreg.REG_EXPAND_SZ
winreg.SetValueEx(hkey, 'PATH', 0, dtype, pathstr)
_broadcast_change(u'Environment')

def _broadcast_change(lparam):
HWND_BROADCAST   = 0x
WM_SETTINGCHANGE = 0x001A
SMTO_ABORTIFHUNG = 0x0002
ERROR_TIMEOUT = 0x05B4
wparam = 0
if not user32.SendMessageTimeoutW(
HWND_BROADCAST, WM_SETTINGCHANGE,
wparam, ctypes.c_wchar_p(lparam),
SMTO_ABORTIFHUNG, 1000, None):
err = ctypes.get_last_error()
if err != ERROR_TIMEOUT:
raise ctypes.WinError(err)
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: asyncio - run coroutine in the background

2016-02-16 Thread Kevin Conway
If you're handling coroutines there is an asyncio facility for "background
tasks". The ensure_future [1] will take a coroutine, attach it to a Task,
and return a future to you that resolves when the coroutine is complete.
The coroutine you schedule with that function will not cause your current
coroutine to wait unless you await the future it returns.

[1]
https://docs.python.org/3/library/asyncio-task.html#asyncio.ensure_future

On Mon, Feb 15, 2016, 23:53 Chris Angelico  wrote:

> On Mon, Feb 15, 2016 at 6:39 PM, Paul Rubin 
> wrote:
> > "Frank Millman"  writes:
> >> The benefit of my class is that it enables me to take the coroutine
> >> and run it in another thread, without having to re-engineer the whole
> >> thing.
> >
> > Threads in Python don't get you parallelism either, of course.
> >
>
> They can. The only limitation is that, in CPython (and some others),
> no two threads can concurrently be executing Python byte-code. The
> instant you drop into a C-implemented function, it can release the GIL
> and let another thread start running. Obviously this happens any time
> there's going to be a blocking API call (eg if one thread waits on a
> socket read, others can run), but it can also happen with
> computational work:
>
> import numpy
> import threading
>
> def thread1():
> arr = numpy.zeros(1, dtype=numpy.int64)
> while True:
> print("1: %d" % arr[0])
> arr += 1
> arr = (arr * arr) % 142957
>
> def thread2():
> arr = numpy.zeros(1, dtype=numpy.int64)
> while True:
> print("2: %d" % arr[0])
> arr += 2
> arr = (arr * arr) % 142957
>
> threading.Thread(target=thread1).start()
> thread2()
>
> This will happily keep two CPU cores occupied. Most of the work is
> being done inside Numpy, which releases the GIL before doing any work.
> So it's not strictly true that threading can't parallelise Python code
> (and as mentioned, it depends on your interpreter - Jython can, I
> believe, do true multithreading), but just that there are limitations
> on what can execute concurrently.
>
> ChrisA
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: extending PATH on Windows?

2016-02-16 Thread Thorsten Kampe
* Ulli Horlacher (Tue, 16 Feb 2016 12:38:44 + (UTC))
> 
> Thorsten Kampe  wrote:
> > * Ulli Horlacher (Tue, 16 Feb 2016 08:30:59 + (UTC))
> > > I need to extend the PATH environment variable on Windows.
> > 
> > 1. Add the path component yourself into HKEY_CURRENT_USER and make 
> > sure it's not there already (pure Python).
> 
> Preferred!
> What is HKEY_CURRENT_USER? Another environment variable?

It's a hive in the Windows registry and the equivalent of `~/.*` in 
Linux terms (HKEY_LOCAL_MACHINE[/Software] being the equivalent of 
`/etc`). The fact that you're asking indicates that you should read 
about that in advance. 

The task itself is definitely not that hard. Maybe someone has 
already asked at StackOverflow. But the devil's in the detail.

Some things to consider

- Python is not by default installed on Windows, so you have to use a 
way to run your script without (PyInstaller for instance).

- by default there is no entry in HKCU, so you have to create it 
first (under HKEY_CURRENT_USER\Environment).

- you need to create the correct type (REG_SZ, null-terminated 
string)

- Windows paths are semicolon separated (not colon).

- Windows only module for the Registry: 
https://docs.python.org/3/library/winreg.html

Thorsten

-- 
https://mail.python.org/mailman/listinfo/python-list


Re: extending PATH on Windows?

2016-02-16 Thread Thorsten Kampe
* Ulli Horlacher (Tue, 16 Feb 2016 12:38:44 + (UTC))

By the way: there is a script called `win_add2path.py` in your Python 
distribution which "is a simple script to add Python to the Windows 
search path. It modifies the current user (HKCU) tree of the 
registry.". That should do most of what you want.

Thorsten

-- 
https://mail.python.org/mailman/listinfo/python-list


Re: asyncio - run coroutine in the background

2016-02-16 Thread Frank Millman
"Kevin Conway"  wrote in message 
news:CAKF=+dim8wzprvm86_v2w5-xsopcchvgm0hy8r4xehdyzy_...@mail.gmail.com...


If you're handling coroutines there is an asyncio facility for "background
tasks". The ensure_future [1] will take a coroutine, attach it to a Task,
and return a future to you that resolves when the coroutine is complete.
The coroutine you schedule with that function will not cause your current
coroutine to wait unless you await the future it returns.

[1]
https://docs.python.org/3/library/asyncio-task.html#asyncio.ensure_future



Thank you Kevin!

That works perfectly, and is much neater than my effort.

Frank


--
https://mail.python.org/mailman/listinfo/python-list


Re: asyncio - run coroutine in the background

2016-02-16 Thread Marko Rauhamaa
Kevin Conway :

> If you're handling coroutines there is an asyncio facility for
> "background tasks". The ensure_future [1] will take a coroutine,
> attach it to a Task, and return a future to you that resolves when the
> coroutine is complete.

Ok, yes, but those "background tasks" monopolize the CPU once they are
scheduled to run.

If your "background task" doesn't need a long time to run, just call the
function in the foreground and be done with it. If it does consume time,
you need to delegate it to a separate process so the other tasks remain
responsive.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: asyncio - run coroutine in the background

2016-02-16 Thread Frank Millman

"Marko Rauhamaa"  wrote in message news:87d1rwpwo2@elektro.pacujo.net...


Kevin Conway :

> If you're handling coroutines there is an asyncio facility for
> "background tasks". The ensure_future [1] will take a coroutine,
> attach it to a Task, and return a future to you that resolves when the
> coroutine is complete.

Ok, yes, but those "background tasks" monopolize the CPU once they are
scheduled to run.

If your "background task" doesn't need a long time to run, just call the
function in the foreground and be done with it. If it does consume time,
you need to delegate it to a separate process so the other tasks remain
responsive.



I will explain my situation - perhaps you can tell me if it makes sense.

My background task does take a long time to run - about 10 seconds - but 
most of that time is spent waiting for database responses, which is handled 
in another thread.


You could argue that the database thread should rather be handled by another 
process, and that is definitely an option if I find that response times are 
affected.


So far my response times have been very good, even with database activity in 
the background. However, I have not simulated a large number of concurrent 
users. That could throw up the kinds of problem that you are concerned 
about.


Frank


--
https://mail.python.org/mailman/listinfo/python-list


Re: asyncio - run coroutine in the background

2016-02-16 Thread Kevin Conway
> Ok, yes, but those "background tasks" monopolize the CPU once they are
scheduled to run.

This is true if the coroutines are cpu bound. If that is the case then a
coroutine is likely the wrong choice for that code to begin with.
Coroutines, in asyncio land, are primarily designed for io bound work.

> My background task does take a long time to run - about 10 seconds - but
most of that time is spent waiting for database responses, which is handled
in another thread.

Something else to look into is an asyncio driver for your database
connections. Threads aren't inherently harmful, but using them to achieve
async networking when running asyncio is a definite code smell since that
is precisely the problem asyncio is supposed to solve for.

On Tue, Feb 16, 2016, 08:37 Frank Millman  wrote:

> "Marko Rauhamaa"  wrote in message news:87d1rwpwo2@elektro.pacujo.net.
> ..
> >
> > Kevin Conway :
> >
> > > If you're handling coroutines there is an asyncio facility for
> > > "background tasks". The ensure_future [1] will take a coroutine,
> > > attach it to a Task, and return a future to you that resolves when the
> > > coroutine is complete.
> >
> > Ok, yes, but those "background tasks" monopolize the CPU once they are
> > scheduled to run.
> >
> > If your "background task" doesn't need a long time to run, just call the
> > function in the foreground and be done with it. If it does consume time,
> > you need to delegate it to a separate process so the other tasks remain
> > responsive.
> >
>
> I will explain my situation - perhaps you can tell me if it makes sense.
>
> My background task does take a long time to run - about 10 seconds - but
> most of that time is spent waiting for database responses, which is handled
> in another thread.
>
> You could argue that the database thread should rather be handled by
> another
> process, and that is definitely an option if I find that response times are
> affected.
>
> So far my response times have been very good, even with database activity
> in
> the background. However, I have not simulated a large number of concurrent
> users. That could throw up the kinds of problem that you are concerned
> about.
>
> Frank
>
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Multiple Assignment a = b = c

2016-02-16 Thread Steven D'Aprano
On Tue, 16 Feb 2016 11:46 pm, srinivas devaki wrote:

> Hi,
> 
> a = b = c
> 
> as an assignment doesn't return anything, i ruled out a = b = c as
> chained assignment, like a = (b = c)
> SO i thought, a = b = c is resolved as
> a, b = [c, c]

That is one way of thinking of it. A better way would be:

a = c
b = c

except that isn't necessarily correct for complex assignments involving
attribute access or item assignment. A better way is:

_temp = c
a = _temp
b = _temp
del _temp


except the name "_temp" isn't actually used.



> at-least i fixed in my mind that every assignment like operation in
> python is done with references and then the references are binded to
> the named variables.
> like globals()['a'] = result()

That's broadly correct.


> but today i learned that this is not the case with great pain(7 hours
> of debugging.)
> 
> class Mytest(object):
> def __init__(self, a):
> self.a = a
> def __getitem__(self, k):
> print('__getitem__', k)
> return self.a[k]
> def __setitem__(self, k, v):
> print('__setitem__', k, v)
> self.a[k] = v
> 
> roots = Mytest([0, 1, 2, 3, 4, 5, 6, 7, 8])
> a = 4
> roots[4] = 6
> a = roots[a] = roots[roots[a]]

`roots[4] = 6` will give "__setitem__ 4 6", as you expect. 

On the right hand side, you have:

roots[roots[a]]

which evaluates `roots[a]` first, giving "__getitem__ 4". That returns 6, as
you expect. So now you have `roots[6]`, which gives "__getitem__ 6", as you
expect, and returns 6.

The left hand side has:

a = roots[a] = ...

which becomes:

a = roots[a] = 6

which behaves like:

a = 6
roots[a] = 6

So you end up with:

a = roots[6] = 6

which gives "__setitem__ 6 6", **not** "__setitem__ 4 6" like you expected.


Here is a simpler demonstration:

py> L = [0, 1, 2, 3, 4, 5, 6]
py> a = L[a//100] = 500
py> print a
500
py> print L
[0, 1, 2, 3, 4, 500, 6]


Let's look at the byte-code generated by the statement:

a = L[a] = x

The exact byte-code used will depend on the version of Python you have, but
for 2.7 it looks like this:

py> from dis import dis
py> code = compile("a = L[a] = x", "", "exec")
py> dis(code)
  1   0 LOAD_NAME0 (x)
  3 DUP_TOP
  4 STORE_NAME   1 (a)
  7 LOAD_NAME2 (L)
 10 LOAD_NAME1 (a)
 13 STORE_SUBSCR
 14 LOAD_CONST   0 (None)
 17 RETURN_VALUE


Translated to English:

- evaluate the expression `x` and push the result onto the stack;

- duplicate the top value on the stack;

- pop the top value off the stack and assign to name `a`;

- evaluate the name `L`, and push the result onto the stack;

- evaluate the name `a`, and push the result onto the stack;

- call setattr with the top three items from the stack.



> SO isn't it counter intuitive from all other python operations.
> like how we teach on how python performs a swap operation???

No. Let's look at the equivalent swap:

py> L = [10, 20, 30, 40, 50]
py> a = 3
py> a, L[a] = L[a], a
Traceback (most recent call last):
  File "", line 1, in 
IndexError: list assignment index out of range


This is equivalent to:

_temp1 = L[a]  # 40 pushed onto the stack
_temp2 = a  # 3 pushed onto the stack
a = _temp1  # 40  # rotate the stack, and pull the top item 40
L[a] = _temp2  # L[40] = 3

which obviously fails. Here's the byte-code:

py> code = compile("a, L[a] = L[a], a", "", "exec")
py> dis(code)
  1   0 LOAD_NAME0 (L)
  3 LOAD_NAME1 (a)
  6 BINARY_SUBSCR
  7 LOAD_NAME1 (a)
 10 ROT_TWO
 11 STORE_NAME   1 (a)
 14 LOAD_NAME0 (L)
 17 LOAD_NAME1 (a)
 20 STORE_SUBSCR
 21 LOAD_CONST   0 (None)
 24 RETURN_VALUE


If you do the swap in the other order, it works:


py> L = [10, 20, 30, 40, 50]
py> a = 3
py> L[a], a = a, L[a]
py> print a
40
py> print L
[10, 20, 30, 3, 50]


In all cases, the same rule applies:

- evaluate the right hand side from left-most to right-most, pushing the
values onto the stack;

- perform assignments on the left hand side, from left-most to right-most.




-- 
Steven

-- 
https://mail.python.org/mailman/listinfo/python-list


Re: asyncio - run coroutine in the background

2016-02-16 Thread Frank Millman
"Kevin Conway"  wrote in message 
news:CAKF=+dhXZ=yax8stawr_gjx3tg8yujprjg-7ym2_brv2kxm...@mail.gmail.com...


> My background task does take a long time to run - about 10 seconds - but
> most of that time is spent waiting for database responses, which is 
> handled

> in another thread.

Something else to look into is an asyncio driver for your database
connections. Threads aren't inherently harmful, but using them to achieve
async networking when running asyncio is a definite code smell since that
is precisely the problem asyncio is supposed to solve for.



Maybe I have not explained very well. I am not using threads to achieve 
async networking. I am using asyncio in a client server environment, and it 
works very well. If a client request involves a database query, I use a 
thread to perform that so that it does not slow down the other users. I 
usually want the originating client to block until I have a response, so I 
use 'await'. However, occasionally the request takes some time, and it is 
not necessary for the client to wait for the response, so I want to unblock 
the client straight away, run the task in the background, and then notify 
the client when the task is complete. This is where your suggestion of 
'ensure_future' does the job perfectly.


I would love to drive the database asynchronously, but of the three 
databases I use, only psycopg2 seems to have asyncio support. As my 
home-grown solution (using queues) seems to be working well so far, I am 
sticking with that until I start to experience responsiveness issues. If 
that happens, my first line of attack will be to switch from threads to 
processes.


Hope this makes sense.

Frank


--
https://mail.python.org/mailman/listinfo/python-list


Re: asyncio - run coroutine in the background

2016-02-16 Thread Steven D'Aprano
On Wed, 17 Feb 2016 01:17 am, Marko Rauhamaa wrote:

> Ok, yes, but those "background tasks" monopolize the CPU once they are
> scheduled to run.

Can you show some code demonstrating this?



-- 
Steven

-- 
https://mail.python.org/mailman/listinfo/python-list


Re: asyncio - run coroutine in the background

2016-02-16 Thread Chris Angelico
On Wed, Feb 17, 2016 at 2:21 AM, Frank Millman  wrote:
> I would love to drive the database asynchronously, but of the three
> databases I use, only psycopg2 seems to have asyncio support. As my
> home-grown solution (using queues) seems to be working well so far, I am
> sticking with that until I start to experience responsiveness issues. If
> that happens, my first line of attack will be to switch from threads to
> processes.

And this is where we demonstrate divergent thought processes. *My*
first line of attack if hybrid async/thread doesn't work would be to
mandate a PostgreSQL backend, not to switch to hybrid async/process :)
Is the added value of "you get three options of database back-end"
worth the added cost of "but now my code is massively more complex"?

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: asyncio - run coroutine in the background

2016-02-16 Thread Frank Millman
"Chris Angelico"  wrote in message 
news:captjjmqmie4groqnyvhwahcn2mwqeyqxt5kvfivotrhqy-s...@mail.gmail.com...


On Wed, Feb 17, 2016 at 2:21 AM, Frank Millman  wrote:
> I would love to drive the database asynchronously, but of the three
> databases I use, only psycopg2 seems to have asyncio support. As my
> home-grown solution (using queues) seems to be working well so far, I am
> sticking with that until I start to experience responsiveness issues. If
> that happens, my first line of attack will be to switch from threads to
> processes.

And this is where we demonstrate divergent thought processes. *My*
first line of attack if hybrid async/thread doesn't work would be to
mandate a PostgreSQL backend, not to switch to hybrid async/process :)
Is the added value of "you get three options of database back-end"
worth the added cost of "but now my code is massively more complex"?


Then we will have to agree to diverge ;-)

If I ever get my app off the ground, it will be an all-purpose, 
multi-company, multi-currency, multi-everything accounting/business system.


There is a massive market out there, and a large percentage of that is 
Microsoft-only shops. I have no intention of cutting myself off from that 
market before I even start.


I am very happy with my choice of 3 databases -

1. sqlite3 - ideal for demo purposes and for one-man businesses

2. Sql Server for those that insist on it

3. PostgreSQL for every one else, and my recommendation if asked

Frank



--
https://mail.python.org/mailman/listinfo/python-list


Re: Multiple Assignment a = b = c

2016-02-16 Thread srinivas devaki
On Tue, Feb 16, 2016 at 6:35 PM, Sven R. Kunze  wrote:
>
> First, the rhs is evaluated.
> Second, the lhs is evaluated from left to right.
Great, I will remember these two lines :)

On Tue, Feb 16, 2016 at 8:46 PM, Steven D'Aprano  wrote:
> _temp = c
> a = _temp
> b = _temp
> del _temp
>
>
> except the name "_temp" isn't actually used.
>
So it is like first right most expression is evaluated and then lhs is
evaluated from left to right.

> py> from dis import dis
> py> code = compile("a = L[a] = x", "", "exec")
> py> dis(code)
>   1   0 LOAD_NAME0 (x)
>   3 DUP_TOP
>   4 STORE_NAME   1 (a)
>   7 LOAD_NAME2 (L)
>  10 LOAD_NAME1 (a)
>  13 STORE_SUBSCR
>  14 LOAD_CONST   0 (None)
>  17 RETURN_VALUE
>
>
> Translated to English:
>
> - evaluate the expression `x` and push the result onto the stack;
>
> - duplicate the top value on the stack;
>
> - pop the top value off the stack and assign to name `a`;
>
> - evaluate the name `L`, and push the result onto the stack;
>
> - evaluate the name `a`, and push the result onto the stack;
>
> - call setattr with the top three items from the stack.
>
thank-you so much, for explaining how to find the underlying details.


>> SO isn't it counter intuitive from all other python operations.
>> like how we teach on how python performs a swap operation???
>
> No. Let's look at the equivalent swap:
>
> In all cases, the same rule applies:
>
> - evaluate the right hand side from left-most to right-most, pushing the
> values onto the stack;
>
> - perform assignments on the left hand side, from left-most to right-most.
>

uhh, i related it with swap because I was thinking variables are
binded, like first of all for all lhs assignments get their references
or names and then put the value of rhs in them.
as `a` is a name, so the rhs reference is copied to the a
`roots[a]` is a reference to an object, so it is initialized with the
reference of rhs.

anyway I got it, and all my further doubts are cleared from that
compiled code. I tried some other examples and understood how it
works.

thanks a lot.

-- 
Regards
Srinivas Devaki
Junior (3rd yr) student at Indian School of Mines,(IIT Dhanbad)
Computer Science and Engineering Department
ph: +91 9491 383 249
telegram_id: @eightnoteight
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: asyncio - run coroutine in the background

2016-02-16 Thread Marko Rauhamaa
Steven D'Aprano :

> On Wed, 17 Feb 2016 01:17 am, Marko Rauhamaa wrote:
>
>> Ok, yes, but those "background tasks" monopolize the CPU once they
>> are scheduled to run.
>
> Can you show some code demonstrating this?

Sure:


#!/usr/bin/env python3

import asyncio, time

def main():
asyncio.get_event_loop().run_until_complete(asyncio.wait([
background_task(),
looping_task() ]))

@asyncio.coroutine
def looping_task():
while True:
yield from asyncio.sleep(1)
print(int(time.time()))

@asyncio.coroutine
def background_task():
yield from asyncio.sleep(4)
t = time.time()
while time.time() - t < 10:
pass

if __name__ == '__main__':
main()


which prints:

   1455642629
   1455642630
   1455642631
   1455642642<== gap
   1455642643
   1455642644
   1455642645


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: asyncio - run coroutine in the background

2016-02-16 Thread Marko Rauhamaa
Steven D'Aprano :

> On Wed, 17 Feb 2016 01:17 am, Marko Rauhamaa wrote:
>
>> Ok, yes, but those "background tasks" monopolize the CPU once they are
>> scheduled to run.
>
> Can you show some code demonstrating this?

Sure:


#!/usr/bin/env python3

import asyncio, time

def main():
asyncio.get_event_loop().run_until_complete(asyncio.wait([
background_task(),
looping_task() ]))

@asyncio.coroutine
def looping_task():
while True:
yield from asyncio.sleep(1)
print(int(time.time()))

@asyncio.coroutine
def background_task():
yield from asyncio.sleep(4)
t = time.time()
while time.time() - t < 10:
pass

if __name__ == '__main__':
main()


which prints:

   1455642629
   1455642630
   1455642631
   1455642642<== gap
   1455642643
   1455642644
   1455642645


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: asyncio - run coroutine in the background

2016-02-16 Thread Marko Rauhamaa
Steven D'Aprano :

> On Wed, 17 Feb 2016 01:17 am, Marko Rauhamaa wrote:
>
>> Ok, yes, but those "background tasks" monopolize the CPU once they are
>> scheduled to run.
>
> Can you show some code demonstrating this?

Sure:


#!/usr/bin/env python3

import asyncio, time

def main():
asyncio.get_event_loop().run_until_complete(asyncio.wait([
background_task(),
looping_task() ]))

@asyncio.coroutine
def looping_task():
while True:
yield from asyncio.sleep(1)
print(int(time.time()))

@asyncio.coroutine
def background_task():
yield from asyncio.sleep(4)
t = time.time()
while time.time() - t < 10:
pass

if __name__ == '__main__':
main()


which prints:

   1455642629
   1455642630
   1455642631
   1455642642<== gap
   1455642643
   1455642644
   1455642645


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: asyncio - run coroutine in the background

2016-02-16 Thread Marko Rauhamaa
Marko Rauhamaa :

> Sure:

Sorry for the multiple copies.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: asyncio - run coroutine in the background

2016-02-16 Thread Marko Rauhamaa
Steven D'Aprano :

> On Wed, 17 Feb 2016 01:17 am, Marko Rauhamaa wrote:
>
>> Ok, yes, but those "background tasks" monopolize the CPU once they are
>> scheduled to run.
>
> Can you show some code demonstrating this?

Sure:


#!/usr/bin/env python3

import asyncio, time

def main():
asyncio.get_event_loop().run_until_complete(asyncio.wait([
background_task(),
looping_task() ]))

@asyncio.coroutine
def looping_task():
while True:
yield from asyncio.sleep(1)
print(int(time.time()))

@asyncio.coroutine
def background_task():
yield from asyncio.sleep(4)
t = time.time()
while time.time() - t < 10:
pass

if __name__ == '__main__':
main()


which prints:

   1455642629
   1455642630
   1455642631
   1455642642
   1455642643
   1455642644
   1455642645


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: asyncio - run coroutine in the background

2016-02-16 Thread Marko Rauhamaa
"Frank Millman" :

> I would love to drive the database asynchronously, but of the three
> databases I use, only psycopg2 seems to have asyncio support.

Yes, asyncio is at its infancy. There needs to be a moratorium on
blocking I/O.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Make a unique filesystem path, without creating the file

2016-02-16 Thread Steven D'Aprano
On Tue, 16 Feb 2016 04:56 pm, Ben Finney wrote:

> An example::
> 
> import io
> import tempfile
> names = tempfile._get_candidate_names()

I'm not sure that calling a private function of the tempfile module is
better than calling a deprecated function.

 
> def test_frobnicates_configured_spungfile():
> """ ‘foo’ should frobnicate the configured spungfile. """
> 
> fake_file_path = os.path.join(tempfile.gettempdir(), names.next())

At this point, you have a valid pathname, but no guarantee whether it refers
to a real file on the file system or not. That's the whole problem with
tempfile.makepath -- it can return a file name which is not in use, but by
the time it returns to you, you cannot guarantee that it still doesn't
exist.

Now, since this is a test which doesn't actually open that file, it doesn't
matter. There's no actual security vulnerability here. So your test doesn't
actually require that the file is unique, or that it doesn't actually
exist. (Which is good, because you can't guarantee that it doesn't exist.)

So why not just pick a random bunch of characters?

chars = list(string.ascii_letters)
random.shuffle(chars)
fake_file_path = ''.join(chars[:10])


> fake_file = io.BytesIO("Lorem ipsum, dolor sit
> amet".encode("utf-8"))
>
> patch_builtins_open(
> when_accessing_path=fake_file_path,
> provide_file=fake_file)

There's nothing apparent in this that requires that fake_file_path not
actually exist, which is good since (as I've pointed out before) you cannot
guarantee that it doesn't exist. One could just as easily, and just as
correctly, write:

patch_builtins_open(
when_accessing_path='/foo/bar/baz',
provide_file=fake_file)

and regardless of whether /foo/bar/baz actually exists or not, you are
guaranteed to get the fake file rather than the real file. So I question
whether you actually need this tempfile.makepath function at all.

*But* having questioned it, for the sake of the argument I'll assume you do
need it, and continue accordingly.


> system_under_test.config.spungfile_path = fake_file_path
> system_under_test.foo()
> assert_correctly_frobnicated(fake_file)
> 
> So the test case creates a fake file, makes a valid filesystem path to
> associate with it, then patches the ‘open’ function so that it will
> return the fake file when that specific path is requested.
> 
> Then the test case alters the system under test's configuration, giving
> it the generated filesystem path for an important file. The test case
> then calls the function about which the unit test is asserting
> behaviour, ‘system_under_test.foo’. When that call returns, the test
> case asserts some properties of the fake file to ensure the system under
> test actually accessed that file.

Personally, I think it would be simpler and easier to understand if, instead
of patching open, you allowed the test to read and write real files:

file_path = '/tmp/spam'
system_under_test.config.spungfile_path = file_path
system_under_test.foo()
assert_correctly_frobnicated(file_path)
os.unlink(file_path)


In practice, I'd want to only unlike the file if the test passes. If it
fails, I'd want to look at the file to see why it wasn't frobnicated.

I think that a correctly-working filesystem is a perfectly reasonable
prerequisite for the test, just like a working CPU, memory, power supply,
operating system and Python interpreter. You don't have to guard against
every imaginable failure ("fixme: test may return invalid results if the
speed of light changes by more than 0.0001%"), and you might as well take
advantage of real files for debugging. But that's my opinion, and if you
have another, that's your personal choice.


> With a supported standard library API for this – ‘tempfile.makepath’ for
> example – the generation of the filesystem path would change from four
> separate function calls, one of which is a private API::
> 
> names = tempfile._get_candidate_names()
> fake_file_path = os.path.join(tempfile.gettempdir(), names.next())
> 
> to a simple public function call::
> 
> fake_file_path = tempfile.makepath()

Nobody doubts that your use of tempfile.makepath is legitimate for your
use-case. But it is *not* legitimate for the tempfile module, and it is a
mistake that it was added in the first place, hence the deprecation.
Assuming that your test suite needs this function, your test library, or
test suite, should provide that function, not tempfile. I believe it is
unreasonable to expect the tempfile module to keep a function which is a
security risk in the context of "temp files" just because it is useful for
some completely unrelated use-cases.

After all, your use of this doesn't actually have anything to do with
temporary files. It is a mocked *permanent* file, not a real temporary one.


> This whole thread began because I expected s

Re: asyncio - run coroutine in the background

2016-02-16 Thread Robin Becker

On 16/02/2016 17:15, Marko Rauhamaa wrote:

Marko Rauhamaa :


Sure:


Sorry for the multiple copies.


Marko



I thought perhaps background jobs were sending them :)
--
Robin Becker

--
https://mail.python.org/mailman/listinfo/python-list


Re: Multiple Assignment a = b = c

2016-02-16 Thread Terry Reedy

On 2/16/2016 7:46 AM, srinivas devaki wrote:

Hi,

a = b = c

as an assignment doesn't return anything, i ruled out a = b = c as
chained assignment, like a = (b = c)
SO i thought, a = b = c is resolved as
a, b = [c, c]


https://docs.python.org/3/reference/simple_stmts.html#assignment-statements
"An assignment statement evaluates the expression list (remember that 
this can be a single expression or a comma-separated list, the latter 
yielding a tuple) and assigns the single resulting object to each of the 
target lists, from left to right."


a = b = c is the same as tem = c; a = tem; b = tem.

This does not work if tem is an iterator.

>>> def g():
yield 1
yield 2

>>> a,b = c,d = g()
Traceback (most recent call last):
  File "", line 1, in 
a,b = c,d = g()
ValueError: not enough values to unpack (expected 2, got 0)
>>> a, b
(1, 2)
>>> c,d
Traceback (most recent call last):
  File "", line 1, in 
c,d
NameError: name 'c' is not defined


at-least i fixed in my mind that every assignment like operation in
python is done with references and then the references are binded to
the named variables.
like globals()['a'] = result()

but today i learned that this is not the case with great pain(7 hours
of debugging.)

class Mytest(object):
 def __init__(self, a):
 self.a = a
 def __getitem__(self, k):
 print('__getitem__', k)
 return self.a[k]
 def __setitem__(self, k, v):
 print('__setitem__', k, v)
 self.a[k] = v

roots = Mytest([0, 1, 2, 3, 4, 5, 6, 7, 8])
a = 4
roots[4] = 6
a = roots[a] = roots[roots[a]]


tem = roots[roots[a]]
a = tem
roots[a] = tem


the above program's output is
__setitem__ 4 6
__getitem__ 4
__getitem__ 6
__setitem__ 6 6


--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list


Re: Make a unique filesystem path, without creating the file

2016-02-16 Thread Ben Finney
Oscar Benjamin  writes:

> If you're going to patch open to return a fake file when asked to open
> fake_file_path why do you care whether there is a real file of that
> name?

I don't, and have been saying explicitly many times in this thread that
I do not care whether the file exists. Somehow that is still not clear?

-- 
 \   “Nothing exists except atoms and empty space; everything else |
  `\is opinion.” —Democritus, c. 460 BCE – 370 BCE |
_o__)  |
Ben Finney

-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Make a unique filesystem path, without creating the file

2016-02-16 Thread Ben Finney
Steven D'Aprano  writes:

> On Tue, 16 Feb 2016 04:56 pm, Ben Finney wrote:
>
> > names = tempfile._get_candidate_names()
>
> I'm not sure that calling a private function of the tempfile module is
> better than calling a deprecated function.

Agreed, which is why I'm seeking a public API that is not deprecated.

> So why not just pick a random bunch of characters?
>
> chars = list(string.ascii_letters)
> random.shuffle(chars)
> fake_file_path = ''.join(chars[:10])

This (an equivalent) is already implemented, internally to ‘tempfile’
and tested and maintained and more robust than me re-inventing the wheel.

> Yes, but the system doesn't try to enforce the filesystem's rules,
> does it?

The test case I'm writing should not be prone to failure if the system
happens to perform some arbitrary validation of filesystem paths.

‘tempfile’ already knows how to generate filesystem paths, I want to use
that and not have to get it right myself.

> and your system shouldn't care.

If it does, this test case should not fail.

> Since your test doesn't know what filesystem your code will be running
> on, you can't make any assumptions about what paths are valid or not
> valid.

That implies that ‘tempfile._get_candidate_names’ would generate paths
that would potentially be invalid. Is that what you intend to imply?

> > Almost. I want the filesystem paths to be valid because the system
> > under test expects them, it may perform its own validation,
>
> If the system tries to validate paths, it is broken.

This is “you don't want what you say you want”, and seeing the
justifications presented I don't agree.

-- 
 \ “I must say that I find television very educational. The minute |
  `\   somebody turns it on, I go to the library and read a book.” |
_o__)—Groucho Marx |
Ben Finney

-- 
https://mail.python.org/mailman/listinfo/python-list


I m facing some problem while opening the interpreter. how can I resolve the issue?

2016-02-16 Thread Chinmaya Choudhury
Please guide me.
#Chinmay

Sent from Mail for Windows 10

-- 
https://mail.python.org/mailman/listinfo/python-list


[Glitch?] Python has just stopped working

2016-02-16 Thread Theo Hamilton
I woke up two days ago to find out that python literally won't work any
more. I have looked everywhere, asked multiple Stack Overflow questions,
and am ready to give up. Whenever I run python (3.5), I get the following
message:

Fatal Python error: Py_initialize: unable to load the file system codec
ImportError: No module named 'encodings'

Current thread 0x2168 (most recent call first):

If there's anything you know that I could do to fix this, then please tell
me. I've tried uninstalling and reparing, so it's not those. Thanks!
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: [Glitch?] Python has just stopped working

2016-02-16 Thread Christian Gollwitzer

Am 16.02.16 um 17:19 schrieb Theo Hamilton:

I woke up two days ago to find out that python literally won't work any
more. I have looked everywhere, asked multiple Stack Overflow questions,
and am ready to give up. Whenever I run python (3.5), I get the following
message:

Fatal Python error: Py_initialize: unable to load the file system codec
ImportError: No module named 'encodings'


Can it be that you have just set a strange locale? What happens if you 
run it as


LANG=C python


?

Christian


--
https://mail.python.org/mailman/listinfo/python-list


Re: I m facing some problem while opening the interpreter. how can I resolve the issue?

2016-02-16 Thread Mark Lawrence

On 16/02/2016 19:55, Chinmaya Choudhury wrote:

Please guide me.
#Chinmay

Sent from Mail for Windows 10



Please read http://catb.org/~esr/faqs/smart-questions.html and possibly 
http://www.sscce.org/, then try asking again.


--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

--
https://mail.python.org/mailman/listinfo/python-list


Re: I m facing some problem while opening the interpreter. how can I resolve the issue?

2016-02-16 Thread Alister
On Wed, 17 Feb 2016 01:25:52 +0530, Chinmaya Choudhury wrote:

> Please guide me.
> #Chinmay
> 
> Sent from Mail for Windows 10

open it correctly





-- 
The temperature of the aqueous content of an unremittingly ogled
culinary vessel will not achieve 100 degrees on the Celsius scale.
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: I m facing some problem while opening the interpreter. how can I resolve the issue?

2016-02-16 Thread Igor Korot
On Tue, Feb 16, 2016 at 2:55 PM, Chinmaya Choudhury
 wrote:
> Please guide me.
> #Chinmay

Dear Cousin Muscle,
I have a serious trouble with Tom.

Need you help at once,
Jerry.

(C)


>
> Sent from Mail for Windows 10
>
> --
> https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: [Glitch?] Python has just stopped working

2016-02-16 Thread eryk sun
On Tue, Feb 16, 2016 at 10:19 AM, Theo Hamilton  wrote:
> Whenever I run python (3.5), I get the following message:
>
> Fatal Python error: Py_initialize: unable to load the file system codec
> ImportError: No module named 'encodings'
>
> Current thread 0x2168 (most recent call first):

The interpreter can't find the standard library, which is a symptom of
having PYTHONHOME set to some other directory:

C:\>set PYTHONHOME=C:\
C:\>py -3.5
Fatal Python error: Py_Initialize: unable to load the file system codec
ImportError: No module named 'encodings'

Current thread 0x0940 (most recent call first):

Generally this environment variable is unnecessary for normal use and
shouldn't be set permanently.
-- 
https://mail.python.org/mailman/listinfo/python-list


Passing data across callbacks in ThreadPoolExecutor

2016-02-16 Thread Joseph L. Casale
What is the pattern for chaining execution of tasks with ThreadPoolExecutor?
Callbacks is not an adequate facility as each task I have will generate new 
output.

Thanks,
jlc
-- 
https://mail.python.org/mailman/listinfo/python-list


Not intalling

2016-02-16 Thread SoNu KuMaR
I'm trying to install the latest version of python . First time it didn't
install successfully ,i tried again now it have installed but not working.
I'm sending screenshot and log file. I tried reinstallation again and again
but result is same.
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Not intalling

2016-02-16 Thread Ben Finney
SoNu KuMaR  writes:

> I'm trying to install the latest version of python . First time it
> didn't install successfully ,i tried again now it have installed but
> not working.

What exactly did you try? What details about the host can you describe
so we know what may be peculiar to the problem?

> I'm sending screenshot and log file. I tried reinstallation again and
> again but result is same.

You will need to copy and paste the actual text; graphic images are not
suitable for this forum.

-- 
 \   “The optimist thinks this is the best of all possible worlds. |
  `\   The pessimist fears it is true.” —J. Robert Oppenheimer |
_o__)  |
Ben Finney

-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Make a unique filesystem path, without creating the file

2016-02-16 Thread Alan Bawden
Ben Finney  writes:

> Cameron Simpson  writes:
>
>> I've been watching this for a few days, and am struggling to
>> understand your use case.
>
> Yes, you're not alone. This surprises me, which is why I'm persisting.
>
>> Can you elaborate with a concrete example and its purpose which would
>> work with a mktemp-ish official function?
>
> An example::

Let me present another example that might strike some as more
straightforward.

If I want to create a temporary file, I can call mkstemp().  
If I want to create a temporary directory, I can call mkdtemp().

Suppose that instead of a file or a directory, I want a FIFO or a
socket.

A FIFO is created by passing a pathname to os.mkfifo().  A socket is
created by passing a pathname to an AF_UNIX socket's bind() method.  In
both cases, the pathname must not name anything yet (not even a symbolic
link), otherwise the call will fail.

So in the FIFO case, I might write something like the following:

def make_temp_fifo(mode=0o600):
while True:
path = tempfile.mktemp()
try:
os.mkfifo(path, mode=mode)
except FileExistsError:
pass
else:
return path

mktemp() is convenient here, because I don't have to worry about whether
I should be using "/tmp" or "/var/tmp" or "c:\temp", or whether the
TMPDIR environment variable is set, or whether I have permission to
create entries in those directories.  It just gives me a pathname
without making me think about the rest of that stuff.  Yes, I have to
defend against the possibility that somebody else creates something with
the same name first, but as you can see, I did that, and it wasn't
rocket science.

So is there something wrong with the above code?  Other than the fact
that the documentation says something scary about mktemp()?

It looks to me like mktemp() provides some real utility, packaged up in
a way that is orthogonal to the type of file system entry I want to
create, the permissions I want to give to that entry, and the mode I
want use to open it.  It looks like a useful, albeit low-level,
primitive that it is perfectly reasonable for the tempfile module to
supply.

And yet the documentation condemns it as "deprecated", and tells me I
should use mkstemp() instead.  (As if that would be of any use in the
situation above!)  It looks like anxiety that some people might use
mktemp() in a stupid way has caused an over-reaction.  Let the
documentation warn about the problem and point to prepackaged solutions
in the common cases of making files and directories, but I see no good
reason to deprecate this useful utility.

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Will file be closed automatically in a "for ... in open..." statement?

2016-02-16 Thread jfong
Thanks for these detailed explanation. Both statements will close file 
automatically sooner or later and, when considering the exceptions, "with" is 
better. Hope my understanding is right.

But, just curious, how do you know the "for" will do it? I can't find any 
document about it from every sources I know. Very depressed:-(

--Jach
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Will file be closed automatically in a "for ... in open..." statement?

2016-02-16 Thread Chris Angelico
On Wed, Feb 17, 2016 at 3:04 PM,   wrote:
> Thanks for these detailed explanation. Both statements will close file 
> automatically sooner or later and, when considering the exceptions, "with" is 
> better. Hope my understanding is right.
>
> But, just curious, how do you know the "for" will do it? I can't find any 
> document about it from every sources I know. Very depressed:-(
>

It's not the 'for' loop that does it. The for loop is kinda like this:

_temp = open("foo.txt")
_temp.read() # do stuff, do stuff
_temp = None

When you stop holding onto an object, Python can get rid of it. When
that happens is not promised, though - and if you have a reference
loop, it might hang around for a long time. But when a file object is
disposed of, the underlying file will get closed.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Will file be closed automatically in a "for ... in open..." statement?

2016-02-16 Thread Raspberry Aether
On 02/16/2016 11:04 PM, jf...@ms4.hinet.net wrote:
> Thanks for these detailed explanation. Both statements will close file 
> automatically sooner or later and, when considering the exceptions, "with" is 
> better. Hope my understanding is right.
> 
> But, just curious, how do you know the "for" will do it? I can't find any 
> document about it from every sources I know. Very depressed:-(
> 
> --Jach
> 

First-- IMO, don't depend on it. Instead, use something like:

with open('foo.txt') as f:
for line in f:
pass # do something here

It's one extra indent and one extra line, but it's cleaner.

To answer your question, technically, it might not-- it really
depends upon your implementation of Python. It just so happens
that the most popular version of Python (CPython, the reference
implementation) will garbage collect the file object right away.

HOWEVER. The reason the "for" will PROBABLY result in file
closure is because as soon as the for loop exits, there is no
reason to hold onto the object returned by "open", so it is
disposed. When file objects are disposed, they are closed.

IMO, don't depend on this behaviour; it's bad form.
-- 
https://mail.python.org/mailman/listinfo/python-list


threading - Doug Hellman stdlib book, Timer() subclassing etc

2016-02-16 Thread Veek. M
In Doug Hellman's book on the stdlib, he does:

import threading
import logging
logging.basicConfig(level=logging.DEBUG,
format=’(%(threadName)-10s) %(message)s’,
)

class MyThreadWithArgs(threading.Thread):
def __init__(self, group=None, target=None, name=None,
args=(), kwargs=None, verbose=None):

threading.Thread.__init__(self, group=group,
target=target,
name=name,
verbose=verbose)

self.args = args
self.kwargs = kwargs
return

def run(self):
logging.debug(’running with %s and %s’,
self.args, self.kwargs)
return

for i in range(5):
t = MyThreadWithArgs(args=(i,), kwargs={’a’:’A’, ’b’:’B’})
t.start()

1. Shouldn't def run() also include a call to the target function?
2. How does a call to a function_target result in a thread being 
created? Normally you'd have to call a function in pthreads (OS call)
One can sort of figure that t.start() hides the actual OS call, but when
we implement run().. somehow, magically there's no OS call? WTH! ??

Then in the Timer example in the next section, how is
the whole delay/canecl bit implemented?
We do t1.start so the 3 second counter starts ticking somewhere
- where? And how does he cancel that?

import threading
import time
import logging

logging.basicConfig(level=logging.DEBUG,
format=’(%(threadName)-10s) %(message)s’,
)

def delayed():
logging.debug(’worker running’)
return

t1 = threading.Timer(3, delayed)
t1.setName(’t1’)

t2 = threading.Timer(3, delayed)
t2.setName(’t2’)

logging.debug(’starting timers’)

t1.start()
t2.start()

logging.debug(’waiting before canceling %s’, t2.getName())
time.sleep(2)

logging.debug(’canceling %s’, t2.getName())
t2.cancel()

logging.debug(’done’)
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: I m facing some problem while opening the interpreter. how can I resolve the issue?

2016-02-16 Thread Steven D'Aprano
On Wednesday 17 February 2016 06:55, Chinmaya Choudhury wrote:

> Please guide me.
> #Chinmay
> 
> Sent from Mail for Windows 10


How can we help you when we don't know what problem you have?


Is the computer turned on? Is the mouse plugged in? Are you double-clicking 
the icon on the desktop? What happens? Do you get an error message or does 
the computer suddenly reboot? What does the error message say?


-- 
Steve
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Will file be closed automatically in a "for ... in open..." statement?

2016-02-16 Thread Steven D'Aprano
On Wednesday 17 February 2016 15:04, jf...@ms4.hinet.net wrote:

> Thanks for these detailed explanation. Both statements will close file
> automatically sooner or later and, when considering the exceptions, "with"
> is better. Hope my understanding is right.
> 
> But, just curious, how do you know the "for" will do it? I can't find any
> document about it from every sources I know. Very depressed:-(

This has nothing to do with "for". You would get exactly the same behaviour 
without "for":


f = open("some file", "r")
x = f.read(20)
x = f.read(30)
x = f.read()


So long as the variable "f" is in-scope, the file will stay open. If the 
above code is in a function, "f" goes out of scope when the function 
returns. If the above code is at the top level of the module, "f" will stay 
in scope forever, or until you either delete the variable from the scope:

del f

or re-assign to something else:

f = "hello world"


At this point, after the function has exited, and the variable has 
completely gone out of scope, what happens?

The garbage collector will:

- reclaim the memory used by the object;
- close the file.

BUT there is no promise WHEN the file will be closed. It might be 
immediately, or it might be when the application shuts down.

If you want the file to be closed immediately, you must:

- use a with statement;

- or explicitly call f.close()


otherwise you are at the mercy of the interpreter, which will close the file 
whenever it wants.


-- 
Steve

-- 
https://mail.python.org/mailman/listinfo/python-list