Re: Tuples and immutability

[Ethan Furman]

On 03/11/2014 08:25 PM, Terry Reedy wrote:

On 3/11/2014 10:01 PM, Rick Johnson wrote:


On Thursday, February 27, 2014 4:18:01 PM UTC-6, Ian wrote:

x += y is meant to be equivalent, except possibly in-place and
more efficient, than x = x + y.


The manual actually says "An augmented assignment expression like x += 1 can be 
rewritten as x = x + 1 to achieve a
similar, but not exactly equal effect. In the augmented version, x is only 
evaluated once. Also, when possible, the
actual operation is performed in-place, meaning that rather than creating a new 
object and assigning that to the target,
the old object is modified instead.


... and reassigned to the target.  :)  (If it doesn't say that last part, it 
should.)

--
~Ethan~
--
https://mail.python.org/mailman/listinfo/python-list

Re: beautiful soup get class info

[Peter Otten]

Christopher Welborn wrote:

> On 03/06/2014 02:22 PM, teddyb...@gmail.com wrote:
>> I am using beautifulsoup to get the title and date of the website.
>> title is working fine but I am not able to pull the date. Here is the
>> code in the url:
>>
>>   October 22, 2011
>>
>> In Python, I am using the following code:
>> date1 = soup.span.text
>> data=soup.find_all(date="value")
>>
>> Results in:
>>
>> []
>> March 5, 2014
>>
>> What is the proper way to get this info?
>> Thanks.
>>
> 
> I believe it's the 'attrs' argument.
> http://www.crummy.com/software/BeautifulSoup/bs4/doc/
> 
> # Workaround the 'class' problem:
> data = soup.find_all(attrs={'class': 'date'})
> 
> I haven't tested it, but it's worth looking into.
 
Yes there are two ways to filtr by class:

>>> soup = bs4.BeautifulSoup("""
... alpha
... beta""")

Use attrs:

>>> soup.find_all(attrs={"class": "one"})
[alpha]

Append an underscore:

>>> soup.find_all(class_="two")
[beta]


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Balanced trees

[Antoon Pardon]

Op 11-03-14 00:24, Roy Smith schreef:

> In article <8761nmrnfk@elektro.pacujo.net>,
>  Marko Rauhamaa  wrote:
>
>> Anyway, this whole debate is rather unnecessary since every developer is
>> supposed to have both weapons in their arsenal.
> The problem with having a choice is that it opens up the possibility of 
> making the wrong one :-)

This is just a standard defense for the status quo. Introducing the decimal
module also added a choice.

> As this discussion has shown, figuring out whether a hash table or a 
> tree is better for a given problem is non-trivial.  My guess is that if 
> you gave 1000 typical developers both data structures and let them pick 
> freely, the number of cases where it really mattered and the developer 
> picked the right one would be approximately equal to the number of cases 
> where they picked the wrong one.

You are only illustrating one part. How about all those cases now where the
wrong choice is more or less forced on the developer for lack of the 
alternative?

-- 
Antoon Pardon

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Tuples and immutability

[Antoon Pardon]

Op 12-03-14 07:28, Steven D'Aprano schreef:
> On Tue, 11 Mar 2014 23:25:19 -0400, Terry Reedy wrote:
>
>> Nope, 'similar' is not 'equivalent'. Evaluating x twice instead of once
>> and possibly allocating a new object versus not take extra time. In a
>> statement like "x.y.z[3*n+m] += 1", calculating the target dominates the
>> time to increment, so this form should be nearly twice as fast.
> Excellent point Terry!
>
> I always forget that the target of an augmented assignment may not be a 
> simple name like "x" but can be an arbitrary complex reference, anything 
> that is a legal assignment target. Because += is documented as only 
> evaluating the expression once it can behave quite differently to the 
> `spam = spam + 1` case. Evaluating the right hand side may have side-
> effects that change what the left hand side evaluates to. This is not the 
> case with the augmented assignment.

The documentation is wrong at that point as the following code illustrates.

| import sys
| write = sys.stdout.write
|
| class logdict:
| def __init__(self):
| self.lst = {}
|
| def __setitem__(self, key, value):
| write('[%s] <= %s\n' % (key, value))
| self.lst[key] = value
|
| def __getitem__(self, key):
| value = self.lst[key]
| write('[%s] => %s\n' % (key, value))
| return value
|
| tab = logdict()
| tab['key'] = 'value'
| tab['key'] += ' with extra tail'
| write('\n')
| tab = logdict()
| tab['key'] = 'value'
| tab['key'] = tab['key'] + ' with extra tail'

If you run this code, you get the following result:

| [key] <= value
| [key] => value
| [key] <= value with extra tail
| 
| [key] <= value
| [key] => value
| [key] <= value with extra tail

As you can see there is no difference here in the evaluations done
between using

| tab['key'] += ' with extra tail'

or

| tab['key'] = tab['key'] + ' with extra tail'
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Tuples and immutability

[Ian Kelly]

On Wed, Mar 12, 2014 at 12:28 AM, Steven D'Aprano  wrote:
> On Tue, 11 Mar 2014 23:25:19 -0400, Terry Reedy wrote:
>> Nope, 'similar' is not 'equivalent'. Evaluating x twice instead of once
>> and possibly allocating a new object versus not take extra time. In a
>> statement like "x.y.z[3*n+m] += 1", calculating the target dominates the
>> time to increment, so this form should be nearly twice as fast.
>
> Excellent point Terry!
>
> I always forget that the target of an augmented assignment may not be a
> simple name like "x" but can be an arbitrary complex reference, anything
> that is a legal assignment target. Because += is documented as only
> evaluating the expression once it can behave quite differently to the
> `spam = spam + 1` case. Evaluating the right hand side may have side-
> effects that change what the left hand side evaluates to. This is not the
> case with the augmented assignment.

Of course one could also do:

z = x.y.z
i = 3*n+m
z[i] = z[i] + 1

which reduces the duplicated work down to storing and loading a couple
of locals, and also prevents side-effects from affecting the LHS
evaluation.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Tuples and immutability

[Ian Kelly]

On Wed, Mar 12, 2014 at 3:39 AM, Antoon Pardon
 wrote:
> The documentation is wrong at that point as the following code illustrates.

Either way it still has to do a getitem and a setitem, but if you have
a more nested structure then the extra getitems are not repeated.  For
example, using your logdict class:

>>> tab = logdict()
>>> tab[1] = logdict()
[1] <= <__main__.logdict object at 0x02A2E430>
>>> tab[1][2] = logdict()
[1] => <__main__.logdict object at 0x02A2E430>
[2] <= <__main__.logdict object at 0x02A2EB10>
>>> tab[1][2][3] = ['value']
[1] => <__main__.logdict object at 0x02A2E430>
[2] => <__main__.logdict object at 0x02A2EB10>
[3] <= ['value']
>>> tab[1][2][3] += [' with extra tail']
[1] => <__main__.logdict object at 0x02A2E430>
[2] => <__main__.logdict object at 0x02A2EB10>
[3] => ['value']
[3] <= ['value', ' with extra tail']

versus:

>>> tab[1][2][3] = ['value']
[1] => <__main__.logdict object at 0x02A2E430>
[2] => <__main__.logdict object at 0x02A2EB10>
[3] <= ['value']
>>> tab[1][2][3] = tab[1][2][3] + [' with extra tail']
[1] => <__main__.logdict object at 0x02A2E430>
[2] => <__main__.logdict object at 0x02A2EB10>
[3] => ['value']
[1] => <__main__.logdict object at 0x02A2E430>
[2] => <__main__.logdict object at 0x02A2EB10>
[3] <= ['value', ' with extra tail']

As you can see the += version does two fewer getitem calls in this case.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: which async framework?

[Ian Kelly]

On Tue, Mar 11, 2014 at 5:38 PM, Chris Angelico  wrote:
> On Wed, Mar 12, 2014 at 10:18 AM, Marko Rauhamaa  wrote:
>> Chris Angelico :
>>
>>> What corner cases are there with threads that you don't have with
>>> anything else?
>>
>> There are numerous. Here's one example: deadlocks due to two threads
>> taking locks in a different order. The problem crops up naturally with
>> two intercommunicating classes. It can sometimes be very difficult to
>> spot or avoid.
>
> Yep. Now how is that not a problem when you use some other model, like
> an event loop? The standard methods of avoiding deadlocks (like
> acquiring locks in strict order) work exactly the same for all models,
> and are just as necessary.

If you don't have threads then the only locks you need to acquire are
for external resources.  You might still deadlock, but at least your
process won't be deadlocking with itself.
-- 
https://mail.python.org/mailman/listinfo/python-list

SMTPHandler and mail subject

[eras . rasmuson]

Hi

I use logging.handlers.SMTPHandler and i have tried to change subject of mail.

Are there exists easy way to do that ?

Subject of mail can change repeatedly depends on content of mail.


Eras
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Tuples and immutability

[Oscar Benjamin]

On 12 March 2014 03:25, Terry Reedy  wrote:
> On 3/11/2014 10:01 PM, Rick Johnson wrote:
>>
>>
>> On Thursday, February 27, 2014 4:18:01 PM UTC-6, Ian wrote:
>>>
>>> x += y is meant to be equivalent, except possibly in-place and
>>> more efficient, than x = x + y.
>
>
> The manual actually says "An augmented assignment expression like x += 1 can
> be rewritten as x = x + 1 to achieve a similar, but not exactly equal
> effect. In the augmented version, x is only evaluated once. Also, when
> possible, the actual operation is performed in-place, meaning that rather
> than creating a new object and assigning that to the target, the old object
> is modified instead.
>
>
>
>> In an ideal world, the speed of these two codes should be the same,
>
>
> Nope, 'similar' is not 'equivalent'. Evaluating x twice instead of once and
> possibly allocating a new object versus not take extra time. In a statement
> like "x.y.z[3*n+m] += 1", calculating the target dominates the time to
> increment, so this form should be nearly twice as fast.

An example where the result is semantically different:

>>> from numpy import array
>>> a = array([1, 2, 3], dtype=int)
>>> a
array([1, 2, 3])
>>> a + 1.1
array([ 2.1,  3.1,  4.1])
>>> a += 1.1
>>> a
array([2, 3, 4])

The reason is that numpy arrays are both mutable and have statically
determined elementary data type:

>>> (a + 1.1).dtype
dtype('float64')
>>> a.dtype
dtype('int64')


Oscar
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: SMTPHandler and mail subject

[Andriy Kornatskyy]

Eras,

You have to override getSubject method of SMTPHandler.

http://hg.python.org/cpython/file/677327810121/Lib/logging/handlers.py#l907

Thanks.

Andriy Kornatskyy

On Mar 12, 2014, at 12:08 PM, eras.rasmu...@gmail.com wrote:

> Hi
> 
> I use logging.handlers.SMTPHandler and i have tried to change subject of mail.
> 
> Are there exists easy way to do that ?
> 
> Subject of mail can change repeatedly depends on content of mail.
> 
> 
> Eras
> -- 
> https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: unittest weirdness

[Ethan Furman]

On 03/11/2014 08:36 PM, Terry Reedy wrote:

On 3/11/2014 6:13 PM, John Gordon wrote:

In  Ethan Furman 
 writes:


  if missing:
  raise ValueError('invoices %r missing from batch' % missing)


It's been a while since I wrote test cases, but I recall using the assert*
methods (assertEqual, assertTrue, etc.) instead of raising exceptions.
Perhaps that's the issue?


Yes. I believe the methods all raise AssertionError on failure, and the test 
methods are wrapped with try:.. except
AssertionError as err:

if missing:
  raise ValueError('invoices %r missing from batch' % missing)

should be "assertEqual(missing, [], 'invoices missing from batch')" and if that 
fails, the non-empty list is printed
along with the message.


I've tried it both ways, and both ways my process is being killed, presumably 
by the O/S.

I will say it's an extra motivating factor to have few failing tests -- if more than two of my tests fail, all I see are 
'.'s, 'E's, and 'F's, with no clues as to which test failed nor why.  Thank goodness for '-v' and being able to specify 
which method of which class to run!


--
~Ethan~
--
https://mail.python.org/mailman/listinfo/python-list

Re: which async framework?

[Neal Becker]

Grant Edwards wrote:

> On 2014-03-11, Antoine Pitrou  wrote:
>> Sturla Molden  gmail.com> writes:
>>> 
>>> Chris Withers  simplistix.co.uk> wrote:
>>> > Hi All,
>>> > 
>>> > I see python now has a plethora of async frameworks and I need to try
>>> > and pick one to use from:
>>> > 
>>> > - asyncio/tulip
>>> > - tornado
>>> > - twisted
>>> 
>>> I'd go for using iocp, epoll and kqueue/kevent directly. Why bother to
>>> learn a framework? You will find epoll and kqueue/kevent in the select
>>> module and iocp in pywin32.
>>
>> Yes, why use a library when you can rewrite it all yourself?
>> Actually, you should probably issue system calls to the kernel directly,
>> the libc is overrated (as is portability, I suppose).
> 
> And don't bother with device drivers for the network adapters either.
> Just map their PCI regions in to user-space and twiddle the reigisters
> directly!  ;)
> 
> [I do that when testing PCI boards with C code, and one of these days
> I'm going to figure out how to do it with Python.]
> 

Use numpy with mmap

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Tuples and immutability

[Antoon Pardon]

Op 12-03-14 10:51, Ian Kelly schreef:
> On Wed, Mar 12, 2014 at 3:39 AM, Antoon Pardon
>  wrote:
>> The documentation is wrong at that point as the following code illustrates.
> Either way it still has to do a getitem and a setitem, but if you have
> a more nested structure then the extra getitems are not repeated.  For
> example, using your logdict class:

Sure, but the documentation doesn't say for sufficiently nested structures
some evaluations are not repeated.

Take the following example.

| tab['key'] = [1]
| tab['key'] += [2]

Now let us rewrite that second statment in two different ways.

| tab['key'] = tab['key'] + [2]

or

| tmp = tab['key']
| tmp += [2]

Now which of these two rewritings is closer to the augmented concatenation?
A natural reading of the documentation would conclude the second one, because
in that case we only need to evaluate tab['key'] once as righthand sided.
However it turns out the augmented concantenation is closer to the first
rewriting here, evaluating tab['key'] twice once a lefthand sided and once
as right hand sided, which IMO will surprise people that rely on the 
documentation.

-- 
Antoon Pardon

-- 
https://mail.python.org/mailman/listinfo/python-list

locale getlocale returns None on OSX

[Albert-Jan Roskam]

Hi,

locale.getlocale() sometimes returns (None, None) under OSX (Python 2, not sure 
about Python 3, but I think so). The problem is outlined here:
http://stackoverflow.com/questions/1629699/locale-getlocale-problems-on-osx

What is the cause of this? Is it limited to just Darwin systes? Does the 
'horrible hack' solution on OS have any drawbacks? I like it better because it 
is not needed to set the LC_ALL environment variable prior to starting the 
Python program.

Regards,

Albert-Jan



~~

All right, but apart from the sanitation, the medicine, education, wine, public 
order, irrigation, roads, a 

fresh water system, and public health, what have the Romans ever done for us?

~~

-- 
https://mail.python.org/mailman/listinfo/python-list

Save to a file, but avoid overwriting an existing file

[zoom]

Hi!

I would like to assure that when writing to a file I do not overwrite an 
existing file, but I'm unsure which is the best way to approach to this 
problem. As I can see, there are at least two possibilities:


1. I could use fd = os.open("x", os.O_WRONLY | os.O_CREAT | os.O_EXCL)
which will fail - if the file exists. However, I would prefer if the 
program would try to save under different name in this case, instead of 
discarding all the calculation done until now - but I' not too well with 
catching exceptions.


2. Alternatively, a unique string could be generated to assure that no 
same file exists. I can see one approach to this is to include date and 
time in the file name. But this seems to me a bit clumsy, and is not 
unique, i.e. it could happen (at least in theory) that two processes 
finish in the same second.


Any suggestions, please?
--
https://mail.python.org/mailman/listinfo/python-list

Re: SMTPHandler and mail subject

[eras . rasmuson]

It works. Thank you :)


Eras
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Save to a file, but avoid overwriting an existing file

[Skip Montanaro]

This seems to be an application-level decision. If so, in your
application, why not just check to see if the file exists, and
implement whatever workaround you deem correct for your needs? For
example (to choose a simple, but rather silly, file naming strategy):

fname = "x"
while os.path.exists(fname):
fname = "%s.%f" % (fname, random.random())
fd = open(fname, "w")

It's clearly not going to be safe from race conditions, but I leave
solving that problem as an exercise for the reader.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

How do we pass default argument value to create thread object?

[Piyush Verma]

Hi,

I am using Thread class to create threads.

thread = threading.Thread(target=Fun, args=[arg1, arg2, arg3="val"])
thread.start()


This code is throwing compilation error(Ipython).
In [19]: import threading

In [20]: def Fun(agr1, arg2, arg3=None):
   : pass
   :

In [21]: thread = threading.Thread(target=Fun, args=[arg1, arg2, arg3="val"
])

   File "", line 1
 thread = threading.Thread(target=Fun, args=[arg1, arg2, arg3="val"])
 ^
SyntaxError: invalid syntax

How do we pass the value to default arguments while creating thread object?

Regards,
~Piyush
Facebook 
Twitter
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Save to a file, but avoid overwriting an existing file

[Tim Chase]

On 2014-03-12 13:29, zoom wrote:
> 2. Alternatively, a unique string could be generated to assure that
> no same file exists. I can see one approach to this is to include
> date and time in the file name. But this seems to me a bit clumsy,
> and is not unique, i.e. it could happen (at least in theory) that
> two processes finish in the same second.

Python offers a "tempfile" module that gives this (and a whole lot
more) to you out of the box.

-tkc






-- 
https://mail.python.org/mailman/listinfo/python-list

Re: unittest weirdness

[Roy Smith]

In article ,
 Ethan Furman  wrote:

> I've tried it both ways, and both ways my process is being killed, presumably 
> by the O/S.

What evidence do you have the OS is killing the process?

Some systems have an oom (Out Of Memory) process killer, which nukes 
(semi-random) process when the system exhausts memory.  Is it possible 
this is happening?  If so, you should see some log message in one of 
your system logs.

You didn't mention (or maybe I misssed it) which OS you're using.  I'm 
assuming you've got some kind of system call tracer (strace, truss, 
dtrace, etc).  Try running your tests under that.  If something is 
sending your process a kill signal, you'll see it:

[gazillions of lines elided]
write(1, ">>> ", 4>>> ) = 4
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
select(1, [0], NULL, NULL, NULL)= ? ERESTARTNOHAND (To be 
restarted)
--- SIGTERM (Terminated) @ 0 (0) ---
+++ killed by SIGTERM +++

Alternatively, maybe something inside your process is just calling 
sys.exit(), or even os._exit().  You'll see the exit() system call in 
the strace output.

And, of course, the standard suggestion to reduce this down to the 
minimum test case.  You posted:

 def test_xxx_1(self):
 p = self.pp.lockbox_payments[0]
 # affirm we have what we're expecting
 self.assertEqual(
 (p.payer, p.ck_num, p.credit),
 ('a customer', '010101', 1),
 )
 self.assertEqual(p.invoices.keys(), ['XXX'])
 self.assertEqual(p.invoices.values()[0].amount, 1)
 # now make sure we get back what we're expecting
 np, b = self.pp._match_invoices(p)
 missing = []
 for inv_num in ('123456', '789012', '345678'):
 if inv_num not in b:
 missing.append(inv_num)
 if missing:
 raise ValueError('invoices %r missing from batch' % missing)

what happens if you reduce that to:

 def test_xxx_1(self):
  self.fail()

do you still get this strange behavior?  What if you get rid of your 
setUpClass()?  Keep hacking away at the test suite until you get down to 
a single line of code which, if run, exhibits the behavior, and if 
commented out, does not.  At that point, you'll have a clue what's 
causing this.  If you're lucky :-)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How do we pass default argument value to create thread object?

[Jean-Michel Pichavant]

- Original Message - 

> Hi,

> I am using Thread class to create threads.
> 
> thread = threading.Thread(target=Fun, args=[arg1, arg2, arg3="val"])
> thread.start()
> 

> This code is throwing compilation error(Ipython).

> In [19]: import threading

> In [20]: def Fun(agr1, arg2, arg3=None):
> : pass
> :

> In [21]: thread = threading.Thread(target=Fun, args=[arg1, arg2,
> arg3="val" ])
> 
> File "", line 1
> thread = threading.Thread(target=Fun, args=[arg1, arg2, arg3="val"])
> ^
> SyntaxError: invalid syntax

> How do we pass the value to default arguments while creating thread
> object?

> Regards,
> ~Piyush
> Facebook Twitter

Hi,

try something like

thread = threading.Thread(target=Fun, args=("val1", "val2"), 
kwargs={"arg3":"val3"})

This will call
Fun("val1", "val2", arg3="val3")

Browse the python tutorial for details on arguments and keyword arguments.

Cheers,

JM


-- IMPORTANT NOTICE: 

The contents of this email and any attachments are confidential and may also be 
privileged. If you are not the intended recipient, please notify the sender 
immediately and do not disclose the contents to any other person, use it for 
any purpose, or store or copy the information in any medium. Thank you.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How do we pass default argument value to create thread object?

[Zachary Ware]

On Wed, Mar 12, 2014 at 8:09 AM, Piyush Verma <114piy...@gmail.com> wrote:
> Hi,
>
> I am using Thread class to create threads.
> 
> thread = threading.Thread(target=Fun, args=[arg1, arg2, arg3="val"])
> thread.start()
> 
>
> This code is throwing compilation error(Ipython).
> In [19]: import threading
>
> In [20]: def Fun(agr1, arg2, arg3=None):
>: pass
>:
>
> In [21]: thread = threading.Thread(target=Fun, args=[arg1, arg2,
> arg3="val"])
> 
>File "", line 1
>  thread = threading.Thread(target=Fun, args=[arg1, arg2, arg3="val"])
>  ^
> SyntaxError: invalid syntax
>
> How do we pass the value to default arguments while creating thread object?

Use the 'kwargs' argument:

   thread = threading.Thread(target=Fun, args=[1, 2], kwargs={'arg3': "val"})

See http://docs.python.org/3.4/library/threading#threading.Thread for more info.

Hope this helps,
-- 
Zach
-- 
https://mail.python.org/mailman/listinfo/python-list

Deep vs. shallow copy?

[Alex van der Spek]

I think I understand the difference between deep vs. shallow copies but 
I was bitten by this:

with open(os.path.join('path', 'foo.txt', 'rb') as txt:
 reader = csv.reader(txt)
 data = [row.append(year) for row in reader]

This does not work although the append does complete. The below works:

with open(os.path.join('path', 'foo.txt', 'rb') as txt:
 reader = csv.reader(txt)
 data = [row + [year] for row in reader]

However in this context I am baffled. If someone can explain what is 
going on here, I would be most grateful.

Alex van der Spek
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Deep vs. shallow copy?

[Skip Montanaro]

On Wed, Mar 12, 2014 at 9:25 AM, Alex van der Spek  wrote:
> with open(os.path.join('path', 'foo.txt', 'rb') as txt:
>  reader = csv.reader(txt)
>  data = [row.append(year) for row in reader]

Forget deep v. shallow copies. What is the value of the variable year?
And why would you expect list.append to return anything?

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Deep vs. shallow copy?

[Zachary Ware]

On Wed, Mar 12, 2014 at 9:25 AM, Alex van der Spek  wrote:
> I think I understand the difference between deep vs. shallow copies but
> I was bitten by this:
>
> with open(os.path.join('path', 'foo.txt', 'rb') as txt:
>  reader = csv.reader(txt)
>  data = [row.append(year) for row in reader]
>
> This does not work although the append does complete. The below works:
>
> with open(os.path.join('path', 'foo.txt', 'rb') as txt:
>  reader = csv.reader(txt)
>  data = [row + [year] for row in reader]
>
> However in this context I am baffled. If someone can explain what is
> going on here, I would be most grateful.

Deep/shallow copying doesn't really come into this.  row.append()
mutates the list (row), it doesn't return a new list.  Like most
in-place/mutating methods in Python, it returns None instead of self
to show that mutation was done, so your listcomp fills `data` with
Nones; there is no copying done at all.  The second example works as
you expected because `row + [year]` results in a new list, which the
listcomp is happy to append to `data`--which does mean that `row` is
copied.

To avoid the copy that the second listcomp is doing (which really
shouldn't be necessary anyway, unless your rows are astronomically
huge), you have a couple of options.  First, you can expand your
listcomp and use append:

   with open(os.path.join('path', 'foo.txt'), 'rb') as txt: # with
your typo fixed ;)
   reader = csv.reader(txt)
   data = []
   for row in reader:
   row.append(year)
   data.append(row)

To me, that's pretty readable and pretty clear about what it's doing.
Then there's this option, which I don't recommend:

   import operator
   with open(os.path.join('path', 'foo.txt'), 'rb') as txt:
   reader = csv.reader(txt)
   data = [operator.iadd(row, [year]) for row in reader]

This works because operator.iadd is basically shorthand for
row.__iadd__([year]), which does return self (otherwise, the
assignment part of `row += [year]` couldn't work).  But, it's not as
clear about what's happening, and only saves a whole two lines (maybe
3 if you already have operator imported).

Hope this helps,
-- 
Zach
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Deep vs. shallow copy?

[Alex van der Spek]

On Wed, 12 Mar 2014 10:00:09 -0500, Zachary Ware wrote:

> On Wed, Mar 12, 2014 at 9:25 AM, Alex van der Spek 
> wrote:
>> I think I understand the difference between deep vs. shallow copies but
>> I was bitten by this:
>>
>> with open(os.path.join('path', 'foo.txt', 'rb') as txt:
>>  reader = csv.reader(txt)
>>  data = [row.append(year) for row in reader]
>>
>> This does not work although the append does complete. The below works:
>>
>> with open(os.path.join('path', 'foo.txt', 'rb') as txt:
>>  reader = csv.reader(txt)
>>  data = [row + [year] for row in reader]
>>
>> However in this context I am baffled. If someone can explain what is
>> going on here, I would be most grateful.
> 
> Deep/shallow copying doesn't really come into this.  row.append()
> mutates the list (row), it doesn't return a new list.  Like most
> in-place/mutating methods in Python, it returns None instead of self to
> show that mutation was done, so your listcomp fills `data` with Nones;
> there is no copying done at all.  The second example works as you
> expected because `row + [year]` results in a new list, which the
> listcomp is happy to append to `data`--which does mean that `row` is
> copied.
> 
> To avoid the copy that the second listcomp is doing (which really
> shouldn't be necessary anyway, unless your rows are astronomically
> huge), you have a couple of options.  First, you can expand your
> listcomp and use append:
> 
>with open(os.path.join('path', 'foo.txt'), 'rb') as txt: # with
> your typo fixed ;)
>reader = csv.reader(txt)
>data = []
>for row in reader:
>row.append(year)
>data.append(row)
> 
> To me, that's pretty readable and pretty clear about what it's doing.
> Then there's this option, which I don't recommend:
> 
>import operator
>with open(os.path.join('path', 'foo.txt'), 'rb') as txt:
>reader = csv.reader(txt)
>data = [operator.iadd(row, [year]) for row in reader]
> 
> This works because operator.iadd is basically shorthand for
> row.__iadd__([year]), which does return self (otherwise, the assignment
> part of `row += [year]` couldn't work).  But, it's not as clear about
> what's happening, and only saves a whole two lines (maybe 3 if you
> already have operator imported).
> 
> Hope this helps,

Thank you, that helped immensely! 

Having been taught programming in Algol60 Python still defeats me at times!
Particularly since Algol60 wasn't very long lived and what came
thereafter (FORTRAN) much worse.

I get it now, the below is equivalent! 

I am perfectly happy with the one copy of the list row + [year]. 
Just wanted to learn something here and I have!


Python 2.6.5 (r265:79063, Feb 27 2014, 19:44:14) 
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> a = [1,2,3]
>>> b = 'val'
>>> a.append(b)
>>> a
[1, 2, 3, 'val']
>>> c = a.append(b)
>>> print c
None
>>> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: unittest weirdness

[Ethan Furman]

On 03/12/2014 06:44 AM, Roy Smith wrote:

In article ,
  Ethan Furman  wrote:


I've tried it both ways, and both ways my process is being killed, presumably
by the O/S.


What evidence do you have the OS is killing the process?


I put a bare try/except around the call to unittest.main, with a print 
statement in the except, and nothing ever prints.



Some systems have an oom (Out Of Memory) process killer, which nukes
(semi-random) process when the system exhausts memory.  Is it possible
this is happening?  If so, you should see some log message in one of
your system logs.


That would explain why my editor windows were being killed.



You didn't mention (or maybe I misssed it) which OS you're using.


Ubuntu 13 something or other.


I'm
assuming you've got some kind of system call tracer (strace, truss,
dtrace, etc).


Sadly, I have no experience with those programs yet, and until now didn't even 
know they existed.


Try running your tests under that.  If something is
sending your process a kill signal, you'll see it:

[gazillions of lines elided]
write(1, ">>> ", 4>>> ) = 4
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
select(1, [0], NULL, NULL, NULL)= ? ERESTARTNOHAND (To be
restarted)
--- SIGTERM (Terminated) @ 0 (0) ---
+++ killed by SIGTERM +++

Alternatively, maybe something inside your process is just calling
sys.exit(), or even os._exit().  You'll see the exit() system call in
the strace output.


My bare try/except would have caught that.



And, of course, the standard suggestion to reduce this down to the
minimum test case.  You posted:

  def test_xxx_1(self):
  p = self.pp.lockbox_payments[0]
  # affirm we have what we're expecting
  self.assertEqual(
  (p.payer, p.ck_num, p.credit),
  ('a customer', '010101', 1),
  )
  self.assertEqual(p.invoices.keys(), ['XXX'])
  self.assertEqual(p.invoices.values()[0].amount, 1)
  # now make sure we get back what we're expecting
  np, b = self.pp._match_invoices(p)
  missing = []
  for inv_num in ('123456', '789012', '345678'):
  if inv_num not in b:
  missing.append(inv_num)
  if missing:
  raise ValueError('invoices %r missing from batch' % missing)

what happens if you reduce that to:

  def test_xxx_1(self):
   self.fail()


I only get the strange behavior if more than two (or maybe three) of my test cases fail.  Less than that magic number, 
and everything works just fine.  It doesn't matter which two or three, either.




do you still get this strange behavior?  What if you get rid of your
setUpClass()?  Keep hacking away at the test suite until you get down to
a single line of code which, if run, exhibits the behavior, and if
commented out, does not.  At that point, you'll have a clue what's
causing this.  If you're lucky :-)


I strongly suspect it's memory.  When I originally wrote the code I tried to include six months worth of EoM data, but 
had to back it down to three as my process kept mysteriously dying at four or more months.  There must be waaay too 
much stuff being kept alive by the stack traces of the failed tests.


Thanks for your help!

--
~Ethan~
--
https://mail.python.org/mailman/listinfo/python-list

Re: How to create an instance of a python class from C++

[Stefan Behnel]

Barry Scott, 11.03.2014 22:37:
> On 5 Mar 2014, at 00:14, Bill wrote:
>> I can't figure out how to create an instance
>> of a python class from 'C++':
> 
> Why not use pycxx from http://sourceforge.net/projects/cxx/?
> 
> This lib does all the heavy lifting for you for both python2 and python3.
> Has docs and examples.

Yes, tool support definitely helps here. I was going to suggest Cython
(also for obvious reasons), where the code that the OP posted would look
like this:

  def RegisterClass(class_decl):
  an = type(class_decl)()
  print(an.description())
  return 0

Clearly substantially simpler than the posted C code (and certainly safer,
faster and more correct) - although that doesn't really help me much with
understanding what the intention of this code is, looks rather weird...

Stefan


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: unittest weirdness

[Roy Smith]

In article ,
 Ethan Furman  wrote:

> > Alternatively, maybe something inside your process is just calling
> > sys.exit(), or even os._exit().  You'll see the exit() system call in
> > the strace output.
> 
> My bare try/except would have caught that.

A bare except would catch sys.exit(), but not os._exit().  Well, no 
that's not actually true.  Calling os._exit() will raise:

TypeError: _exit() takes exactly 1 argument (0 given)

but it won't catch os._exit(0) :-)

> > what happens if you reduce that to:
> >
> >   def test_xxx_1(self):
> >self.fail()
> 
> I only get the strange behavior if more than two (or maybe three) of my test 
> cases fail.  Less than that magic number, 
> and everything works just fine.  It doesn't matter which two or three, 
> either.

OK, well, assuming this is a memory problem, what if you do:

  def test_xxx_1(self):
   l = []
   while True:
   l.append(0)

That should eventually run out of memory.  Does that get you the same 
behavior in a single test case?  If so, that at least would be evidence 
supporting the memory exhaustion theory.

> I strongly suspect it's memory.  When I originally wrote the code I tried to 
> include six months worth of EoM data, but 
> had to back it down to three as my process kept mysteriously dying at four or 
> more months.  There must be waaay too 
> much stuff being kept alive by the stack traces of the failed tests.

One thing you might try is running your tests under nose 
(http://nose.readthedocs.org/).  Nose knows how to run unittest tests, 
and one of the gazillion options it has is to run each test case in an 
isolated process:

  --process-restartworker
If set, will restart each worker process once their
tests are done, this helps control memory leaks from
killing the system. [NOSE_PROCESS_RESTARTWORKER]

that might be what you need.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re:Save to a file, but avoid overwriting an existing file

[Dave Angel]

 zoom  Wrote in message:
> Hi!
> 
> I would like to assure that when writing to a file I do not overwrite an 
> existing file, but I'm unsure which is the best way to approach to this 
> problem. As I can see, there are at least two possibilities:
> 
> 1. I could use fd = os.open("x", os.O_WRONLY | os.O_CREAT | os.O_EXCL)
> which will fail - if the file exists. However, I would prefer if the 
> program would try to save under different name in this case, instead of 
> discarding all the calculation done until now - but I' not too well with 
> catching exceptions.
> 

The tempfile module is your best answer,  but if you really need
 to keep the file afterwards,  you'll have the same problem when
 you rename it later. 

I suggest you learn about try/except.  For simple cases it's not
 that tough, though if you want to ask about it, you'll need to
 specify your Python version. 

-- 
DaveA

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Deep vs. shallow copy?

[Wayne Brehaut]

On 12 Mar 2014 15:29:59 GMT, Alex van der Spek 
wrote:

>On Wed, 12 Mar 2014 10:00:09 -0500, Zachary Ware wrote:
>
>> On Wed, Mar 12, 2014 at 9:25 AM, Alex van der Spek 
>> wrote:

=== 8< ===

>Having been taught programming in Algol60 Python still defeats me at times!
>Particularly since Algol60 wasn't very long lived and what came
>thereafter (FORTRAN) much worse.

Actually, Algol 60 lived on (and lives on, though not so much used now
outside of Scandinavia) in an improved and OOP-extended form in Simula
67 (now just Simula). Most implementations excpt that for DEC-System10
were, however, overpriced and poorly marketed, so we had to wait for
C++ (improved and OOP-extended C) for OOP to catch on.

=== 8< ===
-- 
https://mail.python.org/mailman/listinfo/python-list

What does gc.get_objects() return?

[Jurko Gospodnetić]

  Hi all.

  I was wondering if someone could explain gc.get_objects() in a bit 
more detail to me.


  Does it return a list of 'all objects known to Python'? Only some of 
them? Which does it return? Which it does not?


  For example (done using CPython 3.4 interactive interpreter) here it 
does not contain an object I know exists:



>>> a = object()
>>> a in gc.get_objects()
False


  but here it does:


>>> class Foo: pass
...
>>> a = Foo()
>>> a in gc.get_objects()
True


  It also does not seem to contain string objects. At first I thought 
that perhaps interned strings were not placed under GC control, so I 
tested this:



>>> a = "asdkjfhk23498237$&(*$&*($ksjdhfkjsd"
>>> b = "asdkjfhk23498237$&(*$&*($ksjdhfkjsd"
>>> a is b
False
>>> a in gc.get_objects()
False
>>> b in gc.get_objects()
False


  Which shows that non-interned scripts such as these are not getting 
listed in gc.get_objects() either. :-s


  Many thanks.

  Best regards,
Jurko Gospodnetić

--
https://mail.python.org/mailman/listinfo/python-list

Re: Save to a file, but avoid overwriting an existing file

[Emile van Sebille]

On 3/12/2014 5:29 AM, zoom wrote:


2. Alternatively, a unique string could be generated to assure that no
same file exists. I can see one approach to this is to include date and
time in the file name. But this seems to me a bit clumsy, and is not
unique, i.e. it could happen (at least in theory) that two processes
finish in the same second.


I tend to use this method -- prepending the job name or targeting 
different directories per job precludes duplication.  Unless you're 
running the same job at the same time, in which case tempfile is the way 
to go (which I use for archiving spooled print files which can occur 
simultaneously.)


Emile





--
https://mail.python.org/mailman/listinfo/python-list

Re: locale getlocale returns None on OSX

[Albert-Jan Roskam]

>
> From: Albert-Jan Roskam 
>To: Python  
>Sent: Wednesday, March 12, 2014 1:22 PM
>Subject: locale getlocale returns None on OSX
> 
>
>Hi,
>
>locale.getlocale() sometimes returns (None, None) under OSX (Python 2, not 
>sure about Python 3, but I think so). The problem is outlined here:
>http://stackoverflow.com/questions/1629699/locale-getlocale-problems-on-osx
>
>What is the cause of this? Is it limited to just Darwin systes? Does the 
>'horrible hack' solution on OS have any drawbacks? I like it better because it 
>is not needed to set the LC_ALL environment variable prior to starting the 
>Python program.
>
>Regards,
>
>Albert-Jan


Ok, here are some tests on my own system:

albertjan@debian:~$ uname -a
Linux debian 3.2.0-4-amd64 #1 SMP Debian 3.2.54-2 x86_64 GNU/Linux


Even if locale.setlocale is used first, OSX returns (None, None) when 
locale.getlocale() is called. Another thing that surprises me in the examples 
below is the output of the "python -c" example using Python 2.7. Isn't this 
supposed to be exactly equivalent to the code that follows?



#  Python 2.7 albertjan@debian:~$ python -c "import locale; 
locale.setlocale(locale.LC_ALL, ""); print(locale.getlocale())"

(None, None)   # < why is this?


albertjan@debian:~$ python
Python 2.7.3 (default, Jan  2 2013, 13:56:14) 
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale
>>> locale.setlocale(locale.LC_ALL, "")
'en_US.UTF-8'
>>> locale.getlocale()
('en_US', 'UTF-8')  # <--- OSX (sometimes?) returns (None, None) here.



#  Python 3.3 
albertjan@debian:~$ python3 -c "import locale; locale.setlocale(locale.LC_ALL, 
""); print(locale.getlocale())"
('en_US', 'UTF-8')

albertjan@debian:~$ python3
Python 3.3.4 (default, Feb 17 2014, 19:23:00) 
[GCC 4.7.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale
>>> locale.setlocale(locale.LC_ALL, "")
'en_US.UTF-8'
>>> locale.getlocale()
('en_US', 'UTF-8')



#  Pypy 
albertjan@debian:~$ pypy -c "import locale; locale.setlocale(locale.LC_ALL, 
""); print locale.getlocale()"
(None, None)

albertjan@debian:~$ pypy

Python 2.7.3 (87aa9de10f9ca71da9ab4a3d53e0ba176b67d086, Mar 10 2014, 14:07:15)
[PyPy 2.2.1 with GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
And now for something completely different: ``pypy is a better kind of
foolishness - lac''
 import locale
 locale.setlocale(locale.LC_ALL, "")
'en_US.UTF-8'
 locale.getlocale()
('en_US', 'UTF-8')
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What does gc.get_objects() return?

[Dan Stromberg]

On Wed, Mar 12, 2014 at 12:34 PM, Jurko Gospodnetić
 wrote:
>   Hi all.
>
>   I was wondering if someone could explain gc.get_objects() in a bit more
> detail to me.
>
>   Does it return a list of 'all objects known to Python'? Only some of them?
> Which does it return? Which it does not?
>
>   For example (done using CPython 3.4 interactive interpreter) here it does
> not contain an object I know exists:
>
>> >>> a = object()
>> >>> a in gc.get_objects()
>> False
>
>
>   but here it does:
>
>> >>> class Foo: pass
>> ...
>> >>> a = Foo()
>> >>> a in gc.get_objects()
>> True

I don't know why some objects aren't showing up in get_objects, but
here's a quick note to say it's not just 3.4:

$ ./pythons 'import gc; var="data"; print(var in gc.get_objects())'
/usr/local/cpython-2.4/bin/python
   [6600 refs]
   False
/usr/local/cpython-2.5/bin/python
   False
   [7536 refs]
/usr/local/cpython-2.6/bin/python
   False
   [15233 refs]
/usr/local/cpython-2.7/bin/python
   False
   [18526 refs]
/usr/local/cpython-3.0/bin/python
   Traceback (most recent call last):
 File "", line 1, in 
 File "/usr/local/cpython-3.0/lib/python3.0/_weakrefset.py", line
121, in __eq__
   return self.data == set(ref(item) for item in other)
 File "/usr/local/cpython-3.0/lib/python3.0/_weakrefset.py", line
121, in 
   return self.data == set(ref(item) for item in other)
   TypeError: cannot create weak reference to 'str' object
   [34510 refs]
/usr/local/cpython-3.1/bin/python
   False
   [37270 refs]
/usr/local/cpython-3.2/bin/python
   False
   [37700 refs]
/usr/local/cpython-3.3/bin/python
   False
   [42662 refs]
/usr/local/cpython-3.4/bin/python False
/usr/local/pypy-2.2/bin/pypy True
/usr/local/jython-2.7b1/bin/jython
   Traceback (most recent call last):
 File "", line 1, in 
   NotImplementedError: not applicable to Java GC

Interesting that it is present in Pypy.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What does gc.get_objects() return?

[MRAB]

On 2014-03-12 19:34, Jurko Gospodnetić wrote:

Hi all.

I was wondering if someone could explain gc.get_objects() in a bit
more detail to me.

Does it return a list of 'all objects known to Python'? Only some of
them? Which does it return? Which it does not?


[snip]

gc.is_tracked(...) might be relevant to your question.

--
https://mail.python.org/mailman/listinfo/python-list

Re: locale getlocale returns None on OSX

[Ned Deily]

In article 
<1394626979.46880.yahoomailba...@web163806.mail.gq1.yahoo.com>,
 Albert-Jan Roskam  wrote:
> locale.getlocale() sometimes returns (None, None) under OSX (Python 2, not 
> sure about Python 3, but I think so). The problem is outlined here:
> http://stackoverflow.com/questions/1629699/locale-getlocale-problems-on-osx

Python's locale uses the plaform's locale implementation and POSIX 
locale is an old and somewhat weird animal.  Make sure you thoroughly 
read the description of the locale module, in particular, the caveats 
section:

http://docs.python.org/2/library/locale.html#background-details-hints-tip
s-and-caveats

The first gotcha is that you need to explicitly call 
locale.setlocale("LC_ALL,"") to get the preferred locale environment, as 
either specified by the user, by LANG or other LC_* environment 
variables, or the platform defaults.  In general, OS X does not provide 
a default for your process.  However, various environments do.  The OS X 
Terminal.app has profile settings (Preferences -> Settings -> (Profile 
name) -> Advanced) to specific a character encoding and a checkbox to 
"Set locale environment variables on startup".  With that set properly, 
programs run under Terminal.app will see LANG set.  The user can also 
set an explicit LANG value in a shell profile, like .profile or .bashrc, 
but those only apply when a shell is being used.  Depending on which 
profile it is set in, that might not work under all conditions, like 
under "sudo" or in an "ssh" session.  Setting an env variable in a shell 
profile would also not apply to an double-clickable app bundle since no 
shell is involved in launching it; it is possible to set environment 
variables in the app's Info.plist, though.
 
> What is the cause of this? Is it limited to just Darwin systes? Does the 
> 'horrible hack' solution on OS have any drawbacks? I like it better because 
> it is not needed to set the LC_ALL environment variable prior to starting the 
> Python program.

As the caveats section points out, setting locale env vars may have 
unwanted side effects on other parts of the process it is running in or 
creates.  So, if you are using it in a standalone program, it may be OK.  
If you are using it a module intended to be used by other programs, you 
probably shouldn't be changing something that could break other parts of 
the calling program.

-- 
 Ned Deily,
 n...@acm.org

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What does gc.get_objects() return?

[Jurko Gospodnetić]

  Hi.

On 12.3.2014. 21:02, MRAB wrote:

I was wondering if someone could explain gc.get_objects() in a bit
more detail to me.

Does it return a list of 'all objects known to Python'? Only some of
them? Which does it return? Which it does not?



gc.is_tracked(...) might be relevant to your question.


  Thanks! The documentation for gc.is_tracked() straightened me out. :-)

  So gc.collect() returns a list of all the objects GC is in charge of, 
and which instances are and are not tracked by the GC is, I guess, an 
interpreter implementation detail.


  For CPython 3.4 I guess strings and other atomic types such as ints 
are not, as well as raw object() instances. Custom class instances on 
the other hand seem to be under GC control.


  Thanks again.

  Best regards,
Jurko Gospodnetić


--
https://mail.python.org/mailman/listinfo/python-list

DB API question - where is a stored procedure's return value?

[Skip Montanaro]

I've stumbled on a problem with the python-sybase module. If I have a
stored procedure like this:

create stored procedure test_proc
as
return 1

and call it from Python like this:

curs.callproc("test_proc", {})

it's not clear to me where the return status is stored. Currently, the
python-sybase module provides (I believe) the ability to set output
parameters (those specified as such in the argument dictionary), and
you can iterate over all the result sets the stored procedure
produces.

Looking at the Python database API (PEP 249), I saw no mention of the
return status:

.callproc( procname [, parameters ] )

(This method is optional since not all databases provide stored procedures. [3])

Call a stored database procedure with the given name. The sequence of
parameters must contain one entry for each argument that the procedure
expects. The result of the call is returned as modified copy of the
input sequence. Input parameters are left untouched, output and
input/output parameters replaced with possibly new values.

The procedure may also provide a result set as output. This must then
be made available through the standard.fetch*() methods.

I see no mention of how to handle the return value. The "modified copy
of the input sequence" is just that. In the case of the python-sybase
module, any parameter values in the input dictionary  of type
DataBufType are rewritten with values from the first row of the first
result set. The Cursor class also defines a _status_result() method,
which sounds promising. I haven't had a chance to try it out yet. My
concern is more that the Python database API doesn't mention stored
procedure return values at all.

What do other database adaptors do about stored procedure return
values? Does PEP 249 need revision?

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What does gc.get_objects() return?

[Chris Angelico]

On Thu, Mar 13, 2014 at 8:28 AM, Jurko Gospodnetić
 wrote:
>   So gc.collect() returns a list of all the objects GC is in charge of, and
> which instances are and are not tracked by the GC is, I guess, an
> interpreter implementation detail.

I assume you don't mean collect() there, as that returns the amount of
garbage that it just collected :)

It's not strictly an implementation detail, beyond that there are
certain optimizations. For instance...

>   For CPython 3.4 I guess strings and other atomic types such as ints are
> not, as well as raw object() instances. Custom class instances on the other
> hand seem to be under GC control.

... strings and ints should never be listed, and custom objects should
always be listed, but I'd say the non-tracking of object() would be an
implementation-specific optimization. Definitely the non-tracking of a
dict of nothing but atomic keys and values would be that.

The concept is that the GC tracks (in that sense; everything in
CPython is refcounted, but that's not what these functions look at)
anything that could be a part of a reference cycle. That's all it
concerns itself with, so something that can't have references to
arbitrary objects can't possibly be worth tracking. Interestingly, a
tuple of integers is tracked:

>>> a=1,2,3
>>> gc.is_tracked(a)
True

So not all optimizations are done that could be done.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

TALK.ORIGINS CENSORS THE TRUTH

[I THRINAXODON]

EVIDENCE THAT HUMANS LIVED IN THE DEVONIAN:

https://groups.google.com/group/sci.bio.paleontology/browse_thread/thread/6f501c469c7af24f# 



https://groups.google.com/group/sci.bio.paleontology/browse_thread/thread/3aad75c16afb0b82# 





http://thrinaxodon.wordpress.com/

===

THRINAXODON ONLY HAD THIS TO SAY:

"I..I...I...Can't believe it. This completely disproved Darwinian
orthodoxy."

===

THE BASTARDS AT THE SMITHSONIAN, AND THE LEAKEY FOUNDATION ARE ERODING
WITH FEAR.

===
THESE ASSHOLES ARE GOING TO DIE:
THOMAS AQUINAS;
ALDOUS HUXLEY;
BOB CASANVOVA;
SkyEyes;
DAVID IAIN GRIEG;
MARK ISAAK;
JOHN HARSHAM;
RICHARD NORMAN;
DR. DOOLITTLE;
CHARLES DARWIN;
MARK HORTON;
ERIK SIMPSON;
HYPATIAB7;
PAUL J. GANS;
JILLERY;
WIKI TRIK;
THRINAXODON;
PETER NYIKOS;
RON OKIMOTO;
JOHN S. WILKINS
===

THRINAXODON WAS SCOURING ANOTHER DEVONIAN FOSSIL BED, AND FOUND A
HUMAN SKULL, AND A HUMAN FEMUR. HE ANALYSED THE FINDS, AND SAW THAT
THEY WERE NOT NORMAL ROCKS. THESE WERE FOSSILIZED BONES. THEY EVEN HAD
TOOTH MARKS ON THEM. SO, THRINAXODON BROUGHT THEM TO THE LEAKEY
FOUNDATION, THEY UTTERLY DISMISSED IT, AND SAID, "We want to keep
people thinking that humans evolved 2 Ma." THRINAXODON BROUGHT HIS
SWORD, AND SAID, "SCIENCE CORRECTS ITSELF." RICHARD LEAKEY SAID, "That
is a myth, for people to believe in science." THRINAXODON PLANS TO
BRING DOOM TO SCIENCE, ITSELF.



THRINAXODON IS NOW ON REDDIT

--
---Thrinaxodon

--
https://mail.python.org/mailman/listinfo/python-list

Re: What does gc.get_objects() return?

[Ian Kelly]

On Wed, Mar 12, 2014 at 3:44 PM, Chris Angelico  wrote:
> The concept is that the GC tracks (in that sense; everything in
> CPython is refcounted, but that's not what these functions look at)
> anything that could be a part of a reference cycle. That's all it
> concerns itself with, so something that can't have references to
> arbitrary objects can't possibly be worth tracking. Interestingly, a
> tuple of integers is tracked:
>
 a=1,2,3
 gc.is_tracked(a)
> True
>
> So not all optimizations are done that could be done.

Or is it?

>>> a = 1,2,3
>>> gc.is_tracked(a)
True
>>> gc.collect()
0
>>> gc.is_tracked(a)
False
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What does gc.get_objects() return?

[Ian Kelly]

On Wed, Mar 12, 2014 at 4:35 PM, Ian Kelly  wrote:
>> So not all optimizations are done that could be done.
>
> Or is it?
>
 a = 1,2,3
 gc.is_tracked(a)
> True
 gc.collect()
> 0
 gc.is_tracked(a)
> False

I guess the reason for this is that when PyTuple_New is called, it
knows how many elements it will contain but doesn't know yet what they
are, so it defers the issue by just going ahead and adding itself to
the GC
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Save to a file, but avoid overwriting an existing file

[Cameron Simpson]

On 12Mar2014 13:29, zoom  wrote:
> I would like to assure that when writing to a file I do not
> overwrite an existing file, but I'm unsure which is the best way to
> approach to this problem. As I can see, there are at least two
> possibilities:
> 
> 1. I could use fd = os.open("x", os.O_WRONLY | os.O_CREAT | os.O_EXCL)
> which will fail - if the file exists. However, I would prefer if the
> program would try to save under different name in this case, instead
> of discarding all the calculation done until now - but I' not too
> well with catching exceptions.

Others have menthions tempfile, though of course you have the same collision
issue when you come to rename the temp file if you are keeping it.

I would run with option 1 for your task.

Just iterate until os.open succeeds.

However, you need to distinuish _why_ an open fails. For example,
if you were trying to make files in a directory to which you do not
have write permission, or just a directory that did not exist,
os.open would fail not matter what name you used, so your loop would
run forever.

Therefore you need to continue _only_ if you get EEXIST. Otherwise abort.

So you'd have some code like this (totally untested):

  # at top of script
  import errno

  # where you make the file
  def open_new(primary_name):
try:
  fd = os.open(primary_name, os.O_WRONLY | os.O_CREAT | os.O_EXCL)
except OSError as e:
  if e.errno != errno.EEXIST:
raise
else:
  return primary_name, fd
n = 1
while True:
  secondary_name = "%s.%d" % (primary_name, n)
  try:
fd = os.open(secondary_name, os.O_WRONLY | os.O_CREAT | os.O_EXCL)
  except OSError as e:
if e.errno != errno.EEXIST:
  raise
  else:
return secondary_name, fd
  n += 1

  # where you need the file
  path, fd = open_new("x")

That gets you a function your can reuse which returns the file's
name and the file descriptor.

Cheers,
-- 
Cameron Simpson 

Reason #173 to fear technology:

o   o   o   o   o   o  
   ^|\ ^|^ v|^ v|v |/v |X|  \|  |
/\  >\ /<   >\ /<   >\ /<   >\

o>  o   o   o   o   o   o   o
\   x<\> <)>  |\
   /<   >\ /<   >\ /<   >\  >>  L

 Mr. email does the Macarena.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: DB API question - where is a stored procedure's return value?

[Ian Kelly]

On Wed, Mar 12, 2014 at 3:44 PM, Skip Montanaro  wrote:
> What do other database adaptors do about stored procedure return
> values? Does PEP 249 need revision?

I can't speak generally, but in cx_Oracle you either execute a query like this:

result = cursor.var(cx_Oracle.NUMBER)
cursor.execute(":1 := test_proc()", [result])
print(result.getvalue())

or you use the non-standard callfunc method:

print(cursor.callfunc("test_proc", cx_Oracle.NUMBER))

As a general solution, one might wrap a stored procedure that returns
a value into a stored procedure that has an output parameter and call
it with callproc.  Some implementations might include a return value
in the parameter list anyway.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Save to a file, but avoid overwriting an existing file

[Mark Lawrence]

On 12/03/2014 22:19, Cameron Simpson wrote:

On 12Mar2014 13:29, zoom  wrote:

I would like to assure that when writing to a file I do not
overwrite an existing file, but I'm unsure which is the best way to
approach to this problem. As I can see, there are at least two
possibilities:

1. I could use fd = os.open("x", os.O_WRONLY | os.O_CREAT | os.O_EXCL)
which will fail - if the file exists. However, I would prefer if the
program would try to save under different name in this case, instead
of discarding all the calculation done until now - but I' not too
well with catching exceptions.


Others have menthions tempfile, though of course you have the same collision
issue when you come to rename the temp file if you are keeping it.

I would run with option 1 for your task.

Just iterate until os.open succeeds.

However, you need to distinuish _why_ an open fails. For example,
if you were trying to make files in a directory to which you do not
have write permission, or just a directory that did not exist,
os.open would fail not matter what name you used, so your loop would
run forever.

Therefore you need to continue _only_ if you get EEXIST. Otherwise abort.

So you'd have some code like this (totally untested):

   # at top of script
   import errno

   # where you make the file
   def open_new(primary_name):
 try:
   fd = os.open(primary_name, os.O_WRONLY | os.O_CREAT | os.O_EXCL)
 except OSError as e:
   if e.errno != errno.EEXIST:
 raise
 else:
   return primary_name, fd
 n = 1
 while True:
   secondary_name = "%s.%d" % (primary_name, n)
   try:
 fd = os.open(secondary_name, os.O_WRONLY | os.O_CREAT | os.O_EXCL)
   except OSError as e:
 if e.errno != errno.EEXIST:
   raise
   else:
 return secondary_name, fd
   n += 1

   # where you need the file
   path, fd = open_new("x")

That gets you a function your can reuse which returns the file's
name and the file descriptor.

Cheers,



I haven't looked but would things be easier if the new exception 
hierarchy were used 
http://docs.python.org/3.3/whatsnew/3.3.html#pep-3151-reworking-the-os-and-io-exception-hierarchy 
?


--
My fellow Pythonistas, ask not what our language can do for you, ask 
what you can do for our language.


Mark Lawrence

---
This email is free from viruses and malware because avast! Antivirus protection 
is active.
http://www.avast.com


--
https://mail.python.org/mailman/listinfo/python-list

Re: DB API question - where is a stored procedure's return value?

[Ian Kelly]

On Wed, Mar 12, 2014 at 5:00 PM, Ian Kelly  wrote:
> result = cursor.var(cx_Oracle.NUMBER)
> cursor.execute(":1 := test_proc()", [result])
> print(result.getvalue())

Sorry, that should properly be:

result = cursor.var(cx_Oracle.NUMBER)
cursor.execute("BEGIN :1 := test_proc(); END;", [result])
print(result.getvalue())
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Deep vs. shallow copy?

[Steven D'Aprano]

On Wed, 12 Mar 2014 15:29:59 +, Alex van der Spek wrote:

> Having been taught programming in Algol60 Python still defeats me at
> times! Particularly since Algol60 wasn't very long lived and what came
> thereafter (FORTRAN) much worse.

Fortran came first. Fortran was the first high-level language which 
allowed the programmer to write things that looked rather like the sorts 
of mathematical expressions they were used to.

There were a few higher-level assembly languages that came before 
Fortran, such as SpeedCoding, Fortran's predecessor, but Fortran was the 
first truly high-level programming language, and even in 1957 it came 
with an optimizing compiler.

I'm not really familiar with Algol, but I do know Pascal, and you should 
think of the append method to be like a Pascal procedure. Because Python 
doesn't have true procedures, it follows the convention that returning 
the special object None signals the intention to return nothing at all. 
Hence your example below:


 c = a.append(b)
 print c
> None





-- 
Steven D'Aprano
http://import-that.dreamwidth.org/
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What does gc.get_objects() return?

[Chris Angelico]

On Thu, Mar 13, 2014 at 9:35 AM, Ian Kelly  wrote:
> Or is it?
>
 a = 1,2,3
 gc.is_tracked(a)
> True
 gc.collect()
> 0
 gc.is_tracked(a)
> False

Huh, *that* is interesting!

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: DB API question - where is a stored procedure's return value?

[Petite Abeille]

On Mar 13, 2014, at 12:00 AM, Ian Kelly  wrote:

> As a general solution, one might wrap a stored procedure that returns
> a value into a stored procedure that has an output parameter and call
> it with callproc.  Some implementations might include a return value
> in the parameter list anyway.

Alternatively… if it’s really a function… wrap it in a select statement… such 
as:

select foo() as value from dual

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: unittest weirdness

[Steven D'Aprano]

On Wed, 12 Mar 2014 08:32:29 -0700, Ethan Furman wrote:

> There must
> be waaay too much stuff being kept alive by the stack traces of the
> failed tests.


I believe that unittest does keep stack traces alive until the process 
ends. I thought that there was a recent bug report for it, but the only 
one I can find was apparently fixed more than a decade ago:

http://bugs.python.org/issue451309





-- 
Steven D'Aprano
http://import-that.dreamwidth.org/
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Tuples and immutability

[Steven D'Aprano]

On Tue, 11 Mar 2014 17:06:43 -0600, Ian Kelly wrote:

> On Tue, Mar 11, 2014 at 10:46 AM, Steven D'Aprano
>  wrote:
>>> There are a number of possible solutions.  One possibility would be to
>>> copy the Circle as an Ellipse and return the new object instead of
>>> mutating it. Then you have the situation where, given a mutable object
>>> x that satisfies isinstance(x, Ellipse), the stretch method *may* be
>>> able to update the object in-place, or it *may* not.
>>
>> That is a really lousy design. Of course we are free to create classes
>> with ugly, inconsistent, stupid or unworkable APIs if we like. Python
>> won't stop us:
> 
> That's true but irrelevant to my point, which was to counter the
> assertion that mutable types can always be assumed to be able to perform
> operations in-place.

"Always"? Not so fast.

This is Python. We have freedom to abuse nearly everything, and if you 
want to shoot yourself in the foot, you can. With the exception of a 
handful of things which cannot be overridden (e.g. None, numeric 
literals, syntax) you cannot strictly assume anything about anything. 
Python does not enforce that iterators raise StopIteration when empty, or 
that indexing beyond the boundaries of a sequence raises IndexError, or 
that __setitem__ of a mapping sets the key and value, or that __len__ 
returns a length.

Augmented assignment is no different. The docs describe the intention of 
the designers and the behaviour of the classes that they control, so with 
standard built-in classes like int, str, list, tuple etc. you can safely 
assume that mutable types will perform the operation in place and 
immutable types won't, but with arbitrary types from some arbitrarily 
eccentric or twisted programmer, who knows what it will do?


-- 
Steven D'Aprano
http://import-that.dreamwidth.org/
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: unittest weirdness

[Steven D'Aprano]

On Tue, 11 Mar 2014 13:58:17 -0700, Ethan Furman wrote:

> class Test_wfbrp_20140225(TestCase):
> 
>  @classmethod
>  def setUpClass(cls):
>  cls.pp = wfbrp.PaymentProcessor(
>  '.../lockbox_file',
>  '.../aging_file',
>  [
>  Path('month_end_1'),
>  Path('month_end_2'),
>  Path('month_end_3'),
>  ],
>  )

This has nothing to do with your actual problem, which appears to be the 
Linux(?) OOM killer reaping your applications, just some general 
observations on your test.


>  def test_xxx_1(self):

Having trouble thinking up descriptive names for the test? That's a sign 
that the test might be doing too much. Each test should check one self-
contained thing. That may or may not be a single call to a unittest 
assert* method, but it should be something you can describe in a few 
words:

"it's a regression test for bug 23"
"test that the database isn't on fire"
"invoices should have a valid debtor"
"the foo report ought to report all the foos"
"...and nothing but the foos."

This hints -- its just a hint, mind you, since I lack all specific 
knowledge of your application -- that the following "affirm" tests should 
be pulled out into separate tests.

>  p = self.pp.lockbox_payments[0]
>  # affirm we have what we're expecting 
>  self.assertEqual(
>  (p.payer, p.ck_num, p.credit),
>  ('a customer', '010101', 1),
>  )
>  self.assertEqual(p.invoices.keys(), ['XXX'])
>  self.assertEqual(p.invoices.values()[0].amount, 1)

which would then leave this to be the Actual Thing Being Tested for this 
test, which then becomes test_no_missing_invoices rather than test_xxx_1.

>  # now make sure we get back what we're expecting 
>  np, b = self.pp._match_invoices(p)
>  missing = []
>  for inv_num in ('123456', '789012', '345678'):
>  if inv_num not in b:
>  missing.append(inv_num)
>  if missing:
>  raise ValueError('invoices %r missing from batch' %
>  missing)

Raising an exception directly inside the test function should only occur 
if the test function is buggy. As Terry has already suggested, this 
probably communicates your intention much better:

self.assertEqual(missing, [])



-- 
Steven D'Aprano
http://import-that.dreamwidth.org/
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: unittest weirdness

[Steven D'Aprano]

On Wed, 12 Mar 2014 08:32:29 -0700, Ethan Furman wrote:

>> Some systems have an oom (Out Of Memory) process killer, which nukes
>> (semi-random) process when the system exhausts memory.  Is it possible
>> this is happening?  If so, you should see some log message in one of
>> your system logs.
> 
> That would explain why my editor windows were being killed.


Try opening a second console tab and running top in it. It will show the 
amount of memory being used. Then run the tests in the first, jump back 
to top, and watch to see if memory use goes through the roof:

top -Mm -d 0.5

will sort by memory use, display memory in more sensible human-readable 
units instead of bytes, and update the display every 0.5 second. You can 
then hit the "i" key to toggle display of idle processes and only show 
those that are actually doing something (which presumably will include 
Python running the tests).

This at least will allow you to see whether or not memory is the concern.




-- 
Steven D'Aprano
http://import-that.dreamwidth.org/
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Save to a file, but avoid overwriting an existing file

[Ben Finney]

Cameron Simpson  writes:

> Therefore you need to continue _only_ if you get EEXIST. Otherwise
> abort.

If you target Python 3.3 or later, you can catch “FileExistsError”
http://docs.python.org/3/library/exceptions.html#FileExistsError>
which is far simpler than messing around with ‘errno’ values.

-- 
 \ “I know you believe you understood what you think I said, but I |
  `\ am not sure you realize that what you heard is not what I |
_o__) meant.” —Robert J. McCloskey |
Ben Finney

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: unittest weirdness

[Ethan Furman]

On 03/12/2014 04:47 PM, Steven D'Aprano wrote:


top -Mm -d 0.5


Cool, thanks!

--
~Ethan~
--
https://mail.python.org/mailman/listinfo/python-list

Re: unittest weirdness

[Ethan Furman]

On 03/12/2014 04:38 PM, Steven D'Aprano wrote:


[snip lots of good advice for unit testing]


I was just removing the Personally Identifiable Information.  Each test is pulling a payment from a batch of payments, 
so the first couple asserts are simply making sure I have the payment I think I have, then I run the routine that is 
supposed to match that payment with a bunch of invoices, and then I test to make sure I got back the invoices that I 
have manually verified are the correct ones to be returned.


There are many different tests because there are many different paths through the code, depending on exactly which 
combination of insanities the bank, the customer, and the company choose to inflict at that moment.  ;)


--
~Ethan~
--
https://mail.python.org/mailman/listinfo/python-list

Re: Tuples and immutability

[Ian Kelly]

On Wed, Mar 12, 2014 at 5:20 PM, Steven D'Aprano
 wrote:
> On Tue, 11 Mar 2014 17:06:43 -0600, Ian Kelly wrote:
>
>> That's true but irrelevant to my point, which was to counter the
>> assertion that mutable types can always be assumed to be able to perform
>> operations in-place.
>
> "Always"? Not so fast.
>
> This is Python. We have freedom to abuse nearly everything, and if you
> want to shoot yourself in the foot, you can. With the exception of a
> handful of things which cannot be overridden (e.g. None, numeric
> literals, syntax) you cannot strictly assume anything about anything.
> Python does not enforce that iterators raise StopIteration when empty, or
> that indexing beyond the boundaries of a sequence raises IndexError, or
> that __setitem__ of a mapping sets the key and value, or that __len__
> returns a length.

Thank you; you've stated my point more succinctly than I did.

> Augmented assignment is no different. The docs describe the intention of
> the designers and the behaviour of the classes that they control, so with
> standard built-in classes like int, str, list, tuple etc. you can safely
> assume that mutable types will perform the operation in place and
> immutable types won't, but with arbitrary types from some arbitrarily
> eccentric or twisted programmer, who knows what it will do?

This got me curious about how consistent the standard library is about
this exactly, so I did some grepping.  In the standard library there
are 5 mutable types that support concatenation that I was able to
find: list, deque, array, bytearray, and Counter.  There are none that
support addition, which I find interesting in that the language
provides hooks for in-place addition but never uses them itself.

All of the classes above appear to follow the rule that if you can
concatenate an operand, you can in-place concatenate the same operand.
 The converse however does not hold:  list.__iadd__ and
Counter.__iadd__ are both more permissive in what types they will
accept than their __add__ counterparts, and especially interesting to
me is that deque implements __iadd__ but does not implement __add__ at
all.  This last in particular seems to support the assertion that +=
should be viewed more as a shorthand for an in-place operation, less
as an equivalent for x = x + y.

>>> l = [1,2,3]
>>> l + (4,5,6)
Traceback (most recent call last):
  File "", line 1, in 
TypeError: can only concatenate list (not "tuple") to list
>>> l += (4,5,6)
>>> l
[1, 2, 3, 4, 5, 6]

>>> c = collections.Counter('mississippi')
>>> c + collections.Counter('alabama')
Counter({'s': 4, 'a': 4, 'i': 4, 'p': 2, 'm': 2, 'b': 1, 'l': 1})
>>> c + dict({'a': 4, 'l': 1, 'b': 1, 'm': 1})
Traceback (most recent call last):
  File "", line 1, in 
TypeError: unsupported operand type(s) for +: 'Counter' and 'dict'
>>> c += dict({'a': 4, 'l': 1, 'b': 1, 'm': 1})
>>> c
Counter({'s': 4, 'a': 4, 'i': 4, 'p': 2, 'm': 2, 'b': 1, 'l': 1})

>>> d = collections.deque([1,2,3])
>>> d += [4,5,6]
>>> d
deque([1, 2, 3, 4, 5, 6])
>>> d + [7,8,9]
Traceback (most recent call last):
  File "", line 1, in 
TypeError: unsupported operand type(s) for +: 'collections.deque' and 'list'
>>> d.__add__
Traceback (most recent call last):
  File "", line 1, in 
AttributeError: 'collections.deque' object has no attribute '__add__'
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Tuples and immutability

[Terry Reedy]

On 3/12/2014 9:35 PM, Ian Kelly wrote:

On Wed, Mar 12, 2014 at 5:20 PM, Steven D'Aprano
 wrote:

On Tue, 11 Mar 2014 17:06:43 -0600, Ian Kelly wrote:


That's true but irrelevant to my point, which was to counter the
assertion that mutable types can always be assumed to be able to perform
operations in-place.


"Always"? Not so fast.

This is Python. We have freedom to abuse nearly everything, and if you
want to shoot yourself in the foot, you can. With the exception of a
handful of things which cannot be overridden (e.g. None, numeric
literals, syntax) you cannot strictly assume anything about anything.
Python does not enforce that iterators raise StopIteration when empty, or
that indexing beyond the boundaries of a sequence raises IndexError, or
that __setitem__ of a mapping sets the key and value, or that __len__
returns a length.


Thank you; you've stated my point more succinctly than I did.


Augmented assignment is no different. The docs describe the intention of
the designers and the behaviour of the classes that they control, so with
standard built-in classes like int, str, list, tuple etc. you can safely
assume that mutable types will perform the operation in place and
immutable types won't, but with arbitrary types from some arbitrarily
eccentric or twisted programmer, who knows what it will do?


This got me curious about how consistent the standard library is about
this exactly, so I did some grepping.  In the standard library there
are 5 mutable types that support concatenation that I was able to
find: list, deque, array, bytearray, and Counter.  There are none that
support addition, which I find interesting in that the language
provides hooks for in-place addition but never uses them itself.

All of the classes above appear to follow the rule that if you can
concatenate an operand, you can in-place concatenate the same operand.
  The converse however does not hold:  list.__iadd__ and
Counter.__iadd__ are both more permissive in what types they will
accept than their __add__ counterparts, and especially interesting to
me is that deque implements __iadd__ but does not implement __add__ at
all.  This last in particular seems to support the assertion that +=
should be viewed more as a shorthand for an in-place operation, less
as an equivalent for x = x + y.


l = [1,2,3]
l + (4,5,6)

Traceback (most recent call last):
   File "", line 1, in 
TypeError: can only concatenate list (not "tuple") to list

l += (4,5,6)
l

[1, 2, 3, 4, 5, 6]


Like it or not, one should actually think of 'somelist += iterable' as 
equivalent to 'somelist.extend(iterable)'.  Without looking at the C 
code, I  suspect that the latter is the internal implementation. 
Collections.deque also has .extend. Collections.Counter has .update and 
that is += seems to be doing.




c = collections.Counter('mississippi')
c + collections.Counter('alabama')

Counter({'s': 4, 'a': 4, 'i': 4, 'p': 2, 'm': 2, 'b': 1, 'l': 1})

c + dict({'a': 4, 'l': 1, 'b': 1, 'm': 1})

Traceback (most recent call last):
   File "", line 1, in 
TypeError: unsupported operand type(s) for +: 'Counter' and 'dict'

c += dict({'a': 4, 'l': 1, 'b': 1, 'm': 1})
c

Counter({'s': 4, 'a': 4, 'i': 4, 'p': 2, 'm': 2, 'b': 1, 'l': 1})


d = collections.deque([1,2,3])
d += [4,5,6]
d

deque([1, 2, 3, 4, 5, 6])

d + [7,8,9]

Traceback (most recent call last):
   File "", line 1, in 
TypeError: unsupported operand type(s) for +: 'collections.deque' and 'list'

d.__add__

Traceback (most recent call last):
   File "", line 1, in 
AttributeError: 'collections.deque' object has no attribute '__add__'




--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list

Re: unittest weirdness

[Terry Reedy]

On 3/12/2014 11:32 AM, Ethan Furman wrote:


I strongly suspect it's memory.  When I originally wrote the code I
tried to include six months worth of EoM data, but had to back it down
to three as my process kept mysteriously dying at four or more months.
There must be waaay too much stuff being kept alive by the stack
traces of the failed tests.


There is an issue or two about unittest not releasing memory. Also, 
modules are not cleared from sys.modules, so anything accessible from 
global scope is kept around.


--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list

Re: Save to a file, but avoid overwriting an existing file

[Cameron Simpson]

On 13Mar2014 11:22, Ben Finney  wrote:
> Cameron Simpson  writes:
> > Therefore you need to continue _only_ if you get EEXIST. Otherwise
> > abort.
> 
> If you target Python 3.3 or later, you can catch “FileExistsError”
> http://docs.python.org/3/library/exceptions.html#FileExistsError>
> which is far simpler than messing around with ‘errno’ values.

Yes and no.

Firstly, we Old Farts find these new ways confusing:-)

Secondly, and perhaps more importantly, if I were using the O_EXCL
et al constants in a call to os.open as the OP is doing, I would
want to be equally picky about what I recognise in the exception;
I _want_ to use error explicitly. In particular, the behaviour of
"os.open" is pretty much a bare metal call to the POSIX "open(2)"
OS interface, and so I want my response to be framed directly in
terms of the documented error responses for that API. Thus, errno
values.

Finally, if I were grouping my responses to the exception by type
and perhaps performing a few different actions depending on the
error, then the new class hierachy is a good way to do it. But for
this instance, IMO, we're catching exactly one failure mode as
"retry with another name" and reraising anything else. So I don't
find an errno test any less readable that a class comparison, and
it still feels more precise to me.

Cheers,
-- 
Cameron Simpson 

The road less traveled is FASTER !!
- Luc Marcouiller, marco...@ireq.hydro.qc.ca
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What does gc.get_objects() return?

[Terry Reedy]

On 3/12/2014 3:34 PM, Jurko Gospodnetić wrote:


   I was wondering if someone could explain gc.get_objects() in a bit
more detail to me.

   Does it return a list of 'all objects known to Python'? Only some of
them? Which does it return? Which it does not?


This took about 10 seconds.

>>> import gc
>>> help(gc.get_objects)
Help on built-in function get_objects in module gc:

get_objects(...)
get_objects() -> [...]

Return a list of objects tracked by the collector (excluding the 
list returned).

---

Now, what object does gc track? CPython answer: objects that could be 
involved in and kept alive by reverence cycles and therefore not deleted 
when not needed by reference counting alone. Object instances, numbers, 
and strings cannot be involved in reference cycles, as there are no user 
settable references. Tuples can be:


a = []
b = (a,)
a[0] = b

A tuple of, for instance, ints would not need to be tracked, but if it 
is, someone probably decided that the cost of checking is not worth the 
benefit. Details are subject to change with versions. In 3.4, gc can 
break cycles of objects with delete methods, whereas they previously 
waiting until program shutdown. Some year ago, I remember a suggestion 
to turn off gc while creating lots of tuples that did not need to be 
tracked, but I do not know if that is still useful.


--
Terry Jan Reedy


--
https://mail.python.org/mailman/listinfo/python-list

Re: Deep vs. shallow copy?

[Rustom Mody]

On Thursday, March 13, 2014 4:37:53 AM UTC+5:30, Steven D'Aprano wrote:
> On Wed, 12 Mar 2014 15:29:59 +, Alex van der Spek wrote:

> > Having been taught programming in Algol60 Python still defeats me at
> > times! Particularly since Algol60 wasn't very long lived and what came
> > thereafter (FORTRAN) much worse.


> I'm not really familiar with Algol, but I do know Pascal, and you should 
> think of the append method to be like a Pascal procedure. Because Python 
> doesn't have true procedures, it follows the convention that returning 
> the special object None signals the intention to return nothing at all. 
> Hence your example below:

Yes... Algol I dont know. But to the extent that it corresponds to
Pascal the following may help.

Pascal is remarkably clean and cleaner than what came before -- Lisp
-- and after -- C, all its derivatives, python... almost everything.

First off there are expressions and statements, and no messing with
the distinction.  C started laissez-faire. No void, just default
functions intended to be used as procedures to int.  This was
problematic to enough people that void was introduced soon enough.

But its still messy:
An assignment like x = 3 is both an expression and (like) a statement.
It is mostly used as a statement by appending a semicolon.
It can also be used as an expression as in y = x = 3.

Try explaining this to a first year class and you'd know what a mess it is.
For the most part if you think youve got it you probably havent.

Python is in some respects worse than C, in some better.

Its better in that assignment statements are not re-purposeable as
expressions. Its worse in that procedures are simulated by None
returning functions -- this means syntax errors which C would give
when using a void function as an expression, you wont get in python.

On the whole, whether your language supports it or not, its best to
think of the world as separated into actions and values.

Call the action-world the 'imperative' world.
Call the value-world the 'functional' world ('declarative' would be
better but 'functional' is too entrenched).

[Following table meant to be read with fixed (courier) font]

| | Imperative | Functional  |
| Language entity | Statement  | Expression  |
| Denote (and think with) | Action | Value   |
| Abstracted into | Procedure  | Function|
| Atoms are   | Assignment | Constant/Variable   |
| Data Structures | Mutable| Immutable   |
| Loop primitive  | Recursion  | Iteration   |
| World is| In time| Timeless (Platonic) |

Generally mixing these two worlds is necessary for any non-trivial
programming but is commonly problematic.

Your error was to use a functional form -- list comprehension -- and
embed an imperative construct -- row.append -- into that.  This is
always a gaffe

Legal mixing is done thus
Expression -> Statement by putting the expression on the rhs of an assignment
Statement -> Expression by putting the statement(s) into a function

While most programmers think this unproblematic, Backus' widely-cited
Turing Award lecture is devoted to showing why this is a problem
http://www.thocp.net/biographies/papers/backus_turingaward_lecture.pdf

For an old Algol/Fortran programmer it may be a worthwhile read considering
Backus invented Fortran :-)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: SMTPHandler and mail subject

[Chris Withers]

If you want a much more fully-featured mail handler for the standard 
logging framework, there's this:


https://pypi.python.org/pypi/mailinglogger/3.8.0

cheers,

Chris

On 12/03/2014 12:25, eras.rasmu...@gmail.com wrote:


It works. Thank you :)


Eras



--
Simplistix - Content Management, Batch Processing & Python Consulting
   - http://www.simplistix.co.uk
--
https://mail.python.org/mailman/listinfo/python-list