Re: Efficient way to break up a list into two pieces

2010-02-21 Thread Jonathan Gardner
On Sat, Feb 20, 2010 at 10:48 PM, Steven D'Aprano wrote: > On Sat, 20 Feb 2010 21:21:47 -0800, Jonathan Gardner wrote: >> For ten items, though, is it really faster to muck around with array >> lengths than just copying the data over? Array copies are extremely fast >> on modern processors. > > My

Re: Efficient way to break up a list into two pieces

2010-02-21 Thread Steven D'Aprano
On Sun, 21 Feb 2010 11:07:24 -0800, marwie wrote: >> x = SomeReallyBigListOrString >> for item in x[1:]: >>     process(item) >> >> has to copy the entire list or string (less the first item). But >> honestly, I've never found a situation where it actually mattered. > > Good grief, it copies that

Re: Efficient way to break up a list into two pieces

2010-02-21 Thread Steve Howell
On Feb 20, 5:55 pm, marwie wrote: > On 21 Feb., 02:30, Steven D'Aprano > cybersource.com.au> wrote: > > Python lists are arrays of pointers to objects, so copying a slice is > > fast: it doesn't have to copy the objects, just pointers. Deleting from > > the end of the list is also quick, because

Re: Efficient way to break up a list into two pieces

2010-02-21 Thread MRAB
MRAB wrote: Steven D'Aprano wrote: [snip] I'm sympathetic to your concern: I've often felt offended that doing something like this: x = SomeReallyBigListOrString for item in x[1:]: process(item) has to copy the entire list or string (less the first item). But honestly, I've never found a

Re: Efficient way to break up a list into two pieces

2010-02-21 Thread marwie
On 21 Feb., 07:38, Carl Banks wrote: > Numpy arrays can share underlying data like that when you take > slices.  For instance, this probably works the way you want: > > a = numpy.array([1,2,3,4,5,6]) > b = a[:3] > c = a[3:] > > None of the actual data was copied here. Hmm, that might be worth loo

Re: Efficient way to break up a list into two pieces

2010-02-21 Thread marwie
On 21 Feb., 04:40, Steven D'Aprano wrote: > > Additionally, Python lists are over-allocated so that appends are fast. A > list of (say) 1000 items might be over-allocated to (say) 1024 items, so > that you can do 24 appends before the array is full and the array needs > to be resized. This means t

Re: Efficient way to break up a list into two pieces

2010-02-20 Thread Steven D'Aprano
On Sat, 20 Feb 2010 21:21:47 -0800, Jonathan Gardner wrote: > On Sat, Feb 20, 2010 at 5:41 PM, Steven D'Aprano > wrote: >> >> What the OP wants is: >> >> (1) assign the name l2 to l1[:10] without copying (2) resize l1 in >> place to the first 10 items without affecting l2. >> >> > For ten items,

Re: Efficient way to break up a list into two pieces

2010-02-20 Thread Carl Banks
On Feb 20, 4:55 pm, marwie wrote: > Hello, > > I recently read about augmented assignments and that (with l1, l2 > being lists) > >     l1.extend(l2) > > is more efficient than > >     l1 = l1 + l2 > > because unnecessary copy operations can be avoided. Now my question is > if there's a similar th

Re: Efficient way to break up a list into two pieces

2010-02-20 Thread Jonathan Gardner
On Sat, Feb 20, 2010 at 5:41 PM, Steven D'Aprano wrote: > > What the OP wants is: > > (1) assign the name l2 to l1[:10] without copying > (2) resize l1 in place to the first 10 items without affecting l2. > For ten items, though, is it really faster to muck around with array lengths than just cop

Re: Efficient way to break up a list into two pieces

2010-02-20 Thread MRAB
Steven D'Aprano wrote: [snip] I'm sympathetic to your concern: I've often felt offended that doing something like this: x = SomeReallyBigListOrString for item in x[1:]: process(item) has to copy the entire list or string (less the first item). But honestly, I've never found a situation wh

Re: Efficient way to break up a list into two pieces

2010-02-20 Thread Daniel Stutzbach
On Sat, Feb 20, 2010 at 6:55 PM, marwie wrote: > Now my question is > if there's a similar thing for breaking a list into two parts. Let's > say I want to remove from l1 everything from and including position 10 > and store it in l2. Then I can write > >l2 = l1[10:] >del l1[10:] With Py

Re: Efficient way to break up a list into two pieces

2010-02-20 Thread Steven D'Aprano
On Sat, 20 Feb 2010 17:55:18 -0800, marwie wrote: > On 21 Feb., 02:30, Steven D'Aprano cybersource.com.au> wrote: >> Python lists are arrays of pointers to objects, so copying a slice is >> fast: it doesn't have to copy the objects, just pointers. Deleting from >> the end of the list is also quic

Re: Efficient way to break up a list into two pieces

2010-02-20 Thread marwie
On 21 Feb., 02:41, Steven D'Aprano wrote: > What the OP is doing is quite different: > > (1) copy l1[:10] > (2) assign the name l2 to it > (3) resize l1 in place to the first 10 items. > > What the OP wants is: > > (1) assign the name l2 to l1[:10] without copying > (2) resize l1 in place to the f

Re: Efficient way to break up a list into two pieces

2010-02-20 Thread marwie
On 21 Feb., 02:30, Steven D'Aprano wrote: > Python lists are arrays of pointers to objects, so copying a slice is > fast: it doesn't have to copy the objects, just pointers. Deleting from > the end of the list is also quick, because you don't have to move memory, > just clear some pointers and cha

Re: Efficient way to break up a list into two pieces

2010-02-20 Thread Steven D'Aprano
On Sat, 20 Feb 2010 17:06:36 -0800, Jonathan Gardner wrote: > On Sat, Feb 20, 2010 at 4:55 PM, marwie wrote: [...] >>    l2 = l1[10:] >>    del l1[10:] >> >> But since I'm assigning a slice the elements will be copied. Basically, >> I'm looking for something like l1.pop(10,len(l1)) which returns

Re: Efficient way to break up a list into two pieces

2010-02-20 Thread Steven D'Aprano
On Sat, 20 Feb 2010 16:55:10 -0800, marwie wrote: > Hello, > > I recently read about augmented assignments and that (with l1, l2 being > lists) > > l1.extend(l2) > > is more efficient than > > l1 = l1 + l2 > > because unnecessary copy operations can be avoided. Now my question is > if

Re: Efficient way to break up a list into two pieces

2010-02-20 Thread Shashwat Anand
You can always implement your own data-structures or simply a function if you need so. A language consist of building blocks and not buildings. On Sun, Feb 21, 2010 at 6:36 AM, Jonathan Gardner < jgard...@jonathangardner.net> wrote: > On Sat, Feb 20, 2010 at 4:55 PM, marwie wrote: > > Hello, > >

Re: Efficient way to break up a list into two pieces

2010-02-20 Thread Jonathan Gardner
On Sat, Feb 20, 2010 at 4:55 PM, marwie wrote: > Hello, > > I recently read about augmented assignments and that (with l1, l2 > being lists) > >    l1.extend(l2) > > is more efficient than > >    l1 = l1 + l2 > > because unnecessary copy operations can be avoided. Now my question is > if there's a