Re: Python 3.4.1 installer on Mac links Python to old Tcl/Tk

[Christian Gollwitzer]

Am 03.10.14 00:08, schrieb Ned Deily:

So, to really support
Tk 8.6, the only viable option at the moment would be for us to ship our
own versions of Tk, like the Windows installer does.  But it wouldn't be
acceptable, IMO, to force other projects and users to migrate to 8.6 in
the middle of a release cycle, e.g. 3.4.x or 2.7.x.  Providing it as an
option, though, would be OK.


Hmm, I'm not sure that the other projects would need an update. In Tcl 
world, there is the concept of stub libraries, an extra level of 
indirect linking provided by both the Tcl and Tk core. If the extensions 
link to the stub library (the standard way), they can be loaded into any 
later version of Tcl and do not directly depend on libtcl or libtk. If 
the extensions link directly to libtcl, they are either broken or very 
special (providing one's own interpreter, doing nasty stuff with 
internals etc.)


Christian
--
https://mail.python.org/mailman/listinfo/python-list

Re: Is there a way to display source code for Python function?

[Peter Otten]

Viet Nguyen wrote:

> On Thursday, October 2, 2014 10:34:15 PM UTC-7, Viet Nguyen wrote:
>> Hi,
>> 
>> 
>> 
>> When I am debug mode, is there some command which will help display the
>> source code for a Python function of interest?  Much like you'd use "info
>> proc" to display contents of Tcl proc.
>> 
>> 
>> 
>> Thanks,
>> 
>> Viet
> 
> I tried this:
 def func(a):
> ...   a = 'abcd'
> 
> 
 inspect.getsource(func)
> Traceback (most recent call last):
>   File "", line 1, in 
>   File "/sw/packages/python/current/lib/python2.7/inspect.py", line 701,
>   in getsource
> lines, lnum = getsourcelines(object)
>   File "/sw/packages/python/current/lib/python2.7/inspect.py", line 690,
>   in getsourcelines
> lines, lnum = findsource(object)
>   File "/sw/packages/python/current/lib/python2.7/inspect.py", line 538,
>   in findsource
> raise IOError('could not get source code')
> IOError: could not get source code
> 
 inspect.getsourcelines(func)
> Traceback (most recent call last):
>   File "", line 1, in 
>   File "/sw/packages/python/current/lib/python2.7/inspect.py", line 690,
>   in getsourcelines
> lines, lnum = findsource(object)
>   File "/sw/packages/python/current/lib/python2.7/inspect.py", line 538,
>   in findsource
> raise IOError('could not get source code')
> IOError: could not get source code
> 
> What is wrong?

The source of func is compiled and immediately discarded by the interactive 
interpreter. inspect.getsource() only works if the source code is available 
(as a module):

$ cat ham.py
def spam():
return 42
$ python3
Python 3.4.0 (default, Apr 11 2014, 13:05:11) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ham, inspect
>>> print(inspect.getsource(ham.spam))
def spam():
return 42

>>> 

There is an alternative interactive interpreter called ipython that allows 
you to retrieve a function definition:

In [1]: def foo():
   ...: return bar
   ...: 

In [2]: %psource foo
def foo():
return bar

In [3]: 



-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to show a dictionary sorted on a value within its data?

[Steven D'Aprano]

On Fri, 03 Oct 2014 12:44:32 +1000, Chris Angelico wrote:

> On Fri, Oct 3, 2014 at 9:16 AM, Steven D'Aprano
>  wrote:
>>> Anyway, pylint doesn't complain about a bare use of lambda, but it
>>> does complain about a map applied to a lambda or a filter applied to a
>>> lambda.  Pylint says they could be replaced by a list comprehension,
>>> with the warning "deprecated-lambda".
>>
>> The warning name is misleading, lambda is not deprecated. But
>> stylistically, many people prefer to use a generator expression in
>> place of map or filter:
>>
>> # instead of this
>> map(lambda x: 2*x+1, values)
>>
>> # use this
>> (2*x+1 for x in values)
>>
>> # instead of this
>> filter(lambda x: x > 23, values)
>>
>> # use this
>> (x for x in values if x > 23)
> 
> Don't forget that your equivalencies are based on the Py3 map/filter,

Dan did ask about Python 3.x :-)


> which return non-list iterables. There's no really convenient equivalent
> to the map() usage of "call this function with each of these args, and
> discard the return values", as it looks very odd to do a list comp for
> nothing:

map doesn't discard the return values. It accumulates them in a list, or 
yields them in a list comp.

> [print(x) for x in list_of_strings]

That would be an abuse of a list comprehension, generator expression or 
map since it will produce a whole lot of None values. If you really want 
to discard the return values, put them in a for-loop:

for s in list_of_strings:
print(s)

> Short of borrowing Pike's automap syntax:
> 
> print(list_of_strings[*])


you mean something like this perhaps?

print(*list_of_strings, sep='\n')


> there's really not much that's better than the original map() call. On
> the other hand, there aren't actually all that many times when I've
> needed to do this, and the simple for loop generally suffices:
> 
> for x in list_of_strings: print(x)

Exactly.


-- 
Steven
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to show a dictionary sorted on a value within its data?

[Chris Angelico]

On Fri, Oct 3, 2014 at 5:55 PM, Steven D'Aprano  wrote:
>> which return non-list iterables. There's no really convenient equivalent
>> to the map() usage of "call this function with each of these args, and
>> discard the return values", as it looks very odd to do a list comp for
>> nothing:
>
> map doesn't discard the return values. It accumulates them in a list, or
> yields them in a list comp.

Sure, but I've seen it used in various places to call functions where
you don't care about the return values. Technically it's identical to
the list comp, but somehow it's seen as less of an abuse.

>> Short of borrowing Pike's automap syntax:
>>
>> print(list_of_strings[*])
>
> you mean something like this perhaps?
>
> print(*list_of_strings, sep='\n')

Well, no. In that one specific example, it happens to be possible to
pass all the args to one print() call and get the same effect, but
imagine some function that doesn't have that particular feature. (I
just used print for simplicity.)

>> there's really not much that's better than the original map() call. On
>> the other hand, there aren't actually all that many times when I've
>> needed to do this, and the simple for loop generally suffices:
>>
>> for x in list_of_strings: print(x)
>
> Exactly.

Yeah. That's the normal way to do things. I'm not entirely sure why I
keep seeing map() for that; there's probably a justification in
functional programming languages, or something.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Is there a way to display source code for Python function?

[Viet Nguyen]

On Friday, October 3, 2014 12:48:08 AM UTC-7, Peter Otten wrote:
> Viet Nguyen wrote:
> 
> 
> 
> > On Thursday, October 2, 2014 10:34:15 PM UTC-7, Viet Nguyen wrote:
> 
> >> Hi,
> 
> >> 
> 
> >> 
> 
> >> 
> 
> >> When I am debug mode, is there some command which will help display the
> 
> >> source code for a Python function of interest?  Much like you'd use "info
> 
> >> proc" to display contents of Tcl proc.
> 
> >> 
> 
> >> 
> 
> >> 
> 
> >> Thanks,
> 
> >> 
> 
> >> Viet
> 
> > 
> 
> > I tried this:
> 
>  def func(a):
> 
> > ...   a = 'abcd'
> 
> > 
> 
> > 
> 
>  inspect.getsource(func)
> 
> > Traceback (most recent call last):
> 
> >   File "", line 1, in 
> 
> >   File "/sw/packages/python/current/lib/python2.7/inspect.py", line 701,
> 
> >   in getsource
> 
> > lines, lnum = getsourcelines(object)
> 
> >   File "/sw/packages/python/current/lib/python2.7/inspect.py", line 690,
> 
> >   in getsourcelines
> 
> > lines, lnum = findsource(object)
> 
> >   File "/sw/packages/python/current/lib/python2.7/inspect.py", line 538,
> 
> >   in findsource
> 
> > raise IOError('could not get source code')
> 
> > IOError: could not get source code
> 
> > 
> 
>  inspect.getsourcelines(func)
> 
> > Traceback (most recent call last):
> 
> >   File "", line 1, in 
> 
> >   File "/sw/packages/python/current/lib/python2.7/inspect.py", line 690,
> 
> >   in getsourcelines
> 
> > lines, lnum = findsource(object)
> 
> >   File "/sw/packages/python/current/lib/python2.7/inspect.py", line 538,
> 
> >   in findsource
> 
> > raise IOError('could not get source code')
> 
> > IOError: could not get source code
> 
> > 
> 
> > What is wrong?
> 
> 
> 
> The source of func is compiled and immediately discarded by the interactive 
> 
> interpreter. inspect.getsource() only works if the source code is available 
> 
> (as a module):
> 
> 
> 
> $ cat ham.py
> 
> def spam():
> 
> return 42
> 
> $ python3
> 
> Python 3.4.0 (default, Apr 11 2014, 13:05:11) 
> 
> [GCC 4.8.2] on linux
> 
> Type "help", "copyright", "credits" or "license" for more information.
> 
> >>> import ham, inspect
> 
> >>> print(inspect.getsource(ham.spam))
> 
> def spam():
> 
> return 42
> 
> 
> 
> >>> 
> 
> 
> 
> There is an alternative interactive interpreter called ipython that allows 
> 
> you to retrieve a function definition:
> 
> 
> 
> In [1]: def foo():
> 
>...: return bar
> 
>...: 
> 
> 
> 
> In [2]: %psource foo
> 
> def foo():
> 
> return bar
> 
> 
> 
> In [3]:

Thanks Peter!
-- 
https://mail.python.org/mailman/listinfo/python-list

Returning a List

[Shiva]

Hi All,

I might be doing something really silly here, but I can't seem to spot it:

def front_x(words):
  b=[]
  c=[]
  for a in words:
 if a[0] == 'x':
 b.append(a)
 else:
 c.append(a)

  b = sorted(b)
  c = sorted(c)
  d= b+c
  print('d = ',d)

  #return b+c
  return d

front_x(['bbb', 'ccc', 'axx', 'xzz', 'xaa'])



Why is return d or return b+c not returning anything??

The d's value is confirmed by the print statement.

Thanks,
Shiva.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Returning a List

[Chris Angelico]

On Fri, Oct 3, 2014 at 9:35 PM, Shiva
 wrote:
> Why is return d or return b+c not returning anything??
>

On what basis do you believe it's not returning anything? When you
call it, down below, you're ignoring its return value. I expect it's
returning just fine, but you then do nothing with it.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Returning a List

[Shiva]

Chris Angelico  gmail.com> writes:

> 
> On Fri, Oct 3, 2014 at 9:35 PM, Shiva
>  yahoo.com.dmarc.invalid> wrote:
> > Why is return d or return b+c not returning anything??
> >
> 
> On what basis do you believe it's not returning anything? When you
> call it, down below, you're ignoring its return value. I expect it's
> returning just fine, but you then do nothing with it.
> 
> ChrisA
> 


I wrongly thought calling the function was enough. Looks like the call is
just a placeholder for the returned value. Need to print it out.

Thanks!!

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Is there a way to display source code for Python function?

[Mark Lawrence]

On 03/10/2014 09:16, Viet Nguyen wrote:

[snip normal google stuff]

Would you please access this list via 
https://mail.python.org/mailman/listinfo/python-list or read and action 
this https://wiki.python.org/moin/GoogleGroupsPython to prevent us 
seeing double line spacing and single line paragraphs, thanks.


--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

--
https://mail.python.org/mailman/listinfo/python-list

Re: Recommended hosting

[marco . nawijn]

On Thursday, October 2, 2014 3:30:38 PM UTC+2, writeson wrote:
> Hi all,
> 
> 
> 
> I'd like to build a web site for myself, essentially a "vanity" web site to 
> show off whatever web development skills I have, and perhaps do some 
> blogging. I'm a Python developer, so I'd like to develop the site with the 
> following stack:
> 
> 
> 
> web applications written with Python and Flask, running as uwsgi 
> applications. These would support dynamic HTML where needed, but mostly it 
> would provide REST API's.
> 
> 
> 
> static content delivered by Nginx
> 
> 
> 
> Can anyone give me some recommendations for a good hosting company that would 
> allow me work with the above tool set? I'm US based if that makes a 
> difference.
> 
> 
> 
> Thanks in advance!
> 
> Doug

Doug,

Also don't forget "the new kid on the block" Docker! So, you can build your 
webapp locally on your machine, deploy it (also locally) in a Docker 
container. Test it and then deploy it in the cloud. You have lots of 
options for this.

I am the process of doing this myself and I like it a lot! 

Marco



-- 
https://mail.python.org/mailman/listinfo/python-list

Egg help wanted

[cl]

I have installed (using easy_install) a little utility, it seems to
have installed just the top-level python script in /usr/local/bin
which in turn uses load_entry_point() to run the actual code.

As far as I can see the code and support files remain in the .egg file
in /usr/local/lib/python2.7/dist-packages, is this right?  I.e. is the
egg file unzipped 'on the fly' when the program is called?

Reading the documentation for pkg_resources it sounds like it will
also work if I unpack the .egg file into a directory of the same name
also put in /usr/local/lib/python2.7/dist-packages.  Have I understood
this correctly?  I want to do this to make it easy to do trivial edits
to the program rather than having to move it all back into a proper
development environment.

-- 
Chris Green
·
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Recommended hosting

[Miki Tebeka]

Greetings,

> I'd like to build a web site for myself, essentially a "vanity" web site to 
> show off whatever web development skills I have, and perhaps do some 
> blogging. I'm a Python developer, so I'd like to develop the site with the 
> following stack:
> web applications written with Python and Flask, running as uwsgi 
> applications. These would support dynamic HTML where needed, but mostly it 
> would provide REST API's.
IIRC you can use Flask with AppEngine (static content will be handled by 
AppEngine and not nginx).

It's free for small-ish sites and removes a lot of operations headache.

All the best,
--
Miki
-- 
https://mail.python.org/mailman/listinfo/python-list

Python Basics

[diarmuid . higgins]

Hi

I have just started an introductory course on Python. I have never even seen a 
programming language before so I am struggling a bit. Our lecturer has given us 
a number of excercises to complete and I am stuck on the one I have listed 
below. Please help 

Q2. Implement a function called printNumTriangle. The function should ask the 
user to enter a single integer. It should then print a triangle of that size 
specified by the integer so that each row in the triangle is made up of the 
integer displayed. 

The following is a sample output

Please enter an integer for triangle size:
1
22
333

5
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python Basics

[Chris Angelico]

On Sat, Oct 4, 2014 at 3:35 AM,   wrote:
> Our lecturer has given us a number of excercises to complete and I am stuck 
> on the one I have listed below. Please help

Sure! Show us the code you have so far, and explain what it is you're
stuck on. Then we can help you. But we're not going to write the code
for you.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python Basics

[diarmuid . higgins]

def printNumTriangle():
   num = input("Please enter an integer for triangle size:")
   for i in range(1,num+1):
  for j in range (i):
 print i

Cheers


On Friday, October 3, 2014 6:42:46 PM UTC+1, Chris Angelico wrote:
> On Sat, Oct 4, 2014 at 3:35 AM,   wrote:
> 
> > Our lecturer has given us a number of excercises to complete and I am stuck 
> > on the one I have listed below. Please help
> 
> 
> 
> Sure! Show us the code you have so far, and explain what it is you're
> 
> stuck on. Then we can help you. But we're not going to write the code
> 
> for you.
> 
> 
> 
> ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python Basics

[Chris Angelico]

On Sat, Oct 4, 2014 at 3:44 AM,   wrote:
> def printNumTriangle():
>num = input("Please enter an integer for triangle size:")
>for i in range(1,num+1):
>   for j in range (i):
>  print i
>
> Cheers

Okay! You're very close, and I can see what's going on there. But I'd
like you to learn how to ask questions, as well as how to write Python
code, because both skills are important to your future. :) So I'm
going to get you to take another step in asking the question: What is
it that your code isn't doing right?

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python Basics

[diarmuid . higgins]

On Friday, October 3, 2014 6:47:54 PM UTC+1, Chris Angelico wrote:
> On Sat, Oct 4, 2014 at 3:44 AM,   wrote:
> 
> > def printNumTriangle():
> 
> >num = input("Please enter an integer for triangle size:")
> 
> >for i in range(1,num+1):
> 
> >   for j in range (i):
> 
> >  print i
> 
> >
> 
> > Cheers
> 
> 
> 
> Okay! You're very close, and I can see what's going on there. But I'd
> 
> like you to learn how to ask questions, as well as how to write Python
> 
> code, because both skills are important to your future. :) So I'm
> 
> going to get you to take another step in asking the question: What is
> 
> it that your code isn't doing right?
> 
> 
> 
> ChrisA

Hi Chris
I can't get the code to display the output as it should.
I can get it to display like this:
1223335 or I can get it to display like this:
1
2
2
3
3
3
4
4
4
4
5
5
5
5
5
but not as has been asked in the question.
Cheers
Diarmuid
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python Basics

[Chris Angelico]

On Sat, Oct 4, 2014 at 3:54 AM,   wrote:
> Hi Chris
> I can't get the code to display the output as it should.
> I can get it to display like this:
> 1223335 or I can get it to display like this:
> 1
> 2
> 2
> 3
> 3
> 3
> 4
> 4
> 4
> 4
> 5
> 5
> 5
> 5
> 5
> but not as has been asked in the question.

Good! (It would help if you explained how you had the two different
versions, incidentally.) What you need is like the first version, only
you want to break the lines every time the digit changes. That can be
done pretty easily. All you need to do is print out nothing, followed
by end-of-line; and you need to do this in such a way that it happens
after all of one digit has been displayed, but before the next one is.
Can you do that?

Side point: Google Groups makes a horrible mess of your posts. Please
consider switching to the mailing list, or to a better newsreader. You
can join the mailing list here:

https://mail.python.org/mailman/listinfo/python-list

Thanks!

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python Basics

[Tobiah]

Hi Chris
I can't get the code to display the output as it should.
I can get it to display like this:
1223335 or I can get it to display like this:
1
2
2
3
3
3
4
4
4
4
5
5
5
5
5
but not as has been asked in the question.
Cheers
Diarmuid




Hint:


'a' * 4

''


--
https://mail.python.org/mailman/listinfo/python-list

"High water" Memory fragmentation still a thing?

[Croepha]

Question:

Does python in general terms (apart from extensions or gc manipulation),
exhibit a "high water" type leak of allocated memory in recent python
versions (2.7+)?

Background:


>From the post:
http://chase-seibert.github.io/blog/2013/08/03/diagnosing-memory-leaks-python.html

Begin quote:

Long running Python jobs that consume a lot of memory while running may not
return that memory to the operating system until the process actually
terminates, even if everything is garbage collected properly. That was news
to me, but it's true. What this means is that processes that do need to use
a lot of memory will exhibit a "high water" behavior, where they remain
forever at the level of memory usage that they required at their peak.

Note: this behavior may be Linux specific; there are anecdotal reports that
Python on Windows does not have this problem.

This problem arises from the fact that the Python VM does its own internal
memory management. It's commonly know as memory fragmentation.
Unfortunately, there doesn't seem to be any fool-proof method of avoiding
it.

End Quote

However this paper seems to indicate that that is not a modern problem:
http://www.evanjones.ca/memoryallocator/


--Thanks
Dave Butler
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Returning a List

[Terry Reedy]

On 10/3/2014 7:35 AM, Shiva wrote:

Hi All,

I might be doing something really silly here, but I can't seem to spot it:

def front_x(words):
   b=[]
   c=[]
   for a in words:
  if a[0] == 'x':
  b.append(a)
  else:
  c.append(a)

   b = sorted(b)
   c = sorted(c)


Just sort in place with
 b.sort()
 c.sort()


   d= b+c
   print('d = ',d)

   #return b+c
   return d

front_x(['bbb', 'ccc', 'axx', 'xzz', 'xaa'])


xlist = front_x(['bbb', 'ccc', 'axx', 'xzz', 'xaa'])
print(xlist)

--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list

Re: "High water" Memory fragmentation still a thing?

[Antoine Pitrou]

Hello,

Croepha  gmail.com> writes:
> 
> Question:
> 
> Does python in general terms (apart from extensions or gc manipulation),
exhibit a "high water" type leak of allocated memory in recent python
versions (2.7+)?

It is not a leak. It is a quite common pattern of memory fragmentation.
The article is wrong in assuming that Python doesn't return the memory to
the OS. Python does return its empty memory pools to the OS, however the OS
itself may not be able to release that memory, because of heap fragmentation.

As the article mentions, this was improved (mostly fixed?) in 3.3.

Regards

Antoine.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: "High water" Memory fragmentation still a thing?

[Christian Heimes]

On 03.10.2014 21:16, Antoine Pitrou wrote:
> It is not a leak. It is a quite common pattern of memory fragmentation.
> The article is wrong in assuming that Python doesn't return the memory to
> the OS. Python does return its empty memory pools to the OS, however the OS
> itself may not be able to release that memory, because of heap fragmentation.

The article doesn't state if the writer is referring to virtual memory
or resident set size. For long-running 32bit processes it is quite
common to run out of virtual address space. But that's something totally
different than running out of memory. A 64bit process can have a virtual
address size of several GB but only occupy a few hundred MB of physical
memory. People tend to confuse the meaning of VSZ and RSS and forget
about paging.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 3.4.1 installer on Mac links Python to old Tcl/Tk

[Ned Deily]

In article ,
 Christian Gollwitzer  wrote:
> Hmm, I'm not sure that the other projects would need an update. In Tcl 
> world, there is the concept of stub libraries, an extra level of 
> indirect linking provided by both the Tcl and Tk core. If the extensions 
> link to the stub library (the standard way), they can be loaded into any 
> later version of Tcl and do not directly depend on libtcl or libtk. If 
> the extensions link directly to libtcl, they are either broken or very 
> special (providing one's own interpreter, doing nasty stuff with 
> internals etc.)

Even if there were no incompatibilities, on OS X with Tcl and Tk (and 
other) frameworks, the version number is embedded in the path to the 
shared library and the linker normally creates an absolute path at that.

$ otool -L _tkagg.so
_tkagg.so:
   /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current 
version 60.0.0)
   /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current 
version 1197.1.1)
   /Library/Frameworks/Tcl.framework/Versions/8.5/Tcl (compatibility 
version 8.5.0, current version 8.5.15)
   /Library/Frameworks/Tk.framework/Versions/8.5/Tk (compatibility 
version 8.5.0, current version 8.5.15)

-- 
 Ned Deily,
 n...@acm.org

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: "High water" Memory fragmentation still a thing?

[Skip Montanaro]

On Fri, Oct 3, 2014 at 1:36 PM, Croepha 
wrote:

> Long running Python jobs that consume a lot of memory while
> running may not return that memory to the operating system
> until the process actually terminates, even if everything is
> garbage collected properly.

(I see Antoine replied that this is mostly fixed starting in
3.3.  My response below was written with my Unix/Python 2
rose-colored glasses firmly affixed to my face.)

Unless you are blessed with a long-running program which does
everything exactly right, I'd be surprised if any such program
behaved the way you want. Python may be extreme in this regard,
since it often allcates small objects, and its special object
allocator (obmalloc - is it still used?) might mitigate the
problem by collecting allocations of similar size together
(malloc implementations probably do some of this as well), but
it's still going to have these issues.

The problem boils down to how the program dynamically allocates
and frees memory, and how the malloc subsystem interacts with
the kernel through the brk and sbrk system calls.  (Anywhere I
mention "brk", you can mentally replace it with "sbrk". They do
the same thing - ask for memory from or return memory to the
kernel - using a different set of units, memory addresses or
bytes.)  In the general case, programmers call malloc (or
calloc, or realloc) to allocate a chunk of storage from the
heap.  (I'm ignoring anything special which Python has layered
on top of malloc.  It can mitigate problems, but I don't think
it will fundamentally change the way malloc interacts with the
kernel.)  The malloc subsystem maintains a free list (recently
freed bits of memory) from which it can allocate memory without
traipsing off to the kernel.  If it can't return a chunk of
memory from the free list, it will (in the most simpleminded
malloc implementation) call brk to grab a new (large) chunk of
memory.  The system simply moves the end of the program's
"break", effectively increasing or decreasing the (virtual) size
of the running program.  That memory is then doled out to the
user by malloc.  If, and only if, every chunk of memory in the
last chunk allocated by a call to brk is placed on malloc's free
list, *and* if the particular malloc implementation on your box
is smart enough to coalesce adjacent chunks of freed memory back
into brk-sized memory chunks, can brk be called once again to
reduce the program's footprint.

Example...  I don't know how much memory malloc requests from
brk, so let's make things easy, and make the following
assumptions:

* Assume malloc calls brk to allocate 1MB chunks.

* Assume the program only ever calls malloc(1024).

* Assume malloc's own overhead (free list, etc) is zero.

So, starting afresh, having no free memory, the first time we
call malloc(1024), it calls brk asking for 1MB, then returns its
caller a pointer to that 1024-byte chunk. Now call malloc(1024)
1023 more times.  We have used up that entire 1MB chunk of
memory.  Now free each of those chunks by calling free() 1024
times. We are left, once again, with a 1MB chunk of free memory.
It might be stitched together by malloc into one single block of
memory, or it might appear on the free list as 1024 chunks of
memory.  Should malloc return it to the system with a call to
brk?  Maybe.  Maybe not.  Maybe the malloc subsystem goes
through all the necessary bookkeeping to hand that memory back
to the system, only to find that the next thing the program does
is make another malloc(1024) call.  Whoops. Now you have to call
brk again.  And system calls are expensive.

Now, consider a similar case. You make 1024 calls to
malloc(1024). Then you free all of them except the 512th
one. Now you have a 1MB chunk of memory on malloc's free list
which is entirely free, except for a small chunk in the middle.
That chunk is broken into three fragments, two free fragments,
separated by one chunk which is still in use.  Can't free that.
Well, perhaps you could, but only the top half of it. And you'd
have the same dilemma as before. Should you return it or not?

Final consideration.  Suppose your program makes 1023 calls to
malloc(1024) and frees them all, but sometime during that work,
a low-level library far from the programmer's view also calls
malloc to get a 1KB chunk.  Even if your program (e.g., the
Python runtime) was perfectly well-behaved and returned all of
the 1023 chunks of memory it had requested, it has no control
over this low-level library.  You're still stuck with a
fragmented chunk of memory, free except for a hole somewhere in
the middle.

Long story short, there's only so much you can do to try to
return memory to the system.  I am not up-to-date on the latest
malloc intelligence, but it's a challenging enough problem that
it was a long time before any malloc implementation attempted to
automatically return memory pages to the system.

Finally, as this is not really Python-specific, you might want
to do some reading on how malloc is implemen

Re: Python 3.4.1 installer on Mac links Python to old Tcl/Tk

[Kevin Walzer]

On 10/3/14, 3:55 PM, Ned Deily wrote:

Even if there were no incompatibilities, on OS X with Tcl and Tk (and
other) frameworks, the version number is embedded in the path to the
shared library and the linker normally creates an absolute path at that.


Is this because Python lacks the concept of stubs?

A Tcl library compiled for 8.5 can be loaded into 8.6 with no 
re-compiling required because of stubs.

--
Kevin Walzer
Code by Kevin/Mobile Code by Kevin
http://www.codebykevin.com
http://www.wtmobilesoftware.com
--
https://mail.python.org/mailman/listinfo/python-list

Re: Recommended hosting

[David Hutto]

The best thing, and yes this is self promotion, is to find a "digital
dealer".
The server can be modularized locally, or across the server because you can
work your way up with others.

Just state the service(s) directly that you need, and an account can be
individually modified just for you.

On Fri, Oct 3, 2014 at 12:08 PM, Miki Tebeka  wrote:

> Greetings,
>
> > I'd like to build a web site for myself, essentially a "vanity" web site
> to show off whatever web development skills I have, and perhaps do some
> blogging. I'm a Python developer, so I'd like to develop the site with the
> following stack:
> > web applications written with Python and Flask, running as uwsgi
> applications. These would support dynamic HTML where needed, but mostly it
> would provide REST API's.
> IIRC you can use Flask with AppEngine (static content will be handled by
> AppEngine and not nginx).
>
> It's free for small-ish sites and removes a lot of operations headache.
>
> All the best,
> --
> Miki
> --
> https://mail.python.org/mailman/listinfo/python-list
>



-- 
Best Regards,
David Hutto
*CEO:*
*Hutto Industrial Technologies Inc.http://www.zenfinite.com
*
http://www.payizm.com -Coming Soon!
http://www.mylayawayplan.com  -Coming Soon!
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to set the global variable so that it can be accessed and released inside other methods

[Milson Munakami]

Hi Chris,
I want to remove that CreateNet() part from the test class so that it will not 
create network every time in setup() because all the test cases are run and 
tested in same network!

But the test class method also need to access the net object 

How can I do that?

Please Help.
Thanks,
Milson
On Thursday, October 2, 2014 9:30:21 AM UTC-6, Milson Munakami wrote:
> Hi,
> 
> 
> 
> I am newbie to Python,
> 
> 
> 
> I am trying to use unittest and python. My python script is like this:
> 
> 
> 
> #! /usr/bin/env python
> 
> __author__ = 'Milson Munakami'
> 
> __revision__ = '0.0.2'
> 
> 
> 
> import json
> 
> import urllib
> 
> import httplib
> 
> from scapy.all import *
> 
> 
> 
> import unittest
> 
> 
> 
> import os, sys, socket, struct, select, time 
> 
> from threading import Thread
> 
> 
> 
> import logging
> 
> import traceback
> 
> 
> 
> from mininet.net import Mininet
> 
> from mininet.node import OVSSwitch, OVSKernelSwitch, Controller, 
> RemoteController
> 
> from mininet.log import setLogLevel, info
> 
> from mininet.cli import CLI
> 
> 
> 
> class testFirewallS1( unittest.TestCase ):
> 
> 
> 
>   #I am trying to set net variable to global
> 
>   global net
> 
>   ##
> 
> 
> 
>   def setUp(self):
> 
>   self.controllerIp="127.0.0.1"
> 
>   self.switch = "00:00:00:00:00:00:00:01"
> 
>   self.destinationIp = "10.0.0.1"
> 
>   self.startTime_ = time.time()
> 
>   self.failed = False
> 
>   self.reportStatus_ = True
> 
>   self.name_ = "Firewall"
> 
>   self.log = logging.getLogger("unittest")
> 
>   self.CreateNet()
> 
>   self.SetPrecondition()  
> 
> 
> 
>   def CreateNet(self):
> 
>   "Create an empty network and add nodes to it."
> 
>   net = Mininet( controller=RemoteController )
> 
> 
> 
>   info( '*** Adding controller\n' )
> 
>   net.addController( 'c0' , controller=RemoteController,ip= 
> "127.0.0.1", port=6633)
> 
> 
> 
>   info( '*** Adding hosts\n' )
> 
>   h1 = net.addHost( 'h1', ip='10.0.0.1' )
> 
>   h2 = net.addHost( 'h2', ip='10.0.0.2' )
> 
>   h3 = net.addHost( 'h3', ip='10.0.0.3' )
> 
> 
> 
>   info( '*** Adding switch\n' )
> 
>   s1 = net.addSwitch( 's1' )
> 
> 
> 
>   info( '*** Creating links\n' )
> 
>   net.addLink( h1, s1 )
> 
>   net.addLink( h2, s1 )
> 
>   net.addLink( h3, s1 )
> 
> 
> 
>   info( '*** Starting network\n')
> 
>   net.start()
> 
> 
> 
>   def tearDown(self):
> 
>   if self.failed:
> 
>   return
> 
>   duration = time.time() - self.startTime_
> 
>   self.cleanup(True)
> 
>   if self.reportStatus_:
> 
>   self.log.info("=== Test %s completed normally (%d 
> sec)", self.name_, duration)
> 
> 
> 
>   def cleanup(self, success):
> 
>   sys.excepthook = sys.__excepthook__
> 
>   self.SetFinalcondition()
> 
>   try:
> 
>   return
> 
>   except NameError:
> 
>   self.log.error("Exception hit during cleanup, 
> bypassing:\n%s\n\n" % traceback.format_exc())
> 
>   pass
> 
>   else:
> 
> 
> 
>   fail("Expected a NameError")
> 
>   
> 
>   def StatusFirewall(self):
> 
>   command = "http://%s:8080/wm/firewall/module/status/json"; % 
> self.controllerIp
> 
>   x = urllib.urlopen(command).read()
> 
>   parsedResult = json.loads(x)
> 
>   return parsedResult['result']
> 
>   
> 
>   def CountFirewallRules(self):
> 
>   command = "http://%s:8080/wm/firewall/rules/json"; % 
> self.controllerIp
> 
>   x = urllib.urlopen(command).read()
> 
>   return x
> 
>   
> 
>   def CountFlowRules(self):
> 
>   command = "http://%s:8080/wm/core/switch/%s/flow/json"; % 
> (self.controllerIp, self.switch)
> 
>   x = urllib.urlopen(command).read()
> 
>   parsedResult = json.loads(x)
> 
>   content = parsedResult['00:00:00:00:00:00:00:01']
> 
>   if content is None:
> 
>   return "[]"
> 
>   else:   
> 
>   return str(content)
> 
> 
> 
>   def SetPrecondition(self):
> 
>   command = "http://%s:8080/wm/firewall/module/enable/json"; % 
> self.controllerIp
> 
>   urllib.urlopen(command).read()
> 
>   
> 
>   # cleanup all Firewall rules
> 
>   command = "http://%s:8080/wm/firewall/rules/json"; % 
> self.controllerIp
> 
>   x = urllib.urlopen(command).read()
> 
>   

Re: How to set the global variable so that it can be accessed and released inside other methods

[Mark Lawrence]

On 03/10/2014 22:43, Milson Munakami wrote:

Hi Chris,
I want to remove that CreateNet() part from the test class so that it will not 
create network every time in setup() because all the test cases are run and 
tested in same network!

But the test class method also need to access the net object

How can I do that?

Please Help.
Thanks,
Milson


You are far more likely to get answers if you don't top post and if you 
don't use the obnoxious google groups.  I've snipped roughly 600 lines 
from your reply.  To avoid sending the rubbish that many of us here 
don't want to see would you please access this list via 
https://mail.python.org/mailman/listinfo/python-list or read and action 
this https://wiki.python.org/moin/GoogleGroupsPython to prevent us 
seeing double line spacing and single line paragraphs, thanks.


--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

--
https://mail.python.org/mailman/listinfo/python-list

Re: Python Basics

[Seymore4Head]

On Fri, 3 Oct 2014 10:35:38 -0700 (PDT), diarmuid.higg...@mycit.ie
wrote:

>Hi
>
>I have just started an introductory course on Python. I have never even seen a 
>programming language before so I am struggling a bit. Our lecturer has given 
>us a number of excercises to complete and I am stuck on the one I have listed 
>below. Please help 
>
>Q2. Implement a function called printNumTriangle. The function should ask the 
>user to enter a single integer. It should then print a triangle of that size 
>specified by the integer so that each row in the triangle is made up of the 
>integer displayed. 
>
>The following is a sample output
>
>Please enter an integer for triangle size:
>1
>22
>333
>
>5
for i in range(1,10):
print (str(i)*i)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python Basics

[Chris Angelico]

On Sat, Oct 4, 2014 at 8:54 AM, Seymore4Head
 wrote:
>>Q2. Implement a function called printNumTriangle. The function should ask the 
>>user to enter a single integer. It should then print a triangle of that size 
>>specified by the integer so that each row in the triangle is made up of the 
>>integer displayed.
>>
>>The following is a sample output
>>
>>Please enter an integer for triangle size:
>>1
>>22
>>333
>>
>>5
> for i in range(1,10):
> print (str(i)*i)

Seymour, please don't do this. When you "help" someone by just giving
him the answer to a homework problem, you get him past his immediate
issue of "I need to submit my homework for this problem". That lets
him get through his course without understanding the code he's
creating (because he's not the one creating it). He'll either run into
more trouble before graduating the course, or (far worse) he'll
graduate successfully, without having the competence that the course
is supposed to teach - and the world will be given a qualified
programmer who doesn't know his stuff. That damages the world, damages
the reputation of the course, and gives python-list a reputation as a
homework assignment solver... none of which I want. I'm pretty sure
you don't want it either.

So don't do people's homework for them. PLEASE!!

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re:Python Basics

[Dave Angel]

diarmuid.higg...@mycit.ie Wrote in message:
> Hi
> 
> I have just started an introductory course on Python. I have never even seen 
> a programming language before so I am struggling a bit. Our lecturer has 
> given us a number of excercises to complete and I am stuck on the one I have 
> listed below. Please help 
> 

Which part are you having trouble with? 

> Q2. Implement a function called printNumTriangle.


A function definition starts with the keyword 'def'



-- 
DaveA

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: "High water" Memory fragmentation still a thing?

[Chris Angelico]

On Sat, Oct 4, 2014 at 4:36 AM, Croepha  wrote:
> What this means is that processes that do need to use a lot of memory will
> exhibit a "high water" behavior, where they remain forever at the level of
> memory usage that they required at their peak.

This is almost never true. What you will see, though, is something
like Christian described; pages get allocated, and then partially
used, and you don't always get all that memory back. In theory, a high
level language like Python would be allowed to move objects around to
compact memory, but CPython doesn't do this, and there's no proof that
it'd really help anything anyway. (Look at Christian's comments about
"Should you return it or not?" and the cost of system calls... now
consider the orders-of-magnitude worse cost of actually moving memory
around.)

This is why a lot of long-duration processes are built to be restarted
periodically. It's not strictly necessary, but it can be the most
effective way of solving a problem. I tend to ignore that, though, and
let my processes just keep on running... for 88 wk 4d 23:56:27 so far,
on one of those processes. It's consuming less than half a gig of
virtual memory, quarter gig resident, and it's been doing a fair bit
(it keeps all sorts of things in memory to avoid re-decoding from
disk). So don't worry too much about memory usage until you see that
there's actually a problem; with most Python processes, you'll restart
them to deploy new code sooner than you'll restart them to fix memory
problems. (The above example isn't a Python process, and code updates
happen live.) In fact, I'd advise that as a general policy: Don't
panic about any Python limitation until you've proven that it's
actually a problem. :)

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: "High water" Memory fragmentation still a thing?

[Steven D'Aprano]

Chris Angelico wrote:

> In theory, a high
> level language like Python would be allowed to move objects around to
> compact memory, but CPython doesn't do this, and there's no proof that
> it'd really help anything anyway.

I welcome correction, but I understand that both the JVM and the CLR memory
managers move memory around. That's why Jython and IronPython use
sequential integers as object IDs, since memory locations are not fixed.

Way back in the mid 1980s, Apple Macintoshes used a memory manager which
could move memory around. Given that the Macs of the day had 128K of RAM,
of which something like a third or a half was used for the screen, being
able to move blocks of memory around to avoid fragmentation was critical,
so I guess that proves that it would help at least one thing.


> (Look at Christian's comments about 
> "Should you return it or not?" and the cost of system calls... now
> consider the orders-of-magnitude worse cost of actually moving memory
> around.)
> 
> This is why a lot of long-duration processes are built to be restarted
> periodically. 

Ironically, the cost of restarting the process periodically is likely to be
orders of magnitude more expensive than that of moving a few blocks of
memory around from time to time. Especially on Windows, where starting
processes is expensive, but even on Linux you have to shut the running
application down, then start it up again and rebuild all the internal data
structures that you just tore down...


-- 
Steven

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python Basics

[Steven D'Aprano]

Chris Angelico wrote:

> On Sat, Oct 4, 2014 at 8:54 AM, Seymore4Head
>  wrote:

>> for i in range(1,10):
>> print (str(i)*i)
> 
> Seymour, please don't do this. When you "help" someone by just giving
> him the answer to a homework problem, you get him past his immediate
> issue of "I need to submit my homework for this problem". That lets
> him get through his course without understanding the code he's
> creating
[...]

In fairness to Seymour, at this extremely basic level, it's really hard to
explain to somebody how to solve a problem without giving them the answer.

While I don't condone mindless parroting of work that others have done,
remember that for tens of thousands of years, being shown how to do
something, then imitating that, has been the most effective way for people
to learn. Dropping hints is typically the least effective learning method.



-- 
Steven

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: "High water" Memory fragmentation still a thing?

[Chris Angelico]

On Sat, Oct 4, 2014 at 11:02 AM, Steven D'Aprano
 wrote:
> Chris Angelico wrote:
>
>> In theory, a high
>> level language like Python would be allowed to move objects around to
>> compact memory, but CPython doesn't do this, and there's no proof that
>> it'd really help anything anyway.
>
> I welcome correction, but I understand that both the JVM and the CLR memory
> managers move memory around. That's why Jython and IronPython use
> sequential integers as object IDs, since memory locations are not fixed.

Right; I should have made it clearer that there's no proof that it'd
help anything *in CPython*. Removing the GIL is periodically proposed,
too, but there's no proof that its removal would benefit CPython; it's
not just that nobody's gotten around to writing a memory compactor for
CPython.

> Ironically, the cost of restarting the process periodically is likely to be
> orders of magnitude more expensive than that of moving a few blocks of
> memory around from time to time. Especially on Windows, where starting
> processes is expensive, but even on Linux you have to shut the running
> application down, then start it up again and rebuild all the internal data
> structures that you just tore down...

Maybe. But you deal with a number of things all at once:

1) Code updates (including interpreter updates)
2) Compaction of Python objects
3) Disposal of anything that got "high level leaked" - unintended
longevity caused by a global reference of some sort
4) Cleanup of low-level allocations that don't go through the Python
memory manager
etc etc etc.

So, yes, it's expensive. And sometimes it's not even possible (there's
no way to retain socket connections across a restart, AFAIK). But it's
there if you want it.

Personally, I like to keep processes running, but that's me. :)

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python Basics

[Chris Angelico]

On Sat, Oct 4, 2014 at 11:09 AM, Steven D'Aprano
 wrote:
> In fairness to Seymour, at this extremely basic level, it's really hard to
> explain to somebody how to solve a problem without giving them the answer.
>
> While I don't condone mindless parroting of work that others have done,
> remember that for tens of thousands of years, being shown how to do
> something, then imitating that, has been the most effective way for people
> to learn. Dropping hints is typically the least effective learning method.

Maybe. I'd still like to see the student's existing code before just
giving the answer. Providing code in response to the copy/pasted
problem definition invites the answer to be copy/pasted, which creates
the situation I described.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Returning a List

[Larry Hudson]

On 10/03/2014 04:35 AM, Shiva wrote:

Hi All,

I might be doing something really silly here, but I can't seem to spot it:

def front_x(words):
   b=[]
   c=[]
   for a in words:
  if a[0] == 'x':
  b.append(a)
  else:
  c.append(a)

   b = sorted(b)
   c = sorted(c)
   d= b+c
   print('d = ',d)

   #return b+c
   return d

front_x(['bbb', 'ccc', 'axx', 'xzz', 'xaa'])



Why is return d or return b+c not returning anything??

The d's value is confirmed by the print statement.

Thanks,
Shiva.


Works fine for me...

For a quickie test, I copy/pasted this code into Idle, and it works.

I suspect you are trying to run this as a script -- but remember, when running interactively 
(directly in Python or in Idle) the return values are displayed automatically.  This is NOT true 
when running as a program, you have to specifically print the return values (or save them in a 
variable to access them later).  Change your calling line to:


print(front_x(['bbb', 'ccc', 'axx', 'xzz', 'xaa']))

 -=- Larry -=-

--
https://mail.python.org/mailman/listinfo/python-list

Re: "High water" Memory fragmentation still a thing?

[Steven D'Aprano]

Chris Angelico wrote:

> On Sat, Oct 4, 2014 at 11:02 AM, Steven D'Aprano
>  wrote:
>> Chris Angelico wrote:
>>
>>> In theory, a high
>>> level language like Python would be allowed to move objects around to
>>> compact memory, but CPython doesn't do this, and there's no proof that
>>> it'd really help anything anyway.
>>
>> I welcome correction, but I understand that both the JVM and the CLR
>> memory managers move memory around. That's why Jython and IronPython use
>> sequential integers as object IDs, since memory locations are not fixed.
> 
> Right; I should have made it clearer that there's no proof that it'd
> help anything *in CPython*. Removing the GIL is periodically proposed,
> too, but there's no proof that its removal would benefit CPython; it's
> not just that nobody's gotten around to writing a memory compactor for
> CPython.

I think that you're conflating a couple of different issues, although I
welcome correction.

I don't think that removing the GIL is a requirement for a memory compactor,
or vice versa. I think that PyPy is capable of plugging in various
different garbage collectors, including some without the GIL, which may or
may not include memory compactors. So as far as I can tell, the two are
independent.

As far as the GIL in CPython goes, there have been at least two attempts to
remove it, and they do show strong improvements for multi-threaded code
running on multi-core machines. Alas, they also show significant *slowdown*
for single-core machines, and very little improvement on dual-core
machines.

[Aside: The thing that people fail to understand is that the GIL is not in
fact something which *prevents* multi-tasking, but it *enables* cooperative
multi-tasking:

http://www.dabeaz.com/python/GIL.pdf

although that's not to say that there aren't some horrible performance
characteristics of the GIL. David Beazley has identified issues with the
GIL which suggest room for improving the GIL and avoiding "GIL battles"
which are responsible for much of the overhead of CPU-bound threads. Any C
programmers who want to hack on the interpreter core?]


Nevertheless, you're right that since nobody has actually built a version of
CPython with memory compactor, there's no *proof* that it would help
anything.


-- 
Steven

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 3.4.1 installer on Mac links Python to old Tcl/Tk

[Ned Deily]

In article ,
 Kevin Walzer  wrote:
> On 10/3/14, 3:55 PM, Ned Deily wrote:
> > Even if there were no incompatibilities, on OS X with Tcl and Tk (and
> > other) frameworks, the version number is embedded in the path to the
> > shared library and the linker normally creates an absolute path at that.
> 
> Is this because Python lacks the concept of stubs?
> 
> A Tcl library compiled for 8.5 can be loaded into 8.6 with no 
> re-compiling required because of stubs.

It has nothing to do with Python per se; that's just the way linking 
with OS frameworks work.  Both Apple and ActiveState install their 
versions of Tcl and Tk as frameworks.  What gets installed and what the 
OS X compilers and linkers expect is something like this:

$ cd /Library/Frameworks/Tk.framework/Versions/8.5
$ ls -l
total 5720
drwxr-xr-x+ 3 root  wheel  272 Nov 24  2013 Headers
drwxr-xr-x+ 2 root  wheel  340 Nov 24  2013 PrivateHeaders
drwxr-xr-x+ 4 root  wheel  272 Nov 24  2013 Resources
-rw-r--r--+ 1 root  wheel  2905000 Oct 27  2013 Tk
-rw-r--r--+ 1 root  wheel12240 Oct 27  2013 libtkstub8.5.a
-rw-r--r--+ 1 root  wheel 4622 Oct 27  2013 tkConfig.sh
$ file Tk
Tk: Mach-O universal binary with 2 architectures
Tk (for architecture x86_64): Mach-O 64-bit dynamically linked shared 
library x86_64
Tk (for architecture i386):   Mach-O dynamically linked shared library 
i386
$ file libtkstub8.5.a
libtkstub8.5.a: Mach-O universal binary with 2 architectures
libtkstub8.5.a (for architecture x86_64): current ar archive random 
library
libtkstub8.5.a (for architecture i386):   current ar archive random 
library
$ otool -D Tk
Tk:
/Library/Frameworks/Tk.framework/Versions/8.5/Tk

When Python or other programs link with a framework, they use the cc or 
ld -framework option, rather than -l:
-framework Tcl -framework Tk

That causes the linker to look in the default search paths for 
frameworks, /Library/Frameworks followed by /System/Library/Frameworks.  
The install names of the framework shared libraries used are embedded in 
the Mach-O file (executable, bundle, or shared library) produced by ld.  
So the stub library archive is there but is not used, AFAIK.

There may be other ways to do it but that's how Python has always linked 
to Tcl and Tk.  FWIW, that's how both Apple's and ActiveState's wish 
executables are linked as well:

$ more /usr/local/bin/wish8.5
#!/bin/sh
"$(dirname 
$0)/../../../Library/Frameworks/Tk.framework/Versions/8.5/Resources/Wish.
app/Contents/MacOS/Wish" "$@"
$ otool -L 
/Library/Frameworks/Tk.framework/Versions/8.5/Resources/Wish.app/Contents
/MacOS/Wish
/Library/Frameworks/Tk.framework/Versions/8.5/Resources/Wish.app/Contents
/MacOS/Wish:
   /Library/Frameworks/Tk.framework/Versions/8.5/Tk (compatibility 
version 8.5.0, current version 8.5.15)
   /Library/Frameworks/Tcl.framework/Versions/8.5/Tcl (compatibility 
version 8.5.0, current version 8.5.15)
   /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current 
version 125.2.0)
   /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa 
(compatibility version 1.0.0, current version 15.0.0)
   /System/Library/Frameworks/Carbon.framework/Versions/A/Carbon 
(compatibility version 2.0.0, current version 152.0.0)
   /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit 
(compatibility version 1.0.0, current version 275.0.0)
   /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current 
version 625.0.0)

There's more info about frameworks in various Apple developer docs and 
in the man pages for ld, otool, and install_name_tool.

-- 
 Ned Deily,
 n...@acm.org

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: "High water" Memory fragmentation still a thing?

[Chris Angelico]

On Sat, Oct 4, 2014 at 3:48 PM, Steven D'Aprano
 wrote:
> I think that you're conflating a couple of different issues, although I
> welcome correction.
>
> I don't think that removing the GIL is a requirement for a memory compactor,
> or vice versa. I think that PyPy is capable of plugging in various
> different garbage collectors, including some without the GIL, which may or
> may not include memory compactors. So as far as I can tell, the two are
> independent.
>
> As far as the GIL in CPython goes, there have been at least two attempts to
> remove it, and they do show strong improvements for multi-threaded code
> running on multi-core machines. Alas, they also show significant *slowdown*
> for single-core machines, and very little improvement on dual-core
> machines.

Not conflating, comparing. In both cases, it's perfectly possible *in
theory* to build a CPython with this change. (GIL removal is clearly
possible in theory, as it's been done in practice.) And *in theory*,
there's some benefit to be gained by doing so. But it's not a case of
"why doesn't python-dev just knuckle down and do it already", as
there's no evidence that it's a real improvement. (A "compile to
machine code" feature might well be purely beneficial, and that's
simply a matter of work - how much work, for how many CPU
architectures, to get how much benefit. But at least that's going to
have a fairly good expectation of performance improvement.) A memory
compactor might well help a narrow set of Python programs (namely,
those which allocate heaps and heaps of memory, then throw away most
of it, and keep running for a long time), but the complexity cost will
make it unlikely to be of benefit.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: "High water" Memory fragmentation still a thing?

[dieter]

Croepha  writes:
> Does python in general terms (apart from extensions or gc manipulation),
> exhibit a "high water" type leak of allocated memory in recent python
> versions (2.7+)?

Likely because it is very difficult to avoid. Would you need to
definitely prevent it is "memory compaction": not only is garbage
collected and freed but in addition, the used memory is also
relocated to form contigous blocks of used and free memory.

Without memory compaction, long running processes tend to suffer
from "memory fragmentation": while sufficient free memory is available,
it is available only in small blocks, not large enough for some memory
requests and those requests then call for more memory from the operating
system. When this memory is later released, it may become split up in
smaller blocks and when another large memory request arrives, new memory
may be requested from the operating system (even though the original
block would have been large enough).

Python tries hard to limit the effect of fragmentation (maintaining the
free blocks in bins of given size) but cannot eliminate it completely.


In order to support memory compaction, all C extensions must adhere
to a strict memory access protocol: they cannot freely use C pointers
to access the memory (as the compaction may invalidate those pointers
at any time) but must beforehand announce "I will now be using this pointer"
(such that the compaction does not move the memory) and afterwards
announce "I am no longer using this pointer" (such that memory
compaction becomes again possible for this memory).

As you see from the description, memory compaction presents a heavy burden
for all extension writers.

-- 
https://mail.python.org/mailman/listinfo/python-list