Re: Are threads bad? - was: Future of Pypy?

[Marko Rauhamaa]

Chris Angelico :

> Actually, you can quite happily have multiple threads messing with the
> underlying file descriptors, that's not a problem. (Though you will
> tend to get interleaved output. But if you always produce output in
> single blocks of text that each contain one line with a trailing
> newline, you should see interleaved lines that are each individually
> correct. I'm also not sure of any sane way to multiplex stdin -
> merging output from multiple threads is fine, but dividing input
> between multiple threads is messy.) The problem is *buffers* for stdin
> and stdout, where you have to be absolutely sure that you're not
> trampling all over another thread's data structures. If you unbuffer
> your output, it's probably going to be thread-safe.

Here's an anecdote describing one real-life threading problem. We had a
largish multithreading framework (in Java, but I'm setting it in Python
and in a much simplified form).

We were mindful of deadlocks caused by lock reversal so we had come up
with a policy whereby objects form a layered hierarchy. An object higher
up in the hierarchy was allowed to call methods of objects below while
holding locks. The opposite was not allowed; if an object desired to
call a method of an object above it (through a registered callback), it
had to relinquish all locks before doing so.

However, a situation like this arose:

class App:
def send_stream(self, sock):
with self.lock:
self.register_socket(sock)

class SocketWrapper:
def read(_, count):
return sock.recv(count)
def close(_):
sock.close()
with self.lock:
self.unregister_socket(sock)

self.transport.forward_and_close(SocketWrapper(sock))

class Transport:
def forward_and_close(self, readable):
with self.lock:
more = readable.read(1000)
if more is WOULDBLOCK:
self.reschedule(readable)
elif more:
... # out of scope for the anecdote
else:
# EOF reached
readable.close()

Now the dreaded lock reversal arises when the App object calls
self.transport.forward_and_close() and Transport calls readable.close()
at the same time.

So why lock categorically like that? Java has a handy "synchronized"
keyword that wraps the whole method in "with self.lock". Ideally, that
handy idiom could be employed methodically. More importantly, to avoid
locking problems, the methodology should be rigorous and mindless. If
the developer must perform a deep locking analysis at every turn, they
are bound to make mistakes, especially when more than one developer is
involved, with differing intuitions.

Unfortunately, that deep locking analysis *is* required at every turn,
and mistakes *are* bound to happen.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Concatenate list values

[loial]

Many thanks for those you chose to help me out. Problem solved.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Design thought for callbacks

[Cem Karan]

On Feb 23, 2015, at 7:29 AM, "Frank Millman"  wrote:

> 
> "Cem Karan"  wrote in message 
> news:a3c11a70-5846-4915-bb26-b23793b65...@gmail.com...
>> 
>> 
>> Good questions!  That was why I was asking about 'gotchas' with WeakSets 
>> originally.  Honestly, the only way to know for sure would be to write two 
>> APIs for doing similar things, and then see how people react to them.  The 
>> problem is, how do you set up such a study so it is statistically valid?
>> 
> 
> Just in case you missed Steven's comment on my 'gotcha', and my reply, it is 
> worth repeating that what I reported as a gotcha was not what it seemed.
> 
> If you set up the callback as a weakref, and the listening object goes out 
> of scope, it will wait to be garbage collected. However, as far as I can 
> tell, the weakref is removed at the same time as the object is gc'd, so 
> there is no 'window' where the weakref exists but the object it is 
> referencing does not exist.
> 
> My problem was that I had performed a cleanup operation on the listening 
> object before letting it go out of scope, and it was no longer in a valid 
> state to deal with the callback, resulting in an error. If you do not have 
> that situation, your original idea may well work.

Thank you Frank, I did read Steve's comment to your reply earlier, but what you 
said in your original reply made sense to me.  I don't have control over user 
code.  That means that if someone wants to write code such that they perform 
some kind of cleanup and are no longer able to handle the callback, they are 
free to do so.  While I can't prevent this from happening, I can make it as 
obvious as possible in my code that before you perform any cleanup, you also 
need to unregister from the library.  That is my main goal in developing 
pythonic/obvious methods of registering callbacks.

Thanks,
Cem Karan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Design thought for callbacks

[Cem Karan]

I'm combining two messages into one, 

On Feb 24, 2015, at 12:29 AM, random...@fastmail.us wrote:

> On Tue, Feb 24, 2015, at 00:20, Gregory Ewing wrote:
>> Cem Karan wrote:
>>> I tend to structure my code as a tree or DAG of objects.  The owner refers 
>>> to
>>> the owned object, but the owned object has no reference to its owner.  With
>>> callbacks, you get cycles, where the owned owns the owner.
>> 
>> This is why I suggested registering a listener object
>> plus a method name instead of a callback. It avoids that
>> reference cycle, because there is no long-lived callback
>> object keeping a reference to the listener.
> 
> How does that help? Everywhere you would have had a reference to the
> "callback object", you now have a reference to the listener object.
> You're just shuffling deck chairs around: if B shouldn't reference A
> because A owns B, then removing C from the B->C->A reference chain does
> nothing to fix this.

On Feb 24, 2015, at 12:45 AM, Gregory Ewing  wrote:

> Cem Karan wrote:
>> On Feb 22, 2015, at 5:15 AM, Gregory Ewing 
>> wrote:
>>> Perhaps instead of registering a callback function, you should be
>>> registering the listener object together with a method name.
>> I see what you're saying, but I don't think it gains us too much.  If I store
>> an object and an unbound method of the object, or if I store the bound method
>> directly, I suspect it will yield approximately the same results.
> 
> It would be weird and unpythonic to have to register both
> an object and an unbound method, and if you use a bound
> method you can't keep a weak reference to it.


Greg, random832 said what I was thinking earlier, that you've only increased 
the diameter of your cycle without actually fixing it.  Can you give a code 
example where your method breaks the cycle entirely?

Thanks,
Cem Karan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Newbie question about text encoding

[Chris Angelico]

On Tue, Feb 24, 2015 at 9:49 PM,   wrote:
> Working with pyshp, this is my code :
>
> import shapefile
>
> inFile = shapefile.Reader("blah")
>
> for sr in inFile.shapeRecords():
> rec = sr.record[2]
> print("Output : ", rec, type(rec))
>
> Output:  hippodrome du resto 
> Output:  b'stade de man\xe9 braz' 
>
> Why do I get 2 different types ?
> How to get a string object when I have accented characters ?

I don't know what pyshp is doing here, so you may want to seek a
pyshp-specific mailing list for help. My guess is that it's
automatically decoding to str if it's ASCII-only, and giving you back
the raw bytes if there are any that it can't handle. The question is:
What encoding _is_ that? Do you know what character you're expecting
to see there? Before you can turn that into a string, you have to
figure out whether it's Latin-1 (ISO-8859-1), or some other ISO-8859-x
standard, or a Windows codepage, or an ancient thing off a Mac, or
whatever else it might be. Once you know that, it's easy: you just
decode() the bytes objects. But you MUST figure out the encoding
first.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: Python shell: Arrow keys not working in PuTTY

[David Aldrich]

> >> BUT do *not* run `make install` as that will overwrite your system
> >> Python and Bad Things will happen. Instead, run `make altinstall`.

Thanks for all the warnings. We did use `make altinstall`, so all is ok.

Recompiling, with readline installed, fixed the arrow keys.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Newbie question about text encoding

[Dave Angel]

On 02/24/2015 05:49 AM, pierrick.brih...@gmail.com wrote:

Hello,

Working with pyshp, this is my code :


What version of Python, what version of pyshp, from where, and what OS? 
 These are the first information to supply in any query that goes 
outside of the standard library.


For example, you might be running CPython 3.2.1 on Ubuntu 14.04.1, and 
installed pyshp 1.2.1 from https://pypi.python.org/pypi/pyshp


Or some other combination.



import shapefile

inFile = shapefile.Reader("blah")

for sr in inFile.shapeRecords():
 rec = sr.record[2]
 print("Output : ", rec, type(rec))

Output:  hippodrome du resto 
Output:  b'stade de man\xe9 braz' 

Why do I get 2 different types ?
How to get a string object when I have accented characters ?

Thank you,

p.b.



From my (cursory) reading of the pyshp docs on the above page, I cannot 
see what the [2] element of the record list should look like.  So I'd 
have to guess.


The bytes object is presumably an encoded version of the character 
string.  I don't see anything on that page about unicode, or decode, so 
you might have to guess the encoding.  Anyway, you can decode the 
bytestring into a regular string if you can correctly guess the encoding 
method, such as utf-8.


If that were the right decoding, you could just use
mystring = rec.decode()

But utf-8 does not seem to be the right encoding for that bytestring. 
So you'll need a form like:

mystring = rec.decode(encoding='xxx')

for some value of xxx.







--
DaveA
--
https://mail.python.org/mailman/listinfo/python-list

Re: Python shell: Arrow keys not working in PuTTY

[Laura Creighton]

In a message of Tue, 24 Feb 2015 11:18:38 +, David Aldrich writes:
>> >> BUT do *not* run `make install` as that will overwrite your system
>> >> Python and Bad Things will happen. Instead, run `make altinstall`.
>
>Thanks for all the warnings. We did use `make altinstall`, so all is ok.
>
>Recompiling, with readline installed, fixed the arrow keys.

Great!  Thank you for letting us know.

Laura (who was worried)

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Newbie question about text encoding

[Laura Creighton]

In a message of Tue, 24 Feb 2015 15:55:41 +0100, Laura Creighton writes:
>In a message of Tue, 24 Feb 2015 06:25:24 -0500, Dave Angel writes:
>>But utf-8 does not seem to be the right encoding for that bytestring. 
>>So you'll need a form like:
>> mystring = rec.decode(encoding='xxx')
>>
>>for some value of xxx.
>
>>DaveA
>
>And the xxx you want is "latin1"
>
>Laura

er, latin1.  You don't want an extra set of quotes.
There are many aliases for latin1. i.e. latin_1, iso-8859-1, iso8859-1,
8859, cp819, latin, latin1, L1
see: https://docs.python.org/2.4/lib/standard-encodings.html

and you might want to read
https://docs.python.org/2/howto/unicode.html

to understand the problem better.

Laura
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Newbie question about text encoding

[Laura Creighton]

In a message of Wed, 25 Feb 2015 02:03:16 +1100, Chris Angelico writes:
>On Wed, Feb 25, 2015 at 1:55 AM, Laura Creighton  wrote:
>> In a message of Tue, 24 Feb 2015 06:25:24 -0500, Dave Angel writes:
>>>But utf-8 does not seem to be the right encoding for that bytestring.
>>>So you'll need a form like:
>>> mystring = rec.decode(encoding='xxx')
>>>
>>>for some value of xxx.
>>
>>>DaveA
>>
>> And the xxx you want is "latin1"
>
>Can you be sure it's Latin-1? I'm not certain of that. In any case, I
>never advocate fixing encoding problems by "just do this and it'll all
>go away"; you have to understand your data before you can decode it.
>
>ChrisA

I can, I speak French and I recognise the data.  It's French place names,
places where sporting events are held. :)

Laura

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Newbie question about text encoding

[Chris Angelico]

On Wed, Feb 25, 2015 at 2:07 AM, Laura Creighton  wrote:
>>Can you be sure it's Latin-1? I'm not certain of that. In any case, I
>>never advocate fixing encoding problems by "just do this and it'll all
>>go away"; you have to understand your data before you can decode it.
>>
>>ChrisA
>
> I can, I speak French and I recognise the data.  It's French place names,
> places where sporting events are held. :)

Ah, okay. :) But even with that level of confidence, you still have to
pick between Latin-1 and CP-1252, which you can't tell based on this
one snippet. Welcome to untagged encodings.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Newbie question about text encoding

[Chris Angelico]

On Wed, Feb 25, 2015 at 2:24 AM, Laura Creighton  wrote:
> Ah, yes, you are right about that.  I see CP-1252 about 2 times every 10
> years, and latin1 every minute of my life, so I am biased to assume I
> know what I am seeing.

Fair enough. CP-1252 is still a possibility, but the difference can be
dealt with later.

> ChrisA, you come from an English speaking country, right?

Yes (Australia, to be specific).

> For those of us who come from countries whose language doesn't fit in
> ASCII, the notion of 'understand the data' doesn't work very well.  We
> already understand the data -- its a set of words in our native language.
> The hard part isn't understanding the data, but rather understanding how
> the hell Python could be so stupid as to not understand it. :)  The
> notion that Python normally only understands the subset of the
> characters in your native language than English speakers use in their
> language is not the most obvious thing.

Also a reasonable baseline assumption; but the trouble is that if you
automatically assume that text is encoded in your favourite eight-bit
system, you're taking a huge risk.

Now, you have a huge leg up on me, in that you actually recognize the
*words* in that piece of text. That means you can have MUCH greater
confidence in stating that it's Latin-1 than I can. But that's
precisely what I mean by "understand the data". If you, being a native
French speaker, pick up a file written in (say) Polish, and encoded
Latin-2, you'll recognize by the ASCII characters that it's not French
text, and probably you'd be able to spot that it ought to be Latin-2
rather than Latin-1. That's understanding the data, that's having more
information than just the byte patterns. A computer can't reliably do
that (just look up the "Bush hid the facts" bug if you don't believe
me), but a human often can.

> And having taught countless European kids how to write their very first
> program in Python, I can tell you for certain that the sort of deep
> understanding of encoding methods is not what 10 year olds who just
> want to print out the names of their friends, and their favourite
> music titles, and their favourite musicians want to know. :)

Right, so you should be teaching them to use Python 3, and always
saving everything in UTF-8, and basically ignoring the whole mess of
eight-bit encodings :)

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Newbie question about text encoding

[random832]

On Tue, Feb 24, 2015, at 10:10, Chris Angelico wrote:
> Ah, okay. :) But even with that level of confidence, you still have to
> pick between Latin-1 and CP-1252, which you can't tell based on this
> one snippet. Welcome to untagged encodings.

Or Latin-9 (ISO 8859-15) That was popular on Linux systems for a while
before everyone switched to UTF-8 - it's got the Euro sign, and
(relevant to French) the "oe" ligature, and uppercase Y with diaeresis,
at the expense of "generic currency" and fractions.

Or it could be Latin-3, Latin-5 (8859-9), or Latin-8 (8859-14) - they
are not commonly used for French locales, being primarily intended for
other languages, but they do support all characters (at least all from
Latin-1) used in French names. I assume there are likewise several
Windows codepages it could be.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Newbie question about text encoding

[Chris Angelico]

On Wed, Feb 25, 2015 at 3:20 AM, Laura Creighton  wrote:
> People who are asking for help in getting things to work in their
> native language need a 'do this quick' sort of answer.  The deeper
> problems of supporting all languages and language encodings can very
> much wait.

I'm not so sure about that. When "supporting all languages" is as
simple as "use Python 3 and UTF-8 everywhere", the cost is much lower
than it might be, and the benefit is potentially huge. A "do this
quick" answer might get you by *right now*, but it leaves open the
possibility of subtler errors. That's why Python moved to
Unicode-by-default, even though eight-bit encodings will tend to
produce the right results for simple text.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Newbie question about text encoding

[Dave Angel]

On 02/24/2015 11:20 AM, Laura Creighton wrote:

In a message of Wed, 25 Feb 2015 02:33:30 +1100, Chris Angelico writes:

Also a reasonable baseline assumption; but the trouble is that if you
automatically assume that text is encoded in your favourite eight-bit
system, you're taking a huge risk.


But, you know, I wasn't assuming this.  I actually read latin1.  I
could read it in ascii, know that \xe9  means 'é', a letter combination
that we have in Swedish, so I am rather used to reading, and then
well, I could read all of his strings, know they were in French,
and know that latin1 was what he needed things to be decoded to.


With a sample of one string, how did you read "all his strings".  And 
with one non-ASCII code in that single string, how did you know that 
'latin1' was the only encoding that included a reasonable character at 
that encoding?





Now, you have a huge leg up on me, in that you actually recognize the
*words* in that piece of text. That means you can have MUCH greater
confidence in stating that it's Latin-1 than I can. But that's
precisely what I mean by "understand the data". If you, being a native
French speaker, pick up a file written in (say) Polish, and encoded
Latin-2, you'll recognize by the ASCII characters that it's not French
text, and probably you'd be able to spot that it ought to be Latin-2
rather than Latin-1. That's understanding the data, that's having more
information than just the byte patterns. A computer can't reliably do
that (just look up the "Bush hid the facts" bug if you don't believe
me), but a human often can.


Absolutely correct.  But you must not require that all of the speakers
of non-English languages think about their languages as 'special
encodings'.  Only the monoglot ever think of a foreign language as
a code.


All languages are foreign.  All that can be written to a disk file are 
bytes.  Those have to have been encoded to represent some abstraction 
called a character set, or string.  The question is whether the encoding 
method is specified for the particular file type, or for the particular 
file.


See http://support.esri.com/cn/knowledgebase/techarticles/detail/21106

according to that page, starting at ArcGIS 10.2.1, the default sets the 
code page to UTF-8 (UNICODE) in the shapefile (.DBF)


But in earlier ones, there's supposed to be a reference to the codepage 
used.  From that, one can presumably derive which decoder to use.





That poor guy the original poster just wants to have a nice string
of his sporting event place name.  We should tell him how to get that,
not how to be an expert in all the encodings on the face of this earth.
Chances are, the only thing he needs to talk about are French words.

If not, well, he will come back when things stop working, and have lots
more data to give him.  If, instead, this makes him go away happy, then
this was the very best thing to do.


And having taught countless European kids how to write their very first
program in Python, I can tell you for certain that the sort of deep
understanding of encoding methods is not what 10 year olds who just
want to print out the names of their friends, and their favourite
music titles, and their favourite musicians want to know. :)


Right, so you should be teaching them to use Python 3, and always
saving everything in UTF-8, and basically ignoring the whole mess of
eight-bit encodings :)


Of course this makes sense.  But you seem to be missing the point.
People who are asking for help in getting things to work in their
native language need a 'do this quick' sort of answer.  The deeper
problems of supporting all languages and language encodings can very
much wait.  The OP wants a hunk of bytes that happens to mean
something in French, and is not encodable in the limited English
language to work like a different hunk of bytes that means something
in French but is encodable.

Don't overburden them.


My guess is that this is only appropriate for users who use only locally 
created data.  Since the OP's data is apparently old (if it were current 
versions, it'd have been utf-8), who knows how consistent the encoding is.


--
--
DaveA
--
https://mail.python.org/mailman/listinfo/python-list

Re: How to apply python patch for issue9729

[Grant Edwards]

On 2015-02-24, Chris Angelico  wrote:
> On Tue, Feb 24, 2015 at 8:09 AM, Sada Shirol  wrote:
>> Upon some research found that I need to apply below patch:
>>
>> http://bugs.python.org/issue9729
>>
>> How do I apply a patch to python 2.6.6 on Linux?
>
> Hmm. You have a bit of a problem there: The patch wasn't designed to
> apply to 2.6. So you may find that it doesn't apply at all, or needs
> some tweaking.

Or worse yet, it applies just fine but doesn't work quite right in
particular cases that you won't run across until later when it's much
more embarassing.

-- 
Grant Edwards   grant.b.edwardsYow! ... I want to perform
  at   cranial activities with
  gmail.comTuesday Weld!!
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Newbie question about text encoding

[Laura Creighton]

Dave Angel
are you another Native English speaker living in a world where ASCII
is enough?

Laura

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Newbie question about text encoding

[Dave Angel]

On 02/24/2015 02:57 PM, Laura Creighton wrote:

Dave Angel
are you another Native English speaker living in a world where ASCII
is enough?


I'm a native English speaker, and 7 bits is not nearly enough.  Even if 
I didn't currently care, I have some history:


No.  CDC display code is enough. Who needs lowercase?

No.  Baudot code is enough.

No, EBCDIC is good enough.  Who cares about other companies.

No, the "golf-ball" only holds this many characters.  If we need more, 
we can just get the operator to switch balls in the middle of printing.


No. 2 digit years is enough.  This world won't last till the millennium 
anyway.


No.  2k is all the EPROM you can have.  Your code HAS to fit in it, and 
only 1.5k RAM.


No.  640k is more than anyone could need.

No, you cannot use a punch card made on a model 26 keypunch in the same 
deck as one made on a model 29.  Too bad, many of the codes are 
different.  (This one cost me travel back and forth between two 
different locations with different model keypunches)


No. 8 bits is as much as we could ever use for characters.  Who could 
possibly need names or locations outside of this region?  Or from 
multiple places within it?


35 years ago I helped design a serial terminal that "spoke" Chinese, 
using a two-byte encoding.  But a single worldwide standard didn't come 
until much later, and I cheered Unicode when it was finally unveiled.


I've worked with many printers that could only print 70 or 80 unique 
characters.  The laser printer, and even the matrix printer are 
relatively recent inventions.


Getting back on topic:

According to:
   http://support.esri.com/cn/knowledgebase/techarticles/detail/27345

"""ArcGIS Desktop applications, such as ArcMap, are Unicode based, so 
they support Unicode to a certain level. The level of Unicode support 
depends on the data format."""


That page was written about 2004, so there was concern even then.

And according to another, """In the header of each shapefile (.DBF), a 
reference to a code page is included."""


--
DaveA
--
https://mail.python.org/mailman/listinfo/python-list

Re: Design thought for callbacks

[Gregory Ewing]

random...@fastmail.us wrote:

On Tue, Feb 24, 2015, at 00:20, Gregory Ewing wrote:


This is why I suggested registering a listener object
plus a method name instead of a callback. It avoids that
reference cycle, because there is no long-lived callback
object keeping a reference to the listener.


How does that help? Everywhere you would have had a reference to the
"callback object", you now have a reference to the listener object.


The point is that the library can keep a weak reference
to the listener object, whereas it can't reliably keep
a weak reference to a bound method.

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

Bug in timsort!?

[Roy Smith]

http://envisage-project.eu/proving-android-java-and-python-sorting-algorithm-is-broken-and-how-to-fix-it/
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Bug in timsort!?

[Zachary Ware]

On Tue, Feb 24, 2015 at 3:34 PM, Roy Smith  wrote:
> http://envisage-project.eu/proving-android-java-and-python-sorting-algorithm-is-broken-and-how-to-fix-it/

http://bugs.python.org/issue23515

Note that the article does mention that Python is not vulnerable due
to this bug (and best I can tell has no behavioral issues); no
computer currently in existence can create a large enough array to
cause a problem.

-- 
Zach
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Bug in timsort!?

[sohcahtoa82]

On Tuesday, February 24, 2015 at 1:50:15 PM UTC-8, Mark Lawrence wrote:
> On 24/02/2015 21:34, Roy Smith wrote:
> > http://envisage-project.eu/proving-android-java-and-python-sorting-algorithm-is-broken-and-how-to-fix-it/
> >
> 
> As you can clearly no longer rely on Python it looks like, after a long 
> and loving relationship, I'm forced into moving on.  PHP, Applescript, 
> Javascript or back to C or even CORAL 66, what is your (plural) 
> recommendation(s)?
> 
> -- 
> My fellow Pythonistas, ask not what our language can do for you, ask
> what you can do for our language.
> 
> Mark Lawrence

Personally, I'm going with BASIC being interpreted by a JavaScript script being 
fed from a PHP script running on an out-dated version of Apache.

It includes an in-browser IDE that will be written as an ActiveX control and 
will require Internet Explorer 6.  An Adobe Flash version is in the works to 
allow it to run on modern browsers.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Bug in timsort!?

[Grant Edwards]

On 2015-02-24, Roy Smith  wrote:

> http://envisage-project.eu/proving-android-java-and-python-sorting-algorithm-is-broken-and-how-to-fix-it/

I don't get it. 

3.2 Corrected Python merge_collapse function

merge_collapse(MergeState *ms)
{
struct s_slice *p = ms->pending;

assert(ms);
while (ms->n > 1) {
Py_ssize_t n = ms->n - 2;
if ( n > 0   && p[n-1].len <= p[n].len + p[n+1].len
|| (n-1 > 0 &&  p[n-2].len <= p[n].len + p[n-1].len)) {
if (p[n-1].len < p[n+1].len)
--n;
if (merge_at(ms, n) < 0)
return -1;
}
else if (p[n].len <= p[n+1].len) {
 if (merge_at(ms, n) < 0)
return -1;
}
else
break;
}
return 0;
}

Or does "Python function" mean something else in this context?




-- 
Grant Edwards   grant.b.edwardsYow! I request a weekend in
  at   Havana with Phil Silvers!
  gmail.com
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Bug in timsort!?

[Ian Kelly]

On Tue, Feb 24, 2015 at 3:45 PM, Grant Edwards  wrote:
> On 2015-02-24, Roy Smith  wrote:
>
>> http://envisage-project.eu/proving-android-java-and-python-sorting-algorithm-is-broken-and-how-to-fix-it/
>
> I don't get it.
>
> 3.2 Corrected Python merge_collapse function
>
> merge_collapse(MergeState *ms)
> {
> struct s_slice *p = ms->pending;
>
> assert(ms);
> while (ms->n > 1) {
> Py_ssize_t n = ms->n - 2;
> if ( n > 0   && p[n-1].len <= p[n].len + p[n+1].len
> || (n-1 > 0 &&  p[n-2].len <= p[n].len + p[n-1].len)) {
> if (p[n-1].len < p[n+1].len)
> --n;
> if (merge_at(ms, n) < 0)
> return -1;
> }
> else if (p[n].len <= p[n+1].len) {
>  if (merge_at(ms, n) < 0)
> return -1;
> }
> else
> break;
> }
> return 0;
> }
>
> Or does "Python function" mean something else in this context?

Recall that CPython is implemented in C. ;-)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python Silent Install

[Albert-Jan Roskam]

Personally I find that Python is incomplete without pip and setuptools.-- 
https://mail.python.org/mailman/listinfo/python-list

strip bug?

[baykiwi]

>>> 'http://xthunder'.strip('http://')
'xthunder'
>>> 'http://thunder'.strip('http://')
'under'
>>>

I could understand backslash but forward slash?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: strip bug?

[Ian Kelly]

On Tue, Feb 24, 2015 at 4:05 PM,   wrote:
 'http://xthunder'.strip('http://')
> 'xthunder'
 'http://thunder'.strip('http://')
> 'under'


This removes all leading and trailing occurrences of the characters in
the string 'http://', not the exact substring 'http://'. For that, use
either the str.replace method or slicing.

> I could understand backslash but forward slash?

I don't understand the question.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: strip bug?

[Eduardo]

Well, from the docstring of strip:

--
S.strip([chars]) -> string or unicode

Return a copy of the string S with leading and trailing
whitespace removed.
If chars is given and not None, remove characters in chars instead.
If chars is unicode, S will be converted to unicode before stripping
--

In this case 'http://thunder'.strip('http://') is the same as 
'http://thunder'.strip('htp:/') and any character of the string that is passed 
to the method will be removed. 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: strip bug?

[Irmen de Jong]

On 25-2-2015 0:05, bayk...@gmail.com wrote:
 'http://xthunder'.strip('http://')
> 'xthunder'
 'http://thunder'.strip('http://')
> 'under'

> 
> I could understand backslash but forward slash?
> 




>>> help("".strip)
Help on built-in function strip:

strip(...) method of builtins.str instance
S.strip([chars]) -> str

Return a copy of the string S with leading and trailing
whitespace removed.
If chars is given and not None, remove characters in chars instead.
^^



It strips any chars given in the argument, not as a prefix/suffix to remove.

Irmen

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: strip bug?

[Chris Kaynor]

On Tue, Feb 24, 2015 at 3:05 PM,  wrote:

> >>> 'http://xthunder'.strip('http://')
> 'xthunder'
> >>> 'http://thunder'.strip('http://')
> 'under'
> >>>
>
> I could understand backslash but forward slash?


I believe the issue is that str.strip does not do quite what you are
thinking it does, however your message is very unspecific about what you
expect to get. It seems that you except str.strip to remove a sub-string,
while it actually removes the list of specified characters (see
https://docs.python.org/3.4/library/stdtypes.html?highlight=str.strip#str.strip).
As such, you would get the same result from 'http://thunder'.strip('htp:/')
and 'http://thunder'.strip('/thp:') as from your samples.

To get what I am guessing you want ("xthunder" for the first and "thunder"
for the second), you should use splicing, likely combined with
str.startswith() (untested code):
url = 'http://thunder/
if url.startswith('http://'):
url = url[7:]
else
# Handle case without the prefix in whatever manner is correct for your
case. Possibly continue, possibly error out.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Bug in timsort!?

[Chris Angelico]

On Wed, Feb 25, 2015 at 8:34 AM, Roy Smith  wrote:
> http://envisage-project.eu/proving-android-java-and-python-sorting-algorithm-is-broken-and-how-to-fix-it/

The post links to:

http://svn.python.org/projects/python/trunk/Objects/listobject.c

Is that still valid and current? It says svn, but does it pop through
to the Mercurial repo where the _real_ CPython code is stored?

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Bug in timsort!?

[MRAB]

On 2015-02-24 21:40, Zachary Ware wrote:

On Tue, Feb 24, 2015 at 3:34 PM, Roy Smith  wrote:

http://envisage-project.eu/proving-android-java-and-python-sorting-algorithm-is-broken-and-how-to-fix-it/


http://bugs.python.org/issue23515

Note that the article does mention that Python is not vulnerable due
to this bug (and best I can tell has no behavioral issues); no
computer currently in existence can create a large enough array to
cause a problem.


I think the key word here is "currently".

--
https://mail.python.org/mailman/listinfo/python-list

Re: Bug in timsort!?

[Chris Kaynor]

On Tue, Feb 24, 2015 at 4:07 PM, Chris Angelico  wrote:

> On Wed, Feb 25, 2015 at 10:50 AM, Skip Montanaro
>  wrote:
> > Even if/when we get to the point where machines can hold an array of
> > 2**49 elements, I suspect people won't be using straight Python to
> > wrangle them.
>
> Looking just at CPython, what is the absolute minimum storage space
> for a massive list like that? My guess is that a 64-bit CPython might
> get as good as eight bytes per element; playing around with smaller
> figures (up to circa a million elements) shows it ranging from 8 to 9
> bytes per element, bottoming out at just a smidge over 8, presumably
> at the moments when there's no slack space (there's a fixed overhead,
> which gets diminishingly significant). So an array of 2**49 elements
> would take 2**52 bytes (4 petabytes) of storage - addressable, to be
> sure, but an awful lot to be sorting.
>
> Would it be sufficient to stick a comment into the source saying "This
> may have problems with lists in excess of 2**49 elements", and leave
> it until that's actually likely to happen?
>

I would agree that it will be quite a while as it stands before the bug
appears, so documenting it MAY be appropriate, as it is likely to be far
enough in the future that a number of unforeseen things may block the bug
from being an issue (TimSort may be replaced by something better, CPython
could die, etc).

That said, from what I understand, their proposed fix would be reasonably
easy to use, and have minimal cost, so it may be worthwhile to use just
because the issue is likely to be in use for quite a while before somebody
remembers the issue exists. Once somebody does hit it, it may take a while
to realize that TimSort is at fault, as it will only occur on some data
sets (my understanding is that the elements must be in certain
configurations with at-least the minimum number), so it will likely appear
as just a random crash occurring.

If it were fully up to me, I would do one of the following, in order of
preference:
1) Implement and test the fix provided on the blog. This requires the most
work due to licensing, reviewing, and testing.
2) Add a noisy warning/error to the code when sorting lists large enough to
potentially hit the error (probably 2**48 to leave some wiggle room). This
is much easier to do, but merely pushes the can down the road to be dealt
with later. It does still make it obvious what is going long when it
finally breaks.
3) Add a comment explaining the issue (likely linking back to the blog).
While by far the easiest "solution", again, this merely pushes the can down
the road, and it may be very non-obvious what went wrong when it breaks,
despite the comment.

Chris
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Bug in timsort!?

[Mark Lawrence]

On 24/02/2015 22:36, sohcahto...@gmail.com wrote:

On Tuesday, February 24, 2015 at 1:50:15 PM UTC-8, Mark Lawrence wrote:

On 24/02/2015 21:34, Roy Smith wrote:

http://envisage-project.eu/proving-android-java-and-python-sorting-algorithm-is-broken-and-how-to-fix-it/



As you can clearly no longer rely on Python it looks like, after a long
and loving relationship, I'm forced into moving on.  PHP, Applescript,
Javascript or back to C or even CORAL 66, what is your (plural)
recommendation(s)?

--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence


Personally, I'm going with BASIC being interpreted by a JavaScript script being 
fed from a PHP script running on an out-dated version of Apache.

It includes an in-browser IDE that will be written as an ActiveX control and 
will require Internet Explorer 6.  An Adobe Flash version is in the works to 
allow it to run on modern browsers.



Thanks for your help, much appreciated :)

--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

--
https://mail.python.org/mailman/listinfo/python-list

Re: Are threads bad? - was: Future of Pypy?

[Ryan Stuart]

On Tue Feb 24 2015 at 3:32:47 PM Paul Rubin  wrote:

> Ryan Stuart  writes:
> Sure, the shared memory introduces the possibility of some bad errors,
> I'm just saying that I've found that by staying with a certain
> straightforward style, it doesn't seem difficult in practice to avoid
> those errors.
>

And all I'm saying is that it doesn't matter how careful you are, all it
takes is someone up the stack to not be as careful and you will be bitten.
And even if they are careful, it's still very easy to be bitten because
everything runs in shared memory.


> I don't understand what you mean about malloc.
>

My point is malloc, something further up (down?) the stack, is making
modifications to shared state when threads are involved. Modifying shared
state makes it infinitely more difficult to reason about the correctness of
your software.


> That article was interesting in some ways but confused in others.  One
> way it was interesting is it said various non-thread approaches (such as
> coroutines) had about the same problems as threads.  Some ways it
> was confused were:
>

We clearly got completely different things from the article. My
interpretation was that it was making *the exact opposite* point to what
you stated mainly because non-threading approaches don't share state. It
states that quite clearly. For example "it is – literally – exponentially
more difficult to reason about a routine that may be executed from an
arbitrary number of threads concurrently".


>   1) thinking Haskell threads were like processes with separate address
>   spaces.  In fact they are in the same address space and programming
>   with them isn't all that different from Python threads, though the
>   synchronization primitives are a bit different.  There is also an STM
>   library available that is ingenious though apparently somewhat slow.
>

I don't know Haskell, so I can't comment too much, except to say that by
default Haskell looks to use lightweight threads where only 1 thread can be
executing at a time [1] (sounds similar to the GIL to me). That doesn't
seem to be shared state multithreading, which is what the article is
referring to. But again, they will undoubtedly be someone sharing state
higher up the stack.


>   2) it has a weird story about the brass cockroach, that basically
>   signified that they didn't have a robust enough testing system to be
>   able to reproduce the bug.  That is what they should have worked on.
>

The point was that it wasn't feasible to have a robust testing suite
because, you guessed it, shared state and concurrent threads producing n*n
complexity.


>   3) It goes into various hazards of the balance transfer example not
>   mentioning that STM (available in Haskell and Clojure) completely
>   solves it.
>

This is probably correct. Is there any STM implementations out that that
don't significantly compromise performance?

Anyway, I got one thing out of this, which is that the multiprocessing
> module looks pretty nice and I should try it even when I don't need
> multicore parallelism, so thanks for that.
>

It's 1 real advantage is that it side-steps the GIL. So, if you need to
utilise multiple cores for CPU bound tasks, then it might well be the only
option.

Cheers


> In reality though, Python is still my most productive language for
> throwaway, non-concurrent scripting, but for more complicated concurrent
> programs, alternatives like Haskell, Erlang, and Go all have significant
> attractions.
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Newbie question about text encoding

[Steven D'Aprano]

Laura Creighton wrote:

> Dave Angel
> are you another Native English speaker living in a world where ASCII
> is enough?

ASCII was never enough. Not even for Americans, who couldn't write things
like "I bought a comic book for 10¢ yesterday", let alone interesting
things from maths and science.

I missed the whole 7-bit ASCII period, my first computer (Mac 128K) already
had an extended character set beyond ASCII. But even that never covered the
full range of characters I wanted to write, and then there was the horrible
mess that you got whenever you copied text files from a Mac to a DOS or
Windows PC or visa versa. Yes, even in 1984 we were transferring files and
running into encoding issues.



-- 
Steven

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Newbie question about text encoding

[Steven D'Aprano]

Laura Creighton wrote:

> The idea that the whole world loves utf-8 is nonsense.

I don't think anyone says the whole world loves UTF-8. I think people say
that the whole world *ought to* love UTF-8, and that legacy encodings from
the Windows "code-page" days ought to die.


> Most of europe has been using latin1, latin2 etc. before 
> unicode was invented and will, as far as I know, continue to use it.

And this is why people in Greece cannot transfer text files to people in
France without the content changing (ISO-8859-7 vs ISO-8859-1). And why
Russians cannot even swap text files with other Russians (a plethora of
encodings).

:-(



-- 
Steven

-- 
https://mail.python.org/mailman/listinfo/python-list

Running Idle on Windows 8 (.1)

[Terry Reedy]

On 2/24/2015 3:13 PM, blakemal...@gmail.com wrote:


I too can not get idle to run on win 8.1 using python3.4.2 installed from the 
python-3.4.2.amd64.msi.


What experience have others had with Idle and Windows 8?

The OP for
https://stackoverflow.com/questions/28633690/python-installation-troubleshooting

wrote in a comment "I am invoking it from a win8 startmenu program, so I 
tried to do it via the normal windows 8 interface and it works from there,"


I do not know what 'normal windows 8 interface' means, but does it work 
for anyone else?



pythonw.exe appears in the task manager but no gui is displayed.


pythonw.exe should appear *twice* in task manager.
Please try 3.4.3rc1 and 3.5.0a1
https://www.python.org/downloads/windows/

If the Idle does not start correctly on Win 8, perhaps we could fix it 
before 3.4.3 final (which should be soon).


--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list

py.test can't resolve imports

[Mario Figueiredo]

I cannot seem to solve this problem from either the pytest
documentation or from other places on the web which provide tutorials
on pytest, such as http://pythontesting.net/start-here/

I have the following project setup:

/project
/project/project<-- script files (with __init__.py)
/project/test<-- test files
/project/docs   <-- Sphinx

I cannot seem to be able to have py.test resolve imports by running
the following command from within the /project/project directory:

$ py.test ../


Example scripts:

/project/project/example.py
def func():
pass

/project/test/test_example.py
from example import *
def test_example():
assert 0

I get an error:

$ py.test ../

from example import *
E   ImportError: No module named 'example'

I can run the tests fine with $ python -m pytest ../

So how do I properly setup if I wish to use the less verbose py.test?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Bug in timsort!?

[Dave Angel]

On 02/24/2015 07:07 PM, Chris Angelico wrote:

On Wed, Feb 25, 2015 at 10:50 AM, Skip Montanaro
 wrote:

Even if/when we get to the point where machines can hold an array of
2**49 elements, I suspect people won't be using straight Python to
wrangle them.


Looking just at CPython, what is the absolute minimum storage space
for a massive list like that? My guess is that a 64-bit CPython might
get as good as eight bytes per element; playing around with smaller
figures (up to circa a million elements) shows it ranging from 8 to 9
bytes per element, bottoming out at just a smidge over 8, presumably
at the moments when there's no slack space (there's a fixed overhead,
which gets diminishingly significant). So an array of 2**49 elements
would take 2**52 bytes (4 petabytes) of storage - addressable, to be
sure, but an awful lot to be sorting.

Would it be sufficient to stick a comment into the source saying "This
may have problems with lists in excess of 2**49 elements", and leave
it until that's actually likely to happen?

ChrisA



One could do much better than that.  Put in a test for the length of the 
list, and if it's too large for the current implementation, throw an 
exception.  Much better than a comment.


--
DaveA
--
https://mail.python.org/mailman/listinfo/python-list

Re: Bug in timsort!?

[Terry Reedy]

On 2/24/2015 4:34 PM, Roy Smith wrote:

http://envisage-project.eu/proving-android-java-and-python-sorting-algorithm-is-broken-and-how-to-fix-it/


Tim Peters is aware of this and opened http://bugs.python.org/issue23515


--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list

Re: Newbie question about text encoding

[Marcos Almeida Azevedo]

On Wed, Feb 25, 2015 at 9:19 AM, Steven D'Aprano <
steve+comp.lang.pyt...@pearwood.info> wrote:

> Laura Creighton wrote:
>
> > Dave Angel
> > are you another Native English speaker living in a world where ASCII
> > is enough?
>
> ASCII was never enough. Not even for Americans, who couldn't write things
> like "I bought a comic book for 10¢ yesterday", let alone interesting
> things from maths and science.
>
>
ASCII was a necessity back then because RAM and storage are too small.


> I missed the whole 7-bit ASCII period, my first computer (Mac 128K) already
> had an extended character set beyond ASCII. But even that never covered the
>

I miss the days when I was coding with my XT computer (640kb RAM) too.
Things were so simple back then.


> full range of characters I wanted to write, and then there was the horrible
> mess that you got whenever you copied text files from a Mac to a DOS or
> Windows PC or visa versa. Yes, even in 1984 we were transferring files and
> running into encoding issues.
>
>
>
> --
> Steven
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>



-- 
Marcos | I love PHP, Linux, and Java

-- 
https://mail.python.org/mailman/listinfo/python-list

Pyston 0.3 self-hosting

[Steven D'Aprano]

Pyston 0.3, the latest version of a new high-performance Python 
implementation, has reached self-hosting sufficiency:


http://blog.pyston.org/2015/02/24/pyston-0-3-self-hosting-sufficiency/



-- 
Steve

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Are threads bad? - was: Future of Pypy?

[Marko Rauhamaa]

Marcos Almeida Azevedo :

> Synchronized methods in Java really makes programming life simpler.
> But I think it is standard practice to avoid this if there is a
> lighter alternative as synchronized methods are slow. Worse case I
> used double checked locking.

I have yet to see code whose performance suffers from too much locking.
However, I have seen plenty of code that suffers from anomalies caused
by incorrect locking.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Are threads bad? - was: Future of Pypy?

[Chris Angelico]

On Wed, Feb 25, 2015 at 4:46 PM, Marko Rauhamaa  wrote:
> Marcos Almeida Azevedo :
>
>> Synchronized methods in Java really makes programming life simpler.
>> But I think it is standard practice to avoid this if there is a
>> lighter alternative as synchronized methods are slow. Worse case I
>> used double checked locking.
>
> I have yet to see code whose performance suffers from too much locking.
> However, I have seen plenty of code that suffers from anomalies caused
> by incorrect locking.

Uhh, I have seen *heaps* of code whose performance suffers from too
much locking. At the coarsest and least intelligent level, a database
program that couldn't handle concurrency at all, so I wrote an
application-level semaphore that stopped two people from running it at
once. You want to use that program? Ask the other guy to close it.
THAT is a performance problem. And there are plenty of narrower cases,
where it ends up being a transactions-per-second throughput limiter.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Are threads bad? - was: Future of Pypy?

[Marcos Almeida Azevedo]

On Wed, Feb 25, 2015 at 1:46 PM, Marko Rauhamaa  wrote:

> Marcos Almeida Azevedo :
>
> > Synchronized methods in Java really makes programming life simpler.
> > But I think it is standard practice to avoid this if there is a
> > lighter alternative as synchronized methods are slow. Worse case I
> > used double checked locking.
>
> I have yet to see code whose performance suffers from too much locking.
> However, I have seen plenty of code that suffers from anomalies caused
> by incorrect locking.
>
>
>
Of course code with locking is slower than one without. The locking
mechanism itself is the overhead.  So use it only when necessary.  There is
a reason why double checked locking was invented by clever programmers.



> Marko
> --
> https://mail.python.org/mailman/listinfo/python-list
>



-- 
Marcos | I love PHP, Linux, and Java

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Are threads bad? - was: Future of Pypy?

[Ian Kelly]

On Tue, Feb 24, 2015 at 10:54 PM, Chris Angelico  wrote:
> On Wed, Feb 25, 2015 at 4:46 PM, Marko Rauhamaa  wrote:
>> Marcos Almeida Azevedo :
>>
>>> Synchronized methods in Java really makes programming life simpler.
>>> But I think it is standard practice to avoid this if there is a
>>> lighter alternative as synchronized methods are slow. Worse case I
>>> used double checked locking.
>>
>> I have yet to see code whose performance suffers from too much locking.
>> However, I have seen plenty of code that suffers from anomalies caused
>> by incorrect locking.
>
> Uhh, I have seen *heaps* of code whose performance suffers from too
> much locking. At the coarsest and least intelligent level, a database
> program that couldn't handle concurrency at all, so I wrote an
> application-level semaphore that stopped two people from running it at
> once. You want to use that program? Ask the other guy to close it.
> THAT is a performance problem. And there are plenty of narrower cases,
> where it ends up being a transactions-per-second throughput limiter.

Is the name of that database program "Microsoft Access" perchance?
-- 
https://mail.python.org/mailman/listinfo/python-list

progress bar

[Swapnil Pande]

i want to call another tkinter window after completing the progress bar
an n e one help me
-- 
https://mail.python.org/mailman/listinfo/python-list