minfuds - coding puzzle

[alphonse23]

Would anybody be up to helping me solve this? It's one of the questions on 
Codility. According to them, it should only take 30 mins. 

Two non-empty zero-indexed arrays A and B, each consisting of N integers, are 
given. Four functions are defined based on these arrays:
F(X,K) = A[K]*X + B[K]
U(X)   = max{ F(X,K) : 0 ≤ K < N }
D(X)   = min{ F(X,K) : 0 ≤ K < N }
S(X)   = U(X) − D(X)
Write a function:
double solution(int A[], int B[], int N);
that, given two arrays A and B consisting of N integers each, returns the 
minimum value of S(X) where X can be any real number.
For example, given the following arrays A and B consisting of three elements 
each:
A[0] = -1B[0] = 3
A[1] =  1B[1] = 0
A[2] =  0B[2] = 2
the function should return 0.5 because:
U(X) = −1*X + 3 if X ≤ 1
U(X) =  0*X + 2 if 1 ≤ X ≤ 2
U(X) =  1*X + 0 if 2 ≤ X
and:
D(X) = 1*X + 0  if X ≤ 1.5
D(X) = −1*X + 3 if 1.5 ≤ X
so for X = 1.5, function S(X) is equal to 0.5 and this is the minimum value of 
this function.
Assume that:
N is an integer within the range [1..100,000];
each element of array A is an integer within the range [−1,000..1,000];
each element of array B is an integer within the range [−1,000..1,000].
Complexity:
expected worst-case time complexity is O(N*log(N));
expected worst-case space complexity is O(N), beyond input storage (not 
counting the storage required for input arguments).
Elements of input arrays can be modified.

I'm not sure how to solve it. I feel like it's the equivalent of finding the 
minimum of a function on a graph. I'd know how to do that in algebra/calculus, 
but not with code. Also, I'm not sure why their example skipped A[2] = 0 and 
B[2] = 2 for D(X).

This is as far as I got:

def solution(A, B):
# write your code here...
min = 0
return min

def S(A,B,x):
return U(A,B,x) - D(A,B,x)

def F(a,x,b):
return a*x + b

def U(A,B,x):
max = F(A[0],x,B[0])
for i in range(1,len(A)):
u = F(A[i],x,B[i])
if max < u: max = u
return max

def D(A,B,x):
min = F(A[0],x,B[0])
for i in range(1,len(A)):
d = F(A[i],x,B[i])
if min > d: min = d
return min
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Understanding other people's code

[Azureaus]

On Friday, 12 July 2013 15:22:59 UTC+1, Azureaus  wrote:
> Hi all,
> 
> I've been asked to take over a project from someone else and to extend the 
> functionality of this. The project is written in Python which I haven't had 
> any real experience with (although I do really like it) so I've spent the 
> last week or two settling in, trying to get my head around Python and the way 
> in which this code works.
> 
> 
> 
> The problem is the code was clearly written by someone who is exceptionally 
> good and seems to inherit everything from everywhere else. It all seems very 
> dynamic, nothing is written statically except in some configuration files. 
> 
> Basically the problem is I am new to the language and this was clearly 
> written by someone who at the moment is far better at it than I am!
> 
> 
> 
> I'm starting to get pretty worried about my lack of overall progress and so I 
> wondered if anyone out there had some tips and techniques for understanding 
> other peoples code. There has to be 10/15 different scripts with at least 10 
> functions in each file I would say.
> 
> 
> 
> Literally any idea will help, pen and paper, printing off all the code and 
> doing some sort of highlighting session - anything! I keep reading bits of 
> code and thinking "well where the hell has that been defined and what does it 
> mean" to find it was inherited from 3 modules up the chain. I really need to 
> get a handle on how exactly all this slots together! Any techniques,tricks or 
> methodologies that people find useful would be much appreciated.

Thanks for all the suggestions, I'm afraid I didn't get a chance to view them 
over the weekend but I will get started with them this morning. I'm currently 
using sublime 2 for my text editor and tried to create a UML diagram using 
Pylint to try and get a map overview of what's going on. Unfortunately it 
seemed to map the classes into groups such as StringIO, ThreadPool, GrabOut 
etc.. rather than into the modules they belong go and how they fit together. 
Maybe this is just my inexperience showing through or I'm using the program 
wrong. If anyone has any 'mapping' programs they use to help them visualise 
program flow that would be a great bonus.

To be fair to who programmed it, most functions are commented and I can't 
complain about the messiness of the code, It's actually very tidy. (I suppose 
Python forcing it's formatting is another reason it's an easily readable 
language!) Luckily not blanked import * were used otherwise I really would be 
up the creek without a paddle. 
Thanks!
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: RE Module Performance

[Devyn Collier Johnson]

On 07/14/2013 02:17 PM, 8 Dihedral wrote:

On Saturday, July 13, 2013 1:37:46 PM UTC+8, Steven D'Aprano wrote:

On Fri, 12 Jul 2013 13:58:29 -0400, Devyn Collier Johnson wrote:




I plan to spend some time optimizing the re.py module for Unix systems.
I would love to amp up my programs that use that module.



In my experience, often the best way to optimize a regex is to not use it

at all.



[steve@ando ~]$ python -m timeit -s "import re" \


-s "data = 'a'*100+'b'" \
"if re.search('b', data): pass"

10 loops, best of 3: 2.77 usec per loop



[steve@ando ~]$ python -m timeit -s "data = 'a'*100+'b'" \


"if 'b' in data: pass"

100 loops, best of 3: 0.219 usec per loop



In Python, we often use plain string operations instead of regex-based

solutions for basic tasks. Regexes are a 10lb sledge hammer. Don't use

them for cracking peanuts.







--

Steven

OK, lets talk about the indexed search algorithms of
a character streamor strig which can be buffered and
indexed randomly for RW operations but faster in sequential
block RW operations after some pre-processing.

This was solved long time ago in the suffix array or
suffix tree part and summarized in the famous BWT paper in 199X.

Do we want volunteers to speed up
search operations in the string module in Python?

It would be nice if someone could speed it up.
--
http://mail.python.org/mailman/listinfo/python-list

Python - remote object protocols and security

[Jean-Michel Pichavant]

Hello everyone, 


I'd like to exchange some simple python objects over the internet. 
I initially planned to use Pyro, after reading 
http://pythonhosted.org/Pyro4/security.html I'm still puzzled. 


I don't mind encrypting data, if someone wants to sniff what I'm sending, he's 
welcome. 


What I think I need to care about, is malicious code injections. Because both 
client/server will be in python, would someone capable of executing code by 
changing one side python source ? 


How do I prevent this and still provide the source to everyone ? 


JM 


-- IMPORTANT NOTICE: 

The contents of this email and any attachments are confidential and may also be 
privileged. If you are not the intended recipient, please notify the sender 
immediately and do not disclose the contents to any other person, use it for 
any purpose, or store or copy the information in any medium. Thank you.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python - remote object protocols and security

[Jean-Michel Pichavant]

In text format... sorry for my previous html post

Hello everyone, 

I'd like to exchange some simple python objects over the internet. 
I initially planned to use Pyro, after reading 
http://pythonhosted.org/Pyro4/security.html I'm still puzzled. 

I don't mind encrypting data, if someone wants to sniff what I'm sending, he's 
welcome. 

What I think I need to care about, is malicious code injections. Because both 
client/server will be in python, would someone capable of executing code by 
changing one side python source ? 

How do I prevent this and still provide the source to everyone ? 

JM 


-- IMPORTANT NOTICE: 

The contents of this email and any attachments are confidential and may also be 
privileged. If you are not the intended recipient, please notify the sender 
immediately and do not disclose the contents to any other person, use it for 
any purpose, or store or copy the information in any medium. Thank you.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python - remote object protocols and security

[Dave Angel]

On 07/15/2013 06:20 AM, Jean-Michel Pichavant wrote:

In text format... sorry for my previous html post

Hello everyone,

I'd like to exchange some simple python objects over the internet.
I initially planned to use Pyro, after reading 
http://pythonhosted.org/Pyro4/security.html I'm still puzzled.

I don't mind encrypting data, if someone wants to sniff what I'm sending, he's 
welcome.



I don't think the word you need there is "mind," but I get the idea. 
You don't care who reads the data being sent, but don't want anybody to 
be able to masquerade as somebody else, either by sending with invalid 
credentials or by intercepting and modifying legitimate traffic.


What's needed for that is public/private key encryption, where the 
legitimate user uses his own private key to encode data, and you use 
their public key to decode it.  Anybody can decode it, since the key is 
public, but anyone (especialy you) can be sure who sent it, since they 
presumably keep their private key private.




What I think I need to care about, is malicious code injections. Because both 
client/server will be in python, would someone capable of executing code by 
changing one side python source ?



Even if you have a friendly user sending data, you still need to guard 
against code injection because their system may have been compromised. 
And with code injection, you risk trashing not only their own data, but 
other people's and your own as well.  In other words, encryption 
provides you no assurances.



How do I prevent this and still provide the source to everyone ?



Make sure your deserializing logic (on your own machine) is entirely 
under your control, and impervious to such attacks.  In general, the 
more types that can be encoded, the more likely it's vulnerable.  So 
authors of such libraries have two conflicting goals.


I can't tell you if pyro, or any other particular one is safe.  But if 
you decide to roll your own, which is implied when you say you'll be 
publishing your source, then keep it dirt-simple.  Have a very specific 
and finite set of classes which you'll handle, and write rigorous code 
to make sure the types are limited to those.  For example, you could 
restrict yourself to lists of strings, or lists of strings and floats, 
and you have only three decoders to write.


Note that DOS attacks are possible whatever encoding scheme you have. 
Make sure that self-references within the data are well-defined (or 
impossible), and put limits on size per transaction, and transactions 
per minute per legitimate user.


--
DaveA

--
http://mail.python.org/mailman/listinfo/python-list

Re: Python - remote object protocols and security

[Chris Angelico]

On Mon, Jul 15, 2013 at 8:13 PM, Jean-Michel Pichavant
 wrote:
> I'd like to exchange some simple python objects over the internet.
> I initially planned to use Pyro, after reading
> http://pythonhosted.org/Pyro4/security.html I'm still puzzled.
>
> I don't mind encrypting data, if someone wants to sniff what I'm sending,
> he's welcome.
>
> What I think I need to care about, is malicious code injections. Because
> both client/server will be in python, would someone capable of executing
> code by changing one side python source ?
>
> How do I prevent this and still provide the source to everyone ?

How complicated are the objects you want to transmit? If they're just
strings, integers, floats, and lists or dictionaries of the above,
then you could use JSON instead; that's much safer, but (and because)
it's majorly restricted. Sometimes it's worth warping your data
structure slightly (eg use a dict and global functions instead of a
custom object with methods) to improve security.

ChrisA
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: How to clean up socket connection to printer

[loial]

Well, I certainly suspect the customers network connection to the printer which 
is over a WAN across half of Europe, but proving that is the problem is another 
matter.

I can replicate a "Connection reset by peer" error on own printer by pulling 
the network cable out of the printer. And again I thereafter get the issue of 
"Connection refused" for what seems a variable amount of time.

But at least I am now reassured that the "Connection Refused" is not due to 
something my script has not cleaned up.

Thanks

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: what thread-synch mech to use for clean exit from a thread

[MRAB]

On 15/07/2013 04:04, Steven D'Aprano wrote:

On Mon, 15 Jul 2013 10:27:45 +0800, Gildor Oronar wrote:


A currency exchange thread updates exchange rate once a minute. If the
thread faield to update currency rate for 5 hours, it should inform
main() for a clean exit. This has to be done gracefully, because main()
could be doing something delicate.

I, a newbie, read all the thread sync tool, and wasn't sure which one to
use. In fact I am not sure if there is a need of thread sync, because
there is no racing cond. I thought of this naive way:

class CurrencyExchange():
def __init__(in_case_callback):
   this.callback = in_case_callback


You need to declare the instance parameter, which is conventionally
called "self" not "this". Also, your class needs to inherit from Thread,
and critically it MUST call the superclass __init__.

So:

class CurrencyExchange(threading.Thread):
 def __init__(self, in_case_callback):
 super(CurrencyExchange, self).__init__()
 self.callback = in_case_callback

But I'm not sure that a callback is the right approach here. See below.



def __run__():


Likewise, you need a "self" parameter.



   while time.time() - self.rate_timestamp < 5*3600:
  ... # update exchange rate
  if success:


The "==" in this line should, of course, be "=":


 self.rate_timestamp == time.time()
  time.sleep(60)
   this.callback() # rate not updated 5 hours, a crisis


I think that a cleaner way is to just set a flag on the thread instance.
Initiate it with:

 self.updates_seen = True

in the __init__ method, and then add this after the while loop:

 self.updates_seen = False




def main():
def callback()
   Go_On = False


I don't believe this callback will work, because it will simply create a
local variable call "Go_On", not change the non-local variable.

In Python 3, you can use the nonlocal keyword to get what you want, but I
think a better approach is with a flag on the thread.


agio = CurrencyExchange(in_case = callback)
agio.start()

Go_On = True
while Go_On:
   do_something_delicate(rate_supplied_by=agio)


Change to:

 while agio.updates_seen:
 do_something_delicate...




--
http://mail.python.org/mailman/listinfo/python-list

Re: Python - remote object protocols and security

[Jean-Michel Pichavant]

- Original Message -

> > I don't mind encrypting data, if someone wants to sniff what I'm
> > sending, he's welcome.
> >
> 
> I don't think the word you need there is "mind," but I get the idea.

You're right, I wanted to state actually the opposite, I don't want to encrypt 
data because I don't care if someone sniffs it.
It's pretty meaningless and doesn't include credentials.

Basically, I need to transfer numbers (int). Possibly dictionaries like 
{string: int} in order to structure things a little bit.
I don't think I need a credential system neither, cause if someone is sending 
me crap with the wrong identity it will only mess my statistics, this is pretty 
harmless. 

> Even if you have a friendly user sending data, you still need to
> guard
> against code injection because their system may have been
> compromised.

That is definitively something I'm trying to avoid.

> Make sure your deserializing logic (on your own machine) is entirely
> under your control, and impervious to such attacks.  In general, the
> more types that can be encoded, the more likely it's vulnerable.  So
> authors of such libraries have two conflicting goals.

If I understand correctly any available remote protocols are pretty much of the 
chart.
Since I'm planning to send only int and strings I think I'll follow your advice 
of serializing/deserializing myself.

> DaveA

thanks,

Jean-Michel


-- IMPORTANT NOTICE: 

The contents of this email and any attachments are confidential and may also be 
privileged. If you are not the intended recipient, please notify the sender 
immediately and do not disclose the contents to any other person, use it for 
any purpose, or store or copy the information in any medium. Thank you.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python - remote object protocols and security

[Chris Angelico]

On Mon, Jul 15, 2013 at 10:26 PM, Jean-Michel Pichavant
 wrote:
> Basically, I need to transfer numbers (int). Possibly dictionaries like 
> {string: int} in order to structure things a little bit.

I strongly recommend JSON, then. It's a well-known system, it's
compact, it's secure, and Python comes with a json module.

ChrisA
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: RE Module Performance

[Steven D'Aprano]

On Mon, 15 Jul 2013 06:06:06 -0400, Devyn Collier Johnson wrote:

> On 07/14/2013 02:17 PM, 8 Dihedral wrote:
[...]
>> Do we want volunteers to speed up
>> search operations in the string module in Python?
>
> It would be nice if someone could speed it up.

Devyn,

8 Dihedral is our resident bot, not a human being. Nobody knows who 
controls it, and why they are running it, but we are pretty certain that 
it is a bot responding mechanically to keywords in people's posts.

It's a very clever bot, but still a bot. About one post in four is 
meaningless jargon, the other three are relevant enough to fool people 
into thinking that maybe it is a human being. It had me fooled for a long 
time.



-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python - remote object protocols and security

[Jean-Michel Pichavant]

- Original Message -
> > What I think I need to care about, is malicious code injections.
> > Because
> > both client/server will be in python, would someone capable of
> > executing
> > code by changing one side python source ?
> >
> > How do I prevent this and still provide the source to everyone ?
> 
> How complicated are the objects you want to transmit? If they're just
> strings, integers, floats, and lists or dictionaries of the above,
> then you could use JSON instead; that's much safer, but (and because)
> it's majorly restricted. Sometimes it's worth warping your data
> structure slightly (eg use a dict and global functions instead of a
> custom object with methods) to improve security.
> 
> ChrisA

In the end just strings and Int.
Dave seems to agree with you and JSON is the way to go.

However, I don't want to write net code, I'm lazy and most importantly I'm so 
bad at it.
So how would I send Json strings from one machine to a remote ?
If I'm using http://code.google.com/p/jsonrpclib/, would it still be a Json 
safe way of sending strings and int ?

JM



-- IMPORTANT NOTICE: 

The contents of this email and any attachments are confidential and may also be 
privileged. If you are not the intended recipient, please notify the sender 
immediately and do not disclose the contents to any other person, use it for 
any purpose, or store or copy the information in any medium. Thank you.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python - remote object protocols and security

[Dave Angel]

On 07/15/2013 08:30 AM, Chris Angelico wrote:

On Mon, Jul 15, 2013 at 10:26 PM, Jean-Michel Pichavant
 wrote:

Basically, I need to transfer numbers (int). Possibly dictionaries like 
{string: int} in order to structure things a little bit.


I strongly recommend JSON, then. It's a well-known system, it's
compact, it's secure, and Python comes with a json module.



And presumably has been tested against injection attacks (implied by 
your use of 'secure.')


JM:  That's the flip side.  If you CAN find some open-source that 
exactly meets your needs, it presumably has had lots of eyes on it to 
spot the little bugs that are likely to pop up in any new implementation.


There's a vast grey area between

1) so simple it's safer to do it myself

and

2) so complex the open-source version must have bugs, so I'd better do 
it myself.


in between, you use the open-source code or library.

But this is why I always start by trying to narrow the choice of what 
you *need*.



...the only secure system is one physically contained in a room with a 
padlock, and with a guard.  And only if the guard is yourself...



--
DaveA

--
http://mail.python.org/mailman/listinfo/python-list

Re: Python - remote object protocols and security

[Chris Angelico]

On Mon, Jul 15, 2013 at 10:41 PM, Jean-Michel Pichavant
 wrote:
> - Original Message -
>> > What I think I need to care about, is malicious code injections.
>> > Because
>> > both client/server will be in python, would someone capable of
>> > executing
>> > code by changing one side python source ?
>> >
>> > How do I prevent this and still provide the source to everyone ?
>>
>> How complicated are the objects you want to transmit? If they're just
>> strings, integers, floats, and lists or dictionaries of the above,
>> then you could use JSON instead; that's much safer, but (and because)
>> it's majorly restricted. Sometimes it's worth warping your data
>> structure slightly (eg use a dict and global functions instead of a
>> custom object with methods) to improve security.
>>
>> ChrisA
>
> In the end just strings and Int.
> Dave seems to agree with you and JSON is the way to go.
>
> However, I don't want to write net code, I'm lazy and most importantly I'm so 
> bad at it.
> So how would I send Json strings from one machine to a remote ?
> If I'm using http://code.google.com/p/jsonrpclib/, would it still be a Json 
> safe way of sending strings and int ?

To send JSON-encoded data, you:

1) Encode your data in JSON format and some character encoding (eg UTF-8)
2) Transmit the resulting stream of bytes over the network
3) Decode UTF-8 and then JSON

Python provides all this functionality:

>>> data = {"English":"Hello, world","Russian":"Привет, мир"}
>>> json.dumps(data).encode()
b'{"English": "Hello, world", "Russian":
"\\u041f\\u0440\\u0438\\u0432\\u0435\\u0442, \\u043c\\u0438\\u0440"}'

which happens to look very much like the original input, though this
is more coincidence than design. Note that you could leave the
non-ASCII characters as they are, and transmit them as UTF-8
sequences:

>>> json.dumps(data,ensure_ascii=False).encode()
b'{"English": "Hello, world", "Russian":
"\xd0\x9f\xd1\x80\xd0\xb8\xd0\xb2\xd0\xb5\xd1\x82,
\xd0\xbc\xd0\xb8\xd1\x80"}'

Take your pick, based on what you want to do at the other end. The
second form is (obviously) a lot more compact than the first.

Decoding is just as easy:

>>> data=b'{"English": "Hello, world", "Russian": 
>>> "\xd0\x9f\xd1\x80\xd0\xb8\xd0\xb2\xd0\xb5\xd1\x82, 
>>> \xd0\xbc\xd0\xb8\xd1\x80"}'
>>> json.loads(data.decode())
{'English': 'Hello, world', 'Russian': 'Привет, мир'}

So the only bit you still need is: How do you transmit this across the
network? Since it's now all just bytes, that's easy enough to do, eg
with TCP. But that depends on the rest of your system, and is a quite
separate question - and quite probably one you already have the answer
to.

ChrisA
-- 
http://mail.python.org/mailman/listinfo/python-list

Dihedral

[Devyn Collier Johnson]

On 07/15/2013 08:36 AM, Steven D'Aprano wrote:

On Mon, 15 Jul 2013 06:06:06 -0400, Devyn Collier Johnson wrote:


On 07/14/2013 02:17 PM, 8 Dihedral wrote:

[...]

Do we want volunteers to speed up
search operations in the string module in Python?

It would be nice if someone could speed it up.

Devyn,

8 Dihedral is our resident bot, not a human being. Nobody knows who
controls it, and why they are running it, but we are pretty certain that
it is a bot responding mechanically to keywords in people's posts.

It's a very clever bot, but still a bot. About one post in four is
meaningless jargon, the other three are relevant enough to fool people
into thinking that maybe it is a human being. It had me fooled for a long
time.



Wow! Our mailing list has a pet bot. I bet other mailing lists are so 
jealous of us. Who ever created Dihedral is a genius!


Artificial Intelligence developers put chatbots on mailing lists so that 
the program can learn. I use Python3 to program AI applications. If you 
see my Launchpad account, you will see my two AI projects - Neobot and 
Novabot. (https://launchpad.net/neobot Neo and Nova are still unstable) 
AI developers let their bots loose on the Internet to learn from people. 
Dihedral is learning from us. Dihedral only responses when it feels it 
has sufficient knowledge on the topic. Chatbots want to appear human. 
That is their goal. We should feel honored that Dihedral's botmaster 
feels that this mailinglist would benefit the development of Dihedral's 
knowledge.


Devyn Collier Johnson
--
http://mail.python.org/mailman/listinfo/python-list

Re: Python - remote object protocols and security

[Burak Arslan]

Hi,

On 07/15/13 13:30, Chris Angelico wrote:
> On Mon, Jul 15, 2013 at 10:26 PM, Jean-Michel Pichavant
>  wrote:
>> Basically, I need to transfer numbers (int). Possibly dictionaries like 
>> {string: int} in order to structure things a little bit.
> I strongly recommend JSON, then. It's a well-known system, it's
> compact, it's secure, and Python comes with a json module.
>

Especially for numbers, MessagePack is more efficient. Its API is
identical to Json, so it's almost a drop-in replacement.

A project that I've been working on, Spyne, is designed to implement
public RPC services. It supports both Json and MessagePack. Here's the
json example:
http://spyne.io/#inprot=JsonDocument&outprot=JsonDocument&s=rpc&tpt=WsgiApplication&validator=true

If you choose to use MessagePack, you must HTTP POST the MessagePack
document the same way you'd POST the json document.

Best regards,
Burak

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: GeoIP2 for retrieving city and region ?

[Joel Goldstick]

On Fri, Jul 12, 2013 at 7:04 PM, Dennis Lee Bieber wrote:

> On Sat, 13 Jul 2013 02:47:38 +1000, Chris Angelico 
> declaimed the following:
>
> >
> >Oh, and just for laughs, I tried a few of my recent mobile IP
> >addresses in the GeoIP lookup. All of them quoted Melbourne someplace,
> >some in the CBD and some out in the suburbs, but all vastly wrong, and
> >places I haven't been. But I'd never expect it to be accurate on
> >those.
> >
> Well... the MaxMind demo of "my IP" did get the proper metropolitan
> area... But they list the ISP as "AT&T"... My real ISP is Earthlink
> (piggybacking on AT&T DSL service).
>
> The Lat/Long, however shows as
>
> 42.9634 -85.6681
> whereas a recent GPS readout shows
> 42.9159 -85.5541
>
> or 2m50s too far north, and 6m50s too far west.
>
> Same website, accessed from my Blackberry phone, gave a result of
> "United States, NA" and location 38 -97
> --
> Wulfraed Dennis Lee Bieber AF6VN
> wlfr...@ix.netcom.comHTTP://wlfraed.home.netcom.com/
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>

Speaking more from a political perspective, an important aspect of the
internet is that if you are a publisher who doesn't require registration to
read what you post, your readers are free to be more or less anonymous.
This is a good thing in many ways.  If you have a service that requires
some sort of sign up, then you can require knowing things about them
(location, etc).  If you want to know where your readers are, you need to
make arrangements with their ISPs to get that information, since in many
cases the ISP has a physical link to your machine.  But why should they
provide that to you?  After all you are not their customer.

It might be fun to know, but the repercussions are serious.


-- 
Joel Goldstick
http://joelgoldstick.com
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: GeoIP2 for retrieving city and region ?

[Michael Torrie]

On 07/12/2013 10:32 AM, Νικόλας wrote:
> So, my question now is, if there is some way we can get an accurate Geo 
> City database.

As has been said pretty much by every other poster, there is no way to
do get an accurate location database.  Period.

The databases that do exist were built by hand, and also guessed at
based on routing information.  The best you can really do is region, or
country, and even that fails sometimes.

If you want to know a visitor's city you should ask them using the new
browser location apis available to javascript.

http://diveintohtml5.info/geolocation.html

Since IPs can be dynamic, sometimes even assigned across a region,
there's no way to accurately map ip addresses to a city with the
reliability that you seem to want.  Google is pretty accurate because
they've spent a lot of time building up their own database, and also
convincing users to reveal their locations to them.  Unless you do the
same thing, you have to just get by with what others have provided for you.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Dihedral

[Joel Goldstick]

On Mon, Jul 15, 2013 at 8:52 AM, Devyn Collier Johnson <
devyncjohn...@gmail.com> wrote:

>
> On 07/15/2013 08:36 AM, Steven D'Aprano wrote:
>
>> On Mon, 15 Jul 2013 06:06:06 -0400, Devyn Collier Johnson wrote:
>>
>>  On 07/14/2013 02:17 PM, 8 Dihedral wrote:
>>>
>> [...]
>>
>>> Do we want volunteers to speed up
 search operations in the string module in Python?

>>> It would be nice if someone could speed it up.
>>>
>> Devyn,
>>
>> 8 Dihedral is our resident bot, not a human being. Nobody knows who
>> controls it, and why they are running it, but we are pretty certain that
>> it is a bot responding mechanically to keywords in people's posts.
>>
>> It's a very clever bot, but still a bot. About one post in four is
>> meaningless jargon, the other three are relevant enough to fool people
>> into thinking that maybe it is a human being. It had me fooled for a long
>> time.
>>
>>
>>
>>  Wow! Our mailing list has a pet bot. I bet other mailing lists are so
> jealous of us. Who ever created Dihedral is a genius!
>
> Artificial Intelligence developers put chatbots on mailing lists so that
> the program can learn. I use Python3 to program AI applications. If you see
> my Launchpad account, you will see my two AI projects - Neobot and Novabot.
> (https://launchpad.net/neobot Neo and Nova are still unstable) AI
> developers let their bots loose on the Internet to learn from people.
> Dihedral is learning from us. Dihedral only responses when it feels it has
> sufficient knowledge on the topic. Chatbots want to appear human. That is
> their goal. We should feel honored that Dihedral's botmaster feels that
> this mailinglist would benefit the development of Dihedral's knowledge.
>
> Devyn Collier Johnson
> --
> http://mail.python.org/**mailman/listinfo/python-list
>

I particularly enjoy the misspellings, that seem to be such a human quality
on email messages!

-- 
Joel Goldstick
http://joelgoldstick.com
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: GeoIP2 for retrieving city and region ?

[Benjamin Kaplan]

On Sat, Jul 13, 2013 at 10:43 AM, Νικόλας  wrote:
> Στις 13/7/2013 7:54 μμ, ο/η Dennis Lee Bieber έγραψε:
>>
>> Are you paying for a fixed IP number? I suspect you are if you
>> were
>> running a world-accessible server.
>>
>> Obviously a fixed IP will be tied to a fixed connection and
>> thereby to
>> a fixed location which can be provided to a location database.
>>
>> But most of us have DHCP assigned IP numbers, which change
>> everytime we
>> reboot our connection (or even when the DHCP lease expires -- which may be
>> daily).
>
>
> Same networking scheme for me too, dynamic that is.
>
> Every time the DHCP lease expires or i reboot the router i get a new ip
> address but every time the link i provided states accurately that my ip
> address is from Thessaloníki and not Europe/Athens which is were my ISP
> location is.
>
> Not to mention that in facebook, no matter the way i'am joining, via
> smartphone, tablet, laptop it always pinpoints my exact location.
>
> But yes, i can understand your skepticism.
> An ip address can move anywhere while remaining connected to the same ISP,
> just like a networking device in the house, remains connected to the same
> router while changing rooms or even floors, or even buildings.
>
> But then how do you explain the fact that
> http://www.maxmind.com/en/geoip_demo
> pinpointed Thessaloníki and not Athens and for 2 friends of mine that use
> the same ISP as me but live in different cities also accurately identified
> their locations too?
>

It's not telling you where your ISP is headquartered. It's telling you
where the servers that you're connecting to are. In your case, you're
connecting to servers that your Athens-based ISP has in a Thessaloniki
datacenter. The only way to get an accurate location is to use
something other than IP- phones like to use a combination of their
GPS, a map of the cell phone towers, and a map of wi-fi hotspots (this
is one of the things that Google's StreetView cars log as they drive).
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: GeoIP2 for retrieving city and region ?

[Wayne Werner]

On Sat, 13 Jul 2013, Νικόλας wrote:


But then how do you explain the fact that
http://www.maxmind.com/en/geoip_demo
pinpointed Thessaloníki and not Athens and for 2 friends of mine that 
use the same ISP as me but live in different cities also accurately 
identified their locations too?


If you bothered doing something as simple as read the Wikipedia article on
Geolocation, you could answer this question yourself: Evidently you, and your 
friends have things like cookies, or some other helps that identify your

location, which is why your addresses are close.

-W-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python - remote object protocols and security

[Chris Angelico]

On Mon, Jul 15, 2013 at 10:45 PM, Dave Angel  wrote:
> On 07/15/2013 08:30 AM, Chris Angelico wrote:
>>
>> On Mon, Jul 15, 2013 at 10:26 PM, Jean-Michel Pichavant
>>  wrote:
>>>
>>> Basically, I need to transfer numbers (int). Possibly dictionaries like
>>> {string: int} in order to structure things a little bit.
>>
>>
>> I strongly recommend JSON, then. It's a well-known system, it's
>> compact, it's secure, and Python comes with a json module.
>>
>
> And presumably has been tested against injection attacks (implied by your
> use of 'secure.')

Talking about the json module? I would expect it has, given that JSON
is frequently used in untrusted contexts (unlike, for instance,
pickle, which is specifically *not* for untrusted data). But even if
it has some sort of exploit, that would be a bug to be fixed in the
library; it would be an issue that affects many other users, and
someone will likely report it and get it fixed in the next point
release.

But what I meant was that the protocol itself is designed with
security restrictions in mind. It's designed not to fetch additional
content from the network (as XML can), nor to retrieve named objects
from the environment (as pickle can), etc, etc. That doesn't mean it's
perfect, but it's a lot easier to make a secure protocol based on JSON
than one based on pickle, simply because starting with the basics and
adding safely is easier than starting with massive power and then
protecting around issues.

ChrisA
-- 
http://mail.python.org/mailman/listinfo/python-list

Question regarding building Python Windows installer

[Mcadams, Philip W]

I'm attempting to create a Python 64-bit Windows Installer.  Following the 
instructions here: http://docs.python.org/2/distutils/builtdist.html I'm to 
navigate to my Python folder and user command:

python setup.py build --plat-name=win-amd64 bdist_wininst

I get error: COMPILED_WTH_PYDEBUG = ('-with-pydebug' in 
sysconfig.get_config_var("CONFIG_ARGS"))
TypeError: argument of type 'NoneType' is not iterable

I also have tried:

setup.py build --plat-name=win-amd64 bdist_wininst

and get error:

File "setup.py", line 263
Print "%-*s %-*s %-*s" % (longest, e, longet, f,

SyntaxError: invalid syntax

I followed the instructions here: http://docs.python.org/devguide/setup.html to 
create a PC build for Windows which allows me to run a Python prompt.  Now I 
need to create a Windows Installer to install this Python on a Windows Server 
2008 R2 box.

To explain why I'm attempting to do this instead of just using the Windows 
Installer provided by Python:

I needed to modify a _ssl.c file in the Python source code to deal a Mercurial 
that I'm trying to resolve.

Any help on why I'm hitting these errors would be appreciated.

Thank you.

Philip McAdams
Systems Administrator - NVM Solutions Group Systems Engineering Apps & 
Infrastructure
Desk: (916) 377-6156 Cell: (916) 534-0092 Pole: FM3-1-D7

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: GeoIP2 for retrieving city and region ?

[Νικόλας]

Στις 13/7/2013 9:17 μμ, ο/η Benjamin Kaplan έγραψε:

On Sat, Jul 13, 2013 at 10:43 AM, Νικόλας  wrote:

Στις 13/7/2013 7:54 μμ, ο/η Dennis Lee Bieber έγραψε:


 Are you paying for a fixed IP number? I suspect you are if you
were
running a world-accessible server.

 Obviously a fixed IP will be tied to a fixed connection and
thereby to
a fixed location which can be provided to a location database.

 But most of us have DHCP assigned IP numbers, which change
everytime we
reboot our connection (or even when the DHCP lease expires -- which may be
daily).



Same networking scheme for me too, dynamic that is.

Every time the DHCP lease expires or i reboot the router i get a new ip
address but every time the link i provided states accurately that my ip
address is from Thessaloníki and not Europe/Athens which is were my ISP
location is.

Not to mention that in facebook, no matter the way i'am joining, via
smartphone, tablet, laptop it always pinpoints my exact location.

But yes, i can understand your skepticism.
An ip address can move anywhere while remaining connected to the same ISP,
just like a networking device in the house, remains connected to the same
router while changing rooms or even floors, or even buildings.

But then how do you explain the fact that
http://www.maxmind.com/en/geoip_demo
pinpointed Thessaloníki and not Athens and for 2 friends of mine that use
the same ISP as me but live in different cities also accurately identified
their locations too?



It's not telling you where your ISP is headquartered. It's telling you
where the servers that you're connecting to are. In your case, you're
connecting to servers that your Athens-based ISP has in a Thessaloniki
datacenter. The only way to get an accurate location is to use
something other than IP- phones like to use a combination of their
GPS, a map of the cell phone towers, and a map of wi-fi hotspots (this
is one of the things that Google's StreetView cars log as they drive).


Actually that happens only for my ISP(FORTHnet).
For other ISPs all locations boil down just to Europe/Athens.
This happens to be because my ISP's network scheme is to assign blcoks 
of ip addresses per city in Greek area.


Same thing doesn't apply for others ISPs unfortunately here in Greece.

I have no idea how to implement the solution you proposed.
These are nice ideas we need to have a way of implement them within a 
script.


I have no way of grasping a map of cell towers of a  map of wi-fi hotspots.

--
What is now proved was at first only imagined!
--
http://mail.python.org/mailman/listinfo/python-list

Re: Question regarding building Python Windows installer

[MRAB]

On 15/07/2013 14:11, Mcadams, Philip W wrote:

I’m attempting to create a Python 64-bit Windows Installer.  Following
the instructions here: http://docs.python.org/2/distutils/builtdist.html
I’m to navigate to my Python folder and user command:

python setup.py build --plat-name=win-amd64 bdist_wininst

I get error: COMPILED_WTH_PYDEBUG = (‘—with-pydebug’ in
sysconfig.get_config_var(“CONFIG_ARGS”))

TypeError: argument of type ‘NoneType’ is not iterable

I also have tried:

setup.py build --plat-name=win-amd64 bdist_wininst

and get error:

File “setup.py”, line 263
Print “%-*s %-*s %-*s” % (longest, e, longet, f,

SyntaxError: invalid syntax


Does the line really start with "Print" (initial capital letter)?

Also, are you using Python 2 or Python 3? From the link above it looks
like Python 2.


I followed the instructions here:
http://docs.python.org/devguide/setup.html to create a PC build for
Windows which allows me to run a Python prompt.  Now I need to create a
Windows Installer to install this Python on a Windows Server 2008 R2 box.

To explain why I’m attempting to do this instead of just using the
Windows Installer provided by Python:

I needed to modify a _ssl.c file in the Python source code to deal a
Mercurial that I’m trying to resolve.

Any help on why I’m hitting these errors would be appreciated.



--
http://mail.python.org/mailman/listinfo/python-list

Re: Beginner - GUI devlopment in Tkinter - Any IDE with drag and drop feature like Visual Studio?

[Aseem Bansal]

@CM
Thanks for the suggestion. I'll take a look.
-- 
http://mail.python.org/mailman/listinfo/python-list

[ANN]:JSONStream

[Sol Toure]

I was trying to process a large file containing a number of distinct JSON
object as a stream, but I  couldn't find anything readily available to
that. (maybe I didn't search hard enough)
So I came up with this:
https://github.com/qrtz/JSONStream

I hope you find it useful too.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python - remote object protocols and security

[Burak Arslan]

On 07/15/13 13:51, Chris Angelico wrote:
> So the only bit you still need is: How do you transmit this across the
> network? Since it's now all just bytes, that's easy enough to do, eg
> with TCP. But that depends on the rest of your system, and is a quite
> separate question - and quite probably one you already have the answer
> to.

For Json, you need to have a way of delimiting messages -- to my
knowledge, Python's json library does not support parsing streams.

You can send the json document in the body of a Http POST, or a ZeroMQ
message, or in a UDP datagram (if you can guarantee it fits inside one)
or in a simple TCP-based encapsulation mechanism that e.g. prepends the
length of the message to the document.

e.g.

'\x00\x00\x00\x07{"a":1}'

As MessagePack already does this, you can send MessagePack documents via
an ordinary TCP socket and easily recover them on the other side of the
pipe.

>>> import msgpack; from StringIO import StringIO
>>> s = StringIO(msgpack.dumps({"a":1}) + msgpack.dumps({"b":2}))
>>> for doc in msgpack.Unpacker(s):
... print doc
...
{'a': 1}
{'b': 2}

This won't work with Json:

>>> import json; from StringIO import StringIO
>>> s = StringIO(json.dumps({"a":1}) + json.dumps({"b":2}))
>>> for doc in json.load(s): # or whatever ???
... print doc
...
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib64/python2.7/json/__init__.py", line 290, in load
**kw)
  File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/json/decoder.py", line 368, in decode
raise ValueError(errmsg("Extra data", s, end, len(s)))
ValueError: Extra data: line 1 column 9 - line 1 column 17 (char 8 - 16)

Note that this is a limitation of python's Json parser, not Json itself.

There seems to be a json.scanner module that *sounds* like it provides
this functionality,
but I couldn't find any documentation about it.

Alternatively, PyYaml can also parse streams. yaml.{dump,load}_all()
provide pickle-like unsafe (de)serialization support and
yaml.safe_{dump,load}_all provide msgpack-like safe-but-limited stream
parsing support.


also;

On 07/15/13 13:57, Chris Angelico wrote:
> But what I meant was that the [Json] protocol itself is designed with
> security restrictions in mind. It's designed not to fetch additional
> content from the network (as XML can),

Can you explain how parsing XML can fetch data from the network?


Best,
Burak
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Question regarding building Python Windows installer

[Zachary Ware]

On Mon, Jul 15, 2013 at 8:11 AM, Mcadams, Philip W
 wrote:
> I’m attempting to create a Python 64-bit Windows Installer.  Following the
> instructions here: http://docs.python.org/2/distutils/builtdist.html I’m to
> navigate to my Python folder and user command:
>
>
>
> python setup.py build --plat-name=win-amd64 bdist_wininst
>
>
>
> I get error: COMPILED_WTH_PYDEBUG = (‘—with-pydebug’ in
> sysconfig.get_config_var(“CONFIG_ARGS”))
>
> TypeError: argument of type ‘NoneType’ is not iterable
>
>
>
> I also have tried:
>
>
>
> setup.py build --plat-name=win-amd64 bdist_wininst
>
>
>
> and get error:
>
>
>
> File “setup.py”, line 263
>
> Print “%-*s %-*s %-*s” % (longest, e, longet, f,
>
>
>
> SyntaxError: invalid syntax
>
>
>
> I followed the instructions here: http://docs.python.org/devguide/setup.html
> to create a PC build for Windows which allows me to run a Python prompt.
> Now I need to create a Windows Installer to install this Python on a Windows
> Server 2008 R2 box.
>
>
>
> To explain why I’m attempting to do this instead of just using the Windows
> Installer provided by Python:
>
>
>
> I needed to modify a _ssl.c file in the Python source code to deal a
> Mercurial that I’m trying to resolve.
>
>
>
> Any help on why I’m hitting these errors would be appreciated.
>
>
>
> Thank you.
>
>

Most of your problem is that setup.py is just for the extension
modules of the standard library, not the interpreter and all.  The
rest of your problem is that setup.py is really just not
Windows-friendly, relying on some Makefile vars and Modules/Setup.

Do you really need to install it?  If you're still in the testing
phase, would it be enough to just copy the source tree (with compiled
interpreter) to the box you need it on?  You can still use Mercurial
with it, just stick the hg modules somewhere on PYTHONPATH.

If you really do need an installer, I would suggest trying out
Tools/buildbot/buildmsi.bat and see if you can get it to work for you.
 I will warn you, buildmsi.bat is not well maintained and it may take
quite a bit of effort to make it work.

HTH,

--
Zach
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python - remote object protocols and security

[Chris Angelico]

On Tue, Jul 16, 2013 at 1:42 AM, Burak Arslan
 wrote:
> On 07/15/13 13:57, Chris Angelico wrote:
>> But what I meant was that the [Json] protocol itself is designed with
>> security restrictions in mind. It's designed not to fetch additional
>> content from the network (as XML can),
>
> Can you explain how parsing XML can fetch data from the network?

I haven't looked into the details, but there was one among a list of
exploits that was being discussed a few months ago; it involved XML
schemas, I think, and quite a few generic XML parsers could be tricked
into fetching arbitrary documents. Whether this could be used for
anything more serious than a document-viewed receipt or a denial of
service (via latency) I don't know, but if nothing else, it's a vector
that JSON simply doesn't have.

ChrisA
-- 
http://mail.python.org/mailman/listinfo/python-list

Is this a bug?

[Jack Bates]

Hello,

Is the following code supposed to be an UnboundLocalError?
Currently it assigns the value 'bar' to the attribute baz.foo

   foo = 'bar'
   class baz:
  foo = foo
--
http://mail.python.org/mailman/listinfo/python-list

Re: Is this a bug?

[Zachary Ware]

On Mon, Jul 15, 2013 at 10:50 AM, Jack Bates  wrote:
> Hello,
>
> Is the following code supposed to be an UnboundLocalError?
> Currently it assigns the value 'bar' to the attribute baz.foo
>
>foo = 'bar'
>class baz:
>   foo = foo

No bug.  It's not an error because of differences in the way classes
and functions compile.

--
Zach
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Is this a bug?

[Joshua Landau]

On 15 July 2013 16:50, Jack Bates  wrote:
> Hello,
>
> Is the following code supposed to be an UnboundLocalError?
> Currently it assigns the value 'bar' to the attribute baz.foo
>
>foo = 'bar'
>class baz:
>   foo = foo

I have two responses because I'm not sure what you're saying. Take your pick.

--

If you're getting an error from that, then yes. The code works fine for me.

I assume you're just jumping to conclusions and not testing code and
were doing something like:

foo = "Outside foo"
def this_will_break():
foo
foo = "Inside foo"

and wondering why it breaks. Is this correct?

If so, it's because Python has consistent ideas about whether
something is local to its scope -- some other languages will think
that foo is nonlocal *until* they see the assignment. Python knows
that foo will be assigned later in the function, so is a local
variable the whole time. Being local, the outside version refers to *a
different name*.

If you want them to refer to the same name, use "nonlocal" or
"global", depending on circumstance.

foo = "Outside foo"
def this_will_break():
global foo
foo
foo = "Still outside foo (outside foo is changed too)"



ALTERNATIVELY

Are you saying you think that instead of doing what it does it should
raise an UnboundLocalError?

If so, then no. Assignments inside class bodies are special-cased in
Python. This is because all assignments refer to properties of "self"
on the LHS but external things too on the RHS. This is why you can do
"def x(): ..." instead of "def self.x(): ..." or some other weird
thing. There's also some extra special stuff that goes on.

In order to make this an UnboundLocalError, lots of dramatic and
unhelpful changes would have to take place, hence the current
behaviour. The current behaviour is useful, too.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Is this a bug?

[Chris Angelico]

On Tue, Jul 16, 2013 at 1:50 AM, Jack Bates  wrote:
> Hello,
>
> Is the following code supposed to be an UnboundLocalError?
> Currently it assigns the value 'bar' to the attribute baz.foo
>
>foo = 'bar'
>class baz:
>   foo = foo
> --
> http://mail.python.org/mailman/listinfo/python-list

Unless you're creating that class inside a function, it would be
NameError, not UnboundLocalError. Due to the way class scopes work,
it's actually possible and sometimes useful to do this, as it
"snapshots" the current referent of that name. It's like what happens
with default arguments to a function:

foo = 'bar'
def func(foo=foo):
return foo
foo = 'quux'

print(func())

The newly-defined name isn't "in scope" until its assignment is
complete; until then, the old name is fully in scope.

ChrisA
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python - remote object protocols and security

[Burak Arslan]

On 07/15/13 16:53, Chris Angelico wrote:
> I haven't looked into the details, but there was one among a list of
> exploits that was being discussed a few months ago; it involved XML
> schemas, I think, and quite a few generic XML parsers could be tricked
> into fetching arbitrary documents. Whether this could be used for
> anything more serious than a document-viewed receipt or a denial of
> service (via latency) I don't know, but if nothing else, it's a vector
> that JSON simply doesn't have. ChrisA 

I must have missed that exploit report, can you provide a link?

Parsing arbitrary xml documents and parsing xml schema documents and
applying xml schema semantics to these documents are two very different
operations.

Xml schemas are not "tricked" into fetching arbitrary documents,
xs:include and xs:import fetch external documents, it's a well-known
feature. If you don't want this, you should ship all of the schema
documents together and generate the schemas in a way to not include any
external references. So I'm surprised this was presented as a security
exploit.

Json schemas also have similar functionality:
http://json-schema.org/latest/json-schema-core.html#anchor30

"""
if canonical dereferencing is used, the implementation will dereference
this URI, and fetch the content at this URI;
"""

So I don't understand how you're so sure of yourself, but to me, it
seems like Json schemas have the same attack vectors.

Best regards,
Burak
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python - remote object protocols and security

[Chris Angelico]

On Tue, Jul 16, 2013 at 2:31 AM, Burak Arslan
 wrote:
> On 07/15/13 16:53, Chris Angelico wrote:
>> I haven't looked into the details, but there was one among a list of
>> exploits that was being discussed a few months ago; it involved XML
>> schemas, I think, and quite a few generic XML parsers could be tricked
>> into fetching arbitrary documents. Whether this could be used for
>> anything more serious than a document-viewed receipt or a denial of
>> service (via latency) I don't know, but if nothing else, it's a vector
>> that JSON simply doesn't have. ChrisA
>
> I must have missed that exploit report, can you provide a link?
>
> Parsing arbitrary xml documents and parsing xml schema documents and
> applying xml schema semantics to these documents are two very different
> operations.

I don't remember all the details; it isn't something I took particular
note of, as I don't work much with XML. It was something involving
either a schema declaration or a DTD or something of the sort, where
normally no external lookup is required but there's an HTTP URL in
there and it's possible to force that to be resolved.

> Xml schemas are not "tricked" into fetching arbitrary documents,
> xs:include and xs:import fetch external documents, it's a well-known
> feature. If you don't want this, you should ship all of the schema
> documents together and generate the schemas in a way to not include any
> external references. So I'm surprised this was presented as a security
> exploit.

It was something that parsing a basic XML document could trigger, and
in an environment where you wouldn't normally expect extra HTTP
requests to be going out, hence "tricked".

> Json schemas also have similar functionality:
> http://json-schema.org/latest/json-schema-core.html#anchor30
>
> """
> if canonical dereferencing is used, the implementation will dereference
> this URI, and fetch the content at this URI;
> """
>
> So I don't understand how you're so sure of yourself, but to me, it
> seems like Json schemas have the same attack vectors.

Yes, but normal JSON data doesn't include schema references. Normal
XML data can and often does.

ChrisA
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python - remote object protocols and security

[Irmen de Jong]

On 15-7-2013 13:17, Dave Angel wrote:
> On 07/15/2013 06:20 AM, Jean-Michel Pichavant wrote:
>> In text format... sorry for my previous html post
>>
>> Hello everyone,
>>
>> I'd like to exchange some simple python objects over the internet.
>> I initially planned to use Pyro, after reading
>> http://pythonhosted.org/Pyro4/security.html I'm still puzzled.

Hi, Pyro's author here.
I agree that this chapter of the manual can use some cleanup.
Is there anything in particular that you are puzzled about at this time?

>>
>> I don't mind encrypting data, if someone wants to sniff what I'm sending, 
>> he's welcome.
>>

I don't quite understand what you're saying in this sentence: is it okay if 
someone
eavesdrops on your unencrypted data stream?


>> What I think I need to care about, is malicious code injections. Because both
>> client/server will be in python, would someone capable of executing code by 
>> changing
>> one side python source ?

Pyro since version 4.20 uses a serialization format that is safe against 
arbitrary code
execution: https://pypi.python.org/pypi/serpent
That format only encodes and decodes Python literal expressions, and no 
arbitrary
objects are instantiated. You can also tell Pyro to use JSON (or marshal even), 
both of
which should be impervious to this type of attack/vulnerability as well.

The problem is with older Pyro versions, which allowed only Pickle to be used as
serialization format. It is pickle that causes the remote code execution 
vulnerability.
So as long as you don't explicitly tell Pyro (4.20+) to use pickle (a 
configuration
switch), you should be safe.

> I can't tell you if pyro, or any other particular one is safe.  

Pyro should be, since version 4.20 and provided you don't tell it to use 
pickle. See above.

> Note that DOS attacks are possible whatever encoding scheme you have. Make 
> sure that
> self-references within the data are well-defined (or impossible), and put 
> limits on size
> per transaction, and transactions per minute per legitimate user.

Pyro doesn't provide anything by itself to protect against this.


Cheers
Irmen de Jong

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Beginner - GUI devlopment in Tkinter - Any IDE with drag and drop feature like Visual Studio?

[asimjalis]

Take a look at kivy at http://kivy.org.

On Thursday, July 4, 2013 6:23:41 AM UTC-7, Aseem Bansal wrote:
> I want to start GUI development using Tkinter in Python 2.7.5.
> 
> 
> 
> I have been searching all over google but couldn't find any IDE that has 
> drag-and-drop feature for Python GUI development. Tried to ask on 
> stackoverflow 
> 
> 
> 
> (http://stackoverflow.com/questions/17439620/an-ide-with-drag-and-drop-feature-for-python-2-7-tkinter)
> 
> 
> 
> but couldn't get an answer there. So is there any IDE that can be used for 
> GUI developemnt and has drag-and-drop feature for Python GUI dev?
> 
> 
> 
> I came across somewhere that eclipse's pydev plugin can be used but couldn't 
> find anything on its website.
> 
> 
> 
> Any advice about this?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python - remote object protocols and security

[Irmen de Jong]

On 15-7-2013 18:57, Irmen de Jong wrote:

>> Note that DOS attacks are possible whatever encoding scheme you have. Make 
>> sure that
>> self-references within the data are well-defined (or impossible), and put 
>> limits on size
>> per transaction, and transactions per minute per legitimate user.
> 
> Pyro doesn't provide anything by itself to protect against this.

I'm sorry to follow up on myself, but there is actually one thing: Pyro's 
choice of
serializers (except pickle, again) don't allow self-references. So that type of 
DOS
attack (infinite recursion) is ruled out.


Irmen

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Ideal way to separate GUI and logic?

[asimjalis]

fron...@gmail.com wrote:
> So as a general idea, I should at the very least separate the GUI from the 
> program logic by defining the logic as a function, correct? And the next 
> level of separation is to define the logic as a class in one or more separate 
> files, and then import it to the file with the GUI, correct? 
> 
> My next question is, to what degree should I 'slice' my logic into functions? 
> How small or how large should one function be, as a rule of thumb? 

The way I do this is to write unit tests against the class and the functions 
(take a look at the unittest module). The functions methods (take a look at the 
unittest module). Each function should contain the smallest bit of testable 
logic.

Another way to think about this is that each function should contain the 
smallest piece of logic that you can describe as one action.

-
Asim Jalis
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python - remote object protocols and security

[Jean-Michel Pichavant]

- Original Message -
> On 15-7-2013 13:17, Dave Angel wrote:
> > On 07/15/2013 06:20 AM, Jean-Michel Pichavant wrote:
> >> In text format... sorry for my previous html post
> >>
> >> Hello everyone,
> >>
> >> I'd like to exchange some simple python objects over the internet.
> >> I initially planned to use Pyro, after reading
> >> http://pythonhosted.org/Pyro4/security.html I'm still puzzled.
> 
> Hi, Pyro's author here.
> I agree that this chapter of the manual can use some cleanup.
> Is there anything in particular that you are puzzled about at this
> time?

Nothing wrong with the manual, just my poor knowledge of security issues.

> >>
> >> I don't mind encrypting data, if someone wants to sniff what I'm
> >> sending, he's welcome.
> >>
> 
> I don't quite understand what you're saying in this sentence: is it
> okay if someone
> eavesdrops on your unencrypted data stream?

It's okay is someone eavesdrops, my English is as bad as my net code.

> Pyro since version 4.20 uses a serialization format that is safe
> against arbitrary code
> 
> 
> Cheers
> Irmen de Jong

Thanks for the clarifications (and writing Pyro), I'll make sure I'll be using 
4.20+.

JM


-- IMPORTANT NOTICE: 

The contents of this email and any attachments are confidential and may also be 
privileged. If you are not the intended recipient, please notify the sender 
immediately and do not disclose the contents to any other person, use it for 
any purpose, or store or copy the information in any medium. Thank you.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Tutorials on Jinja

[asimjalis]

On Wednesday, June 24, 2009 11:46:55 AM UTC-7, Saurabh wrote:
> Hi All,
> 
> I am trying to move my application on a MVC architecture and plan to
> use Jinja for the same. Can anyone provide me with few quick links
> that might help me to get started with Jinja?
> 
> Thanks,
> Saby

The documentation at http://jinja.pocoo.org/docs/templates is pretty good.

Also this link contains an end-to-end example which might be helpful.

https://gist.github.com/warren-runk/1317933

-
Asim
-- 
http://mail.python.org/mailman/listinfo/python-list

RE: Question regarding building Python Windows installer

[Mcadams, Philip W]

Thanks for the reply Zachery. We have decided to just use another solution. Out 
of curiosity though I wanted to clarification on your statement:

just stick the hg modules somewhere on PYTHONPATH.

Are you saying that I would just map hg modules i.e.: 
C:\Users\pwmcadam\Downloads\Python-2.7.4\Python-2.7.4\Modules to my environment 
variables in Windows.

Wasn't exactly following your comment.

Thank you.

Philip McAdams
Systems Administrator - NVM Solutions Group Systems Engineering Apps & 
Infrastructure
Desk: (916) 377-6156 Cell: (916) 534-0092 Pole: FM3-1-D7



-Original Message-
From: zachary.w...@gmail.com [mailto:zachary.w...@gmail.com] On Behalf Of 
Zachary Ware
Sent: Monday, July 15, 2013 8:47 AM
To: Mcadams, Philip W
Cc: python-list@python.org
Subject: Re: Question regarding building Python Windows installer

On Mon, Jul 15, 2013 at 8:11 AM, Mcadams, Philip W  
wrote:
> I’m attempting to create a Python 64-bit Windows Installer.  Following 
> the instructions here: 
> http://docs.python.org/2/distutils/builtdist.html I’m to navigate to my 
> Python folder and user command:
>
>
>
> python setup.py build --plat-name=win-amd64 bdist_wininst
>
>
>
> I get error: COMPILED_WTH_PYDEBUG = (‘—with-pydebug’ in
> sysconfig.get_config_var(“CONFIG_ARGS”))
>
> TypeError: argument of type ‘NoneType’ is not iterable
>
>
>
> I also have tried:
>
>
>
> setup.py build --plat-name=win-amd64 bdist_wininst
>
>
>
> and get error:
>
>
>
> File “setup.py”, line 263
>
> Print “%-*s %-*s %-*s” % (longest, e, longet, f,
>
>
>
> SyntaxError: invalid syntax
>
>
>
> I followed the instructions here: 
> http://docs.python.org/devguide/setup.html
> to create a PC build for Windows which allows me to run a Python prompt.
> Now I need to create a Windows Installer to install this Python on a 
> Windows Server 2008 R2 box.
>
>
>
> To explain why I’m attempting to do this instead of just using the 
> Windows Installer provided by Python:
>
>
>
> I needed to modify a _ssl.c file in the Python source code to deal a 
> Mercurial that I’m trying to resolve.
>
>
>
> Any help on why I’m hitting these errors would be appreciated.
>
>
>
> Thank you.
>
>

Most of your problem is that setup.py is just for the extension modules of the 
standard library, not the interpreter and all.  The rest of your problem is 
that setup.py is really just not Windows-friendly, relying on some Makefile 
vars and Modules/Setup.

Do you really need to install it?  If you're still in the testing phase, would 
it be enough to just copy the source tree (with compiled
interpreter) to the box you need it on?  You can still use Mercurial with it, 
just stick the hg modules somewhere on PYTHONPATH.

If you really do need an installer, I would suggest trying out 
Tools/buildbot/buildmsi.bat and see if you can get it to work for you.
 I will warn you, buildmsi.bat is not well maintained and it may take quite a 
bit of effort to make it work.

HTH,

--
Zach
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Question regarding building Python Windows installer

[Zachary Ware]

On Mon, Jul 15, 2013 at 1:02 PM, Mcadams, Philip W
 wrote:
> Thanks for the reply Zachery. We have decided to just use another solution. 
> Out of curiosity though I wanted to clarification on your statement:
>
> just stick the hg modules somewhere on PYTHONPATH.
>
> Are you saying that I would just map hg modules i.e.: 
> C:\Users\pwmcadam\Downloads\Python-2.7.4\Python-2.7.4\Modules to my 
> environment variables in Windows.
>
> Wasn't exactly following your comment.
>

Did I understand correctly that in your initial message you said you
needed an installer to try to resolve a Mercurial issue?  If not,
ignore that whole sentence of mine :)

Otherwise, I meant that you can just copy the mercurial and hgext
directories from \Lib\site-packages to \Lib\site-packages,
which would add that dir to the start of sys.path.  I did somewhat
misuse 'PYTHONPATH' in my earlier message.

--
Zach
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Understanding other people's code

[CM]

On Monday, July 15, 2013 6:02:30 AM UTC-4, Azureaus wrote:

> To be fair to who programmed it, most functions are commented and I can't 
> complain about the messiness of the code, It's actually very tidy. (I suppose 
> Python forcing it's formatting is another reason it's an easily readable 
> language!) Luckily not blanked import * were used otherwise I really would be 
> up the creek without a paddle. 

Oh, good!  OK, so then what you can think in terms of, in terms of a simple 
strategy for getting clear without any fancy tools:

Learn what each module is for.  In my own application programming, I don't just 
put random classes and functions in any old module--the modules have some order 
to them.  So, for example, one module may represent one panel in the 
application, or all the database stuff, or all the graphing stuff, or some 
other set of logic, or whatever.  One might be the main GUI frame.  Etc.  So 
I'd get a notebook or file and make notes for yourself about what each module 
is for, and the name.  Even tack a piece of paper above your workstation with 
the module names and a one line note about what they do, like:

MODULES:

Map_panel:  Displays a panel with the map of the city, with a few buttons.
Dbases:  Has all utility functions relevant to the database.
Utils:  Has a collection of utility functions to format time, i18n, etc.

Now, there's a cheat sheet.  So, if you come across a line in your code like:

pretty_time = Utils.GetPrettyTime(datetime)

You can quickly look at Utils module and read more about that function.

Does this approach make sense to at least clear the cobwebs?

-- 
http://mail.python.org/mailman/listinfo/python-list

RE: Question regarding building Python Windows installer

[Mcadams, Philip W]

Yes.  My goal was to create the installer to put the modified python on my 
Mercurial server.  So I could have effectively copied over the \Lib\site-packages on the server?  What I was trying to resolve was the 
issue with large Mercurial pushes.  I instead am using the IIS Crypto tool to 
resolve the issue.  I'd found a link that stated that modification to the 
_ssl.c module in Python could also fix the issue but that the python source 
would to be recompiled.  Since we are going with IISCrypto the Python change is 
no longer needed.  But out curiosity, and in case I ran into a situation where 
did indeed need to make a fix to Python I've wondered what's the best way to do 
that.  Hopefully this gives you a little insight on what I'm trying to do.  
Thanks for your replies.

Thank you.

Philip McAdams
Systems Administrator - NVM Solutions Group Systems Engineering Apps & 
Infrastructure
Desk: (916) 377-6156 Cell: (916) 534-0092 Pole: FM3-1-D7



-Original Message-
From: zachary.w...@gmail.com [mailto:zachary.w...@gmail.com] On Behalf Of 
Zachary Ware
Sent: Monday, July 15, 2013 12:07 PM
To: Mcadams, Philip W
Cc: python-list@python.org
Subject: Re: Question regarding building Python Windows installer

On Mon, Jul 15, 2013 at 1:02 PM, Mcadams, Philip W  
wrote:
> Thanks for the reply Zachery. We have decided to just use another solution. 
> Out of curiosity though I wanted to clarification on your statement:
>
> just stick the hg modules somewhere on PYTHONPATH.
>
> Are you saying that I would just map hg modules i.e.: 
> C:\Users\pwmcadam\Downloads\Python-2.7.4\Python-2.7.4\Modules to my 
> environment variables in Windows.
>
> Wasn't exactly following your comment.
>

Did I understand correctly that in your initial message you said you needed an 
installer to try to resolve a Mercurial issue?  If not, ignore that whole 
sentence of mine :)

Otherwise, I meant that you can just copy the mercurial and hgext directories 
from \Lib\site-packages to \Lib\site-packages, which would add that dir to the start of 
sys.path.  I did somewhat misuse 'PYTHONPATH' in my earlier message.

--
Zach
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: UTF-EBCDIC encoding?

[Kev Dwyer]

Joel Goldstick wrote:

> On Fri, Jul 12, 2013 at 3:12 PM, Skip Montanaro  wrote:
> 
>> > I can't help you.  I'm astonished.  Trying to imagine the work
>> environment
>> > where this technology would be necessary
>>
>> http://www.iseriespython.com/app/ispMain.py/Start?job=Home
>>
>> Skip
>>
> I remember the AS400 series.. although I never worked with one.  What kind
> of business still use that stuff? Is it for large corporation accounting,
> MIS stuff?
> 
> 

Some banks still run legacy systems on AS/400s, and I've seen them used for 
airline booking systems and retail POS.  



-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Question regarding building Python Windows installer

[Zachary Ware]

(Side note: Please avoid top-posting in future.  Bottom-posting keeps
context more clearly)

On Mon, Jul 15, 2013 at 2:27 PM, Mcadams, Philip W
 wrote:
> Yes.  My goal was to create the installer to put the modified python on my 
> Mercurial server.  So I could have effectively copied over the  with your compiled interpreter\Lib\site-packages to the  dir>\Lib\site-packages on the server?  What I was trying to resolve was the 
> issue with large Mercurial pushes.  I instead am using the IIS Crypto tool to 
> resolve the issue.  I'd found a link that stated that modification to the 
> _ssl.c module in Python could also fix the issue but that the python source 
> would to be recompiled.  Since we are going with IISCrypto the Python change 
> is no longer needed.  But out curiosity, and in case I ran into a situation 
> where did indeed need to make a fix to Python I've wondered what's the best 
> way to do that.  Hopefully this gives you a little insight on what I'm trying 
> to do.  Thanks for your replies.
>

Hmmm, not quite.  \Lib\site-packages would be empty.  I meant the other way
around, copying the installed site-packages dir into the source tree
to use mercurial from the source tree.  I think you'd also have to
copy hg and hg.bat from \Scripts as well, though.  You
might have been able to get away with just copying the newly compiled
_ssl.pyd from your source tree to \DLLs, but I can't
guarantee that.

I think the solution you've gone for is a much better solution in the
long run, though.  Building your own Python (on Windows) should
probably be a last resort.

--
Zach
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Dihedral

[Wayne Werner]

On Mon, 15 Jul 2013, Devyn Collier Johnson wrote:



On 07/15/2013 08:36 AM, Steven D'Aprano wrote:

On Mon, 15 Jul 2013 06:06:06 -0400, Devyn Collier Johnson wrote:


On 07/14/2013 02:17 PM, 8 Dihedral wrote:

[...]

Do we want volunteers to speed up
search operations in the string module in Python?

It would be nice if someone could speed it up.

Devyn,

8 Dihedral is our resident bot, not a human being. Nobody knows who
controls it, and why they are running it, but we are pretty certain that
it is a bot responding mechanically to keywords in people's posts.

It's a very clever bot, but still a bot. About one post in four is
meaningless jargon, the other three are relevant enough to fool people
into thinking that maybe it is a human being. It had me fooled for a long
time.



Wow! Our mailing list has a pet bot. I bet other mailing lists are so jealous 
of us. Who ever created Dihedral is a genius!


Artificial Intelligence developers put chatbots on mailing lists so that the 
program can learn. I use Python3 to program AI applications. If you see my 
Launchpad account, you will see my two AI projects - Neobot and Novabot. 
(https://launchpad.net/neobot Neo and Nova are still unstable) AI developers 
let their bots loose on the Internet to learn from people. Dihedral is 
learning from us. Dihedral only responses when it feels it has sufficient 
knowledge on the topic. Chatbots want to appear human. That is their goal. We 
should feel honored that Dihedral's botmaster feels that this mailinglist 
would benefit the development of Dihedral's knowledge.


Are *you* a bot? ~_^

That post felt surprisingly like Dihedral...

-W
--
http://mail.python.org/mailman/listinfo/python-list

Re: UTF-EBCDIC encoding?

[Wayne Werner]

On Mon, 15 Jul 2013, Kev Dwyer wrote:


Joel Goldstick wrote:


On Fri, Jul 12, 2013 at 3:12 PM, Skip Montanaro  wrote:


I can't help you.  I'm astonished.  Trying to imagine the work

environment

where this technology would be necessary


http://www.iseriespython.com/app/ispMain.py/Start?job=Home

Skip


I remember the AS400 series.. although I never worked with one.  What kind
of business still use that stuff? Is it for large corporation accounting,
MIS stuff?




Some banks still run legacy systems on AS/400s, and I've seen them used for
airline booking systems and retail POS.


Sadly there are many larger corporations that have oodles of legacy code 
on the Mainframe. We run z/OS and IBM DB2 here. In my off time I fiddle 
around with Python in an attempt to make life more enjoyable - and one of 
these forays has led me to attempt to unpack some packed data. Which is 
also an interesting (if not terribly useful) project.


-W
--
http://mail.python.org/mailman/listinfo/python-list

Re: Dihedral

[Fábio Santos]

> On 07/15/2013 08:36 AM, Steven D'Aprano wrote:
>>
>> Devyn,
>>
>> 8 Dihedral is our resident bot, not a human being. Nobody knows who
>> controls it, and why they are running it, but we are pretty certain that
>> it is a bot responding mechanically to keywords in people's posts.
>>
>> It's a very clever bot, but still a bot. About one post in four is
>> meaningless jargon, the other three are relevant enough to fool people
>> into thinking that maybe it is a human being. It had me fooled for a long
>> time.
>>

Does this mean he passes the Turing test?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Dihedral

[Chris Angelico]

On Tue, Jul 16, 2013 at 8:54 AM, Fábio Santos  wrote:
>
>> On 07/15/2013 08:36 AM, Steven D'Aprano wrote:
>>>
>>> Devyn,
>>>
>>> 8 Dihedral is our resident bot, not a human being. Nobody knows who
>>> controls it, and why they are running it, but we are pretty certain that
>>> it is a bot responding mechanically to keywords in people's posts.
>>>
>>> It's a very clever bot, but still a bot. About one post in four is
>>> meaningless jargon, the other three are relevant enough to fool people
>>> into thinking that maybe it is a human being. It had me fooled for a long
>>> time.
>>>
>
> Does this mean he passes the Turing test?

Yes, absolutely. The original Turing test was defined in terms of five
minutes of analysis, and Dihedral and jmf have clearly been
indistinguishably human across that approximate period.

ChrisA
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Ideal way to separate GUI and logic?

[fronagzen]

On Tuesday, July 16, 2013 1:06:30 AM UTC+8, asim...@gmail.com wrote:
> fron...@gmail.com wrote:
> 
> > So as a general idea, I should at the very least separate the GUI from the 
> > program logic by defining the logic as a function, correct? And the next 
> > level of separation is to define the logic as a class in one or more 
> > separate files, and then import it to the file with the GUI, correct? 
> 
> > 
> 
> > My next question is, to what degree should I 'slice' my logic into 
> > functions? How small or how large should one function be, as a rule of 
> > thumb? 
> 
> 
> 
> The way I do this is to write unit tests against the class and the functions 
> (take a look at the unittest module). The functions methods (take a look at 
> the unittest module). Each function should contain the smallest bit of 
> testable logic.
> 
> 
> 
> Another way to think about this is that each function should contain the 
> smallest piece of logic that you can describe as one action.
> 
> 
> 
> -
> 
> Asim Jalis

Again, thanks for all the responses. I'm curious, though, what exactly is the 
rationale for making functions so small? (I've heard that the function calling 
of Python has relatively high overhead?)
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Ideal way to separate GUI and logic?

[Owen Marshall]

On 2013-07-16, fronag...@gmail.com  wrote:
> On Tuesday, July 16, 2013 1:06:30 AM UTC+8, asim...@gmail.com wrote:
>> fron...@gmail.com wrote:
>>
>> > So as a general idea, I should at the very least separate the GUI
>> > from the program logic by defining the logic as a function,
>> > correct? And the next level of separation is to define the logic as
>> > a class in one or more separate files, and then import it to the
>> > file with the GUI, correct?
>>
>> >
>>
>> > My next question is, to what degree should I 'slice' my logic into
>> > functions? How small or how large should one function be, as a rule
>> > of thumb?
>>
>>
>>
>> The way I do this is to write unit tests against the class and the
>> functions (take a look at the unittest module). The functions methods
>> (take a look at the unittest module). Each function should contain
>> the smallest bit of testable logic.
>>
>>
>>
>> Another way to think about this is that each function should contain
>> the smallest piece of logic that you can describe as one action.
>>
>>
>>
>> -
>>
>> Asim Jalis
>
> Again, thanks for all the responses. I'm curious, though, what exactly
> is the rationale for making functions so small? (I've heard that the
> function calling of Python has relatively high overhead?)

Small functions are _always_ encouraged for every language. This is best
practice for everything, not just Python. Has nothing to do with
overhead.

--
-owen
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: UTF-EBCDIC encoding?

[Owen Marshall]

On 2013-07-12, Joel Goldstick  wrote:
> --047d7bdc8be492d67804e154c580 Content-Type: text/plain; charset=UTF-8
>
> On Fri, Jul 12, 2013 at 2:11 PM, Wayne Werner 
> wrote:
>
>> Is anyone aware of a UTF-EBCDIC[1] decoder?
>>
>> While Python does have a few EBCDIC dialects in the codecs, it does
>> not have the (relatively new?) UTF-EBCDIC one.
>>
>> Additionally, if anyone is aware of a Python tool that can unpack a
>> mainframe PDS file, that would also be worthwhile.
>>
>>
>> Thanks, Wayne
>>
>> [1]:
>> https://en.wikipedia.org/wiki/**UTF-EBCDIC
>> --
>> http://mail.python.org/**mailman/listinfo/python-list
>>
>
>
> I can't help you.  I'm astonished.  Trying to imagine the work
> environment where this technology would be necessary
>

Ask any poor shmuck who has ever had to consume files created by the
government -- especially stuff from the Social Security Administration
-- and they'll tell horror stories about EBCDIC.

Typically I've seen ``iconv'' used for this task, so I'd love to see a
native solution...

--
-owen
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: UTF-EBCDIC encoding?

[Roy Smith]

In article ,
 Owen Marshall  wrote:

> On 2013-07-12, Joel Goldstick  wrote:
> > --047d7bdc8be492d67804e154c580 Content-Type: text/plain; charset=UTF-8
> >
> > On Fri, Jul 12, 2013 at 2:11 PM, Wayne Werner 
> > wrote:
> >
> >> Is anyone aware of a UTF-EBCDIC[1] decoder?
> >>
> >> While Python does have a few EBCDIC dialects in the codecs, it does
> >> not have the (relatively new?) UTF-EBCDIC one.
> >>
> >> Additionally, if anyone is aware of a Python tool that can unpack a
> >> mainframe PDS file, that would also be worthwhile.
> >>
> >>
> >> Thanks, Wayne
> >>
> >> [1]:
> >> https://en.wikipedia.org/wiki/**UTF-EBCDIC >> F-EBCDIC>
> >> --
> >> http://mail.python.org/**mailman/listinfo/python-list >> g/mailman/listinfo/python-list>
> >>
> >
> >
> > I can't help you.  I'm astonished.  Trying to imagine the work
> > environment where this technology would be necessary
> >
> 
> Ask any poor shmuck who has ever had to consume files created by the
> government -- especially stuff from the Social Security Administration
> -- and they'll tell horror stories about EBCDIC.
> 
> Typically I've seen ``iconv'' used for this task, so I'd love to see a
> native solution...

Unix command-line:

dd conv=ascii

or perhaps

dd conv=ascii,unblock cbs=80 ibs=80 if=/dev/cr of=/dev/lp (*)

(*) My dd-fu has tarnished with age.  Forgive me if I've munged the 
incantation somewhat.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Ideal way to separate GUI and logic?

[Asim Jalis]

On Mon, Jul 15, 2013 at 5:25 PM,  wrote:

> Again, thanks for all the responses. I'm curious, though, what exactly is
> the rationale for making functions so small? (I've heard that the function
> calling of Python has relatively high overhead?)
>

There is a small overhead, but it makes the code easier to read and
understand. You can look at the function name and get and idea of _what_
the function is doing instead of having to figure out _how_ it is doing it.

Regarding optimization, after you have written your application if you see
performance issues you can surgically optimize the spots that have the
issues and leave most of the code untouched.

To quote Don Knuth, "premature optimization is the root of all evil". Also
the article at http://en.wikipedia.org/wiki/Program_optimization makes some
good points.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Understanding other people's code

[asimjalis]

On Friday, July 12, 2013 7:22:59 AM UTC-7, Azureaus wrote:
> Hi all,
> I've been asked to take over a project from someone else and to extend the 
> functionality of this. The project is written in Python which I haven't had 
> any real experience with (although I do really like it) so I've spent the 
> last week or two settling in, trying to get my head around Python and the way 
> in which this code works.

Here are some techniques I use in these situations.

1. Do a superficial scan of the code looking at names of classes, functions, 
variables, and speculate where the modification that I have to make will go. 
Chances are you don't need to understand the entire system to make your change.

2. Build some hypotheses about how the system works and use print statements or 
some other debugging technique to run the program and see if you get the result 
you expect.

3. Insert your code into a separate class and function and see if you can 
inject a call to your new code from the existing code so that it now works with 
the new functionality.

If you have to understand the details of some code, one approach is to try to 
summarize blocks of code with a single comment to wrap your mind around it.

Asim
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: GeoIP2 for retrieving city and region ?

[Michael Torrie]

On 07/15/2013 06:34 PM, Dennis Lee Bieber wrote:
>> I have no idea how to implement the solution you proposed.
>> These are nice ideas we need to have a way of implement them within a 
>> script.
>>
>> I have no way of grasping a map of cell towers of a  map of wi-fi hotspots.
>>
>   You don't... The phone company knows where their towers are, THEY do
> the triangulation based on signal strength from cell phones on their
> network, and they provide that position to the phone. The phone can then
> use that data to respond to applications running ON the phone that request
> location information using the phone's OS API (which is different for an
> Android phone vs Blackberry vs Apple).

I've posted a link to detailed information on this no less than three
times, yet Nikos has not read any of it, sadly.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Ideal way to separate GUI and logic?

[Chris Angelico]

On Tue, Jul 16, 2013 at 10:25 AM,   wrote:
> Again, thanks for all the responses. I'm curious, though, what exactly is the 
> rationale for making functions so small? (I've heard that the function 
> calling of Python has relatively high overhead?)

A function should be as long as it needs to be - neither longer nor
shorter. If you can name a function appropriately, and it's doing
exactly what its name suggests, it's the right length. This generally
produces short functions rather than long ones, but if the right
length for a function exceeds some arbitrary limit, let the function
be longer. For instance, I have a single function that's a bit over a
page in length, because it's one big switch block (this isn't in
Python, obviously), doing one thing fairly cleanly. Larger than that
would have to be called code smell, but there's certainly nothing
wrong with having the odd function here or there that's over Steven's
dozen-line estimate. There'll also be plenty of really short functions
- even one-liners.

The largest single function in any of my code, I think, is a gigantic
double-nested switch block in PHP .In any decent language, that would
be divided up not just into functions but into files, but PHP has some
stupid restrictions on its include directive that make that
impractical. So syntactically it's one massive function, but logically
it's about seven or eight separate sub-blocks, and the code is laid
out in those blocks. It's just that the technical nesting level never
actually hits zero in between them :) Have to confess, though, I've
had some fairly large functions in C++ (not as big as the
aforementioned, but still fairly large - what's the next one down from
gigantic, megantic? [1] would suggest so), which in some cases could
be split if I felt like putting in the time to do it.

ChrisA

[1] http://gatherer.wizards.com/Pages/Card/Details.aspx?multiverseid=370794
-- 
http://mail.python.org/mailman/listinfo/python-list

a little more explicative error message?

[Vito De Tullio]

Hi

I was writing a decorator and lost half an hour for a stupid bug in my code, 
but honestly the error the python interpreter returned to me doesn't 
helped...

$ python3
Python 3.3.0 (default, Feb 24 2013, 09:34:27)
[GCC 4.7.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from functools import wraps
>>> def dec(fun):
...  @wraps
...  def ret(*args, **kwargs):
...   return fun(*args, **kwargs)
...  return ret
...
>>> @dec
... def fun(): pass
...
>>> fun()
Traceback (most recent call last):
  File "", line 1, in 
TypeError: update_wrapper() missing 1 required positional argument: 
'wrapper'
>>>
$


Soo... at a first glance, no tricks... can you tell where is the error? :D



As I said, the error is totally mine, I just forgot to pass the function as 
parameter to wraps. But... what is "update_wrapper()"? and "wrapper"? There 
is no useful traceback or something... just... this.

Ok, the documentation clearly says:

This is a convenience function to simplify applying partial() to
update_wrapper().

So, again, shame on me... I just read carefully the doc *after* 20 minutes 
trying everything else...  still... I think should be useful if wraps() 
intercept this error saying something more explicit about the missing fun 
parameter...

-- 
ZeD

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Dihedral

[Tim Delaney]

On 16 July 2013 08:59, Chris Angelico  wrote:

> On Tue, Jul 16, 2013 at 8:54 AM, Fábio Santos 
> wrote:
> >
> >> On 07/15/2013 08:36 AM, Steven D'Aprano wrote:
> >>>
> >>> Devyn,
> >>>
> >>> 8 Dihedral is our resident bot, not a human being. Nobody knows who
> >>> controls it, and why they are running it, but we are pretty certain
> that
> >>> it is a bot responding mechanically to keywords in people's posts.
> >>>
> >>> It's a very clever bot, but still a bot. About one post in four is
> >>> meaningless jargon, the other three are relevant enough to fool people
> >>> into thinking that maybe it is a human being. It had me fooled for a
> long
> >>> time.
> >>>
> >
> > Does this mean he passes the Turing test?
>
> Yes, absolutely. The original Turing test was defined in terms of five
> minutes of analysis, and Dihedral and jmf have clearly been
> indistinguishably human across that approximate period.
>

The big difference between them is that the jmfbot does not appear to
evolve its routines in response to external sources - it seems to be stuck
in a closed feedback loop.

Tim Delaney
-- 
http://mail.python.org/mailman/listinfo/python-list