Re: Can anybody explain the '-' in a 2-D creation code?

[Gary Herron]

On 06/25/2015 06:07 PM, fl wrote:

Hi,

I read Ned's tutorial on Python. It is very interesting. On its last
example, I cannot understand the '_' in:



board=[[0]*8 for _ in range(8)]


I know  '_' is the precious answer, but it is still unclear what it is
in the above line. Can you explain it to me?


Thanks,


He uses _ to indicate a variable whose name and value he does not care 
about, but it *is* a valid variable name.


He could have used
  ... for i in range ...
or
  ... for unused_variable in range ...

This is a valid, though probably unclear, use of that same name:

>>> _ = 123
>>> print(_)
123
>>>


Gary Herron



--
https://mail.python.org/mailman/listinfo/python-list

Re: Can anybody explain the '-' in a 2-D creation code?

[Atnakus Arzah]

On Thu, Jun 25, 2015 at 06:07:30PM -0700, fl wrote:

Hi,

I read Ned's tutorial on Python. It is very interesting. On its last
example, I cannot understand the '_' in:



board=[[0]*8 for _ in range(8)]


I know  '_' is the precious answer, but it is still unclear what it is
in the above line. Can you explain it to me?


Its just a variable whose value isn't used in this specific expression, you 
could use 'x', 'y', or any other valid variable
name there and it should work.


--
Atnakus Arzah 

When in doubt, have a cookie!

--
https://mail.python.org/mailman/listinfo/python-list

Re: enumerate XML tags (keys that will become headers) along with text (values) and write to CSV in one row (as opposed to "stacked" values with one header)

[Denis McMahon]

On Thu, 25 Jun 2015 11:39:53 -0700, kbtyo wrote:

> My question can be found here:
> 
> http://stackoverflow.com/questions/31058100/enumerate-column-headers-in-
csv-that-belong-to-the-same-tag-key-in-python

I suggest you look on stack overflow for the answer.

You appear to have failed to comprehend (again) the fact that the xml 
data is an unordered list, and are trying to assign an order to it.

If the xml data was ordered, either each tag would be different, or each 
tag would have an attribute specifying a sequence number.

If the original data is ordered, then you need to go back to the code 
that creates the xml and make it add the ordering information to the xml 
(ideally as an attribute of the tags concerned) to specify the order.

You can not reliably and with any certainty of being accurate try and 
assign some sequence to the list of similarly named elements simply from 
their position in the list.

You don't know if the code that creates the xml is walking the array like 
this:

i = sizeof(arr);
while (i--) addXmlElement(arr[i]);

or like this:

for (i = 0; i < arr.len; i++) addXmlElement(arr[i]);

or even like this:

while (sizeof(arr)) addXmlElement(popArrElement(arr[sizeof(arr)/2]));

Granted the latter is unlikely and perverse, but still not wholly 
impossible. More perverse and less likely mechanisms are also available.

But my point remains - that if the ordering data is not passed into the 
xml, then you can not magically add it when you later use the xml.

-- 
Denis McMahon, denismfmcma...@gmail.com
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Pure Python Data Mangling or Encrypting

[Jon Ribbens]

On 2015-06-26, Chris Angelico  wrote:
> On Fri, Jun 26, 2015 at 1:26 AM, Jon Ribbens
> wrote:
>> Well, it means you need to send 256 times as much data, which is a
>> start. If you're instead using a 256-byte translation table then
>> an attack becomes utterly impractical.
>
> Utterly impractical? Maybe, if you attempt a pure brute-force approach
> - there are 256! possible translation tables, which is roughly e500
> attempts [1], and at roughly four a microsecond [2] that'd still take
> a ridiculously long time. But there are two gigantic optimizations you
> could do. Firstly, there are frequency-based attacks,

No, there aren't. As I already said, the attacker does not have the
ciphertext. He can't do anything related to frequency analysis.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: [ANN] PySWITCH 0.2

[Chris Warrick]

On 26 June 2015 at 04:42, Godson Gera  wrote:
>
> =
> pyswitch 0.2
> =
>
> PySWITCH 0.2 is released
>
> Please, note that PySWITCH 0.2 is not available on PyPI because of name 
> conflict

This is not a good idea. You should just change your name, or upload
under a slightly different name to PyPI. Everyone is on PyPI, everyone
expects to be able to type `pip install`, so please work on that.

> [snip]
> Project website: http://pyswitch.sf.net
> Download Page: http://pyswitch.sourceforge.net/pages/download.html

You are not going to get a lot of downloads with that. SourceForge
spreads malware, and lots of people avoid them nowadays.

Upload to PyPI and switch to GitHub, or you won’t be successful in this world.

-- 
Chris Warrick 
PGP: 5EAAEA16
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: [ANN] PySWITCH 0.2

[Chris Angelico]

On Fri, Jun 26, 2015 at 9:16 PM, Chris Warrick  wrote:
>> Project website: http://pyswitch.sf.net
>> Download Page: http://pyswitch.sourceforge.net/pages/download.html
>
> You are not going to get a lot of downloads with that. SourceForge
> spreads malware, and lots of people avoid them nowadays.
>
> Upload to PyPI and switch to GitHub, or you won’t be successful in this world.

Well, GitHub isn't the only option by any means, but I agree that SF
isn't going to inspire confidence.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Pure Python Data Mangling or Encrypting

[Steven D'Aprano]

On Fri, 26 Jun 2015 11:01 am, Ian Kelly wrote:

> You're making the same mistake that Steven did in misunderstanding the
> threat model.

I don't think I'm misunderstanding the threat, I think I'm pointing out a
threat which the OP is hoping to just ignore.

In an earlier post, I suggested that the threat model should involve at
least *three* different attacks, apart from the usual man-in-the-model
attacks of data in transit.

One is that the attacker is the person sending the data. E.g. I want to send
a nasty payload (say, malware, or an offensive image). Another is that the
attacker is the recipient of the file, who wants to read the sender's data.

As far as I can tell, the OP's plan to defend the sender's privacy is to
dump responsibility for encrypting the files in the sender's lap. As far as
I'm concerned, perhaps as many as one user in 2 will pre-encrypt their
files. (Early adopters will be unrepresentative of the eventual user base
of this system. If this takes off, the user base will likely end up
dominated by people who think that "qwerty" is the epitome of unguessable
passwords.)

Users just don't use crypto unless their applications do it for them.

My opinion is that the application ought to do so, and not expect Aunt
Tillie to learn how to correctly use encryption software before uploading
her files. 

http://www.catb.org/jargon/html/A/Aunt-Tillie.html

It is the OP's prerogative to disagree, of course, but to me, if the OP's
app doesn't use strong crypto to encrypt users' data, that's tantamount to
saying they don't care about their users' data privacy. Using a
monoalphabetic substitution cipher to obfuscate the data is not strong
crypto.


> The goal isn't to prevent the attacker from working out 
> the key for a file that has already been obfuscated. Any real data
> that might be exposed by a vulnerability in the server is presumed to
> have already been strongly encrypted by the user.

I think that's a ridiculously unrealistic presumption, unless your user-base
is entirely taken from a very small subset of security savvy and
pedantically careful users.


> The goal is to prevent the attacker from guessing a key that hasn't
> even been generated yet, which could be exploited to engineer the
> obfuscated content into something malicious.

They don't need to predict the key exactly. If they can predict that the key
will be, lets say, one of these thousand values, then they can generate one
thousand files and upload them. One of them will match the key, and there's
your exploit. That's one attack.

A second attack is to force the key. The attacker controls the machine the
application is running on, they control /dev/urandom and can feed your app
whatever not-so-random numbers they like, so potentially they can force the
app to use the key of their choosing. Then they don't need 1000 files, they
just need one.

That's two. Does anyone think that I've thought of all the possible attacks?

(Well, hypothetical attacks. I acknowledge that I don't know the
application, and cannot be sure that it *actually is* vulnerable to these
attacks.)

The problem here is that a monoalphabetic substitution cipher is not
resistant to preimage attacks. Your only defence is that the key is
unknown. If the attacker can force the key, or predict the key, or guess a
small range of keys, they can exploit your weak cipher.

(Technically, "preimage attack" is usually used to refer to attacks on hash
functions. I'm not sure if the same name is used for attacks on ciphers.)

https://en.wikipedia.org/wiki/Preimage_attack

With a strong crypto cipher, there are no known preimage attacks. Even if
the attacker knows exactly what key you are using, they cannot predict what
preimage they need to supply in order to generate the malicious payload
they want after encryption. (As far as I know.)

That is the critical issue right there. The sort of simple monoalphabetic
substitution cipher using bytes.translate that the OP is using is
vulnerable to preimage attacks. Strong crypto is not.


> There are no 
> frequency-based attacks possible here, because you can't do frequency
> analysis on the result of a key that hasn't even been generated yet.

Frequency-based attacks apply to a different threat. I'm referring to at
least two different attacks here, with different attackers and different
victims. Don't mix them up.


> Assuming that you have no attack on the key generation itself, the

Not a safe assumption!


> best you can do is send a file deobfuscated with a random key and hope
> that the recipient randomly chooses the same key; the odds of that
> happening are 1 in 256!.

It's easy to come up with attacks which are no better than brute force. It's
the attacks which are better than brute force that you have to watch out
for.


-- 
Steven

-- 
https://mail.python.org/mailman/listinfo/python-list

how to replace some methods with instrumented ones

[georgeryoung]

[python 2.7, linux]
I have a python app. I cannot modify the file.  But I can import it and mess 
with it.  I need to perform brief tasks before and after some of the member 
functions.  
I'd like to do this in as clear and maintainable way as possible (no third 
party imports).  Here's what I have now(which works):

from newvbr import *  #get all the stuff from the newvbr app.

def perf_time(tag):  # I do a little more than this, but you get the idea.
print >>sys.stderr, datetime.datetime.now().isoformat() + tag

def perf(tag):
def decorator(fn):
def wrapper(*args):
perf_time('start %s' % tag)
ret = fn(*args)
perf_time('end %s' % tag)
return ret
return wrapper
return decorator


I use this on various members of various classes in the newvbr app:

old_database_snapshot = VerticaSession.database_snapshot
@perf('db_snap')
def new_database_snapshot(self, name):
old_database_snapshot(self, name)
VerticaSession.database_snapshot = new_database_snapshot

old_object_snapshot = VerticaSession.object_snapshot
@perf('obj_snap')
def new_object_snapshot(self, name):
old_object_snapshot(self, name)
VerticaSession.object_snapshot = new_object_snapshot

main()  #do the usual stuff of the imported app, but with annotations.


This is still pretty laborious.  How can I make something roughly like:

   super_tagger(VerticaSession.database_snapshot) 

that does all the above?  I need to tag a bunch of function members like this.  
I don't insist on using decorators, it just looked like that kind of problem.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: HOPE: A Python just-in-time compiler for astrophysical computations

[Michael Torrie]

On 06/23/2015 10:53 AM, Laurent Pointal wrote:
> Mark Lawrence wrote:
> 
>> Another beasty I've just stumbled across which you may find interesting
>> http://www.sciencedirect.com/science/article/pii/S2213133714000687
>  
> Why use a JIT complation when you could use some C++ generation then 
> compilation as Python module, like with Pythran ?
> 
> https://github.com/serge-sans-paille/pythran
> 
> For heavy computing, you may loose some time before running computation, to 
> have (generate) the right tool, then do the job with real bonus.

Not really. I suspect Hope caches the compiled subroutine so only the
very first running of the function on the very first execution of the
pthon program will be slow.  After that, the program will run quickly
each time.  It's a pretty good idea.

I've never heard of pythran; I'll have to check it out and see how it
compares to the ever-growing crop of Python dialect compilers.  Cython
could be used in a similar way as you suggest, and in PyPy we have the
RPython dialect that compiles to C.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: how to replace some methods with instrumented ones

[Peter Otten]

georgeryo...@gmail.com wrote:

> [python 2.7, linux]
> I have a python app. I cannot modify the file.  But I can import it and
> mess with it.  I need to perform brief tasks before and after some of the
> member functions.
> I'd like to do this in as clear and maintainable way as possible (no third
> party imports).  Here's what I have now(which works):
> 
> from newvbr import *  #get all the stuff from the newvbr app.
> 
> def perf_time(tag):  # I do a little more than this, but you get the idea.
> print >>sys.stderr, datetime.datetime.now().isoformat() + tag
> 
> def perf(tag):
> def decorator(fn):
> def wrapper(*args):
> perf_time('start %s' % tag)
> ret = fn(*args)
> perf_time('end %s' % tag)
> return ret
> return wrapper
> return decorator
> 
> 
> I use this on various members of various classes in the newvbr app:
> 
> old_database_snapshot = VerticaSession.database_snapshot
> @perf('db_snap')
> def new_database_snapshot(self, name):
> old_database_snapshot(self, name)
> VerticaSession.database_snapshot = new_database_snapshot
> 
> old_object_snapshot = VerticaSession.object_snapshot
> @perf('obj_snap')
> def new_object_snapshot(self, name):
> old_object_snapshot(self, name)
> VerticaSession.object_snapshot = new_object_snapshot
> 
> main()  #do the usual stuff of the imported app, but with annotations.
> 
> 
> This is still pretty laborious.  How can I make something roughly like:
> 
>super_tagger(VerticaSession.database_snapshot)
> 
> that does all the above?  I need to tag a bunch of function members like
> this.  I don't insist on using decorators, it just looked like that kind
> of problem.

Remember that

@foo
def bar(...):
   ...

is just syntactic sugar for

def bar(...): ...
bar = foo(bar)

In your case the latter form is more convenient, so instead of

> old_object_snapshot = VerticaSession.object_snapshot
> @perf('obj_snap')
> def new_object_snapshot(self, name):
> old_object_snapshot(self, name)
> VerticaSession.object_snapshot = new_object_snapshot

use

VerticalSession.object_snapshot = 
perf("obj_snap")(VerticalSession.object_snapshot)

If you like you can write a helper function along these lines

def wrap_attr(obj, attrname, name=None):
if name is None: 
name = attrname
setattr(obj, attrname
perf(name)(getattr(obj, attrname)))

wrap_attr(VerticalSession, "object_snapshot")

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: how to replace some methods with instrumented ones

[MRAB]

On 2015-06-26 18:12, georgeryo...@gmail.com wrote:

[python 2.7, linux]
I have a python app. I cannot modify the file.  But I can import it and mess 
with it.  I need to perform brief tasks before and after some of the member 
functions.
I'd like to do this in as clear and maintainable way as possible (no third 
party imports).  Here's what I have now(which works):

from newvbr import *  #get all the stuff from the newvbr app.

def perf_time(tag):  # I do a little more than this, but you get the idea.
 print >>sys.stderr, datetime.datetime.now().isoformat() + tag

def perf(tag):
 def decorator(fn):
   a  def wrapper(*args):
 perf_time('start %s' % tag)
 ret = fn(*args)
 perf_time('end %s' % tag)
 return ret
 return wrapper
 return decorator


I use this on various members of various classes in the newvbr app:

old_database_snapshot = VerticaSession.database_snapshot
@perf('db_snap')
def new_database_snapshot(self, name):
 old_database_snapshot(self, name)
VerticaSession.database_snapshot = new_database_snapshot

old_object_snapshot = VerticaSession.object_snapshot
@perf('obj_snap')
def new_object_snapshot(self, name):
 old_object_snapshot(self, name)
VerticaSession.object_snapshot = new_object_snapshot

main()  #do the usual stuff of the imported app, but with annotations.


This is still pretty laborious.  How can I make something roughly like:

super_tagger(VerticaSession.database_snapshot)

that does all the above?  I need to tag a bunch of function members like this.  
I don't insist on using decorators, it just looked like that kind of problem.


You could do something like this:

def super_tagger(module, function_name, tag):
original_function = getattr(module, function_name)

@perf(tag)
def wrapper(*args, **kwargs):
return original_function(*args, **kwargs)

setattr(module, function_name, wrapper)

and then:

super_tagger(VerticaSession, 'database_snapshot', 'db'snap')


It might be cleaner to merge the functionality of 'perf' into it:

def super_tagger(module, function_name, tag):
original_function = getattr(module, function_name)

def wrapper(*args, **kwargs):
perf_time('start %s' % tag)
result = original_function(*args, **kwargs)
perf_time('end %s' % tag)
return result

setattr(module, function_name, wrapper)

--
https://mail.python.org/mailman/listinfo/python-list

Re: windows and file names > 256 bytes

[Albert-Jan Roskam]

On Thu, 25 Jun 2015 14:37:55 +0100, Tim Golden wrote:

> On 25/06/2015 14:35, Michael Torrie wrote:
>> On 06/25/2015 06:34 AM, Tim Golden wrote:
>>> On 25/06/2015 13:04, Joonas Liik wrote:
 It sounds to me more like it is possible to use long file names on
 windows but it is a pain and in python, on windows it is basically
 impossible.
>>>
>>> Certainly not impossible: you could write your own wrapper function:
>>>
>>> def extended_path(p):
>>> return r"\\?\%s" % os.path.abspath(p)
>>>
>>> where you knew that there was a possibility of long paths and that an
>>> absolute path would work.
>> 
>> The OP mentions that even when he manually supplies extended paths,
>> os.mkdir, os.getsize, and shutil.rmtree return errors for him in Python
>> 2.7.  So there's more to this problem.
>> 
>> 
> He's probably not passing unicode strings: the extended path only works
> for unicode string. For 3.x that's what you do by default.

Hi all,

Thanks for your replies. I've been messing with this a bit more. I 
created a little test script (see below). However, this only works with 
drive letters, not with UNC paths. I tried using os.chdir, DOS pushd, 
subst, net use but they all don't seem to work with with long paths. 
I finally managed to remove an absurdly long dir with shutil.rmtree, 
after changing sys.setrecursionlimit. But my main goal was to get the 
file size (and, actually, also the file owner) of a long file name on XP.

import os
import shutil
import sys
 
# create an insanely long directory tree
p = os.getenv("TEMP")
#p = ur"\\server\share\blah\temp"
tmpdir = p
os.chdir(tmpdir)
for i in xrange(1000):
tmpdir = os.path.join(tmpdir, "sub")
os.mkdir("?\\" + tmpdir)
#os.mkdir(u"?\\UNC" + tmpdir[1:])
 
# write a file to it
deep = "?\\" + os.path.join(tmpdir, "deep.txt")
assert os.path.exists(deep)
with open(deep, "w") as f:
f.write("Deep!\r\n")
 
# try if the file size can be determined (requires special \\?\ notation)
print " %d bytes" % os.path.getsize(deep)
 
# now delete the whole directory and its contents.
path = "?\\" + os.path.join(p, "sub")
path = path.decode(sys.getfilesystemencoding())
sys.setrecursionlimit(10 ** 7)  # net use, pushd, subst will not work
shutil.rmtree(path)

Any feedback is welcome. I will post the solution somewhere so somebody 
else will be spared this nuisance. :-)

Regards,
Albert-Jan


---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: windows and file names > 256 bytes

[Albert-Jan Roskam]

 

> import os import shutil import sys
>  
> # create an insanely long directory tree p = os.getenv("TEMP")
> #p = ur"\\server\share\blah\temp"
> tmpdir = p os.chdir(tmpdir)
> for i in xrange(1000):
> tmpdir = os.path.join(tmpdir, "sub") os.mkdir("?\\" + tmpdir)
> #os.mkdir(u"?\\UNC" + tmpdir[1:])
>  
> # write a file to it deep = "?\\" + os.path.join(tmpdir, "deep.txt")
> assert os.path.exists(deep)

sorry, this "assert" should of course follow 'with open(..' 

---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Pure Python Data Mangling or Encrypting

[Randall Smith]

On 06/26/2015 12:06 PM, Steven D'Aprano wrote:

On Fri, 26 Jun 2015 11:01 am, Ian Kelly wrote:


You're making the same mistake that Steven did in misunderstanding the
threat model.


I don't think I'm misunderstanding the threat, I think I'm pointing out a
threat which the OP is hoping to just ignore.


I'm not hoping to ignore anything.  I didn't explain the entire system, 
as it was not necessary to find a solution to the problem at hand.  But 
since you want to make negative assumptions about what I didn't tell 
you, I'll gladly address your accusations of negligence.




In an earlier post, I suggested that the threat model should involve at
least *three* different attacks, apart from the usual man-in-the-model
attacks of data in transit.


All communication is secured using TLS and authentication handled by 
X.509 certificates.  This prevents man in the middle attacks. 
Certificates are signed by CAs I control.




One is that the attacker is the person sending the data. E.g. I want to send
a nasty payload (say, malware, or an offensive image). Another is that the
attacker is the recipient of the file, who wants to read the sender's data.


The only person who can read a file is the owner.   AES encryption is 
built into the client software.  The only way data can be uploaded 
unencrypted is if encryption is intentionally disabled.




As far as I can tell, the OP's plan to defend the sender's privacy is to
dump responsibility for encrypting the files in the sender's lap. As far as
I'm concerned, perhaps as many as one user in 2 will pre-encrypt their
files. (Early adopters will be unrepresentative of the eventual user base
of this system. If this takes off, the user base will likely end up
dominated by people who think that "qwerty" is the epitome of unguessable
passwords.)


Making assumptions again.  See above.  The client software encrypts by 
default.  You're also assuming there is no password strength checking.




Users just don't use crypto unless their applications do it for them.


And it does.



My opinion is that the application ought to do so, and not expect Aunt
Tillie to learn how to correctly use encryption software before uploading
her files.

http://www.catb.org/jargon/html/A/Aunt-Tillie.html

It is the OP's prerogative to disagree, of course, but to me, if the OP's
app doesn't use strong crypto to encrypt users' data, that's tantamount to
saying they don't care about their users' data privacy. Using a
monoalphabetic substitution cipher to obfuscate the data is not strong
crypto.


You've gone on a rampage about nothing.  My original description said 
the client was supposed to encrypt the data, but you want to assume the 
opposite for some unknown reason.






The goal isn't to prevent the attacker from working out
the key for a file that has already been obfuscated. Any real data
that might be exposed by a vulnerability in the server is presumed to
have already been strongly encrypted by the user.


I think that's a ridiculously unrealistic presumption, unless your user-base
is entirely taken from a very small subset of security savvy and
pedantically careful users.


The difference is he's not assuming I'm a moron.  He's giving me the 
benefit of the doubt.  That plus I actually said, "data senders are 
supposed to encrypt data".


In a networked system, you can't make assumptions about what the other 
peers are doing.  You have to handle what comes across the wire.  You 
also have to consider that you may come under attack.  That's what this 
is about.






The goal is to prevent the attacker from guessing a key that hasn't
even been generated yet, which could be exploited to engineer the
obfuscated content into something malicious.


They don't need to predict the key exactly. If they can predict that the key
will be, lets say, one of these thousand values, then they can generate one
thousand files and upload them. One of them will match the key, and there's
your exploit. That's one attack.


Thousand Values ???  Isn't it 256!, which is just freaking huge!  import 
math; math.factorial(256)




A second attack is to force the key. The attacker controls the machine the
application is running on, they control /dev/urandom and can feed your app
whatever not-so-random numbers they like, so potentially they can force the
app to use the key of their choosing. Then they don't need 1000 files, they
just need one.



If the attacker controlled the machine the app was on, why would it fool 
with /dev/urandom?  I think he'd just plant the files he wanted to plant 
and be done.  This is non-nonsensical anyway.



That's two. Does anyone think that I've thought of all the possible attacks?

(Well, hypothetical attacks. I acknowledge that I don't know the
application, and cannot be sure that it *actually is* vulnerable to these
attacks.)

The problem here is that a monoalphabetic substitution cipher is not
resistant to preimage attacks. Your only defence is that the key is
unknown. I

Re: Pure Python Data Mangling or Encrypting

[Johannes Bauer]

On 26.06.2015 22:09, Randall Smith wrote:

> You've gone on a rampage about nothing.  My original description said
> the client was supposed to encrypt the data, but you want to assume the
> opposite for some unknown reason.

While you seem to think that Steven is rampaging about nothing, he does
have a fair point: You consistently were vague about wheter you want to
have encryption, authentication or obfuscation of data. This suggests
that you may not be so sure yourself what it is you actually want.

All Steven is doing is pointing out that people do good crypto for a
reason. It's 2015 and we're still discussion "substitution ciphers",
really? Good crypto is available, it's fast, it has awesome
cryptanalysis. All Steven is pointing out is that when ten crypto-laymen
meet in a Python newsgroup and think they have invented a soooper secure
scheme, it may still be complete and utter crap. Just not everone can
see it.

You always play around with the 256! which would be a ridiculously high
security margin (1684 bits of security, w!). You totally ignore that
the system can be broken in a linear fashion. I don't need to know all
256 characters to do damage, sometimes even a handful will already give
me part of what I need and the option to crack more and more. This is
something that would ultimately and instantly disqualify your
"crypto"system as utterly insecure.

Nobody assumes you're a moron. But it's safe to assume that you're a
crypto layman, because only laymen have no clue on how difficult it is
to get cryptography even remotely right. Everyone who knows the trade
uses proven constructions not because it's inconvenient, but because
it's one of the very few ways to achieve a secure system.

That said, for your solution this type of obfuscation may be fine. And
chances are that nobody will ever notice. But don't claim you weren't
warned about the abyss when you designed your solution and people break
this stuff. Because then you might *look* like a moron (even if you're
not), since the first question people will ask will be: "Why? Why on
earth?" It's a blatantly obvious bad idea(tm).

That people in 2015 actually defend inventing a substitution-cipher
"crypto"system sends literally shivers down my spine.

Cheers,
Johannes

-- 
>> Wo hattest Du das Beben nochmal GENAU vorhergesagt?
> Zumindest nicht öffentlich!
Ah, der neueste und bis heute genialste Streich unsere großen
Kosmologen: Die Geheim-Vorhersage.
 - Karl Kaos über Rüdiger Thomas in dsa 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Pure Python Data Mangling or Encrypting

[Johannes Bauer]

On 26.06.2015 22:09, Randall Smith wrote:

> And that's why we're having this discussion.  Do you know of an attack
> in which you can control the output (say at least 100 consecutive bytes)
> for data which goes through a 256 byte translation table, chosen
> randomly from 256! permutations after the data is sent.  If you do, I'm
> all ears!  But at this point you're just setting up straw men and
> knocking them down.

Oh and I wanted to comment on this as well, but sent my reply too soon.

You misunderstand. This is now how it works, this is not how any of this
works. Steven does not *at all* have to prove to you your system is
breakable or show actual attacks. YOU have to prove that your system is
secure. Either analytically or you wait until you have peer review and
cryptanalysis by actual experts.

It's *very* easy to set up a badly flawed obfuscation system that can't
be broken by laymen in a Python newsgroup and which appers to be secure.
This does not imply one bit that it is even remotely secure.

Cheers,
Johannes

-- 
>> Wo hattest Du das Beben nochmal GENAU vorhergesagt?
> Zumindest nicht öffentlich!
Ah, der neueste und bis heute genialste Streich unsere großen
Kosmologen: Die Geheim-Vorhersage.
 - Karl Kaos über Rüdiger Thomas in dsa 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Pure Python Data Mangling or Encrypting

[Jon Ribbens]

On 2015-06-26, Johannes Bauer  wrote:
> On 26.06.2015 22:09, Randall Smith wrote:
>> You've gone on a rampage about nothing.  My original description said
>> the client was supposed to encrypt the data, but you want to assume the
>> opposite for some unknown reason.
>
> While you seem to think that Steven is rampaging about nothing, he does
> have a fair point: You consistently were vague about wheter you want to
> have encryption, authentication or obfuscation of data. This suggests
> that you may not be so sure yourself what it is you actually want.

He hasn't been vague, you and Steven just haven't been paying
attention.

> You always play around with the 256! which would be a ridiculously high
> security margin (1684 bits of security, w!). You totally ignore that
> the system can be broken in a linear fashion.

No, it can't, because the attacker does not have access to the
ciphertext.

> Nobody assumes you're a moron. But it's safe to assume that you're a
> crypto layman, because only laymen have no clue on how difficult it is
> to get cryptography even remotely right.

Amateur crypto is indeed a bad idea. But what you're still not getting
is that what he's doing here *isn't crypto*. He's just trying to avoid
letting third parties write completely arbitrary data to the disk. You
know what would be a perfectly good solution to his problem? Base 64
encoding. That would solve the issue pretty much completely, the only
reason it's not an ideal solution is that it of course increases the
size of the data.

> That people in 2015 actually defend inventing a substitution-cipher
> "crypto"system sends literally shivers down my spine.

Nobody is defending such a thing, you just haven't understood what
problem is being solved here.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Pure Python Data Mangling or Encrypting

[Mark Lawrence]

On 26/06/2015 22:29, Jon Ribbens wrote:

On 2015-06-26, Johannes Bauer  wrote:

On 26.06.2015 22:09, Randall Smith wrote:

You've gone on a rampage about nothing.  My original description said
the client was supposed to encrypt the data, but you want to assume the
opposite for some unknown reason.


While you seem to think that Steven is rampaging about nothing, he does
have a fair point: You consistently were vague about wheter you want to
have encryption, authentication or obfuscation of data. This suggests
that you may not be so sure yourself what it is you actually want.


He hasn't been vague, you and Steven just haven't been paying
attention.


You always play around with the 256! which would be a ridiculously high
security margin (1684 bits of security, w!). You totally ignore that
the system can be broken in a linear fashion.


No, it can't, because the attacker does not have access to the
ciphertext.


Nobody assumes you're a moron. But it's safe to assume that you're a
crypto layman, because only laymen have no clue on how difficult it is
to get cryptography even remotely right.


Amateur crypto is indeed a bad idea. But what you're still not getting
is that what he's doing here *isn't crypto*. He's just trying to avoid
letting third parties write completely arbitrary data to the disk. You
know what would be a perfectly good solution to his problem? Base 64
encoding. That would solve the issue pretty much completely, the only
reason it's not an ideal solution is that it of course increases the
size of the data.


That people in 2015 actually defend inventing a substitution-cipher
"crypto"system sends literally shivers down my spine.


Nobody is defending such a thing, you just haven't understood what
problem is being solved here.



To be perfectly blunt I gave up days ago trying to follow what was being 
said, just too many words from all angles and too few diagrams for me to 
follow.  I sincerely hope it doesn't end in tears.


--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

--
https://mail.python.org/mailman/listinfo/python-list

Re: Pure Python Data Mangling or Encrypting

[Ian Kelly]

On Fri, Jun 26, 2015 at 3:07 PM, Johannes Bauer  wrote:
> That people in 2015 actually defend inventing a substitution-cipher
> "crypto"system sends literally shivers down my spine.

I think that the people defending this have been reasonably consistent
about using the word "obfuscation", not "crypto".
-- 
https://mail.python.org/mailman/listinfo/python-list

Why pay DICE When TheGongzuo.com is !! FREE !!

[trentonwesley10]

Greetings!
You been Invited as a Beta User for TheGongzuo.com ( Absolutely Extended Trial).
We bring to you TheGongzuo.com, Top notch highly talented IT (Information 
Technology / Software Industry) skilled Professional Candidates, where you can 
find more than a resume.

For a NO COST TRIAL all you have to do is:
*   Register & Post  only 1 Active Job.
*   Start interacting with more 100,000 IT professionals and 
Innovators/Ideators.

Before you get started , there are few things you should know,
1.   We are getting better  :- We will be updating our site with new 
material daily basis and we welcome your feedback on newly added material. 
Please send us your feedback by using Feedback option on the site or send us an 
email @ supp...@thegongzuo.com.
2.   We like Criticism   :-  As a fresh site, We are sure you can find lots 
to Criticize - remember , though we would prefer constructive criticism, so we 
can make site better. Please send us your valuable criticism by email @ 
supp...@thegongzuo.com.


 
Please make us a part of your distribution list,
We will start sending resumes to you.
If you do not wish to be a part of our mailing list just reply to us with 
unsubscribe from this list or else it will consider it as your consent to be 
part of our distribution list.


Thanks for trying TheGongzuo.com and please let us know , What you think !

Thanks,
TheGongzuo.com Team !

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Pure Python Data Mangling or Encrypting

[Johannes Bauer]

On 26.06.2015 23:29, Jon Ribbens wrote:

>> While you seem to think that Steven is rampaging about nothing, he does
>> have a fair point: You consistently were vague about wheter you want to
>> have encryption, authentication or obfuscation of data. This suggests
>> that you may not be so sure yourself what it is you actually want.
> 
> He hasn't been vague, you and Steven just haven't been paying
> attention.

Bullshit. Even the topic indicates that he doesn't know what he wants:
"data mangling" or "encryption", which one is it?

>> You always play around with the 256! which would be a ridiculously high
>> security margin (1684 bits of security, w!). You totally ignore that
>> the system can be broken in a linear fashion.
> 
> No, it can't, because the attacker does not have access to the
> ciphertext.

Or so you claim.

I could go into detail about how the assumtion that the ciphertext is
secret is not a smart one in the context of cryptography. And how side
channels and other leakage may affect overall system security. But I'm
going to save my time on that. I do get paid to review cryptographic
systems and part of the job is dealing with belligerent people who have
read Schneier's blog and think they can outsmart anyone else. Since I
don't get paid to convice you, it's absolutely fine that you think your
substitution scheme is the grand prize.

>> Nobody assumes you're a moron. But it's safe to assume that you're a
>> crypto layman, because only laymen have no clue on how difficult it is
>> to get cryptography even remotely right.
> 
> Amateur crypto is indeed a bad idea. But what you're still not getting
> is that what he's doing here *isn't crypto*. 

So the topic says "Encrypting". If you look really closely at the word,
the part "crypt" might give away to you that cryptography is involved.

> He's just trying to avoid
> letting third parties write completely arbitrary data to the disk.

There's your requirement. Then there's obviously some kind of
implication when a third party *can* write arbitrary data to disk. And
your other solution to that problem...

> You
> know what would be a perfectly good solution to his problem? Base 64
> encoding. That would solve the issue pretty much completely, the only
> reason it's not an ideal solution is that it of course increases the
> size of the data.

...wow.

That's a nice interpretation of not letting a third party write
completely arbitrary data. According to your definition, this would be:
It's okay if the attacker can control 6 of 8 bits.

>> That people in 2015 actually defend inventing a substitution-cipher
>> "crypto"system sends literally shivers down my spine.
> 
> Nobody is defending such a thing, you just haven't understood what
> problem is being solved here.

Oh I understand your "solutions" plenty well. The only thing I don't
understand is why you don't own a Fields medal yet for your
groundbreaking work on bulletproof obfuscation.

Cheers,
Johannes

-- 
>> Wo hattest Du das Beben nochmal GENAU vorhergesagt?
> Zumindest nicht öffentlich!
Ah, der neueste und bis heute genialste Streich unsere großen
Kosmologen: Die Geheim-Vorhersage.
 - Karl Kaos über Rüdiger Thomas in dsa 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Pure Python Data Mangling or Encrypting

[Devin Jeanpierre]

Johannes, I agree with a lot of what you say, but can you please have
less of a mean attitude?

-- Devin

On Fri, Jun 26, 2015 at 3:42 PM, Johannes Bauer  wrote:
> On 26.06.2015 23:29, Jon Ribbens wrote:
>
>>> While you seem to think that Steven is rampaging about nothing, he does
>>> have a fair point: You consistently were vague about wheter you want to
>>> have encryption, authentication or obfuscation of data. This suggests
>>> that you may not be so sure yourself what it is you actually want.
>>
>> He hasn't been vague, you and Steven just haven't been paying
>> attention.
>
> Bullshit. Even the topic indicates that he doesn't know what he wants:
> "data mangling" or "encryption", which one is it?
>
>>> You always play around with the 256! which would be a ridiculously high
>>> security margin (1684 bits of security, w!). You totally ignore that
>>> the system can be broken in a linear fashion.
>>
>> No, it can't, because the attacker does not have access to the
>> ciphertext.
>
> Or so you claim.
>
> I could go into detail about how the assumtion that the ciphertext is
> secret is not a smart one in the context of cryptography. And how side
> channels and other leakage may affect overall system security. But I'm
> going to save my time on that. I do get paid to review cryptographic
> systems and part of the job is dealing with belligerent people who have
> read Schneier's blog and think they can outsmart anyone else. Since I
> don't get paid to convice you, it's absolutely fine that you think your
> substitution scheme is the grand prize.
>
>>> Nobody assumes you're a moron. But it's safe to assume that you're a
>>> crypto layman, because only laymen have no clue on how difficult it is
>>> to get cryptography even remotely right.
>>
>> Amateur crypto is indeed a bad idea. But what you're still not getting
>> is that what he's doing here *isn't crypto*.
>
> So the topic says "Encrypting". If you look really closely at the word,
> the part "crypt" might give away to you that cryptography is involved.
>
>> He's just trying to avoid
>> letting third parties write completely arbitrary data to the disk.
>
> There's your requirement. Then there's obviously some kind of
> implication when a third party *can* write arbitrary data to disk. And
> your other solution to that problem...
>
>> You
>> know what would be a perfectly good solution to his problem? Base 64
>> encoding. That would solve the issue pretty much completely, the only
>> reason it's not an ideal solution is that it of course increases the
>> size of the data.
>
> ...wow.
>
> That's a nice interpretation of not letting a third party write
> completely arbitrary data. According to your definition, this would be:
> It's okay if the attacker can control 6 of 8 bits.
>
>>> That people in 2015 actually defend inventing a substitution-cipher
>>> "crypto"system sends literally shivers down my spine.
>>
>> Nobody is defending such a thing, you just haven't understood what
>> problem is being solved here.
>
> Oh I understand your "solutions" plenty well. The only thing I don't
> understand is why you don't own a Fields medal yet for your
> groundbreaking work on bulletproof obfuscation.
>
> Cheers,
> Johannes
>
> --
>>> Wo hattest Du das Beben nochmal GENAU vorhergesagt?
>> Zumindest nicht öffentlich!
> Ah, der neueste und bis heute genialste Streich unsere großen
> Kosmologen: Die Geheim-Vorhersage.
>  - Karl Kaos über Rüdiger Thomas in dsa 
> --
> https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Pure Python Data Mangling or Encrypting

[Randall Smith]

On 06/26/2015 04:55 PM, Mark Lawrence wrote:



To be perfectly blunt I gave up days ago trying to follow what was being
said, just too many words from all angles and too few diagrams for me to
follow.  I sincerely hope it doesn't end in tears.



Mark.

There's not much to follow.  The solution was simple and complete.

The original description was limited to a small part of a large, more 
complex system.  The reason you've had trouble following is because 
several people made (very bad) assumptions about what the rest of the 
system did.  Everything required for the solution was present in the 
original post.


-Randall

--
https://mail.python.org/mailman/listinfo/python-list

Re: Pure Python Data Mangling or Encrypting

[Randall Smith]

On 06/26/2015 04:07 PM, Johannes Bauer wrote:

You consistently were vague about wheter you want to
have encryption, authentication or obfuscation of data.


I knew (possibly extra) encryption wasn't necessary at this stage, but I 
also knew that encryption would provide good obfuscation.  Problem is, I 
didn't want an extra C library to install. See the original post.


"... I'd like to protect the recipient against exposure to nefarious 
data by mangling or encrypting the data before it is written to disk. My 
original idea was for the recipient to encrypt using AES.  But I want to 
keep this software pure Python "batteries included" and not require 
installation of other platform-dependent software ... I don't know that 
I really need encryption here, but some type of fast mangling algorithm 
where a bad actor sending a payload can't guess the output ahead of time."


-Randall



--
https://mail.python.org/mailman/listinfo/python-list

Re: Pure Python Data Mangling or Encrypting

[Jon Ribbens]

On 2015-06-26, Johannes Bauer  wrote:
> On 26.06.2015 23:29, Jon Ribbens wrote:
>>> While you seem to think that Steven is rampaging about nothing, he does
>>> have a fair point: You consistently were vague about wheter you want to
>>> have encryption, authentication or obfuscation of data. This suggests
>>> that you may not be so sure yourself what it is you actually want.
>> 
>> He hasn't been vague, you and Steven just haven't been paying
>> attention.
>
> Bullshit. Even the topic indicates that he doesn't know what he wants:
> "data mangling" or "encryption", which one is it?

He wants data mangling and he was asking whether he needed encryption
to achieve it. The answer is no, he doesn't.

> I could go into detail about how the assumtion that the ciphertext is
> secret is not a smart one in the context of cryptography.

But, and I've already pointed this out and you don't seem to have
quite got the picture yet, we're not in the context of cryptography.

> And how side channels and other leakage may affect overall system
> security. But I'm going to save my time on that. I do get paid to
> review cryptographic systems and part of the job is dealing with
> belligerent people who have read Schneier's blog and think they can
> outsmart anyone else.

You seem to be describing your own attitude to a tee.

> Since I don't get paid to convice you, it's absolutely fine that you
> think your substitution scheme is the grand prize.

"My" scheme? It wasn't my suggestion.

> So the topic says "Encrypting". If you look really closely at the word,
> the part "crypt" might give away to you that cryptography is involved.

If you were to actually read past the subject line and continue on to
read the text of the articles, you would discover that cryptography is
not involved. No wonder you're confused if you're disengaging your
brain the instant you get past the subject line.

>> He's just trying to avoid letting third parties write completely
>> arbitrary data to the disk.
>
> There's your requirement.

"My" requirement?

>> You know what would be a perfectly good solution to his problem?
>> Base 64 encoding. That would solve the issue pretty much
>> completely, the only reason it's not an ideal solution is that it
>> of course increases the size of the data.
>
> ...wow.
>
> That's a nice interpretation of not letting a third party write
> completely arbitrary data.

It's an accurate interpretation. Something that seems not to be your
forte.

> According to your definition, this would be: It's okay if the
> attacker can control 6 of 8 bits.

Yes, it probably is ok. Add a bit of random gunk at the top and tail
of the file and it's almost certainly ok. Why do you think it's not?

> Oh I understand your "solutions" plenty well.

Evidently not.

> The only thing I don't understand is why you don't own a Fields
> medal yet for your groundbreaking work on bulletproof obfuscation.

That is clearly a very long way from the only thing you don't
understand.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Pure Python Data Mangling or Encrypting

[Randall Smith]

On 06/26/2015 05:42 PM, Johannes Bauer wrote:

On 26.06.2015 23:29, Jon Ribbens wrote:


While you seem to think that Steven is rampaging about nothing, he does
have a fair point: You consistently were vague about wheter you want to
have encryption, authentication or obfuscation of data. This suggests
that you may not be so sure yourself what it is you actually want.


He hasn't been vague, you and Steven just haven't been paying
attention.


Bullshit. Even the topic indicates that he doesn't know what he wants:
"data mangling" or "encryption", which one is it?


I knew exactly what I wanted and spelled it out.

"protect the recipient against exposure to nefarious data ... before it 
is written to disk"


You shouldn't need to make assumptions about other parts of the system. 
 Just prevent potential malware from hitting the disk as such.


Before this thread, I knew that encryption would definitely work and 
data mangling might.  Now I know that data mangling is a really nice 
solution for the given requirements.






You always play around with the 256! which would be a ridiculously high
security margin (1684 bits of security, w!). You totally ignore that
the system can be broken in a linear fashion.


No, it can't, because the attacker does not have access to the
ciphertext.


Or so you claim.


No the attacker does not have access to the ciphertext.  What would lead 
you to think they did?


This statement is central to the problem:

"I'd like to protect the recipient against exposure to nefarious data by 
mangling or encrypting the data before it is written to disk"


This makes it clear I'm not trying to encrypt data to protect the data. 
 I'm trying to protect the recipient (storage server) from an attack. 
This specific attack being malware.  Yes, AES encryption would have 
worked here, but encryption is not the objective.




I could go into detail about how the assumtion that the ciphertext is
secret is not a smart one in the context of cryptography. And how side
channels and other leakage may affect overall system security. But I'm
going to save my time on that. I do get paid to review cryptographic
systems and part of the job is dealing with belligerent people who have
read Schneier's blog and think they can outsmart anyone else. Since I
don't get paid to convice you, it's absolutely fine that you think your
substitution scheme is the grand prize.



All of which has nothing to do with this thread.  Actual encryption is 
handled using AES and TLS.  This is not about encryption.





Nobody assumes you're a moron. But it's safe to assume that you're a
crypto layman, because only laymen have no clue on how difficult it is
to get cryptography even remotely right.


Amateur crypto is indeed a bad idea. But what you're still not getting
is that what he's doing here *isn't crypto*.


So the topic says "Encrypting". If you look really closely at the word,
the part "crypt" might give away to you that cryptography is involved.




This isn't about encrypting data to protect the data.  All the 
encryption I do uses standard AES and TLS.  Yes, I do understand that 
crypto is best left to experts.  The topic says "Encrypting" because I 
knew that encrypting the data would properly obfuscate it.




He's just trying to avoid
letting third parties write completely arbitrary data to the disk.


There's your requirement. Then there's obviously some kind of
implication when a third party *can* write arbitrary data to disk. And
your other solution to that problem...


It's a network protocol.  Just like when writing a web app, you have to 
deal with bad actors.  That's what I'm doing here.  The entire service 
is about handling arbitrary data.  Just like Amazon S3 handles people's 
"arbitrary data".  Not sure what you mean by "third party".  It would be 
a registered peer.  But registration doesn't prevent the scenario in 
discussion.





--
https://mail.python.org/mailman/listinfo/python-list

Re: Pure Python Data Mangling or Encrypting

[Chris Angelico]

On Sat, Jun 27, 2015 at 6:09 AM, Randall Smith  wrote:
> Give me one plausible scenario where an attacker can cause malware to hit
> the disk after bytearray.translate with a 256 byte translation table and
> I'll be thankful to you.

The entire 256-byte translation table is significant ONLY if you need
all 256 possible bytes. Suppose I want to generate the following byte
sequence:

"\xCD\x19"

(Okay, this is a slightly oversimplified example, as this attack
doesn't work on a modern Windows. But back in the days of DOS, this
program would reboot your computer.)

How many truly different translation tables are there if I'm trying to
produce this? Just 256*255, or 65280. If I send random two-byte files,
there is one chance in that of my "malware" successfully landing. Once
I've sent about 45,000 of those files, I have a fifty-fifty chance of
having hit it. Send twice as many, I have a 75% chance of success,
etc.

Malware can be crafted to fit within certain restrictions. I saw a
proof-of-concept and analysis document detailing a particular remote
code execution/privilege escalation attack that involved stuffing
"text" into an entry field and then inducing the program to read that
into its stack, finally triggering it by some sort of buffer overflow,
I think. The text had to be no more than X bytes long (because that's
all the entry field was set to accept - it'd truncate after that), and
had to not contain any NUL bytes, and there might have been other
restrictions too. Sure, it makes it harder to write your malware...
but imagine if you can write something in just a handful of different
bytes, which then goes and triggers something else. You could have an
extremely plausible attack that might need only a day's uploading to
deliver.

It makes no difference that there are 256! possible encryption keys,
if most of them have the same result.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Pure Python Data Mangling or Encrypting

[Steven D'Aprano]

On Sat, 27 Jun 2015 06:09 am, Randall Smith wrote:

> On 06/26/2015 12:06 PM, Steven D'Aprano wrote:
>> On Fri, 26 Jun 2015 11:01 am, Ian Kelly wrote:
>>
>>> You're making the same mistake that Steven did in misunderstanding the
>>> threat model.
>>
>> I don't think I'm misunderstanding the threat, I think I'm pointing out a
>> threat which the OP is hoping to just ignore.
> 
> I'm not hoping to ignore anything.  I didn't explain the entire system,
> as it was not necessary to find a solution to the problem at hand.  But
> since you want to make negative assumptions about what I didn't tell
> you, I'll gladly address your accusations of negligence.

"Negligence" is *your* word, not mine. I've never said that. And I'm not
*assuming* anything, everything I've stated has been based on the evidence
of what you have written. I've even gone so far as to EXPLICITLY say that I
cannot know for a fact that your application is vulnerable to these
threats, since I'm only going from a description rather than the app
itself. But your responses don't suggest that you have these threats under
control, on the contrary, they indicate that you are *far* underestimating
the seriousness of them and overestimating the difficulty of running a
secure application on a machine you cannot trust.

If your application has any saving grace, it is that there are easier ways
to get malware onto somebody else's computer. There are a hundred million
unsecured Windows boxen out there, if I were malicious I would just hire a
bot net rather than spend the time trying to hack your system. But maybe
somebody else will do it just for the lulz, or to prove it can be done.
Some black hats like a challenge, and yours appears to fall nicely into
that middle ground of hard enough to be interesting but not hard enough to
be really difficult.


>> In an earlier post, I suggested that the threat model should involve at
>> least *three* different attacks, apart from the usual man-in-the-model
>> attacks of data in transit.
> 
> All communication is secured using TLS and authentication handled by
> X.509 certificates.  This prevents man in the middle attacks.
> Certificates are signed by CAs I control.

You control the CAs? Presumably that means they're self-signed (unless you
mean you get to choose the CA). I don't know if that makes a difference or
not.


>> One is that the attacker is the person sending the data. E.g. I want to
>> send a nasty payload (say, malware, or an offensive image). Another is
>> that the attacker is the recipient of the file, who wants to read the
>> sender's data.
> 
> The only person who can read a file is the owner.   AES encryption is
> built into the client software.  The only way data can be uploaded
> unencrypted is if encryption is intentionally disabled.

With respect Randall, you contradict yourself. Is there any wonder that some
of us (well, me at least) is suspicious and confused, when your story
changes as often as the weather?

Sometimes you say that the client software uses AES encryption. Sometimes
you say that you don't want to use AES encryption because you want the
client to be pure Python, and a pure-Python implementation would be too
slow. Your very first post says:

My original idea was for the recipient to encrypt using AES.  But
I want to keep this software pure Python "batteries included" and
not require installation of other platform-dependent software.  
Pure Python AES and even DES are just way too slow.


Sometimes you say the user is supposed to encrypt the data themselves:

While the data senders are supposed to encrypt data, that's not
guaranteed


Now you say that the application encrypts the data, except that the user can
turn that option off.

Just make the AES encryption mandatory, not optional. Then the user cannot
upload unencrypted malicious data, and the receiver cannot read the data.
That's two problems solved.

Making AES or similarly strong encryption mandatory protects both the sender
of data and the receiver of data. I cannot imagine why you are considering
making it optional, since that only adds more work for you and reduces the
security of your users.

Oh, and DES is not good enough.


>> As far as I can tell, the OP's plan to defend the sender's privacy is to
>> dump responsibility for encrypting the files in the sender's lap. As far
>> as I'm concerned, perhaps as many as one user in 2 will pre-encrypt
>> their files. (Early adopters will be unrepresentative of the eventual
>> user base of this system. If this takes off, the user base will likely
>> end up dominated by people who think that "qwerty" is the epitome of
>> unguessable passwords.)
> 
> Making assumptions again.  See above.  The client software encrypts by
> default.  You're also assuming there is no password strength checking.

My comment about "qwerty" as a password was a comment on the majority of
people on the internet, not an assumption about your application.


>> Users just don't us

Re: Pure Python Data Mangling or Encrypting

[Devin Jeanpierre]

On Fri, Jun 26, 2015 at 8:38 PM, Steven D'Aprano  wrote:
> Now you say that the application encrypts the data, except that the user can
> turn that option off.
>
> Just make the AES encryption mandatory, not optional. Then the user cannot
> upload unencrypted malicious data, and the receiver cannot read the data.
> That's two problems solved.

No, because another application could pretend to be the file-sending
application, but send unencrypted data instead of encrypted data.

-- Devin
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Turning string into object (name)

[Rustom Mody]

On Friday, June 26, 2015 at 9:30:38 AM UTC+5:30, Chris Angelico wrote:
> Incidentally, I would suggest not having the try/except at all, since
> all it does is print an error and terminate (which is the same result
> you'd get if that error bubbled all the way to top level). But if you
> are going to use it, then I strongly recommend using the newer syntax:
> 
> except errors.ConnectionFailure as e:

I wonder Chris if you know anything about this [yeah a bit of a hijack]:

There is this git plugin https://github.com/felipec/git-remote-hg
that allows one to bi-direct talk to hg

I tried to add a little better error checking so that it doesnt barf with a 
backtrace when connecting to git (instead of hg which is the intent):

My modifs: 
https://github.com/rusimody/git-remote-hg/commit/51cc665eff174e0d6bbf2e1b61ca332af345e76a

I got some response (from github) to the effect that travis-continuous failed 
or some such.
Yeah I know about nothing about github pull requests but it just could be
this -- obsolete except syntax.

Any thoughts? Suggestions?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Turning string into object (name)

[Chris Angelico]

On Sat, Jun 27, 2015 at 2:11 PM, Rustom Mody  wrote:
> On Friday, June 26, 2015 at 9:30:38 AM UTC+5:30, Chris Angelico wrote:
>> Incidentally, I would suggest not having the try/except at all, since
>> all it does is print an error and terminate (which is the same result
>> you'd get if that error bubbled all the way to top level). But if you
>> are going to use it, then I strongly recommend using the newer syntax:
>>
>> except errors.ConnectionFailure as e:
>
> I wonder Chris if you know anything about this [yeah a bit of a hijack]:

Well yes, that's a complete thread hijack. Whatever.

> There is this git plugin https://github.com/felipec/git-remote-hg
> that allows one to bi-direct talk to hg
>
> I tried to add a little better error checking so that it doesnt barf with a 
> backtrace when connecting to git (instead of hg which is the intent):
>
> My modifs: 
> https://github.com/rusimody/git-remote-hg/commit/51cc665eff174e0d6bbf2e1b61ca332af345e76a
>
> I got some response (from github) to the effect that travis-continuous failed
> or some such.
> Yeah I know about nothing about github pull requests but it just could be
> this -- obsolete except syntax.
>
> Any thoughts? Suggestions?

Standard principle when asking for help with a Python program: Quote
the error text! You say you got a response back - quote it!

Standard debugging technique: If you have an idea, try it! Most code
is deterministic enough that you can simply change to the modern
except syntax and try again. Maybe it won't have any effect, but maybe
it'll tell you exactly what the problem is.

Beyond that, I don't think I can much help you.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Pure Python Data Mangling or Encrypting

[Johannes Bauer]

On 27.06.2015 02:55, Randall Smith wrote:

> No the attacker does not have access to the ciphertext.  What would lead
> you to think they did?

Years of practical experience in the field of applied cryptography.
Knowledge of how side channels work and how easily they can be
constructed for bad schemes.

Rest snipped, explanation futile.
Cheers,
Johannes

-- 
>> Wo hattest Du das Beben nochmal GENAU vorhergesagt?
> Zumindest nicht öffentlich!
Ah, der neueste und bis heute genialste Streich unsere großen
Kosmologen: Die Geheim-Vorhersage.
 - Karl Kaos über Rüdiger Thomas in dsa 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Pure Python Data Mangling or Encrypting

[Ian Kelly]

On Fri, Jun 26, 2015 at 9:38 PM, Steven D'Aprano  wrote:
> With respect Randall, you contradict yourself. Is there any wonder that some
> of us (well, me at least) is suspicious and confused, when your story
> changes as often as the weather?
>
> Sometimes you say that the client software uses AES encryption. Sometimes
> you say that you don't want to use AES encryption because you want the
> client to be pure Python, and a pure-Python implementation would be too
> slow. Your very first post says:
>
> My original idea was for the recipient to encrypt using AES.  But
> I want to keep this software pure Python "batteries included" and
> not require installation of other platform-dependent software.
> Pure Python AES and even DES are just way too slow.

In the context of the initial post, this was referring to the data
mangling done by the receiver; it has no bearing on the form of the
data sent by the application.

> Sometimes you say the user is supposed to encrypt the data themselves:
>
> While the data senders are supposed to encrypt data, that's not
> guaranteed

Whereas this clearly describes the behavior of the application itself.

> Now you say that the application encrypts the data, except that the user can
> turn that option off.
>
> Just make the AES encryption mandatory, not optional. Then the user cannot
> upload unencrypted malicious data, and the receiver cannot read the data.
> That's two problems solved.

And what if somebody else writes a competing version of the client
software that doesn't bother with the encryption step at all? The
point was that while encryption is expected, it cannot be assumed by
the receiver, and in fact if the data is actually malicious, then it
likely is not even being sent by the client software in the first
place.

> If the app does encrypt the data with AES before sending, then you don't
> gain any benefit by obfuscating an encrypted file with a classical
> monoalphabetic substitution cipher.

Only if the recipient can *trust* the sender to have performed the
encryption, which it can't, no matter how mandatory the OP tries to
make it.

> Suppose that you hire an intern to write the "choose key" function, and not
> knowing any better, he simply iterates through the keys in numeric order,
> one after the other. So the first upload will use key 0, the second key 1,
> the third key 2, and so on, until key 256! - 1, then start again. In that
> case, predicting the next key is *trivial*. If I can work out what key you
> send now (I just upload a file containing "\x00\x01\x02...\xFF" to myself
> and see what I get), then I know what key the app will use next.

If you upload a file to yourself, the result that you get will have no
bearing on what key might be chosen when you upload a file to somebody
else.

> Even if I can't do that, I might be able to guess the seed: I know what time
> the application started up, to within a few milliseconds,

How?

> and I know (or
> can guess) how many random numbers you have used,

How?

> Except... you're getting your random numbers from a system *I* control.

No you don't. If you did already control the target system, then as
already suggested, you have no need to attack the data upload; you can
just write whatever data you want to disk. This is like suggesting
that the sudoers file is insecure because a user with root access
would be able to add themselves to it.

>> If the attacker controlled the machine the app was on, why would it fool
>> with /dev/urandom?  I think he'd just plant the files he wanted to plant
>> and be done.  This is non-nonsensical anyway.
>
> No, you don't understand the nature of the attack. In this scenario, the
> sender is the attacker. I want to upload malicious files to the receiver.
> You are trying to stop me, that's the whole point of "mangling or
> encrypting" the files. (Your words.) So I, the sender, prepare a file such
> that when you mangle it, the resulting mangled content is the malicious
> content I want.
>
> If you use a substitution cipher, I can do this if I can guess or force the
> key. If you use strong crypto, I can't.
>
> However, I can hack the application. The client sits on my computer, it's
> pure Python, even if it isn't I can still hack the application, I don't
> need access to the source code.

If the recipient system is using the system random to generate the
key, then you can hack the application all you want, and it will give
you precisely zero information about the state of the entropy pool on
the remote system.

> Yes. Do you think that's hard for an attacker who has access to your
> application, possibly including the source code, and controls all the
> sources of entropy on the system your application is running on?
>
> I don't have to *randomly* guess. I control what time your application
> starts, I control what randomness you get from /dev/urandom, I control how
> many keys you go through, I might even be able to read the source code of
> the ap

Re: Pure Python Data Mangling or Encrypting

[Ian Kelly]

On Fri, Jun 26, 2015 at 7:21 PM, Chris Angelico  wrote:
> On Sat, Jun 27, 2015 at 6:09 AM, Randall Smith  wrote:
>> Give me one plausible scenario where an attacker can cause malware to hit
>> the disk after bytearray.translate with a 256 byte translation table and
>> I'll be thankful to you.
>
> The entire 256-byte translation table is significant ONLY if you need
> all 256 possible bytes. Suppose I want to generate the following byte
> sequence:
>
> "\xCD\x19"
>
> (Okay, this is a slightly oversimplified example, as this attack
> doesn't work on a modern Windows. But back in the days of DOS, this
> program would reboot your computer.)

Nice! When I suggested the possibility of a two byte value malicious
payload, I thought it an extreme example of the hypothetical attack. I
didn't expect that somebody might actually produce one.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Pure Python Data Mangling or Encrypting

[Steven D'Aprano]

On Sat, 27 Jun 2015 02:05 pm, Devin Jeanpierre wrote:

> On Fri, Jun 26, 2015 at 8:38 PM, Steven D'Aprano 
> wrote:
>> Now you say that the application encrypts the data, except that the user
>> can turn that option off.
>>
>> Just make the AES encryption mandatory, not optional. Then the user
>> cannot upload unencrypted malicious data, and the receiver cannot read
>> the data. That's two problems solved.
> 
> No, because another application could pretend to be the file-sending
> application, but send unencrypted data instead of encrypted data.

Did you stop reading my post when you got to that? Because I went on to say:

"Actually, the more I think about this, the more I come to think that the
only way this can be secure is for both the sending client application and
the receiving client appl to both encrypt the data. The sender can't
trust the receiver not to read the files, so the sender has to encrypt; the
receiver can't trust the sender not to send malicious files, so the
receiver has to encrypt too."




-- 
Steven

-- 
https://mail.python.org/mailman/listinfo/python-list

EuroPython 2015: Standard rates deadline extended

[M.-A. Lemburg]

We had originally planned switching to On-Desk Rates yesterday, but
because ticket sales are going well, we are extending the deadline for
more than a week.

   *** Buy your EuroPython 2015 tickets ***

 https://ep2015.europython.eu/en/registration/


On-Desk Rates
-

On-Desk Rates will be active starting on Monday, 2015-07-06, 00:00:00 CEST:

 * Student: 180 EUR
 * Personal: 440 EUR
 * Business: 690 EUR
   (all incl. 10% Spanish VAT)

This means that you can save up to 30% on ticket prices, if you buy
your tickets before:

   Sunday, 2015-07-05, 23:59:59 CEST

and still benefit from our Standard Rates:

 * Student: 120 EUR
 * Personal: 340 EUR
 * Business: 530 EUR
   (all incl. 10% Spanish VAT)

Social Event


Please also remember to book your Social Event Ticket. The price for
this, EUR 40, won’t change, but we only have a limited number of
tickets available.

   https://ep2015.europython.eu/en/events/social-event/

For more details on the social event ticket, please see our announcement.

Enjoy,
--
EuroPython 2015 Team
http://ep2015.europython.eu/
http://www.europython-society.org/
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Pure Python Data Mangling or Encrypting

[Chris Angelico]

On Sat, Jun 27, 2015 at 3:59 PM, Ian Kelly  wrote:
> On Fri, Jun 26, 2015 at 7:21 PM, Chris Angelico  wrote:
>> On Sat, Jun 27, 2015 at 6:09 AM, Randall Smith  wrote:
>>> Give me one plausible scenario where an attacker can cause malware to hit
>>> the disk after bytearray.translate with a 256 byte translation table and
>>> I'll be thankful to you.
>>
>> The entire 256-byte translation table is significant ONLY if you need
>> all 256 possible bytes. Suppose I want to generate the following byte
>> sequence:
>>
>> "\xCD\x19"
>>
>> (Okay, this is a slightly oversimplified example, as this attack
>> doesn't work on a modern Windows. But back in the days of DOS, this
>> program would reboot your computer.)
>
> Nice! When I suggested the possibility of a two byte value malicious
> payload, I thought it an extreme example of the hypothetical attack. I
> didn't expect that somebody might actually produce one.

I'm fairly sure this won't actually work on a modern system (I tried
it and all that happened was that debug.exe terminated), but it's
entirely possible there are other attacks. Or attacks that require
only a small number of bytes - maybe create a gzip bomb that will
expand to petabytes of data, that probably wouldn't need many unique
byte values.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list