Re: [OT] Security question
On Thu, 22 Dec 2016 09:10 pm, Frank Millman wrote: > If this is the standard of security out there, it is no wonder we hear of > so many attacks (and how many don't we hear of?) Everything is broken: https://medium.com/message/everything-is-broken-81e5f33a24e1 -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Another security question
Hi all
This is a follow-up to my recent 'security question' post.
I am starting a new thread, for 2 reasons -
1) I sent a link to the previous thread to my ISP for their information. It
is up to them whether they do anything with it, but I wanted to keep that
thread focused on the original issue raised.
2) This one is more on-topic, as it is to do with my python project.
Having read the previous thread and various links, I want to review the way
I handle passwords in my accounting application.
At present I just store a SHA-1 hash of the password for each user. Here are
my thoughts on improving this.
1. Generate a 'salt' for each password. There seem to be two ways in the
standard library to do this -
import os
salt = os.urandom(16)
import secrets
salt = secrets.token_bytes(16)
My guess is that it will not make much difference which I use.
2. Store the salt in the database along with the user-id and hashed password
for each user.
3. Generate the password from the string supplied by the user as follows -
from hashlib import blake2b
password = blake2b('my_password'.encode('utf-8'), salt=salt).digest()
The hashlib docs have the following warning -
"Salted hashing (or just hashing) with BLAKE2 or any other general-purpose
cryptographic hash function, such as SHA-256, is not suitable for hashing
passwords. See BLAKE2 FAQ for more information."
I propose to ignore this warning. I feel that, for my purposes, the above
procedure is adequate.
Does all this sound reasonable?
Any comments appreciated.
Frank Millman
--
https://mail.python.org/mailman/listinfo/python-list
[RELEASE] Python 3.6.0 is released!
On behalf of the Python development community and the Python 3.6 release team, I am pleased to announce the availability of Python 3.6.0. Python 3.6.0 is the newest major release of the Python language, and it contains many new features and optimizations. See the "What’s New In Python 3.6" document for more information: https://docs.python.org/3.6/whatsnew/3.6.html You can download Python 3.6.0 here: https://www.python.org/downloads/release/python-360/ Also, most third-party distributors of Python should be making 3.6.0 packages available soon. Maintenance releases for the 3.6 series will follow at regular intervals starting in the first quarter of 2017. We hope you enjoy Python 3.6.0! P.S. As a volunteer-staffed open source project, we could not bring Python releases to you without the enormous contributions of many, many people. Thank you to all who have contributed and reviewed code and documentation changes, documented and investigated bugs, tested Python and third-party packages, and provided and supported the infrastructure needed to support Python development and testing. Please consider supporting the work of the Python Software Foundation. More at: https://www.python.org/psf-landing/ -- Ned Deily n...@python.org -- [] -- https://mail.python.org/mailman/listinfo/python-list
Re: Another security question
On Fri, 23 Dec 2016 09:19 pm, Frank Millman wrote:
[...]
> Having read the previous thread and various links, I want to review the
> way I handle passwords in my accounting application.
>
> At present I just store a SHA-1 hash of the password for each user. Here
> are my thoughts on improving this.
SHA-1 hashes are (I believe) vulnerable to pre-computed rainbow tables. This
is where the salt comes in: the salt is not a secret itself, but if
prevents attackers pre-computing rainbow tables.
The question is, is SHA-1 (plus salting) strong enough? I'm not sure.
> 1. Generate a 'salt' for each password. There seem to be two ways in the
> standard library to do this -
> import os
> salt = os.urandom(16)
>
> import secrets
> salt = secrets.token_bytes(16)
>
> My guess is that it will not make much difference which I use.
secrets is officially the preferred mechanism. os.urandom is the low-level
operating system routine, secrets.* is the high-level interface.
At the moment it is true that there's very little difference in practice,
but that's an implementation detail which could change.
> 2. Store the salt in the database along with the user-id and hashed
> password for each user.
>
> 3. Generate the password from the string supplied by the user as follows -
> from hashlib import blake2b
> password = blake2b('my_password'.encode('utf-8'), salt=salt).digest()
>
> The hashlib docs have the following warning -
>
> "Salted hashing (or just hashing) with BLAKE2 or any other general-purpose
> cryptographic hash function, such as SHA-256, is not suitable for hashing
> passwords. See BLAKE2 FAQ for more information."
Why are using Blake2 when the docs explicitly say not to use them in this
way? Have you read the FAQ to see what it says?
--
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Another security question
"Steve D'Aprano" wrote in message
news:585d009f$0$1599$c3e8da3$54964...@news.astraweb.com...
On Fri, 23 Dec 2016 09:19 pm, Frank Millman wrote:
>
> 3. Generate the password from the string supplied by the user as
> follows -
> from hashlib import blake2b
> password = blake2b('my_password'.encode('utf-8'),
> salt=salt).digest()
>
> The hashlib docs have the following warning -
>
> "Salted hashing (or just hashing) with BLAKE2 or any other
> general-purpose
> cryptographic hash function, such as SHA-256, is not suitable for
> hashing
> passwords. See BLAKE2 FAQ for more information."
Why are using Blake2 when the docs explicitly say not to use them in this
way? Have you read the FAQ to see what it says?
Why am I using Blake2? Well, before today I had not heard of it. However, in
the past, when I needed to create a hashed password, I used the built-in
hashlib module. Today, when I look at the docs for hashlib in Python 3.6,
this is the new sub-heading -
"15.2. hashlib — BLAKE2 hash functions"
So it appears that this is the new preferred way of doing it.
This is what the Blake2 FAQ says -
"You shouldn't use *any* general-purpose hash function for user passwords,
not BLAKE2, and not MD5, SHA-1, SHA-256, or SHA-3. Instead you should use a
password hashing function such as the PHC winner Argon2 with appropriate
time and memory cost parameters, to mitigate the risk of bruteforce
attacks—Argon2's core uses a variant of BLAKE2's permutation"
I see that there is a Python implementation of Argon2 in PyPi, but I don't
particularly want to add another dependency to my app. My gut-feel says that
this is overkill for my requirement. However, I am not sure. That is partly
why I started this thread, to get some counter-arguments.
Frank
--
https://mail.python.org/mailman/listinfo/python-list
Re: Metaclasses - magic functions
W dniu 21.12.2016 o 02:51, Ethan Furman pisze:
On 12/20/2016 03:39 PM, Ben Finney wrote:
"Mr. Wrobel" writes:
Quick question, can anybody tell me when to use __init__ instead of
__new__ in meta programming?
Use ‘__new__’ to do the work of *creating* one instance from nothing;
allocating the storage, determining the type, etc. — anything that will
be *the same* for every instance. ‘__new__’ is the constructor.
Use ‘__init__’ to do the work of *configuring* one instance, after it
already exists. Any attributes that are special to an instance should be
manipulated in the ‘__init__’ method. ‘__init__’ is the initialiser.
That sounds like general object creation/class advice, which as a general
guideline is okay, but don't feel like it's the most important thing.
I only use `__new__` when the object being created is (or is based on)
an immutable type; otherwise I use `__init__`. Likewise, if I'm using
`__new__` then I do all my configuration in `__new__` unless I have a
really good reason not to (such as making it easier for subclasses to
modify/skip `__init__`).
As far as metaclasses go... the only time I recall writing an `__init__`
for a metaclass was to strip off the extra arguments so `type.__init__`
wouldn't fail.
--
~Ethan~
Hi,thanx for answers, let's imagine that we want to add one class
attribute for newly created classess with using __init__ in metaclass,
here's an example:
#!/usr/bin/env python
class MetaClass(type):
# __init__ manipulation:
def __init__(cls, name, bases, dct):
dct['added_in_init'] = 'test'
super(MetaClass, cls).__init__(name, bases, dct)
class BaseClass(object):
__metaclass__ = MetaClass
class NewBaseClass(BaseClass):
pass
print("Lets print attributes added in __init__ in base classes:")
print(BaseClass.added_in_init)
print(NewBaseClass.added_in_init)
after running it: AttributeError: type object 'BaseClass' has no
attribute 'added_in_init'
Adding the same in __new__ works. Can anyone explain me please what's wrong?
Cheers,
M
--
https://mail.python.org/mailman/listinfo/python-list
Re: Another security question
On Fri, Dec 23, 2016 at 9:19 PM, Frank Millman wrote:
> At present I just store a SHA-1 hash of the password for each user. Here are
> my thoughts on improving this.
>
> 1. Generate a 'salt' for each password. There seem to be two ways in the
> standard library to do this -
>import os
>salt = os.urandom(16)
>
>import secrets
>salt = secrets.token_bytes(16)
>
>My guess is that it will not make much difference which I use.
The main difference is that the 'secrets' module is new in Python 3.6.
If you use anything older - and there are a lot of 3.5s and 3.4s out
there - you can't use it (unless there's a PyPI backport or
something). So if you need compatibility with older Pythons, use
os.urandom; if you're okay with 3.6+, use secrets.
> 2. Store the salt in the database along with the user-id and hashed password
> for each user.
Yep. I generally work with a single database field containing the salt
and the hash as a combined "encrypted password", as it's convenient to
work that way. It's also often worth storing some kind of signature so
you know what scheme you used; in the future, you will eventually
decide that your passwords aren't really secure enough for long-term,
and you'll want to progressively change over. You can't do that by
decrypting and re-encrypting the passwords (since you can't decrypt
them), so you have to introduce a new scheme while still supporting
the old one. Technically you could detect the scheme by the encrypted
length, but it's easier and safer to have a signature.
> 3. Generate the password from the string supplied by the user as follows -
>from hashlib import blake2b
>password = blake2b('my_password'.encode('utf-8'), salt=salt).digest()
>
> The hashlib docs have the following warning -
>
> "Salted hashing (or just hashing) with BLAKE2 or any other general-purpose
> cryptographic hash function, such as SHA-256, is not suitable for hashing
> passwords. See BLAKE2 FAQ for more information."
>
> I propose to ignore this warning. I feel that, for my purposes, the above
> procedure is adequate.
>
> Does all this sound reasonable?
Check out some prior art. When I build a web app using Flask, I
generally use Werkzeug's password management features:
http://werkzeug.pocoo.org/docs/0.11/utils/#werkzeug.security.generate_password_hash
http://werkzeug.pocoo.org/docs/0.11/utils/#werkzeug.security.check_password_hash
As well as doing everything I said above about salting and hashing and
having signatures, it pushes the responsibility onto someone else. You
just give it a password and get back an ASCII string that you stash in
the database. If there's a security flaw, Werkzeug can push a new
version that fixes it - it's not your problem.
At very least, be aware of what these kinds of libraries are doing.
I'm not saying you should blindly trust them or automatically reach
for a dependency, but they're worth looking at.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: [RELEASE] Python 3.6.0 is released!
On 23/12/2016 10:34, Ned Deily wrote: On behalf of the Python development community and the Python 3.6 release team, I am pleased to announce the availability of Python 3.6.0. Python 3.6.0 is the newest major release of the Python language, and it contains many new features and optimizations. See the "What’s New In Python 3.6" document for more information: https://docs.python.org/3.6/whatsnew/3.6.html You can download Python 3.6.0 here: https://www.python.org/downloads/release/python-360/ Also, most third-party distributors of Python should be making 3.6.0 packages available soon. Maintenance releases for the 3.6 series will follow at regular intervals starting in the first quarter of 2017. We hope you enjoy Python 3.6.0! ... Thanks for this release; makes my Christmas just a bit more stressed :( or perhaps happier :) Anyhow I am trying to figure out this error when building windows versions C:\ux\XB33\py36_x86\lib\site-packages\wheel\pep425tags.py:77: RuntimeWarning: Config variable 'Py_DEBUG' is unset, Python ABI tag may be incorrect warn=(impl == 'cp')): C:\ux\XB33\py36_x86\lib\site-packages\wheel\pep425tags.py:81: RuntimeWarning: Config variable 'WITH_PYMALLOC' is unset, Python ABI tag may be incorrect warn=(impl == 'cp')): I guess this must mean I need to set something somewhere, but what? -- Robin Becker -- https://mail.python.org/mailman/listinfo/python-list
not able to run python.exe file successsfully
i have installed 3.5.2 version of python my system windows 8.1 64 bit. after successful installation it is not executing python.exe file.whenever i try a dialof box pop up with an error message "python has stopped working". i hav tried reinstalling it several times.please help -- https://mail.python.org/mailman/listinfo/python-list
Garbage collection problem with generators
Hi, everyone
I'm building a http long polling client for our company's discovery service
and something weird happened in the following code:
```python
while True:
try:
r = requests.get("url", stream=True, timeout=3)
for data in r.iter_lines():
processing_data...
except TimeoutException:
time.sleep(10)
```
When I deliberately times out the request and then check the connections
with `lsof -p process`, I discover that there are *two active
connections*(ESTABLISH)
instead of one. After digging around, it turns out it might not be the
problem with `requests` at all, but gc related to generators.
So I write this script to demonstrate the problem:
https://gist.github.com/wooparadog/766f8007d4ef1227f283f1b040f102ef
Function `A.a` will return a generator which will raise an exception. And
in function `b`, I'm building new a new instance of `A` and iterate over
the exception-raising generator. In the exception handler, I'll close the
generator, delete it, delete the `A` instance, call `gc.collect()` and do
the whole process all over again.
There's another greenlet checking the `A` instances by using
`gc.get_objects()`. It turns out there are always two `A` instances.
This is reproducible with python2.7, but not in python3.5. I've also tried
with `thread` instead of `gevent`, it still happens. I'm guessing it's
related to garbage collection of generators.
Did I bump into a python2 bug? Or am I simply wrong about the way to close
generators...?
Thanks
--
https://mail.python.org/mailman/listinfo/python-list
Re: PLEASE READ - information on (Case 58158) [RELEASE] Python 3.6.0 is released!
Thank you for your email. Your web change 58158[RELEASE] Python 3.6.0 is released!request has been received and will be dealt with shortly. This service desk only covers minor changes to the legacy NHS Digital website (content.digital.nhs.uk) that are made on a web request form. For changes relating to the new beta NHS Digital website or any other communications request please refer to the intranet and complete a communications support request form. This message may contain confidential information. If you are not the intended recipient please inform the sender that you have received the message in error before deleting it. Please do not disclose, copy or distribute information in this e-mail or take any action in reliance on its contents: to do so is strictly prohibited and may be unlawful. Thank you for your co-operation. NHSmail is the secure email and directory service available for all NHS staff in England and Scotland NHSmail is approved for exchanging patient data and other sensitive information with NHSmail and GSi recipients NHSmail provides an email address for your career in the NHS and can be accessed anywhere For more information and to find out how you can switch, visit www.nhsdigital.nhs.uk/nhsmail -- https://mail.python.org/mailman/listinfo/python-list
Re: Another security question
"Frank Millman" writes:
> ... Here are my thoughts on improving this.
>
> 1. Generate a 'salt' for each password. There seem to be two ways in
> the standard library to do this -
>import os
>salt = os.urandom(16)
>
>import secrets
>salt = secrets.token_bytes(16)
>
>My guess is that it will not make much difference which I use.
>
> 2. Store the salt in the database along with the user-id and hashed
> password for each user.
>
> 3. Generate the password from the string supplied by the user as follows -
>from hashlib import blake2b
>password = blake2b('my_password'.encode('utf-8'), salt=salt).digest()
>
> The hashlib docs have the following warning -
>
> "Salted hashing (or just hashing) with BLAKE2 or any other
> general-purpose cryptographic hash function, such as SHA-256, is not
> suitable for hashing passwords. See BLAKE2 FAQ for more information."
As stated, this is confusing as BLAKE2's site lists several password
hashing schemes that use it! The point is that you should not use
*only* a simple salted hash because it's too efficient and therefore
open to brute-force attacks. The hashing schemes that use BLAKE2 are
deliberately designed to be costly.
> I propose to ignore this warning. I feel that, for my purposes, the
> above procedure is adequate.
>
> Does all this sound reasonable?
That depends on the purposes, of course, so it's hard to offer advice.
--
Ben.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Another security question
"Chris Angelico" wrote in message
news:captjjmpppgm+_ut_amtnb7vgo0vrgptu6iagyjqwvpxg5yp...@mail.gmail.com...
On Fri, Dec 23, 2016 at 9:19 PM, Frank Millman wrote:
> 3. Generate the password from the string supplied by the user as
> follows -
>from hashlib import blake2b
>password = blake2b('my_password'.encode('utf-8'), salt=salt).digest()
>
> The hashlib docs have the following warning -
>
> "Salted hashing (or just hashing) with BLAKE2 or any other
> general-purpose
> cryptographic hash function, such as SHA-256, is not suitable for
> hashing
> passwords. See BLAKE2 FAQ for more information."
>
> I propose to ignore this warning. I feel that, for my purposes, the
> above
> procedure is adequate.
>
> Does all this sound reasonable?
Check out some prior art. When I build a web app using Flask, I
generally use Werkzeug's password management features:
http://werkzeug.pocoo.org/docs/0.11/utils/#werkzeug.security.generate_password_hash
http://werkzeug.pocoo.org/docs/0.11/utils/#werkzeug.security.check_password_hash
As well as doing everything I said above about salting and hashing and
having signatures, it pushes the responsibility onto someone else. You
just give it a password and get back an ASCII string that you stash in
the database. If there's a security flaw, Werkzeug can push a new
version that fixes it - it's not your problem.
At very least, be aware of what these kinds of libraries are doing.
I'm not saying you should blindly trust them or automatically reach
for a dependency, but they're worth looking at.
All excellent advice - thanks very much.
It seems that Werkzeug (which looks great, by the way) uses something called
pbkdf2.
The new kid on the block seems to be Argon2. A python implementation called
argon2_cffi has been released by Hynek Schlawack, who has written this
article -
https://hynek.me/articles/storing-passwords/
This is his preamble - "if you’re hashing your passwords with
bcrypt/scrypt/PBKDF2 today, there’s nothing to worry about in the immediate
future. This article is for you if you’re choosing a password hash today and
want a future-proof solution."
I eventually got argon2_cffi installed, and it works very nicely, so I will
run with that for now.
Thanks again
Frank
--
https://mail.python.org/mailman/listinfo/python-list
Re: Metaclasses - magic functions
On Fri, Dec 23, 2016 at 5:14 AM, Mr. Wrobel wrote:
> Hi,thanx for answers, let's imagine that we want to add one class attribute
> for newly created classess with using __init__ in metaclass, here's an
> example:
>
> #!/usr/bin/env python
>
> class MetaClass(type):
> # __init__ manipulation:
>
> def __init__(cls, name, bases, dct):
> dct['added_in_init'] = 'test'
> super(MetaClass, cls).__init__(name, bases, dct)
>
> class BaseClass(object):
> __metaclass__ = MetaClass
>
> class NewBaseClass(BaseClass):
> pass
>
> print("Lets print attributes added in __init__ in base classes:")
>
> print(BaseClass.added_in_init)
>
> print(NewBaseClass.added_in_init)
>
> after running it: AttributeError: type object 'BaseClass' has no attribute
> 'added_in_init'
>
> Adding the same in __new__ works. Can anyone explain me please what's wrong?
When __init__ is called, the class has already been constructed by
__new__, and the 'dct' argument has already been copied into the class
dict. The base __init__ method does nothing, so adding the item to dct
and calling up doesn't accomplish anything.
Instead, the 'cls' argument that gets passed into __init__ is the
newly constructed class, so just use that to set the attributes:
cls.added_in_init = 'test'
--
https://mail.python.org/mailman/listinfo/python-list
dateutil timezone question
I have a datetime that looks like this: '2016-11-11T18:10:09-05:00' and when I pass it to dateutil.parser.parse I get back this: datetime.datetime(2016, 11, 11, 18, 10, 9, tzinfo=tzoffset(None, -18000)) And I have other datetimes like this: '2016-04-27T00:00:00', which went passed to dateutil.parser.parse of course does not return a datetime with the tzinfo. I need to compare these datetimes, and if I do that I get the dreaded "can't compare offset-naive and offset-aware datetimes" error. Is there a way I can get it back without the tzinfo, but instead with the offset applied to the date, so I can compare these 2? In other words I want it to return datetime.datetime(2016, 11, 11, 13, 10, 9) -- https://mail.python.org/mailman/listinfo/python-list
Re: Another security question
On Fri, 23 Dec 2016 10:08 pm, Frank Millman wrote:
> "Steve D'Aprano" wrote in message
> news:585d009f$0$1599$c3e8da3$54964...@news.astraweb.com...
>>
>> On Fri, 23 Dec 2016 09:19 pm, Frank Millman wrote:
>>
>> >
>> > 3. Generate the password from the string supplied by the user as
>> > follows -
>> > from hashlib import blake2b
>> > password = blake2b('my_password'.encode('utf-8'),
>> > salt=salt).digest()
>> >
>> > The hashlib docs have the following warning -
>> >
>> > "Salted hashing (or just hashing) with BLAKE2 or any other
>> > general-purpose
>> > cryptographic hash function, such as SHA-256, is not suitable for
>> > hashing
>> > passwords. See BLAKE2 FAQ for more information."
>>
>> Why are using Blake2 when the docs explicitly say not to use them in this
>> way? Have you read the FAQ to see what it says?
>>
>
> Why am I using Blake2? Well, before today I had not heard of it. However,
> in the past, when I needed to create a hashed password, I used the
> built-in hashlib module. Today, when I look at the docs for hashlib in
> Python 3.6, this is the new sub-heading -
> "15.2. hashlib — BLAKE2 hash functions"
>
> So it appears that this is the new preferred way of doing it.
What makes a good hash function for passwords is not the same as a good
general purpose hash function, cryptographic or not.
You can read more about this:
http://security.blogoverflow.com/2013/09/about-secure-password-hashing/
https://crackstation.net/hashing-security.htm
http://www.darkreading.com/safely-storing-user-passwords-hashing-vs-encrypting/a/d-id/1269374
but the TL;DR is that any of the general-purpose hash functions -- md5,
sha-1, sha-2 (sha-256 or sha-512), sha-3 or BLAKE2 -- are poor choices
because they're *too fast*, or have other vulnerabilities, or both.
> This is what the Blake2 FAQ says -
>
> "You shouldn't use *any* general-purpose hash function for user passwords,
> not BLAKE2, and not MD5, SHA-1, SHA-256, or SHA-3. Instead you should use
> a password hashing function such as the PHC winner Argon2 with appropriate
> time and memory cost parameters, to mitigate the risk of bruteforce
> attacks—Argon2's core uses a variant of BLAKE2's permutation"
>
> I see that there is a Python implementation of Argon2 in PyPi, but I don't
> particularly want to add another dependency to my app. My gut-feel says
> that this is overkill for my requirement. However, I am not sure. That is
> partly why I started this thread, to get some counter-arguments.
I have no opinion about Argon2, but I too would be reluctant to use an
external dependency of unknown quality.
The tried and tested password hashing functions are PBKDF2, bcrypt and
scrypt, with bcrypt generally considered the "boring, reliable" solution.
But there's no Python standard library implementation, which is sad.
crypt is also said to be sufficiently strong, but only some versions, and I
believe it is Unix/Linux only.
https://docs.python.org/3.5/library/crypt.html#module-crypt
There is a stdlib PBKDF2. If you want to avoid third-party dependencies, use
that.
https://docs.python.org/3.4/library/hashlib.html#hashlib.pbkdf2_hmac
By the way, thanks for raising this interesting question! This is exactly
the sort of thing that the secrets module is supposed to make a "no
brainer", so I expect that it will get a password hash function.
--
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.
--
https://mail.python.org/mailman/listinfo/python-list
Re: dateutil timezone question
On Sat, Dec 24, 2016 at 3:30 AM, Larry Martell wrote: > I have a datetime that looks like this: '2016-11-11T18:10:09-05:00' > and when I pass it to dateutil.parser.parse I get back this: > > datetime.datetime(2016, 11, 11, 18, 10, 9, tzinfo=tzoffset(None, -18000)) > > And I have other datetimes like this: '2016-04-27T00:00:00', which > went passed to dateutil.parser.parse of course does not return a > datetime with the tzinfo. > > I need to compare these datetimes, and if I do that I get the dreaded > "can't compare offset-naive and offset-aware datetimes" error. Some of your timestamps have timezones and others don't. That's a fundamental problem. Are you absolutely certain that the ones without them are in UTC? If so, the easiest fix would be to slap a "Z" on them before you parse, which would give you exclusively aware datetimes. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Another security question
On Sat, Dec 24, 2016 at 3:58 AM, Steve D'Aprano wrote: > By the way, thanks for raising this interesting question! This is exactly > the sort of thing that the secrets module is supposed to make a "no > brainer", so I expect that it will get a password hash function. +1. Please can we see something like Werkzeug's functions added? They're the no-brainer option for anything that already has Werkzeug as a dependency (eg Flask apps), and if that were in the stdlib, they'd be the no-brainer option for 3.7+ programs. Should I take this to -ideas? ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: dateutil timezone question
> I need to compare these datetimes, and if I do that I get the dreaded
> "can't compare offset-naive and offset-aware datetimes" error.
If you're sure the naive datetimes are UTC, this should work:
import pytz
dt = pytz.utc.localize(dateutil.parser.parse('2016-04-27T00:00:00'))
You can then compare it with other tz-containing datetime objects.
Skip
--
https://mail.python.org/mailman/listinfo/python-list
Re: US/Eastern offset
Okay, problem solved. My thanks to Chris Barker over on the Anaconda group
for help. (I originally thought there might be something amiss with the
pytz package in Anaconda, as our older non-Anaconda Python seemed not to
have the problem.) It turns out to be a problem I solved several years ago
at my previous employer (but forgot I had solved, and for perhaps obvious
reasons, I wasn't able to take my solution with me - another advantage for
open source...). It all boils down to the difference between these two
operations:
now.replace(tzinfo=eastern_tz)
eastern_tz.astimezone(now)
where now is datetime.datetime.now() and eastern_tz is
pytz.timezone("America/Eastern").
So, my problem has (again) been solved. A colleague here has suggested arrow
as an alternative to
datetime+pytz, which I will look into during the slow period next week.
Skip
--
https://mail.python.org/mailman/listinfo/python-list
Re: Metaclasses - magic functions
W dniu 23.12.2016 o 15:14, Ian Kelly pisze:
(...)
cls.added_in_init = 'test'
Man, you are awsome genius! Finally somebody was able to explain me what
is the power of __new__ and difference between __init__ !!!
So what I wanted to achieve was adding some new attributes to the class
instances (objects) like this:
#!/usr/bin/env python
class MetaClass(type):
# __init__ manipulation:
def __init__(cls, name, bases, dct):
cls.added_in_init = 'test'
super(MetaClass, cls).__init__(name, bases, dct)
class BaseClass(object):
__metaclass__ = MetaClass
class NewBaseClass(BaseClass):
def __init__(self):
self.inst_attr = self.added_in_init
print("Lets print attributes added in __init__ in base classes:")
b = NewBaseClass()
print(b.added_in_init)
print(b.inst_attr)
And finally it works!
Man I owe you a galaxy of thanks, because I was so frustrated! The
examples taken from internet never explained me it in so clearly like
you did!
Really, really thank you!!!
--
https://mail.python.org/mailman/listinfo/python-list
Re: dateutil timezone question
On Fri, Dec 23, 2016 at 2:18 PM, Chris Angelico wrote: > On Sat, Dec 24, 2016 at 3:30 AM, Larry Martell > wrote: >> I have a datetime that looks like this: '2016-11-11T18:10:09-05:00' >> and when I pass it to dateutil.parser.parse I get back this: >> >> datetime.datetime(2016, 11, 11, 18, 10, 9, tzinfo=tzoffset(None, -18000)) >> >> And I have other datetimes like this: '2016-04-27T00:00:00', which >> went passed to dateutil.parser.parse of course does not return a >> datetime with the tzinfo. >> >> I need to compare these datetimes, and if I do that I get the dreaded >> "can't compare offset-naive and offset-aware datetimes" error. > > Some of your timestamps have timezones and others don't. That's a > fundamental problem. Are you absolutely certain that the ones without > them are in UTC? If so, the easiest fix would be to slap a "Z" on them > before you parse, which would give you exclusively aware datetimes. What I ended up doing, which I think is somewhat kludgy is this: def add_tz(datestr): if datestr is None or len(datestr) < 6: return datestr if datestr[-6] == ':': return datestr+'-00:00' else: return datestr Then I pass all my dates through that before calling dateutil.parser.parse -- https://mail.python.org/mailman/listinfo/python-list
Re: dateutil timezone question
On Fri, Dec 23, 2016 at 2:27 PM, Skip Montanaro
wrote:
>> I need to compare these datetimes, and if I do that I get the dreaded
>> "can't compare offset-naive and offset-aware datetimes" error.
>
> If you're sure the naive datetimes are UTC, this should work:
>
> import pytz
>
> dt = pytz.utc.localize(dateutil.parser.parse('2016-04-27T00:00:00'))
>
> You can then compare it with other tz-containing datetime objects.
I did mess around with pytz a bit but I was getting a lot of
exceptions - something related to the TZ already being set or
something like that. I don't recall exactly, and I can't scroll back
far enough to find it.
--
https://mail.python.org/mailman/listinfo/python-list
Re: US/Eastern offset
On Thursday, December 22, 2016 at 5:57:42 PM UTC-8, Chris Angelico wrote: > On Fri, Dec 23, 2016 at 12:54 PM, wrote: > > Wouldn't most users prefer that modern time zones be the default > > information returned by pytz, instead of 150 year-old historical time zones? > > They're the *same* time zones. > > ChrisA Apparently, they're not quite the same. The four-minute discrepancy between New York local (mean solar?) time and the modern time zone is what got Skip Montanaro asking questions. -- https://mail.python.org/mailman/listinfo/python-list
Re: dateutil timezone question
> I did mess around with pytz a bit but I was getting a lot of
> exceptions - something related to the TZ already being set or
> something like that. I don't recall exactly, and I can't scroll back
> far enough to find it.
Yes, if the tzinfo attribute has already been set, you will get errors from
localize(). Once a datetime object has a valid timezone, you can compare it
with other tz-aware datetime objects. They need not be in the same timezone.
For visual purposes (for instance, when writing to logfiles), it can be handy
to display all timestamps in the same timezone. For that, you can use the
normalize() method of timezone objects. Here's a simple example:
>>> import datetime, pytz
Here's a datetime in UTC, but still naive (no tzinfo attribute).
>>> utcnow = datetime.datetime.utcnow()
>>> print utcnow.tzinfo
None
Localize to UTC et voila!
>>> utcnow = pytz.utc.localize(utcnow)
>>> print utcnow.tzinfo
UTC
>>> chicago = pytz.timezone("America/Chicago")
>>> print utcnow
2016-12-23 20:04:51.295127+00:00
>>> print chicago.normalize(utcnow)
2016-12-23 14:04:51.295127-06:00
Different timezones, but the same time.
>>> print utcnow - chicago.normalize(utcnow)
0:00:00
Skip
--
https://mail.python.org/mailman/listinfo/python-list
data frame
Hi all,
#!/usr/bin/env python
import sys
import csv
import numpy as np
import pandas as pd
a= pd.read_csv("s1.csv")
print(a)
size w1 h1
0 512 214 26
1 123 250 34
2 234 124 25
3 334 213 43
4 a45 223 32
5 a12 214 26
I wanted to create a new column by adding the two column values
as follows
a['test'] = a['w1'] + a['h1']
Traceback (most recent call last):
File
"/data/apps/Intel/intelpython35/lib/python3.5/site-packages/pandas/indexes/base.py",
line 2104, in get_loc
return self._engine.get_loc(key)
File "pandas/index.pyx", line 139, in pandas.index.IndexEngine.get_loc
(pandas/index.c:4152)
File "pandas/index.pyx", line 161, in pandas.index.IndexEngine.get_loc
(pandas/index.c:4016)
File "pandas/src/hashtable_class_helper.pxi", line 732, in
pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13153)
File "pandas/src/hashtable_class_helper.pxi", line 740, in
pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13107)
KeyError: 'w1'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "tt.py", line 16, in
a['test']=a['w1'] + a['h1']
File "pandas/src/hashtable_class_helper.pxi", line 740, in
pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13107)
KeyError: 'w1'
Can someone help me what the problem is?
Thank you in advance
--
https://mail.python.org/mailman/listinfo/python-list
Re: US/Eastern offset
On Sat, Dec 24, 2016 at 6:51 AM, wrote: > On Thursday, December 22, 2016 at 5:57:42 PM UTC-8, Chris Angelico wrote: >> On Fri, Dec 23, 2016 at 12:54 PM, wrote: >> > Wouldn't most users prefer that modern time zones be the default >> > information returned by pytz, instead of 150 year-old historical time >> > zones? >> >> They're the *same* time zones. >> >> ChrisA > > Apparently, they're not quite the same. The four-minute discrepancy between > New York local (mean solar?) time and the modern time zone is what got Skip > Montanaro asking questions. Is "US/Eastern" the same thing as "America/New_York"? According to my system, they are symlinks to the same content. So they are actually the same time zone. It's a time zone that has different UTC offsets at different points in time, both historic (the change from local to standard time) and modern (a twice-yearly shift in clocks), but it's the same timezone. If you look at any other form that means "New_York" (eg "EST"), and then set the date way back, you should see the same phenomenon. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: data frame
On Sat, Dec 24, 2016 at 7:39 AM, Val Krem via Python-list
wrote:
> a= pd.read_csv("s1.csv")
> File "pandas/src/hashtable_class_helper.pxi", line 740, in
> pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13107)
> KeyError: 'w1'
>
> Can someone help me what the problem is?
Can you post the first line of the CSV file? Maybe it isn't 'w1' exactly.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: data frame
Val Krem via Python-list wrote:
> Hi all,
>
> #!/usr/bin/env python
> import sys
> import csv
> import numpy as np
> import pandas as pd
>
> a= pd.read_csv("s1.csv")
> print(a)
>
> size w1 h1
> 0 512 214 26
> 1 123 250 34
> 2 234 124 25
> 3 334 213 43
> 4 a45 223 32
> 5 a12 214 26
>
> I wanted to create a new column by adding the two column values
> as follows
>
> a['test'] = a['w1'] + a['h1']
>
> Traceback (most recent call last):
> File
> "/data/apps/Intel/intelpython35/lib/python3.5/site-
packages/pandas/indexes/base.py",
> line 2104, in get_loc return self._engine.get_loc(key) File
> "pandas/index.pyx", line 139, in pandas.index.IndexEngine.get_loc
> (pandas/index.c:4152) File "pandas/index.pyx", line 161, in
> pandas.index.IndexEngine.get_loc (pandas/index.c:4016) File
> "pandas/src/hashtable_class_helper.pxi", line 732, in
> pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13153)
> File "pandas/src/hashtable_class_helper.pxi", line 740, in
> pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13107)
> KeyError: 'w1'
>
> During handling of the above exception, another exception occurred:
>
> Traceback (most recent call last):
> File "tt.py", line 16, in
> a['test']=a['w1'] + a['h1']
>
> File "pandas/src/hashtable_class_helper.pxi", line 740, in
> pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13107)
> KeyError: 'w1'
>
> Can someone help me what the problem is?
>
> Thank you in advance
Have a look at a.keys(). I suspect that the column name has extra space:
>>> pd.read_csv("s1.csv").keys()
Index([u'size', u' w1', u' h1'], dtype='object')
I that's what you see you can fix it by reading the csv with
skipinitialspace=True:
>>> pd.read_csv("s1.csv", skipinitialspace=True).keys()
Index([u'size', u'w1', u'h1'], dtype='object')
--
https://mail.python.org/mailman/listinfo/python-list
Multiprocessing queues receiving from wrong process
I was looking to avoid using a upd connection to transfer messages
between processes, so I thought I'd use multiprocessing (which I expect
would be faster), but...I sure would appreciate an explanation of this
problem.
When I run the code (below) instead of messages receiving messages from
the correct process I get:
(where the first number of a line identifies the index assigned to the
process.)
waiting for completion
1=Process-2 received: at time 0.001243: Process-1 says Hi to 0
0=Process-1 received: at time 0.001211: Process-2 says Hi to 1
0=Process-1 received: at time 0.001144: Process-3 says Hi to 2
4=Process-5 received: at time 0.002324: Process-1 says Hi to 0
0=Process-1 received: at time 0.000953: Process-4 says Hi to 3
0=Process-1 received: at time 0.000674: Process-5 says Hi to 4
3=Process-4 received: at time 0.002114: Process-1 says Hi to 0
3=Process-4 received: at time 0.001864: Process-2 says Hi to 1
4=Process-5 received: at time 0.002094: Process-2 says Hi to 1
2=Process-3 received: at time 0.001711: Process-1 says Hi to 0
4=Process-5 received: at time 0.001885: Process-3 says Hi to 2
4=Process-5 received: at time 0.001586: Process-4 says Hi to 3
1=Process-2 received: at time 0.001456: Process-3 says Hi to 2
3=Process-4 received: at time 0.001734: Process-3 says Hi to 2
2=Process-3 received: at time 0.00158: Process-2 says Hi to 1
2=Process-3 received: at time 0.001444: Process-4 says Hi to 3
2=Process-3 received: at time 0.001088: Process-5 says Hi to 4
3=Process-4 received: at time 0.001221: Process-5 says Hi to 4
1=Process-2 received: at time 0.001212: Process-4 says Hi to 3
1=Process-2 received: at time 0.000885: Process-5 says Hi to 4
##Test multiprocessing queues
importmultiprocessingasmp
importtime
frommultiprocessingimportProcess
frommultiprocessingimportQueue
fromqueueimportEmpty
fromqueueimportFull
fromrandomimportrandom
class TestMPQ:
""" Class doc """
def __init__ (self, ndx):
""" Class initialiser """
self.name=mp.current_process().name
self.ndx=ndx
defsendHi (self):
for i in range(5):
if i != self.ndx:
qs[i].put ("{} says Hi to {}".format(self.name, self.ndx))
defprocessHi (self):
while (True):
time.sleep(random() + 0.001)
try:
msg=qs[self.ndx].get_nowait()
print ("{}={} received: {}".format(self.ndx, self.name,
msg) )
exceptEmpty:
break
except Exception as ex:
print ("Exception: ", repr(ex))
break
defprocHandler (ndx, qs):
p=TestMPQ(ndx)
p.sendHi()
p.processHi()
if "__main__" == __name__:
qs=[]
for i in range(5):
qs.append(Queue())
ps=[]
for i in range(5):
ps.append(Process(target = procHandler, args = (i, qs) ) )
ps[i].start()
print ("waiting for completion")
for i in range(5):
ps[i].join()
--
https://mail.python.org/mailman/listinfo/python-list
Re: data frame
Here is the first few lines of the data
s1.csv
size,w1,h1
512,214,26
123,250,34
234,124,25
334,213,43
and the script
a=pd.read_csv("s1.csv", skipinitialspace=True).keys()
print(a)
i see the following
Index(['size', 'w1', 'h1'], dtype='object')
when I wanted to add the two columns; then I get the following message.
a=pd.read_csv("s1.csv", skipinitialspace=True).keys()
a['test']=a['w1'] + a['h1']
print(a)
data/apps/Intel/intelpython35/lib/python3.5/site-packages/pandas/indexes/base.py:1393:
VisibleDeprecationWarning: using a non-integer number instead of an integer
will result in an error in the future
return getitem(key)
Traceback (most recent call last):
File "tt.py", line 12, in
a['test']=a['w1'] + a['h1']
File
"/data/apps/Intel/intelpython35/lib/python3.5/site-packages/pandas/indexes/base.py",
line 1393, in __getitem__
return getitem(key)
IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis
(`None`) and integer or boolean arrays are valid indices
On Friday, December 23, 2016 3:09 PM, Peter Otten <__pete...@web.de> wrote:
Val Krem via Python-list wrote:
> Hi all,
>
> #!/usr/bin/env python
> import sys
> import csv
> import numpy as np
> import pandas as pd
>
> a= pd.read_csv("s1.csv")
> print(a)
>
> size w1 h1
> 0 512 214 26
> 1 123 250 34
> 2 234 124 25
> 3 334 213 43
> 4 a45 223 32
> 5 a12 214 26
>
> I wanted to create a new column by adding the two column values
> as follows
>
> a['test'] = a['w1'] + a['h1']
>
> Traceback (most recent call last):
> File
> "/data/apps/Intel/intelpython35/lib/python3.5/site-
packages/pandas/indexes/base.py",
> line 2104, in get_loc return self._engine.get_loc(key) File
> "pandas/index.pyx", line 139, in pandas.index.IndexEngine.get_loc
> (pandas/index.c:4152) File "pandas/index.pyx", line 161, in
> pandas.index.IndexEngine.get_loc (pandas/index.c:4016) File
> "pandas/src/hashtable_class_helper.pxi", line 732, in
> pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13153)
> File "pandas/src/hashtable_class_helper.pxi", line 740, in
> pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13107)
> KeyError: 'w1'
>
> During handling of the above exception, another exception occurred:
>
> Traceback (most recent call last):
> File "tt.py", line 16, in
> a['test']=a['w1'] + a['h1']
>
> File "pandas/src/hashtable_class_helper.pxi", line 740, in
> pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13107)
> KeyError: 'w1'
>
> Can someone help me what the problem is?
>
> Thank you in advance
Have a look at a.keys(). I suspect that the column name has extra space:
>>> pd.read_csv("s1.csv").keys()
Index([u'size', u' w1', u' h1'], dtype='object')
I that's what you see you can fix it by reading the csv with
skipinitialspace=True:
>>> pd.read_csv("s1.csv", skipinitialspace=True).keys()
Index([u'size', u'w1', u'h1'], dtype='object')
--
https://mail.python.org/mailman/listinfo/python-list
/data/apps/Intel/intelpython35/lib/python3.5/site-packages/pandas/indexes/base.py:1393:
VisibleDeprecationWarning: using a non-integer number instead of an integer
will result in an error in the future
return getitem(key)
Traceback (most recent call last):
File "tt.py", line 12, in
a['test']=a['w1'] + a['h1']
File
"/data/apps/Intel/intelpython35/lib/python3.5/site-packages/pandas/indexes/base.py",
line 1393, in __getitem__
return getitem(key)
IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis
(`None`) and integer or boolean arrays are valid indices
On Friday, December 23, 2016 3:09 PM, Peter Otten <__pete...@web.de> wrote:
Val Krem via Python-list wrote:
> Hi all,
>
> #!/usr/bin/env python
> import sys
> import csv
> import numpy as np
> import pandas as pd
>
> a= pd.read_csv("s1.csv")
> print(a)
>
> size w1 h1
> 0 512 214 26
> 1 123 250 34
> 2 234 124 25
> 3 334 213 43
> 4 a45 223 32
> 5 a12 214 26
>
> I wanted to create a new column by adding the two column values
> as follows
>
> a['test'] = a['w1'] + a['h1']
>
> Traceback (most recent call last):
> File
> "/data/apps/Intel/intelpython35/lib/python3.5/site-
packages/pandas/indexes/base.py",
> line 2104, in get_loc return self._engine.get_loc(key) File
> "pandas/index.pyx", line 139, in pandas.index.IndexEngine.get_loc
> (pandas/index.c:4152) File "pandas/index.pyx", line 161, in
> pandas.index.IndexEngine.get_loc (pandas/index.c:4016) File
> "pandas/src/hashtable_class_helper.pxi", line 732, in
> pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13153)
> File "pandas/src/hashtable_class_helper.pxi", line 740, in
> pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13107)
> KeyError: 'w1'
>
> During handling of the above exception, another exception occurred:
>
> Traceback (most recent call last):
> File "tt.py", line 16, in
> a['test']=a['w1'] + a['h1']
>
> File "pandas/src/hashtable_class_helper.pxi", line 740, in
> pandas.
Python 3.6 on Centos 6
I'm trying to build Python 3.6 on Centos 6, and am successful in doing so,
except for the sqlite3 library. I started with a brand new install of Centos 6
and installed devtoolset-2 to build with a newer compiler. But whether with
default compiler or 4.82, I get the following errors when building the sqlite3
module. sqlite & sqlite-devel are installed.
[code]
building '_sqlite3' extension
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall
-Wstrict-prototypes -fPIC -DMODULE_NAME="sqlite3" -IModules/_sqlite
-I/usr/include -I./Include -I. -I/usr/local/include -I/tmp/Python-3.6.0/Include
-I/tmp/Python-3.6.0 -c /tmp/Python-3.6.0/Modules/_sqlite/cache.c -o
build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/cache.o
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall
-Wstrict-prototypes -fPIC -DMODULE_NAME="sqlite3" -IModules/_sqlite
-I/usr/include -I./Include -I. -I/usr/local/include -I/tmp/Python-3.6.0/Include
-I/tmp/Python-3.6.0 -c /tmp/Python-3.6.0/Modules/_sqlite/connection.c -o
build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/connection.o
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall
-Wstrict-prototypes -fPIC -DMODULE_NAME="sqlite3" -IModules/_sqlite
-I/usr/include -I./Include -I. -I/usr/local/include -I/tmp/Python-3.6.0/Include
-I/tmp/Python-3.6.0 -c /tmp/Python-3.6.0/Modules/_sqlite/cursor.c -o
build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/cursor.o
/tmp/Python-3.6.0/Modules/_sqlite/cursor.c: In function
‘_pysqlite_query_execute’:
/tmp/Python-3.6.0/Modules/_sqlite/cursor.c:517:5: warning: implicit declaration
of function ‘sqlite3_stmt_readonly’ [-Wimplicit-function-declaration]
if (self->connection->begin_statement &&
!sqlite3_stmt_readonly(self->statement->st) && !self->statement->is_ddl) {
^
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall
-Wstrict-prototypes -fPIC -DMODULE_NAME="sqlite3" -IModules/_sqlite
-I/usr/include -I./Include -I. -I/usr/local/include -I/tmp/Python-3.6.0/Include
-I/tmp/Python-3.6.0 -c /tmp/Python-3.6.0/Modules/_sqlite/microprotocols.c -o
build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/microprotocols.o
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall
-Wstrict-prototypes -fPIC -DMODULE_NAME="sqlite3" -IModules/_sqlite
-I/usr/include -I./Include -I. -I/usr/local/include -I/tmp/Python-3.6.0/Include
-I/tmp/Python-3.6.0 -c /tmp/Python-3.6.0/Modules/_sqlite/module.c -o
build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/module.o
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall
-Wstrict-prototypes -fPIC -DMODULE_NAME="sqlite3" -IModules/_sqlite
-I/usr/include -I./Include -I. -I/usr/local/include -I/tmp/Python-3.6.0/Include
-I/tmp/Python-3.6.0 -c /tmp/Python-3.6.0/Modules/_sqlite/prepare_protocol.c -o
build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/prepare_protocol.o
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall
-Wstrict-prototypes -fPIC -DMODULE_NAME="sqlite3" -IModules/_sqlite
-I/usr/include -I./Include -I. -I/usr/local/include -I/tmp/Python-3.6.0/Include
-I/tmp/Python-3.6.0 -c /tmp/Python-3.6.0/Modules/_sqlite/row.c -o
build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/row.o
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall
-Wstrict-prototypes -fPIC -DMODULE_NAME="sqlite3" -IModules/_sqlite
-I/usr/include -I./Include -I. -I/usr/local/include -I/tmp/Python-3.6.0/Include
-I/tmp/Python-3.6.0 -c /tmp/Python-3.6.0/Modules/_sqlite/statement.c -o
build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/statement.o
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall
-Wstrict-prototypes -fPIC -DMODULE_NAME="sqlite3" -IModules/_sqlite
-I/usr/include -I./Include -I. -I/usr/local/include -I/tmp/Python-3.6.0/Include
-I/tmp/Python-3.6.0 -c /tmp/Python-3.6.0/Modules/_sqlite/util.c -o
build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/util.o
gcc -pthread -shared -Wl,--rpath=/usr/lib
build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/cache.o
build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/connection.o
build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/cursor.o
build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/microprotocols.o
build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/module.o
build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/prepare_protocol.o
build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/row.o
build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/statement.o
build/temp.linux-x86_64-3.6/tmp/Python-3.6.0/Modules/_sqlite/util.o -L.
-L/usr/local/lib -lsqlite3 -lpython3.6m -o
build/lib.linux-x86_64-3.6/_sqlite3.cpython-36m-x86_64-linux-gnu.so
warning: building with the bundled copy of libffi is deprecated on this
Re: Multiprocessing queues receiving from wrong process
On 2016-12-23 21:56, Charles Hixson wrote:
I was looking to avoid using a upd connection to transfer messages
between processes, so I thought I'd use multiprocessing (which I expect
would be faster), but...I sure would appreciate an explanation of this
problem.
When I run the code (below) instead of messages receiving messages from
the correct process I get:
(where the first number of a line identifies the index assigned to the
process.)
waiting for completion
1=Process-2 received: at time 0.001243: Process-1 says Hi to 0
0=Process-1 received: at time 0.001211: Process-2 says Hi to 1
0=Process-1 received: at time 0.001144: Process-3 says Hi to 2
4=Process-5 received: at time 0.002324: Process-1 says Hi to 0
0=Process-1 received: at time 0.000953: Process-4 says Hi to 3
0=Process-1 received: at time 0.000674: Process-5 says Hi to 4
3=Process-4 received: at time 0.002114: Process-1 says Hi to 0
3=Process-4 received: at time 0.001864: Process-2 says Hi to 1
4=Process-5 received: at time 0.002094: Process-2 says Hi to 1
2=Process-3 received: at time 0.001711: Process-1 says Hi to 0
4=Process-5 received: at time 0.001885: Process-3 says Hi to 2
4=Process-5 received: at time 0.001586: Process-4 says Hi to 3
1=Process-2 received: at time 0.001456: Process-3 says Hi to 2
3=Process-4 received: at time 0.001734: Process-3 says Hi to 2
2=Process-3 received: at time 0.00158: Process-2 says Hi to 1
2=Process-3 received: at time 0.001444: Process-4 says Hi to 3
2=Process-3 received: at time 0.001088: Process-5 says Hi to 4
3=Process-4 received: at time 0.001221: Process-5 says Hi to 4
1=Process-2 received: at time 0.001212: Process-4 says Hi to 3
1=Process-2 received: at time 0.000885: Process-5 says Hi to 4
I don't see a bug, but perhaps that's because it's not clear to me what
you expected to see.
##Test multiprocessing queues
importmultiprocessingasmp
importtime
frommultiprocessingimportProcess
frommultiprocessingimportQueue
fromqueueimportEmpty
fromqueueimportFull
fromrandomimportrandom
class TestMPQ:
""" Class doc """
def __init__ (self, ndx):
""" Class initialiser """
self.name=mp.current_process().name
self.ndx=ndx
defsendHi (self):
for i in range(5):
if i != self.ndx:
qs[i].put ("{} says Hi to {}".format(self.name, self.ndx))
defprocessHi (self):
while (True):
time.sleep(random() + 0.001)
try:
msg=qs[self.ndx].get_nowait()
print ("{}={} received: {}".format(self.ndx, self.name,
msg) )
exceptEmpty:
break
except Exception as ex:
print ("Exception: ", repr(ex))
break
defprocHandler (ndx, qs):
p=TestMPQ(ndx)
p.sendHi()
p.processHi()
if "__main__" == __name__:
qs=[]
for i in range(5):
qs.append(Queue())
ps=[]
for i in range(5):
ps.append(Process(target = procHandler, args = (i, qs) ) )
ps[i].start()
print ("waiting for completion")
for i in range(5):
ps[i].join()
--
https://mail.python.org/mailman/listinfo/python-list
Re: data frame
Val Krem via Python-list wrote:
> Here is the first few lines of the data
>
>
> s1.csv
> size,w1,h1
> 512,214,26
> 123,250,34
> 234,124,25
> 334,213,43
Did you put these lines here using copy and paste? The fix below depends on
the assumption that your data is more like
size, w1, h1
512, 214, 26
123, 250, 34
...
> a=pd.read_csv("s1.csv", skipinitialspace=True).keys()
You should use the keys() method call for diagnosis only. The final script
that might work if your problem is actually space after the commas is
import pandas as pd
a = pd.read_csv("s1.csv", skipinitialspace=True)
a["test"] = a["h1"] + a["w1"]
print(a)
--
https://mail.python.org/mailman/listinfo/python-list
Re: Another security question
> "Salted hashing (or just hashing) with BLAKE2 or any other > general-purpose cryptographic hash function, such as SHA-256, is not > suitable for hashing passwords. See BLAKE2 FAQ for more information." > > I propose to ignore this warning. I feel that, for my purposes, the > above procedure is adequate. > > Does all this sound reasonable? The basic problem is those functions are fast enough to make dictionary attacks feasible. The preferred password hashing function these days is Argon2, which has some tunable security parameters: https://en.wikipedia.org/wiki/Argon2 Also in use are bcrypt and later scrypt (I think there are wikipedia articles about both). What is it that you are trying to secure? If it's something important, set up 2-factor authentication (such as TOTP) and encourage your users to use it. There are cheap hardware tokens and cost-free smartphone apps that implement it on the user side. Linotp.org has a free implementation for the server side, though I haven't looked at it closely or tried it yet. -- https://mail.python.org/mailman/listinfo/python-list
Re: Multiprocessing queues receiving from wrong process
On 12/23/2016 01:56 PM, Charles Hixson wrote:
I was looking to avoid using a upd connection to transfer messages
between processes, so I thought I'd use multiprocessing (which I
expect would be faster), but...I sure would appreciate an explanation
of this problem.
When I run the code (below) instead of messages receiving messages
from the correct process I get:
(where the first number of a line identifies the index assigned to the
process.)
waiting for completion
The receiving process should be the one sent to.
receiving sending sent to
ndx process nameprocess
name ndx
1=Process-2 received: at time 0.001243: Process-1 says Hi to 0
0=Process-1 received: at time 0.001211: Process-2 says Hi to 1
0=Process-1 received: at time 0.001144: Process-3 says Hi to 2
4=Process-5 received: at time 0.002324: Process-1 says Hi to 0
0=Process-1 received: at time 0.000953: Process-4 says Hi to 3
0=Process-1 received: at time 0.000674: Process-5 says Hi to 4
3=Process-4 received: at time 0.002114: Process-1 says Hi to 0
3=Process-4 received: at time 0.001864: Process-2 says Hi to 1
4=Process-5 received: at time 0.002094: Process-2 says Hi to 1
2=Process-3 received: at time 0.001711: Process-1 says Hi to 0
4=Process-5 received: at time 0.001885: Process-3 says Hi to 2
4=Process-5 received: at time 0.001586: Process-4 says Hi to 3
1=Process-2 received: at time 0.001456: Process-3 says Hi to 2
3=Process-4 received: at time 0.001734: Process-3 says Hi to 2
2=Process-3 received: at time 0.00158: Process-2 says Hi to 1
2=Process-3 received: at time 0.001444: Process-4 says Hi to 3
2=Process-3 received: at time 0.001088: Process-5 says Hi to 4
3=Process-4 received: at time 0.001221: Process-5 says Hi to 4
1=Process-2 received: at time 0.001212: Process-4 says Hi to 3
1=Process-2 received: at time 0.000885: Process-5 says Hi to 4
##Test multiprocessing queues
importmultiprocessingasmp
importtime
frommultiprocessingimportProcess
frommultiprocessingimportQueue
fromqueueimportEmpty
fromqueueimportFull
fromrandomimportrandom
class TestMPQ:
""" Class doc """
def __init__ (self, ndx):
""" Class initialiser """
self.name=mp.current_process().name
self.ndx=ndx
defsendHi (self):
for i in range(5):
if i != self.ndx:
qs[i].put ("{} says Hi to {}".format(self.name,
self.ndx))
defprocessHi (self):
while (True):
time.sleep(random() + 0.001)
try:
msg=qs[self.ndx].get_nowait()
print ("{}={} received: {}".format(self.ndx,
self.name, msg) )
exceptEmpty:
break
except Exception as ex:
print ("Exception: ", repr(ex))
break
defprocHandler (ndx, qs):
p=TestMPQ(ndx)
p.sendHi()
p.processHi()
if "__main__" == __name__:
qs=[]
for i in range(5):
qs.append(Queue())
ps=[]
for i in range(5):
ps.append(Process(target = procHandler, args = (i, qs) ) )
ps[i].start()
print ("waiting for completion")
for i in range(5):
ps[i].join()
--
https://mail.python.org/mailman/listinfo/python-list
Re: Another security question
On Sat, Dec 24, 2016 at 11:20 AM, Paul Rubin wrote:
> The basic problem is those functions are fast enough to make dictionary
> attacks feasible. The preferred password hashing function these days is
> Argon2, which has some tunable security parameters:
Solution: Don't use dictionary-attackable passwords. Here's a quick
and stupid password verifier that uses MD5:
import os, base64, hashlib, timeit
def encrypt_password(pw):
salt = base64.b64encode(os.urandom(3))
return salt + b":" + hashlib.md5(salt + pw).hexdigest().encode("ascii")
def verify_password(pw, enc):
salt, hash = enc.split(b":")
return hash == hashlib.md5(salt + pw).hexdigest().encode("ascii")
crackme = encrypt_password(b"elven-their-cure-planning")
assert not verify_password(b"figure-dagger-personal-excited", crackme)
assert verify_password(b"elven-their-cure-planning", crackme)
print(timeit.repeat('verify_password(b"figure-dagger-personal-excited",
crackme)', globals=globals()))
# [0.9219889058731496, 0.9255746062844992, 0.9231259850785136,
0.9204203351400793, 0.9239354520104825]
Okay, so that's about as insecure as salted hashes can get, right? I
brute-forced a million of them a second, one core. Now, suppose you
know for sure that I used that format of password (four words, all
lower-case, separated by hyphens), and you have a copy of my 2000-word
dictionary from which I select words. It'll still take you 185 days to
exhaustively try every possible password. So on average, it'll take
you a day of searching with 93 computers to crack this kind of
password. And that's just using four characters of salt and a single
iteration of MD5. That's borderline infeasible, without needing any
sort of top-notch hashing system. Switch to werkzeug's functions
and... well, I had to tweak timeit.repeat, because it did about two
*thousand* per second. And that's with default parameters. You're
going to need five hundred times as much processing to crack those
things.
Use XKCD 936 passwords. Even if someone has your entire password
generation system, they're still pretty secure. If anything, they'll
be even harder to crack, because of the many variations you could use
(capitalization, separators, precise list of words), not to mention
that you can easily add a fifth word to make them even more secure.
(But only if you're paranoid. Memorizing a fifth word is harder.) It
works.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Another security question
On Sat, 24 Dec 2016 11:20 am, Paul Rubin wrote: > What is it that you are trying to secure? If it's something important, > set up 2-factor authentication (such as TOTP) and encourage your users > to use it. You say that as if two-factor auth was a panacea. That's the sort of thinking that leads to: https://www.schneier.com/blog/archives/2009/09/hacking_two-fac.html https://www.schneier.com/blog/archives/2005/10/scandinavian_at_1.html http://resources.infosecinstitute.com/two-factor-authentication/ http://www.securityweek.com/two-factor-authentication-bypassed-simple-attacks not to mention the abomination of "one factor authentication, twice", like that used by the Australian government unified web portal. To log in, you have to provide something you know (username and password), plus something else you know (answer to a low-security question like "what was your mother's maiden name?"). -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Re: Another security question
On Sat, Dec 24, 2016 at 12:32 PM, Steve D'Aprano wrote: > not to mention the abomination of "one factor authentication, twice", like > that used by the Australian government unified web portal. To log in, you > have to provide something you know (username and password), plus something > else you know (answer to a low-security question like "what was your > mother's maiden name?"). My mother's maiden name was here-campaigns-your-really. My first pet was expensive-items-know-thats. The street I grew up on was sorry-days-standard-just. My date of birth is notes-wondering-laptop-think. :D ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Multiprocessing queues receiving from wrong process
On 2016-12-24 01:17, Charles Hixson wrote:
On 12/23/2016 01:56 PM, Charles Hixson wrote:
I was looking to avoid using a upd connection to transfer messages
between processes, so I thought I'd use multiprocessing (which I
expect would be faster), but...I sure would appreciate an explanation
of this problem.
When I run the code (below) instead of messages receiving messages
from the correct process I get:
(where the first number of a line identifies the index assigned to the
process.)
waiting for completion
The receiving process should be the one sent to.
receiving sending sent to
ndx process nameprocess
name ndx
1=Process-2 received: at time 0.001243: Process-1 says Hi to 0
0=Process-1 received: at time 0.001211: Process-2 says Hi to 1
0=Process-1 received: at time 0.001144: Process-3 says Hi to 2
4=Process-5 received: at time 0.002324: Process-1 says Hi to 0
0=Process-1 received: at time 0.000953: Process-4 says Hi to 3
0=Process-1 received: at time 0.000674: Process-5 says Hi to 4
3=Process-4 received: at time 0.002114: Process-1 says Hi to 0
3=Process-4 received: at time 0.001864: Process-2 says Hi to 1
4=Process-5 received: at time 0.002094: Process-2 says Hi to 1
2=Process-3 received: at time 0.001711: Process-1 says Hi to 0
4=Process-5 received: at time 0.001885: Process-3 says Hi to 2
4=Process-5 received: at time 0.001586: Process-4 says Hi to 3
1=Process-2 received: at time 0.001456: Process-3 says Hi to 2
3=Process-4 received: at time 0.001734: Process-3 says Hi to 2
2=Process-3 received: at time 0.00158: Process-2 says Hi to 1
2=Process-3 received: at time 0.001444: Process-4 says Hi to 3
2=Process-3 received: at time 0.001088: Process-5 says Hi to 4
3=Process-4 received: at time 0.001221: Process-5 says Hi to 4
1=Process-2 received: at time 0.001212: Process-4 says Hi to 3
1=Process-2 received: at time 0.000885: Process-5 says Hi to 4
##Test multiprocessing queues
importmultiprocessingasmp
importtime
frommultiprocessingimportProcess
frommultiprocessingimportQueue
fromqueueimportEmpty
fromqueueimportFull
fromrandomimportrandom
class TestMPQ:
""" Class doc """
def __init__ (self, ndx):
""" Class initialiser """
self.name=mp.current_process().name
self.ndx=ndx
defsendHi (self):
for i in range(5):
if i != self.ndx:
qs[i].put ("{} says Hi to {}".format(self.name,
self.ndx))
"self.ndx" is the sender's (my) index, "i" is the receiver's (your
index), so the message you're building is:
says Hi to
which you then put in the receiver's (your) queue.
defprocessHi (self):
while (True):
time.sleep(random() + 0.001)
try:
msg=qs[self.ndx].get_nowait()
print ("{}={} received: {}".format(self.ndx,
self.name, msg) )
exceptEmpty:
break
except Exception as ex:
print ("Exception: ", repr(ex))
break
defprocHandler (ndx, qs):
p=TestMPQ(ndx)
p.sendHi()
p.processHi()
if "__main__" == __name__:
qs=[]
for i in range(5):
qs.append(Queue())
ps=[]
for i in range(5):
ps.append(Process(target = procHandler, args = (i, qs) ) )
ps[i].start()
print ("waiting for completion")
for i in range(5):
ps[i].join()
--
https://mail.python.org/mailman/listinfo/python-list
Re: Another security question
Chris Angelico writes: > Solution: Don't use dictionary-attackable passwords. If you allow people to choose their own passwords, they'll too-often pick dictionary-attackable ones; or even if they choose difficult ones, they'll use them in more than one place, and eventually the weakest of those places will eventually leak it. At that point it can be tried against whatever other hashes the attacker collected. The -real- right thing to do is use a secret-keyed hash function like HMAC-whatever, but that gives you a chicken-and-egg problem of how to get the secret into the system and prevent it from escaping, when you've postulated that the hashed passwords might escape. That's doable through hardware approaches like external crypto modules, TPM, secure enclaves in the CPU, etc.; but none of that is very widely deployed at the moment, and it brings its own unattractiveness. -- https://mail.python.org/mailman/listinfo/python-list
Re: Another security question
On Sat, Dec 24, 2016 at 6:18 PM, Paul Rubin wrote: > Chris Angelico writes: >> Solution: Don't use dictionary-attackable passwords. > > If you allow people to choose their own passwords, they'll too-often > pick dictionary-attackable ones; or even if they choose difficult ones, > they'll use them in more than one place, and eventually the weakest of > those places will eventually leak it. At that point it can be tried > against whatever other hashes the attacker collected. Correct. However, weak passwords are ultimately the user's responsibility, where the hashing is the server's responsibility. The one thing that you _can_ do as server admin is to make appropriate recommendations, including that if you have one of those "password is weak" warnings, make sure it favours length over apparent alphabet. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
