Re: OAuth 2.0 implementation

[Stuart Bishop]

On Tue, Mar 27, 2012 at 10:11 AM, Ben Finney  wrote:
> Demian Brecht  writes:
>
>> I'm getting close to an alpha release of an OAuth 2.0 implementation
>> (https://github.com/demianbrecht/py-sanction).
>
> Thank you for doing this work.
>
> As someone who uses OpenID, what can I read about why OAuth is better?

They are different, and often you need to use both.

OpenID allows web sites to authenticate someone. It is not really
useful for anything not an interactive web site. The consuming site
never gets your keys, it just gets confirmation from the provider that
the user is who they claim they are and maybe some details that the
provider chooses to provide such as an email address.

OAuth is for generating authentication keys that allow a program to
authenticate as someone and perform operations on their behalf. You
use OAuth to generate a key so that Foursquare can send messages via
Twitter on your behalf, or so the Facebook client on your phone can
access your account without storing your password. You also get
authentication here, as you can't generate a key without being
authenticated, but the real reason it is used instead of OpenID is so
you can keep the key and keep using it to act as the user; you can
keep using that key until it expires or it is revoked.

Authentication providers that don't provide a webapi just implement
OpenID. Big sites like Google and Facebook implement both OpenID (for
'log in with your GMail account') and OAuth ('post this message to
your Facebook wall').

-- 
Stuart Bishop 
http://www.stuartbishop.net/
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: how to write thread-safe module ? and pytz

[Stuart Bishop]

nicolas_riesch wrote:
> Does someone know if the module pytz
> (http://sourceforge.net/projects/pytz/) is thread-safe ?
> I have not seen it explicitely stated, and just wanted to be sure, as I
> want to use it.

pytz is thread safe.

> That's because in the file pytz/tzinfo.py, I see global variables
> _timedelta_cache, _datetime_cache, _ttinfo_cache, which are
> dictionnaries and are used as cache.

> I always thought that you must protect shared data with locks when
> multithreading, but I don't see any lock anywhere in pytz.
> However, pytz seems to work well with multiple threads creating various
> timezone objects at the same time.
> I don't understand where the trick is, that allows multiple threads to
> access this module without any locking and that all this seems to work
> without any problem...

Thanks to the global interpreter lock, with the Python builtin types you
only need to maintain a lock if there is a race condition, or if you care
about the race condition. For example, the following is thread safe code:

>>> from threading import Thread
>>> import time
>>> stack = []
>>> stack2 = []
>>> def doit(i):
... stack.append(i)
... time.sleep(0.1)
... stack2.append(stack.pop())
...
>>> threads = [Thread(target=doit, args=(i,)) for i in range(0,100)]
>>> for t in threads: t.start()
...
>>> for t in threads: t.join()
...
>>> len(stack2)
100
>>> stack2
[99, 95, 98, 94, 93, 97, 92, 91, 96, 88, 87, 86, 85, 84, 83, 90, 79, 78, 77,
76, 74, 73, 72, 71, 70, 75, 82, 81, 80, 89, 69, 67, 66, 65, 64, 68, 60, 59,
58, 57, 56, 55, 63, 62, 61, 49, 54, 53, 52, 51, 46, 45, 44, 50, 48, 47, 29,
28, 35, 34, 33, 43, 42, 41, 40, 39, 38, 32, 37, 31, 30, 36, 27, 26, 25, 24,
23, 22, 21, 20, 19, 18, 17, 12, 16, 15, 14, 13, 11, 10, 9, 8, 7, 6, 4, 3, 2,
1, 0, 5]

Note that the value being appended to 'stack2' might not be the value that
was appended to 'stack' in any particular thread - in this case, we don't
care (but is the sort of thing you might need to watch out for).

In the code you mention in pytz, there *is* a race condition. However, if
this condition occurs the side effects are so trivial as to not worry about
locking. ie. if a number of threads call memorized_timedelta(seconds=60)
simultaneously, there is a slight chance that each thread will get a
different timedelta instance. This is extremely unlikely, and the rest of
the code doesn't care at all. If pytz compared the timedeltas using 'is'
instead of '==' at any point, it would be a bug (but it doesn't, so it isn't).

So you can write thread safe Python code without locks provided you are
using the builtin types, and keep a close eye out for race conditions. This
might sound error prone, but it is quite doable provided the critical areas
that are accessing shared objects are kept isolated, short and simple.

Here is an thread unsafe example. Here the mistake is made that the length
of stack will not change after checking it. Also because we don't use the
atomic stack.pop(), two threads might add the same value to stack2:

>>> from threading import Thread
>>> import time
>>> stack = range(0, 50)
>>> stack2 = []
>>> def doit():
... if len(stack) > 0:
... stack2.append(stack[-1])
... time.sleep(0.1)
... del stack[-1]
...
>>> threads = [Thread(target=doit) for i in range(0, 100)]
>>> for t in threads: t.start()
...
Exception in thread Thread-249:
Traceback (most recent call last):
  File "/usr/lib/python2.4/threading.py", line 442, in __bootstrap
self.run()
  File "/usr/lib/python2.4/threading.py", line 422, in run
self.__target(*self.__args, **self.__kwargs)
  File "", line 5, in doit
IndexError: list assignment index out of range



-- 
Stuart Bishop <[EMAIL PROTECTED]>
http://www.stuartbishop.net/


signature.asc
Description: OpenPGP digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Postgres PL/Python

[Stuart Bishop]

Ksenia Marasanova wrote:

> I wonder if anyone on this list is using  Python as Postgres
> procedural language. I can't find a place for it i my mind. How would
> a typical MVC web application benefit from it  (besides performance)?
> I understand from the docs that Postgres 7.4 has PL/PythonU -
> unlimited functionality. It sounds like I have the whole Python
> available in Postgres. That means big parts of application logic can
> be moved to stored procedures, and dummy SQL layer becomes something
> else... sounds scary. Any opinions on this?

To start with, PL/PythonU procedures share the same use cases as any other
server side procedure:
- Implementing triggers
- Implementing non-trivial database constraints
- Cases where code will be much faster when run in the RDBMS
  environment rather than in the client. (as an obvious example,
  you are better off running 'SELECT sum(foo) FROM bar' than
  sucking all the results to your client and summing them there.
- Cases where the code can't be rolled out onto the clients. eg.
  it contains secrets, is proprietary and the client runs in
  untrusted environment.
- You need to access a resource only available on the database
  server, such as reading a secret from a file, logging
  information to a central log, making use of cached information
  (if each of your clients implements their own caching, it can
  be inconsistant between them)
- Code you want to run on transaction commit without having to
  put all that transactional logic in your client.
- Providing an interface to data that should not be available
  to clients. eg. validating a password is correct without
  needing to send the plaintext, encrypted or hashed password
  to the client.
- Wierd environments where it is easier to change code in the
  central database than on the clients (eg. the clients are
  installed on laptops that are not always on site or network
  connected, or internal beurocracy makes updating clients
  non trivial).
- Allows logic to be coded and maintained in one place
  and in one language by someone who understands the database
  and the data, rather than having it reimplemented for each
  different client environment, probably incorrectly and
  possibly dangerously

Of course, only some or none of these may be appropriate to your
environment. Or you might have other use cases. I doubt anyone could
actually agree on what 'a typical MVC web application' is. If you have a
trivial system where the database and a single web app run on the same
server and are maintained and developed by the same people, I doubt there is
any reason to use stored procedures. But when you go beyond that you might
find you do.

PL/PythonU procedures have the extra nice use case of being able to share
code between database and clients when the clients are written in Python
(and similarly PL/Perl, PL/Tcl or PL/Java when the clients share the
language). This means your database constraints can use the same algorithms
that the clients are supposed to be using, providing a safety net in case
someone forgets in their code to validate the data before tossing it into
the database or a client for some reason ends up running an old version of
the validatation code.

In general it is less work to keep code client side because it is easier to
modify, test and debug. testing can be done once you have created suitable
fixtures (not trivial, but doable). Debugging is much easier with PL/Python
than, for examplle, PL/SQL because if you structure things correctly you can
run much of your code outside of the database. However, for debugging code
that talks to the internal database APIs you are still stuck with just the
equivalent of 'print' as your toolkit.

-- 
Stuart Bishop <[EMAIL PROTECTED]>
http://www.stuartbishop.net/


signature.asc
Description: OpenPGP digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Tabnanny really useful?

[Stuart Bishop]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Franz Steinhaeusler wrote:
| Not really soo useful, because most syntax and also indentation errors
| are actually detected by invoking python, i.e. the command compile.
| But as combination for this: yes why not.
| I looked for Stanis spe editor, which uses a combination of these two.
Just because Python compiles and runs something does not mean that there
are no indentation errors. tabnanny works great because it removes all
ambiguous cases so you never have to worry about it. We use it as part
of our test suite to guard against misconfigured editors.
(again using '.' instead of space) Why would you want to allow '.\tprint
foo' in Python source when you could clean it up to be the totally
unambigous 'print foo'. As a more rational example, would you
want to allow '\tprint foo', which will totally confuse someone with
their editor misconfigured with 4 char tabs?
- --
Stuart Bishop <[EMAIL PROTECTED]>
http://www.stuartbishop.net/
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.2.5 (GNU/Linux)
iD8DBQFByp4nAfqZj7rGN0oRAoq9AJ4scAe0A3aFnx7jRZuqtRXXaxAcRgCfYkVf
FbH2HBFA4/44KgZfpiPuvNU=
=nyQt
-END PGP SIGNATURE-
--
http://mail.python.org/mailman/listinfo/python-list

Re: I look for a list to convert time zone abbreviation to full time zone in python

[Stuart Bishop]

On Mon, Dec 2, 2013 at 11:18 PM, Stéphane Klein
 wrote:

> * are there the same list somewhere (I didn't found in pytz) ?

Not that I know of.

> * is it possible to append this list in pytz or in standard python date 
> module ?

It could go into pytz (but generated from the IANA database, not from
the list you quote). Whether it should go into pytz is debatable.

If you need to map an abbreviation back to a single timezone you are
solving the wrong problem, because you can only map an abbreviation
back to a list of possible timezones (And that list might change when
the database is corrected). Also, to correctly represent this you need
to specify the point in time. EST in 'Tue Dec  3 20:44:00 EST 2013'
maps to about 3 timezones. EST in 'Tue Dec  3 20:44:00 EST 2011' maps
to about 6.

-- 
Stuart Bishop 
http://www.stuartbishop.net/
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Encoding sniffer?

[Stuart Bishop]

[EMAIL PROTECTED] wrote:
> Andreas> Does anyone know of a Python module that is able to sniff the
> Andreas> encoding of text?
> 
> I have such a beast.  Search here:
> 
> http://orca.mojam.com/~skip/python/
> 
> for "decode".
> 
> Skip

We have similar code. It looks functionally the same except that we also:

Check if the string starts with a BOM.
Detects probable ISO-8859-15 using a set of characters common
is ISO-8859-15 but uncommon in ISO-8859-1
Doctests :-)

# Detect BOM
_boms = [
(codecs.BOM_UTF16_BE, 'utf_16_be'),
(codecs.BOM_UTF16_LE, 'utf_16_le'),
(codecs.BOM_UTF32_BE, 'utf_32_be'),
(codecs.BOM_UTF32_LE, 'utf_32_le'),
]

try:
for bom, encoding in _boms:
if s.startswith(bom):
return unicode(s[len(bom):], encoding)
except UnicodeDecodeError:
pass

[...]

# If we have characters in this range, it is probably ISO-8859-15
if re.search(r"[\xa4\xa6\xa8\xb4\xb8\xbc-\xbe]", s) is not None:
try:
return unicode(s, 'ISO-8859-15')
except UnicodeDecodeError:
pass

Feel free to update your available code. Otherwise, I can probably post ours
somewhere if necessary.

-- 
Stuart Bishop <[EMAIL PROTECTED]>
http://www.stuartbishop.net/


signature.asc
Description: OpenPGP digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: psycopg2 faster way to retrieve last x records

[Stuart Bishop]

Luis P. Mendes wrote:
> Hi,
> 
> I would like to know if there is a better way to do what I'm already doing
> as stated in the following example, when using psycopg2 with PostgresQL.
> ...
> nr_bars_before = 200
> tabela = 'seconds'
> sqlString = "SELECT * FROM " + tabela + " ORDER BY tempounix;"
> curs = self.conn.cursor()
> curs.execute(sqlString)
> try:
>   while 1:
>   curs.scroll(1,mode='relative')
> except: pass
> curs.scroll(-int(math.fabs(nr_bars_before)),mode='relative')
> row = curs.fetchone()
> curs.close()
> ...
> 
> What I need is to get the last 200
> records from the table, each couple minutes.  As stated,
> what I do is to go all the way through the table records until the end,
> then going back 200 in order to select all of them (those 200) forward
> down to the last one.
> 
> But it takes a lot of time to do it.  I mean some seconds.  And it brings
> some 'heavy' work on disk.  The table 'seconds' has 54+ lines right
> now. 

The following SQL statement will return the last 200 rows in reverse order:

SELECT * FROM seconds ORDER BY tempounix DESC LIMIT 200

This will only send 200 rows from the server to the client (your existing
approach will send all of the rows). Also, if you have an index on tempounix
it will be really fast.


If you really need the results in tempounix order, then:

SELECT * FROM (
SELECT * FROM seconds ORDER BY tempounix DESC LIMIT 200
) AS whatever
ORDER BY tempounix;


-- 
Stuart Bishop <[EMAIL PROTECTED]>
http://www.stuartbishop.net/



signature.asc
Description: OpenPGP digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: A critique of cgi.escape

[Stuart Bishop]

Jon Ribbens wrote:
> In article <[EMAIL PROTECTED]>, Georg Brandl wrote:
>>> I'm sorry, that's not good enough. How, precisely, would it break
>>> "existing code"? Can you come up with an example, or even an
>>> explanation of how it *could* break existing code?
>> Is that so hard to see? If cgi.escape replaced "'" with an entity reference,
>> code that expects it not to do so would break.
> 
> Sorry, that's still not good enough. Why would any code expect such a
> thing?

Plenty of test suites for a start. A non-backwards compatible change such as
being suggested can create a huge maintenance burden on lots of people.
People also use that function to escape non-HTML too - if they are using it
as documented, and it produces the correct results for them, great. Note
that the documentation doesn't say that input has to be HTML, nor that
output must be used as HTML. It just describes the transformation that it
does clearly and unambiguously and can quite happily be used for generating
 quoted text for use in, say, XML documents. Also, because Python has a
conservative policy on backwards incompatible changes, you are protected
from some wanker going and changing the HTML safe mappings arbitrarily, say
using numerical entity references instead of >, < and &. This
policy allows large software projects to be developed in Python and
maintained with less pain than if they were written in languages with a less
conservative policy.

If you want to improve the situation, join the WEB-SIG to help design new
and improved APIs so that the existing ones like the ancient cgi module can
be deprecated. Or maybe just some helpers can be added to the existing
htmllib module? There are better approaches than making non-backwards
compatible changes to functions people have been relying on since Python 1.5.


-- 
Stuart Bishop <[EMAIL PROTECTED]>
http://www.stuartbishop.net/



signature.asc
Description: OpenPGP digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: PostgreSQL, psycopg2 and OID-less tables

[Stuart Bishop]

Frank Millman wrote:

> I used to use 'select lastval()', but I hit a problem. If I insert a
> row into table A, I want the id of the row inserted. If it is a complex
> insert, which triggers inserts into various other tables, some of which
> may also be auto-incrementing, lastval() returns the id of the row last
> inserted into any auto-incrementing table.
> 
> I therefore use the following -
> cur.execute("select currval('%s_%s_Seq')" % (tableid, columnid)
> 
> where tableid is the specific table I want (in this example, table A),
> and columnid is the column specified as the auto-incrementing one.

If you are using a modern PostgreSQL (8.1 for sure, maybe 8.0), this is
better spelt:

cur.execute("SELECT currval(pg_get_serial_sequence(%s, %s))" % (
tableid, columnid))

(Assuming of course your table name and column name don't contain odd
characters like = or ' or ., in which case you need to properly quote them).

The reason it is better is that in odd cases the sequence name will not
always be %(tableid)s_%(columnid)s_seq, such as after you have renamed a table.

-- 
Stuart Bishop <[EMAIL PROTECTED]>
http://www.stuartbishop.net/



signature.asc
Description: OpenPGP digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: does anybody earn a living programming in python?

[Stuart Bishop]

walterbyrd wrote:

> My research of this subject was very limited, just looked at the major
> job boards, and compared demand for python developers to demand for
> other languages, such as java, c++, visual basic, or php. An
> unscientific test, I realize. But, it's not easy to get good data on
> such a subject: everybody has an agenda, it seems.
> 
> I don't count jobs where python is an "also ran" skill. For example,
> sys-admin jobs where python is listed with a dozen other skills.

I think you are getting skewed results. The companies I have been involved
in that use Python tend to be after top-tier (or as close as they can
afford) programmers rather than the sort of applications you get by the
truckload if you advertise a job on 'the major job boards'. You get a much
better quality of applicant if you target niche boards (such as
http://www.python.org/community/jobs/), use word of mouth, or posts to
highly targeted mailing lists like the ones run by your companies local
users groups or direct contact with visible community members with the
skillsets you need. The results get further skewed on 'the major job boards'
when one real job results in 100 job notices from 100 different recruitment
companies.

My personal experience is that there is a shortage of good Python
programmers. In Melbourne, Australia for example there is a continual need
for about 2 more - one Python shop there just hires clueful developers and
makes their first task 'learn Python'. We generally have a few positions
open at any particular moment too - http://www.ubuntu.com/employment (but we
are picky - it is hard to hand hold when the hand is several time zones away).

On a slight tangent, one of the appealing things about Python (and other
'non-mainstream' languages) in the past has been that the people who knew
Python also tended to be the people you wanted to employ - they generally
had experience in other languages but moved onto something they perceived as
'better' either at work or home. It indicated a level of care and pride
about their profession rather than someone who just treated cutting code as
a day job. That might be changing now that Python is becoming more visible
on peoples radar.

-- 
Stuart Bishop <[EMAIL PROTECTED]>
http://www.stuartbishop.net/



signature.asc
Description: OpenPGP digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: DB API 2.0 and transactions

[Stuart Bishop]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Magnus Lycka wrote:

> Another option would be to investigate if any of the other postgreSQL
> drivers have a more correct behaviour. The non-standard behaviour that
> you describe it obvious from the pgdb source. See:
> http://www.pygresql.org/cvsweb.cgi/pygresql/module/pgdb.py?rev=1.27

fwiw psycopg 1.1.18 has the correct behavior (as usual).

I can add this test to the DB-API 2.0 Anal Compliance Test Suite if we want,
although it sounds like it is only an issue with PostgreSQL drivers.

- --
Stuart Bishop <[EMAIL PROTECTED]>
http://www.stuartbishop.net/
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.2.5 (GNU/Linux)

iD8DBQFCsk3oAfqZj7rGN0oRAjHAAJ4kQzxJXFW6hX6Q1t896fMzT0EUjACgkBhw
X4wB17+4FwO9TsKpiIBJB50=
=mYfn
-END PGP SIGNATURE-
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: pytz2007c error

[Stuart Bishop]

looping wrote:

On Mar 6, 9:51 am, "looping" <[EMAIL PROTECTED]> wrote:

Hi,
Why this error ?


from pytz import timezone
eastern = timezone('US/Eastern')

Traceback (most recent call last):
  File "", line 1, in 
  File "C:\Python25\lib\site-packages\pytz-2007c-py2.5.egg\pytz
\__init__.py", line 93, in timezone
  File "C:\Python25\lib\site-packages\pytz-2007c-py2.5.egg\pytz
\tzfile.py", line 33, in build_tzinfo
for trans in data[:timecnt]]
  File "C:\Python25\lib\site-packages\pytz-2007c-py2.5.egg\pytz
\tzinfo.py", line 27, in memorized_datetime
dt = datetime.utcfromtimestamp(seconds)
ValueError: timestamp out of range for platform localtime()/gmtime()
function

I'm running python 2.5 on WinXP French with the egg from CheeseShop.

Same error with all timezone (like timezone('Europe/Zurich'),
timezone('Europe/Amsterdam'), ...) except a few one:>>> print timezone('UTC')

UTC

Is this a pytz problem or something I didn't understand ?


OK, the error come from datetime.utcfromtimestamp that doesn't support
negative value.
pytz try to pass -1633280400 to this function.

Is this a problem from Windows ?


Looks like it. I've opened a bug.

https://bugs.beta.launchpad.net/pytz/+bug/90096

Current workaround is to use an earlier release. This will be fixed as pytz 
needs to remain cross platform.


--
Stuart Bishop <[EMAIL PROTECTED]>
http://www.stuartbishop.net/



signature.asc
Description: OpenPGP digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list