urllib2 timeout not working - stalls for an hour or so
Hi,
I use urllib2 to grab google.com webpages on my Mac over my Comcast home
network.
I see about 1 error for every 50 pages grabbed. Most exceptions are
ssl.SSLError, very few are socket.error and urllib2.URLError.
The problem is - after a first exception, urllib2 occasionally stalls for upto
an hour (!), at either the urllib2.urlopen or response.read stages.
Apparently the urllib2 and socket timeouts are not effective here - how do I
fix this?
import urllib2
import socket
from sys import exc_info as sysExc_info
timeout = 2
socket.setdefaulttimeout(timeout)
try :
req = urllib2.Request(url,None,headers)
response = urllib2.urlopen(req,timeout=timeout)
html = response.read()
except :
e = sysExc_info()[0]
open(logfile,'a').write('Exception: %s \n' % e)
< code that follows this : after the first exception, I try again for a few
tries >
--
https://mail.python.org/mailman/listinfo/python-list
Re: urllib2 timeout not working - stalls for an hour or so
Sumeet Sandhu wrote:
> Hi,
>
> I use urllib2 to grab google.com webpages on my Mac over my Comcast home
> network.
>
> I see about 1 error for every 50 pages grabbed. Most exceptions are
> ssl.SSLError, very few are socket.error and urllib2.URLError.
>
> The problem is - after a first exception, urllib2 occasionally stalls for
> upto an hour (!), at either the urllib2.urlopen or response.read stages.
>
> Apparently the urllib2 and socket timeouts are not effective here - how do
> I fix this?
>
>
> import urllib2
> import socket
> from sys import exc_info as sysExc_info
> timeout = 2
> socket.setdefaulttimeout(timeout)
>
> try :
> req = urllib2.Request(url,None,headers)
> response = urllib2.urlopen(req,timeout=timeout)
> html = response.read()
> except :
> e = sysExc_info()[0]
> open(logfile,'a').write('Exception: %s \n' % e)
> < code that follows this : after the first exception, I try again for a
> few tries >
I'd use separate try...except-s for response = urlopen() and
response.read(). If the problem originates with read() you could try to
replace it with select.select([response.fileno()], [], [], timeout) calls in
a loop.
--
https://mail.python.org/mailman/listinfo/python-list
*args and **kwargs
Hello to all,
I'm trying to understand the concept of * args and ** kwarg with python3
But I can not understand why I returns the error message "SyntaxError:
positional argument follows the keyword argument" when I insert values.
You can help me?
def start(data=None, *args, **kwargs):
print(' Arguments ===')
print('data is: ', data)
print('args is:', args)
print('kwargs is:', kwargs)
In [22]: def start(data=None, *args, **kwargs):
print(' Arguments ===')
print('data is: ', data)
print('args is:', args)
print('kwargs is:', kwargs)
In [23]: start(data="Fish",2,3,tox="tux")
File "", line 1
start(data="Fish",2,3,tox="tux")
^
SyntaxError: positional argument follows keyword argument
In [24]: start(data="Fish",tox="tux")
Arguments ===
data is: Fish
args is: ()
kwargs is: {'tox': 'tux'}
--
https://mail.python.org/mailman/listinfo/python-list
Re: *args and **kwargs
Smith writes: > I'm trying to understand the concept of * args and ** kwarg with > python3 Welcome. Your questions are fine in this forum; but you may also want to participate in our collaborative learning forum for Python beginners, https://mail.python.org/mailman/listinfo/tutor>. > But I can not understand why I returns the error message "SyntaxError: > positional argument follows the keyword argument" when I insert > values. It's fairly simple (though it may not be obvious!): > In [23]: start(data="Fish",2,3,tox="tux") > File "", line 1 > start(data="Fish",2,3,tox="tux") > ^ > SyntaxError: positional argument follows keyword argument Exactly. Note that this has nothing to do with how the function is defined; in the definition of the function, parameters are neither positional nor keyword. You name each of them, and you define an order for them; and neither of those makes any of them “positional” or “keyword”. Rather, “positional argument and “keyword argument” are characteristics of the arguments you *supply* in a particular call to the function. You have specified four arguments, in this order: * A keyword argument, ‘data="Fish"’. * A positional argument, ‘2’. * A positional argument, ‘3’. * A keyword argument, ‘tox="tux"’. After specifying a keyword argument, you may not then specify any positional arguments. Hence the SyntaxError. -- \ “I know that we can never get rid of religion …. But that | `\ doesn’t mean I shouldn’t hate the lie of faith consistently and | _o__) without apology.” —Paul Z. Myers, 2011-12-28 | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
Re: *args and **kwargs
On Fri, 02 Sep 2016 23:44:50 +1000, Ben Finney wrote: > Smith writes: > >> I'm trying to understand the concept of * args and ** kwarg with >> python3 > > Welcome. Your questions are fine in this forum; but you may also want to > participate in our collaborative learning forum for Python beginners, > https://mail.python.org/mailman/listinfo/tutor>. > >> But I can not understand why I returns the error message "SyntaxError: >> positional argument follows the keyword argument" when I insert values. > > It's fairly simple (though it may not be obvious!): > >> In [23]: start(data="Fish",2,3,tox="tux") >> File "", line 1 >> start(data="Fish",2,3,tox="tux") >> ^ >> SyntaxError: positional argument follows keyword argument > > Exactly. > > Note that this has nothing to do with how the function is defined; in > the definition of the function, parameters are neither positional nor > keyword. You name each of them, and you define an order for them; and > neither of those makes any of them “positional” or “keyword”. > > Rather, “positional argument and “keyword argument” are characteristics > of the arguments you *supply* in a particular call to the function. > > You have specified four arguments, in this order: > > * A keyword argument, ‘data="Fish"’. > * A positional argument, ‘2’. > * A positional argument, ‘3’. > * A keyword argument, ‘tox="tux"’. > > After specifying a keyword argument, you may not then specify any > positional arguments. Hence the SyntaxError. & the solution is to change the order of the definition def start( must_have,*args,**kwargs,data=none): -- Pretend to spank me -- I'm a pseudo-masochist! -- https://mail.python.org/mailman/listinfo/python-list
Re: Python slang
On Wed, Aug 10, 2016, at 12:19, Random832 wrote: > On Wed, Aug 10, 2016, at 07:59, Dennis Lee Bieber wrote: > > The use of = also has a long history... FORTRAN (where the > > comparison was .EQ.), BASIC (granted, K&K required assignment to > > start with the keyword LET, so the use of = was mainly a > > delimiter between target and expression being assigned). > > Visual Basic actually uses = for both assignment and comparison > *without* the Let keyword, it gets by on the fact that assignment is > not an expression. I just discovered something interesting. It turns out, Python 0.9.1 did the same thing (i.e. = for both assignment and comparison), with the ambiguity resolved by not allowing comparison operators generally in some expression contexts. >>> a, b = 1, 2 a < b Parsing error: file , line 1: a < b ^ Unhandled exception: run-time error: syntax error >>> # wait, what? I was doing this, originally, to test that comparisons >>> # still worked after a change to cmp_outcome to silence a compiler >>> # warning. I thought I'd somehow broken the parser. >>> (a < b) 1 >>> (a = b) 0 >>> if a = b: 1 # It doesn't always need parentheses ... else: 0 ... 0 >>> a, b (1, 2) >>> (a == b) # Turns out the modern operator doesn't even exist Parsing error: file , line 1: (a == b) ^ Unhandled exception: run-time error: syntax error >>> (a != b) # And now something for Barry Warsaw Parsing error: file , line 1: (a != b) ^ Unhandled exception: run-time error: syntax error >>> (a <> b) 1 It's gone in Python 1.0 - all of the modern behavior we're familiar with, except of course the boolean type, is fully present, with the continued existence [for a while] of the <> operator the only trace of days gone by. -- https://mail.python.org/mailman/listinfo/python-list
Strange behaviour with numbers in exponential notation
>>> 10**26 - 1 99 >>> 1e26 - 1 1e+26 Why? -- https://mail.python.org/mailman/listinfo/python-list
Re: *args and **kwargs
After specifying a keyword argument, you may not then specify any positional arguments. Hence the SyntaxError. thanks a lot -- https://mail.python.org/mailman/listinfo/python-list
Re: *args and **kwargs
On 02/09/2016 16:52, alister wrote: On Fri, 02 Sep 2016 23:44:50 +1000, Ben Finney wrote: Smith writes: I'm trying to understand the concept of * args and ** kwarg with python3 Welcome. Your questions are fine in this forum; but you may also want to participate in our collaborative learning forum for Python beginners, https://mail.python.org/mailman/listinfo/tutor>. But I can not understand why I returns the error message "SyntaxError: positional argument follows the keyword argument" when I insert values. It's fairly simple (though it may not be obvious!): In [23]: start(data="Fish",2,3,tox="tux") File "", line 1 start(data="Fish",2,3,tox="tux") ^ SyntaxError: positional argument follows keyword argument Exactly. Note that this has nothing to do with how the function is defined; in the definition of the function, parameters are neither positional nor keyword. You name each of them, and you define an order for them; and neither of those makes any of them “positional” or “keyword”. Rather, “positional argument and “keyword argument” are characteristics of the arguments you *supply* in a particular call to the function. You have specified four arguments, in this order: * A keyword argument, ‘data="Fish"’. * A positional argument, ‘2’. * A positional argument, ‘3’. * A keyword argument, ‘tox="tux"’. After specifying a keyword argument, you may not then specify any positional arguments. Hence the SyntaxError. & the solution is to change the order of the definition def start( must_have,*args,**kwargs,data=none): thanks a lot -- https://mail.python.org/mailman/listinfo/python-list
Re: Strange behaviour with numbers in exponential notation
On Fri, Sep 2, 2016, at 11:51, Marco Sulla wrote: > >>> 10**26 - 1 > 99 > >>> 1e26 - 1 > 1e+26 > > > Why? Exponential notation creates floating point numbers, which have a limited amount of precision in binary. Specifically (on my system which, as most modern computers do, use IEEE double precision numbers for the python float type), 0x52b7d2dcc80cd4 is the value of 1e26, whereas. 0x52b7d2dcc80cd2e400 is the valoe of 10**26. Trying to add 1 gets it rounded off again, and the value is simply printed as 1e+26 by default because this is the shortest representation that gives the same number, even if "14764729344.0" would be more accurate. -- https://mail.python.org/mailman/listinfo/python-list
Re: Strange behaviour with numbers in exponential notation
Am 02.09.16 um 17:51 schrieb Marco Sulla: 10**26 - 1 99 1e26 - 1 1e+26 10**26 is computed with integer arithmetics. Python has bigints (i.e. as big as the memory allows) 1e26 denotes a *floating point number* Floating point has finite precision, in CPython it is a 64bit IEEE number. The largest exact integer there is 2**53 (~10^16), everything beyond cannot be accurately represented. Christian -- https://mail.python.org/mailman/listinfo/python-list
Re: urllib2 timeout not working - stalls for an hour or so
On Friday, September 2, 2016 at 6:05:05 AM UTC-7, Peter Otten wrote:
> Sumeet Sandhu wrote:
>
> > Hi,
> >
> > I use urllib2 to grab google.com webpages on my Mac over my Comcast home
> > network.
> >
> > I see about 1 error for every 50 pages grabbed. Most exceptions are
> > ssl.SSLError, very few are socket.error and urllib2.URLError.
> >
> > The problem is - after a first exception, urllib2 occasionally stalls for
> > upto an hour (!), at either the urllib2.urlopen or response.read stages.
> >
> > Apparently the urllib2 and socket timeouts are not effective here - how do
> > I fix this?
> >
> >
> > import urllib2
> > import socket
> > from sys import exc_info as sysExc_info
> > timeout = 2
> > socket.setdefaulttimeout(timeout)
> >
> > try :
> > req = urllib2.Request(url,None,headers)
> > response = urllib2.urlopen(req,timeout=timeout)
> > html = response.read()
> > except :
> > e = sysExc_info()[0]
> > open(logfile,'a').write('Exception: %s \n' % e)
> > < code that follows this : after the first exception, I try again for a
> > few tries >
>
> I'd use separate try...except-s for response = urlopen() and
> response.read(). If the problem originates with read() you could try to
> replace it with select.select([response.fileno()], [], [], timeout) calls in
> a loop.
thanks Peter. I will try this.
However, I suspect Comcast is rate limiting my home use. Is there a workaround
for that?
What I really need is to somehow monitor the full url-read loop and break it if
Comcast is stalling me too long...
--
https://mail.python.org/mailman/listinfo/python-list
Re: Strange behaviour with numbers in exponential notation
Excuse me, I forgot to include the python list mail addess. I repost the mail. On Fri, Sep 2, 2016 at 6:18 PM, Christian Gollwitzer wrote: > 1e26 denotes a *floating point number* Floating point has finite precision, > in CPython it is a 64bit IEEE number. The largest exact integer there is > 2**53 (~10^16), everything beyond cannot be accurately represented. I see. So python float type is the IEEE 754 double. Why is it not automatically converted to long double? And I want to add to my original question: indeed I read from the docs: https://docs.python.org/3/reference/lexical_analysis.html#floating-point-literals that float has an 'exponentfloat' syntax. Why integers does not have an equivalent syntax? It's not too difficult to lexically distinguish between a integer and a float written in exponential notation, I think. -- https://mail.python.org/mailman/listinfo/python-list
Re: Strange behaviour with numbers in exponential notation
On Fri, Sep 2, 2016, at 13:02, Marco Sulla wrote:
> On Fri, Sep 2, 2016 at 6:17 PM, Random832 wrote:
> > Trying to add 1 gets it rounded off again, and the value is simply
> > printed as 1e+26 by default because this is the shortest representation
> > that gives the same number, even if "14764729344.0"
> > would be more accurate.
>
> It seems it's not simply a representation. It seems the numbers are just
> equal:
Yes. The number 1e+16 is represented as a floating point number
equal to 14764729344. Then when you add 1 to it, the
1 gets rounded off and you get a floating point number that is still
equal to 14764729344:
>>> a = float(10**16)
>>> int(a)
14764729344
>>> b = a + 1
>>> int(b)
14764729344
You switched examples. 10**22 is the largest power of 10 that can be
represented exactly, but 10**15+1 is te largest "power of 10 plus
one" that can be represented exactly.
Float can only represent numbers whose binary representation
contains 53 bits or less from the first to the last "1".
>>> '{:b}'.format(10**15)
'1110001101011010100100110001101000'
# 35 bits before the run of zeros
>>> '{:b}'.format(10**16)
'1000111110001001101101'
# 38 bits, works fine in float
>>> '{:b}'.format(10**15+1)
'1110001101011010100100110001101001'
# 50 bits, barely fits in a float
>>> '{:b}'.format(10**16+1)
'10001111100010011011010001'
# 54 bits, too many.
Anything else gets rounded off. It's the same as if you tried to add
a tiny number to 1.
>>> 1+1e-15 == 1
False
>>> 1+1e-16 == 1
True
> This is really weird. I tried to do it just in C:
You should be using double instead of long double; double is the
C equivalent of the Python float type.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Strange behaviour with numbers in exponential notation
Am 02.09.16 um 19:24 schrieb Marco Sulla: Excuse me, I forgot to include the python list mail addess. I repost the mail. On Fri, Sep 2, 2016 at 6:18 PM, Christian Gollwitzer wrote: 1e26 denotes a *floating point number* Floating point has finite precision, in CPython it is a 64bit IEEE number. The largest exact integer there is 2**53 (~10^16), everything beyond cannot be accurately represented. I see. So python float type is the IEEE 754 double. Why is it not automatically converted to long double? Because Python has no long double type? And "long double" (assuming 80bit or 128bit) does not solve the problem, it just shifts the bound to some higher arbitrary limit. In fact, it is impossible to make a bigfloat class which acts like a float, but always carries enough bits. A simple counterexample is 0.3, a periodic fraction in binary. Or, if you include arbitrary precision fractions, then e.g. sqrt(2). And I want to add to my original question: indeed I read from the docs: https://docs.python.org/3/reference/lexical_analysis.html#floating-point-literals that float has an 'exponentfloat' syntax. Why integers does not have an equivalent syntax? Tradition? All languages I know of treat a number with an exponent as floating point. Christian -- https://mail.python.org/mailman/listinfo/python-list
Re: Strange behaviour with numbers in exponential notation
Christian Gollwitzer : > Am 02.09.16 um 19:24 schrieb Marco Sulla: >> float has an 'exponentfloat' syntax. Why integers does not have an >> equivalent syntax? > > Tradition? All languages I know of treat a number with an exponent as > floating point. Approximate real numbers are mostly needed by scientific applications, and floats do the job marvelously. Unlike many programming languages, Python has adopted big integers, which have immediate applications in cryptography. Those big integers hardly ever involve powers of ten, however, and they cannot be approximated. So the scientific notation doesn't offer any notational advantage. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Strange behaviour with numbers in exponential notation
On Fri, Sep 2, 2016, at 15:12, Christian Gollwitzer wrote: > Tradition? All languages I know of treat a number with an exponent as > floating point. Scheme does allow you to give integers (and rationals) in decimal and/or exponential notation with the "#e" prefix. -- https://mail.python.org/mailman/listinfo/python-list
Re: Strange behaviour with numbers in exponential notation
On 2016-09-02 20:47, Random832 wrote: On Fri, Sep 2, 2016, at 15:12, Christian Gollwitzer wrote: Tradition? All languages I know of treat a number with an exponent as floating point. Scheme does allow you to give integers (and rationals) in decimal and/or exponential notation with the "#e" prefix. In Ada, integers can have an exponent, provided that it's non-negative. -- https://mail.python.org/mailman/listinfo/python-list
Re: Strange behaviour with numbers in exponential notation
On 2 September 2016 at 21:12, Christian Gollwitzer wrote: > Am 02.09.16 um 19:24 schrieb Marco Sulla: > Because Python has no long double type? Python no of course, but C++ yes, and CPython is written in C++. However, I think the answer is here: https://en.wikipedia.org/wiki/Long_double#Implementations Briefly, long double is not implemented in all hardwares, or its implementation is really different from machine to machine. I suppose this is why long double is not supported. -- https://mail.python.org/mailman/listinfo/python-list
Re: Strange behaviour with numbers in exponential notation
On 3 September 2016 at 02:31, Marco Sulla wrote: > Python no of course, but C++ yes, and CPython is written in C++. Sorry, I just founded CppPython... -- https://mail.python.org/mailman/listinfo/python-list
Re: *args and **kwargs
Ben Finney wrote: in the definition of the function, parameters are neither positional nor keyword. In Python 3 that's not quite true -- it's possible to define "keyword only" parameters that can't be passed positionally. However, it's true that any of the "ordinary" parameters (ones that aren't keyword-only) can be specified either positionally or by keyword in a call. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: Pythons for .Net
On Sunday, July 24, 2016 at 11:30:09 PM UTC-5, Steven D'Aprano wrote: > Yes, I said Pythons plural :-) > > For those wanting to use Python on .Net or Mono, there is some good news. > > Firstly, the venerable old "Python for .Net" project is still alive, and now > supports up to Python 3.5 on .Net or Mono. PythonNet, as this is known, > integrates the regular CPython interpreter with .Net or Mono. > > Secondly, we can also expect that IronPython will have a new lease of life. > Continuing on the work of their predecessor, Jeff Hardy, the IronPython > project now has two tech leads who will carry it forward: Alex Earl and > Benedikt Eggers. > > https://thelasttechie.com/2016/07/24/its-back-python-for-net/ > > > IronPython is a reimplementation of Python, written in C# as managed code > for .Net. It doesn't use a GIL, and is sometimes considered a little faster > than CPython, so this is good news for the Python ecosystem. > > IronPython: > http://ironpython.net/ > > Python for .Net: > https://github.com/pythonnet/pythonnet > http://pythonnet.github.io/ > https://pypi.python.org/pypi/pythonnet > > > > > -- > Steven > “Cheer up,” they said, “things could be worse.” So I cheered up, and sure > enough, things got worse. Hello Steven, I'm one of the contributors to pythonnet. We strive for compatibility with IronPython, but also have a simplified embedding interface and support Python 3 + C-Extensions such as numpy, scipy, sklearn. If anyone is interested in learning pythonnet, then try it out from PYPI or github and use this tutorial: https://pythonnet.github.io/readme.html There is also a mailing list and a tag on stackoverflow: https://mail.python.org/mailman/listinfo/pythondotnet http://stackoverflow.com/questions/tagged/python.net Finally if anyone can contact Christian Heimes (Python Core Developer), then please ask him to reply on request to update the license to MIT: https://github.com/pythonnet/pythonnet/issues/234 He is the only contributor that prevents updating to MIT license. -- https://mail.python.org/mailman/listinfo/python-list
Re: Strange behaviour with numbers in exponential notation
Am 03.09.16 um 02:31 schrieb Marco Sulla: On 2 September 2016 at 21:12, Christian Gollwitzer wrote: Am 02.09.16 um 19:24 schrieb Marco Sulla: Because Python has no long double type? Python no of course, but C++ yes, and CPython is written in C++. However, I think the answer is here: https://en.wikipedia.org/wiki/Long_double#Implementations Briefly, long double is not implemented in all hardwares, or its implementation is really different from machine to machine. Yes - 64 bit IEEE floats are today the universally available floating point data type across different machines. If you want arbitrary precision floats, you can have them in Python, too: https://pypi.python.org/pypi/bigfloat/ Of course this is a "software emulation" and is consequently slower then hardware floats. It does not, howver take away the principle problem that you must set the precision from the beginning of your calculation. Christian -- https://mail.python.org/mailman/listinfo/python-list
