Re: Fastest way to retrieve and write html contents to file

2016-05-01 Thread DFS
On 5/2/2016 2:05 AM, Steven D'Aprano wrote: On Monday 02 May 2016 15:00, DFS wrote: I tried the 10-loop test several times with all versions. The results were 100% consistent: VBSCript xmlHTTP was always 2x faster than any python method. Are you absolutely sure you're comparing the same job

Re: You gotta love a 2-line python solution

2016-05-01 Thread Terry Reedy
On 5/2/2016 12:31 AM, Stephen Hansen wrote: On Sun, May 1, 2016, at 08:39 PM, DFS wrote: To save a webpage to a file: - 1. import urllib 2. urllib.urlretrieve("http://econpy.pythonanywhere.com /ex/001.html","D:\file.html")

Re: What should Python apps do when asked to show help?

2016-05-01 Thread Terry Reedy
On 5/1/2016 9:48 PM, Steven D'Aprano wrote: On Mon, 2 May 2016 03:04 am, Grant Edwards wrote: On 2016-05-01, Steven D'Aprano wrote: On Mon, 2 May 2016 02:30 am, Grant Edwards wrote: In discussions like these, it would be important to draw from precedents. Are there commands that have such a

Re: Code Opinion - Enumerate

2016-05-01 Thread Sayth Renshaw
Thanks for the opinion. I should add that is not my code in first post it's the code from Rosetta on how to do Conway's GOL. I thought it looked ugly. Sayth -- https://mail.python.org/mailman/listinfo/python-list

Re: Fastest way to retrieve and write html contents to file

2016-05-01 Thread Stephen Hansen
On Sun, May 1, 2016, at 10:59 PM, DFS wrote: > startTime = time.clock() > for i in range(loops): > r = urllib2.urlopen(webpage) > f = open(webfile,"w") > f.write(r.read()) > f.close > endTime = time.clock() > print "Finished urllib2 in %.2g seconds" %(endTi

Re: You gotta love a 2-line python solution

2016-05-01 Thread DFS
On 5/2/2016 1:37 AM, Stephen Hansen wrote: On Sun, May 1, 2016, at 10:23 PM, DFS wrote: Trying the rawstring thing (say it fast 3x): webpage = "http://econpy.pythonanywhere.com/ex/001.html"; webfile = "D:\\econpy001.html" urllib.urlretrieve(

Re: Fastest way to retrieve and write html contents to file

2016-05-01 Thread Steven D'Aprano
On Monday 02 May 2016 15:00, DFS wrote: > I tried the 10-loop test several times with all versions. > > The results were 100% consistent: VBSCript xmlHTTP was always 2x faster > than any python method. Are you absolutely sure you're comparing the same job in two languages? Is VB using a local

Re: Fastest way to retrieve and write html contents to file

2016-05-01 Thread Steven D'Aprano
On Monday 02 May 2016 15:04, DFS wrote: > 0.2 is half as fast as 0.1, here. > > And two small numbers turn into bigger numbers when the webpage is big, > and soon the download time differences are measured in minutes, not half > a second. It takes twice as long to screw a screw into timber than

Re: Fastest way to retrieve and write html contents to file

2016-05-01 Thread DFS
On 5/2/2016 1:15 AM, Stephen Hansen wrote: On Sun, May 1, 2016, at 10:00 PM, DFS wrote: I tried the 10-loop test several times with all versions. Also how, _exactly_, are you testing this? C:\Python27>python -m timeit "filename='C:\\test.txt'; webpage='http://econpy.pythonanywhere.com/ex/001.

Re: You gotta love a 2-line python solution

2016-05-01 Thread Steven D'Aprano
On Monday 02 May 2016 15:21, Stephen Hansen wrote: > On Sun, May 1, 2016, at 10:08 PM, DFS wrote: >> On 5/2/2016 1:02 AM, Stephen Hansen wrote: >> >> I actually use "D:\\file.html" in my code. >> > >> > Or you can do that. But the whole point of raw strings is not having to >> > escape slashes :)

Re: You gotta love a 2-line python solution

2016-05-01 Thread Stephen Hansen
On Sun, May 1, 2016, at 10:23 PM, DFS wrote: > Trying the rawstring thing (say it fast 3x): > > webpage = "http://econpy.pythonanywhere.com/ex/001.html"; > > > webfile = "D:\\econpy001.html" > urllib.urlretrieve(webpage,webfile) WORKS > ---

Re: You gotta love a 2-line python solution

2016-05-01 Thread DFS
On 5/2/2016 1:02 AM, Stephen Hansen wrote: On Sun, May 1, 2016, at 09:51 PM, DFS wrote: On 5/2/2016 12:31 AM, Stephen Hansen wrote: On Sun, May 1, 2016, at 08:39 PM, DFS wrote: To save a webpage to a file: - 1. import urllib 2. urllib.urlretrieve("http://eco

Re: Code Opinion - Enumerate

2016-05-01 Thread Stephen Hansen
On Sun, May 1, 2016, at 08:17 PM, Sayth Renshaw wrote: > Just looking for your opinion on style would you write it like this > continually calling range or would you use enumerate instead, or neither > (something far better) ? I can't comment on your specific code because there's too much noise to

Re: You gotta love a 2-line python solution

2016-05-01 Thread Stephen Hansen
On Sun, May 1, 2016, at 10:08 PM, DFS wrote: > On 5/2/2016 1:02 AM, Stephen Hansen wrote: > >> I actually use "D:\\file.html" in my code. > > > > Or you can do that. But the whole point of raw strings is not having to > > escape slashes :) > > > Nice. Where/how else is 'r' used? Raw strings are

Re: Fastest way to retrieve and write html contents to file

2016-05-01 Thread Stephen Hansen
On Sun, May 1, 2016, at 10:04 PM, DFS wrote: > And two small numbers turn into bigger numbers when the webpage is big, > and soon the download time differences are measured in minutes, not half > a second. Are you sure of that? Have you determined that the time is not a constant overhead verses

Re: Fastest way to retrieve and write html contents to file

2016-05-01 Thread Stephen Hansen
On Sun, May 1, 2016, at 10:00 PM, DFS wrote: > I tried the 10-loop test several times with all versions. Also how, _exactly_, are you testing this? C:\Python27>python -m timeit "filename='C:\\test.txt'; webpage='http://econpy.pythonanywhere.com/ex/001.html'; import urllib2; r = urllib2.urlopen(we

Re: Fastest way to retrieve and write html contents to file

2016-05-01 Thread Chris Angelico
On Mon, May 2, 2016 at 3:04 PM, DFS wrote: > And two small numbers turn into bigger numbers when the webpage is big, and > soon the download time differences are measured in minutes, not half a > second. > > So, any ideas? So, measure with bigger web pages, and find out whether it's really a 2:1

Re: You gotta love a 2-line python solution

2016-05-01 Thread DFS
On 5/2/2016 1:02 AM, Stephen Hansen wrote: On Sun, May 1, 2016, at 09:51 PM, DFS wrote: On 5/2/2016 12:31 AM, Stephen Hansen wrote: On Sun, May 1, 2016, at 08:39 PM, DFS wrote: To save a webpage to a file: - 1. import urllib 2. urllib.urlretrieve("http://eco

Re: Fastest way to retrieve and write html contents to file

2016-05-01 Thread DFS
On 5/2/2016 1:00 AM, Stephen Hansen wrote: On Sun, May 1, 2016, at 09:50 PM, DFS wrote: On 5/2/2016 12:40 AM, Chris Angelico wrote: On Mon, May 2, 2016 at 2:34 PM, Stephen Hansen wrote: On Sun, May 1, 2016, at 09:06 PM, DFS wrote: Then I tested them in loops - the VBScript is MUCH faster: 0.

Re: Fastest way to retrieve and write html contents to file

2016-05-01 Thread DFS
On 5/2/2016 12:49 AM, Ben Finney wrote: DFS writes: Then I tested them in loops - the VBScript is MUCH faster: 0.44 for 10 iterations, vs 0.88 for python. […] urllib2 and requests were about the same speed as urllib.urlretrieve, while pycurl was significantly slower (1.2 seconds). Network

Re: Fastest way to retrieve and write html contents to file

2016-05-01 Thread Stephen Hansen
On Sun, May 1, 2016, at 09:50 PM, DFS wrote: > On 5/2/2016 12:40 AM, Chris Angelico wrote: > > On Mon, May 2, 2016 at 2:34 PM, Stephen Hansen wrote: > >> On Sun, May 1, 2016, at 09:06 PM, DFS wrote: > >>> Then I tested them in loops - the VBScript is MUCH faster: 0.44 for 10 > >>> iterations, vs 0

Re: You gotta love a 2-line python solution

2016-05-01 Thread Stephen Hansen
On Sun, May 1, 2016, at 09:51 PM, DFS wrote: > On 5/2/2016 12:31 AM, Stephen Hansen wrote: > > On Sun, May 1, 2016, at 08:39 PM, DFS wrote: > >> To save a webpage to a file: > >> - > >> 1. import urllib > >> 2. urllib.urlretrieve("http://econpy.pythonanywhere.com

Re: You gotta love a 2-line python solution

2016-05-01 Thread DFS
On 5/2/2016 12:31 AM, Stephen Hansen wrote: On Sun, May 1, 2016, at 08:39 PM, DFS wrote: To save a webpage to a file: - 1. import urllib 2. urllib.urlretrieve("http://econpy.pythonanywhere.com /ex/001.html","D:\file.html")

Re: Fastest way to retrieve and write html contents to file

2016-05-01 Thread DFS
On 5/2/2016 12:40 AM, Chris Angelico wrote: On Mon, May 2, 2016 at 2:34 PM, Stephen Hansen wrote: On Sun, May 1, 2016, at 09:06 PM, DFS wrote: Then I tested them in loops - the VBScript is MUCH faster: 0.44 for 10 iterations, vs 0.88 for python. ... I know it's asking a lot, but is there a r

Re: Fastest way to retrieve and write html contents to file

2016-05-01 Thread Chris Angelico
On Mon, May 2, 2016 at 2:49 PM, Ben Finney wrote: > One simple way to do that: Run the exact same test many times (say, > 10 000 or so) on the same machine, and then compute the average of all > the durations. > > Do the same for each different program, and then you may have more > meaningfully co

Re: Fastest way to retrieve and write html contents to file

2016-05-01 Thread Ben Finney
DFS writes: > Then I tested them in loops - the VBScript is MUCH faster: 0.44 for 10 > iterations, vs 0.88 for python. > > […] > > urllib2 and requests were about the same speed as urllib.urlretrieve, > while pycurl was significantly slower (1.2 seconds). Network access is notoriously erratic in

Re: Fastest way to retrieve and write html contents to file

2016-05-01 Thread Chris Angelico
On Mon, May 2, 2016 at 2:34 PM, Stephen Hansen wrote: > On Sun, May 1, 2016, at 09:06 PM, DFS wrote: >> Then I tested them in loops - the VBScript is MUCH faster: 0.44 for 10 >> iterations, vs 0.88 for python. > ... >> I know it's asking a lot, but is there a really fast AND really short >> python

Re: Fastest way to retrieve and write html contents to file

2016-05-01 Thread Stephen Hansen
On Sun, May 1, 2016, at 09:06 PM, DFS wrote: > Then I tested them in loops - the VBScript is MUCH faster: 0.44 for 10 > iterations, vs 0.88 for python. ... > I know it's asking a lot, but is there a really fast AND really short > python solution for this simple thing? 0.88 is not fast enough for

Re: You gotta love a 2-line python solution

2016-05-01 Thread Stephen Hansen
On Sun, May 1, 2016, at 08:39 PM, DFS wrote: > To save a webpage to a file: > - > 1. import urllib > 2. urllib.urlretrieve("http://econpy.pythonanywhere.com > /ex/001.html","D:\file.html") > - Note, for paths on windows y

Fastest way to retrieve and write html contents to file

2016-05-01 Thread DFS
I posted a little while ago about how short the python code was: - 1. import urllib 2. urllib.urlretrieve(webpage, filename) - Which is very sweet compared to the VBScript version: --

Re: Code Opinion - Enumerate

2016-05-01 Thread Sayth Renshaw
Also not using enumerate but no ugly for i range implementation this one from code review uses a generator on live cells only. http://codereview.stackexchange.com/a/108121/104381 def neighbors(cell): x, y = cell yield x - 1, y - 1 yield x, y - 1 yield x + 1, y - 1 yield

You gotta love a 2-line python solution

2016-05-01 Thread DFS
To save a webpage to a file: - 1. import urllib 2. urllib.urlretrieve("http://econpy.pythonanywhere.com /ex/001.html","D:\file.html") - That's it! Coming from VB/A background, some of the stuff you can do with python -

Re: How to fill in abbreviation in one column based on state name in another column?

2016-05-01 Thread Rustom Mody
Your code (below) is too garbled to be able to read On Monday, May 2, 2016 at 12:00:59 AM UTC+5:30, David Shi wrote: > Hello, I am back.  Thank you very much for your positive response. > I am trying to use Pandas apply to execute a lookup function, so that we can > put abbreviation in a new colu

Re: web facing static text db

2016-05-01 Thread sum abiut
Django is an excellent framework. you can use it with sqlite. cheers On Sat, Apr 30, 2016 at 7:17 PM, Gordon Levi wrote: > "Fetchinson ." wrote: > > >Hi folks,go > > >I have a vo ery specific set of requirements for a task and was > >wondering if anyone had good suggestions for the best set of

Code Opinion - Enumerate

2016-05-01 Thread Sayth Renshaw
Looking at various Python implementations of Conway's game of life. I came across one on rosetta using defaultdict. http://rosettacode.org/wiki/Conway%27s_Game_of_Life#Python Just looking for your opinion on style would you write it like this continually calling range or would you use enumerate

Re: What should Python apps do when asked to show help?

2016-05-01 Thread Steven D'Aprano
On Mon, 2 May 2016 03:04 am, Grant Edwards wrote: > On 2016-05-01, Steven D'Aprano wrote: >> On Mon, 2 May 2016 02:30 am, Grant Edwards wrote: >> In discussions like these, it would be important to draw from precedents. Are there commands that have such an option? >>> >>> It's pretty r

Re: What should Python apps do when asked to show help?

2016-05-01 Thread cs
On 01May2016 17:04, Grant Edwards wrote: On 2016-05-01, Steven D'Aprano wrote: On Mon, 2 May 2016 02:30 am, Grant Edwards wrote: In discussions like these, it would be important to draw from precedents. Are there commands that have such an option? It's pretty rare.  It is assumed that Unix

Re: What should Python apps do when asked to show help?

2016-05-01 Thread cs
On 01May2016 21:23, Chris Angelico wrote: On Sun, May 1, 2016 at 8:55 PM, Steven D'Aprano wrote: Is there an environment variable to tell the application what you consider "short", or should it read your mind? How about $LINES? If it's less than that, it'll fit on one screen. Of course, that

Re: What should Python apps do when asked to show help?

2016-05-01 Thread cs
On 01May2016 20:55, Steven D'Aprano wrote: On Sun, 1 May 2016 05:28 pm, c...@zip.com.au wrote: On 01May2016 16:44, Chris Angelico wrote: So you expect the environment variable to say which of multiple pagers you might want, but only when you already want a pager. Okay. How is an app supposed

Re: do_POST not working on http.server with python

2016-05-01 Thread Pierre Quentel
Le jeudi 28 avril 2016 10:36:27 UTC+2, Rahul Raghunath a écrit : > 0 > down vote > favorite > > > I'm trying to create a simple http server with basic GET and POST > functionality. The program is supposed to GET requests by printing out a > simple webpage that greets a user and askes how

How to fill in abbreviation in one column based on state name in another column?

2016-05-01 Thread David Shi via Python-list
Hello, I am back.  Thank you very much for your positive response. I am trying to use Pandas apply to execute a lookup function, so that we can put abbreviation in a new column, in accordance to a state name in another column. Does anyone knows how to make this to work? Regards.DavidLook up funct

Re: What should Python apps do when asked to show help?

2016-05-01 Thread Ethan Furman
On 05/01/2016 09:36 AM, Steven D'Aprano wrote: On Mon, 2 May 2016 02:30 am, Grant Edwards wrote: It's pretty rare. It is assumed that Unix uses can type " | less" Is nobody except me questioning the assumption that we're only talking about Unix users? Even Windows has "more". -- ~Ethan~

Re: What should Python apps do when asked to show help?

2016-05-01 Thread Random832
On Sun, May 1, 2016, at 13:04, Grant Edwards wrote: > On 2016-05-01, Steven D'Aprano wrote: > > Is nobody except me questioning the assumption that we're only > > talking about Unix users? > > Didn't the OP specify that he was writing a command-line utility for > Linux/Unix? We've been talking a

Re: What should Python apps do when asked to show help?

2016-05-01 Thread Gene Heskett
On Sunday 01 May 2016 12:36:48 Steven D'Aprano wrote: > On Mon, 2 May 2016 02:30 am, Grant Edwards wrote: > >> In discussions like these, it would be important to draw from > >> precedents. Are there commands that have such an option? > > > > It's pretty rare.  It is assumed that Unix uses can typ

Re: What should Python apps do when asked to show help?

2016-05-01 Thread Grant Edwards
On 2016-05-01, Steven D'Aprano wrote: > On Mon, 2 May 2016 02:30 am, Grant Edwards wrote: > >>> In discussions like these, it would be important to draw from >>> precedents. Are there commands that have such an option? >> >> It's pretty rare.  It is assumed that Unix uses can type " | less" > >

Re: Python3 html scraper that supports javascript

2016-05-01 Thread Bob Gailer
On May 1, 2016 10:20 AM, wrote: > > Hi, > > can you please recommend to me a python3 library that I can use for scrapping JS I'm not sure what you mean by that. The tool I use is Splinter. Install it using pip. that works on windows as well as linux? -- https://mail.python.org/mailman/listinfo/py

Re: What should Python apps do when asked to show help?

2016-05-01 Thread Steven D'Aprano
On Mon, 2 May 2016 02:30 am, Grant Edwards wrote: >> In discussions like these, it would be important to draw from >> precedents. Are there commands that have such an option? > > It's pretty rare.  It is assumed that Unix uses can type " | less" Is nobody except me questioning the assumption t

Re: What should Python apps do when asked to show help?

2016-05-01 Thread Grant Edwards
On 2016-05-01, Marko Rauhamaa wrote: > Grant Edwards : > >> On 2016-05-01, Chris Angelico wrote: >>> Okay. How is an app supposed to know whether or not to use a pager? >> Command line option. >> >>> How do you expect them to mindread? >> Nope, just recognize '-p' or somesuch. > > In discussions

Re: What should Python apps do when asked to show help?

2016-05-01 Thread Marko Rauhamaa
Grant Edwards : > On 2016-05-01, Chris Angelico wrote: >> Okay. How is an app supposed to know whether or not to use a pager? > Command line option. > >> How do you expect them to mindread? > Nope, just recognize '-p' or somesuch. In discussions like these, it would be important to draw from pre

Python3 html scraper that supports javascript

2016-05-01 Thread zljubisic
Hi, can you please recommend to me a python3 library that I can use for scrapping JS that works on windows as well as linux? Regards. -- https://mail.python.org/mailman/listinfo/python-list

Re: What should Python apps do when asked to show help?

2016-05-01 Thread Grant Edwards
On 2016-05-01, Chris Angelico wrote: > On Sun, May 1, 2016 at 3:24 PM, wrote: >> Yes, PAGER=cat would make "man" also not page, and likely almost everything. >> And yet I am unwilling to do so. Why? >> >> On reflection, my personal problems with this approach are twofold: >> >> - I want $PAGER t

Re: What should Python apps do when asked to show help?

2016-05-01 Thread Chris Angelico
On Sun, May 1, 2016 at 8:55 PM, Steven D'Aprano wrote: > On Sun, 1 May 2016 05:28 pm, c...@zip.com.au wrote: > >> On 01May2016 16:44, Chris Angelico wrote: > >>>So you expect the environment variable to say which of multiple pagers >>>you might want, but only when you already want a pager. Okay.

Re: What should Python apps do when asked to show help?

2016-05-01 Thread Steven D'Aprano
On Sun, 1 May 2016 05:28 pm, c...@zip.com.au wrote: > On 01May2016 16:44, Chris Angelico wrote: >>So you expect the environment variable to say which of multiple pagers >>you might want, but only when you already want a pager. Okay. How is >>an app supposed to know whether or not to use a pager?

Re: What should Python apps do when asked to show help?

2016-05-01 Thread Steven D'Aprano
On Sun, 1 May 2016 04:44 pm, Chris Angelico wrote: > On Sun, May 1, 2016 at 3:24 PM, wrote: >> Yes, PAGER=cat would make "man" also not page, and likely almost >> everything. And yet I am unwilling to do so. Why? >> >> On reflection, my personal problems with this approach are twofold: >> >> - I

Re: What should Python apps do when asked to show help?

2016-05-01 Thread alister
On Sun, 01 May 2016 17:28:53 +1000, cs wrote: > On 01May2016 16:44, Chris Angelico wrote: >>On Sun, May 1, 2016 at 3:24 PM, wrote: >>> Yes, PAGER=cat would make "man" also not page, and likely almost >>> everything. >>> And yet I am unwilling to do so. Why? >>> >>> On reflection, my personal pr

Re: What should Python apps do when asked to show help?

2016-05-01 Thread cs
On 01May2016 16:44, Chris Angelico wrote: On Sun, May 1, 2016 at 3:24 PM, wrote: Yes, PAGER=cat would make "man" also not page, and likely almost everything. And yet I am unwilling to do so. Why? On reflection, my personal problems with this approach are twofold: - I want $PAGER to specify