Re: Selenium script - stuck - could someone take a look?
On 2021-05-29, Veek M wrote: fixed div './/' vs '//' -- https://mail.python.org/mailman/listinfo/python-list
Selenium script - stuck - could someone take a look?
Script: http://paste.debian.net/1199271/
It mostly works but line 78 is supposed to extract
100 pieces / lot No matter what I
try it's failed and I DON'T KNOW WHY? It's a simple div.classname
match..
Could someone take a look and figure it out - I'm stuck.
import re, sys, time
from selenium.common.exceptions import NoSuchElementException
from selenium.common.exceptions import StaleElementReferenceException
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
url = 'https://www.aliexpress.com'
caps = DesiredCapabilities().FIREFOX; caps["pageLoadStrategy"] = 'eager'
ignored_exceptions=(NoSuchElementException,StaleElementReferenceException,)
fh = open('/tmp/log.html', 'w')
fh.write(' parts\n\n')
def convert(m):
money = m.group()
return str(round(float(money) * 72.4, 3))
import re
def process_fields(txt):
if '$' in txt:
txt = txt.replace('+', '')
txt = txt.replace('$', '')
txt = txt.replace('US', '')
txt = txt.replace('Shipping:', '')
r = re.sub(r'(\s*[0-9]+\.[0-9]+)', convert, txt)
return str(r)
def ali_search(url, txt):
driver.get(url)
assert 'AliExpress' in driver.title
try:
srch_elem = WebDriverWait(driver, 3600,
ignored_exceptions=ignored_exceptions).until(
EC.presence_of_element_located((By.XPATH, '//div[@class="search-key-box"]')))
print('search')
x = driver.find_element_by_id('search-key')
if 'input' in x.tag_name:
print 'success'
finally:
for c in list(txt):
time.sleep(1)
x.send_keys(c)
x.send_keys(Keys.RETURN)
try:
element = WebDriverWait(driver, 3600,
ignored_exceptions=ignored_exceptions).until(
EC.presence_of_element_located((By.XPATH, '//div[@class="product-container"]')))
finally:
print('product-container')
x = driver.find_element_by_xpath('//body')
x.send_keys(Keys.HOME)
for i in range(1,10):
print('send END')
time.sleep(1)
x.send_keys(Keys.PAGE_DOWN)
time.sleep(1)
#driver.execute_script("window.scrollTo(0,
document.body.scrollHeight);")
# EC.presence_of_element_located((By.XPATH, '//div[contains(@class, "
product-list")]')))
divs = element.find_elements_by_xpath('//li[@class="list-item
packaging_sale"]')
for c, div in enumerate(divs):
fh.write('')
for param in ['price-current', 'item-price-row packaging-sale',
'shipping-value', 'store-name']:
try:
if 'store' in param:
fh.write('' +
div.find_elements_by_class_name(param)[0].text + '')
elif 'sale' in param:
print param
lot = div.find_elements_by_class_name(param)
fh.write('' + str(lot) + '')
else:
fh.write('' +
process_fields(div.find_elements_by_class_name(param).text) + '')
except Exception as e:
fh.write('' + str(e) + '')
fh.write('\n')
fh.write('\n')
fh.close()
def part_lookup():
global driver
with webdriver.Firefox(executable_path=r'/mnt/sdb1/root/geckodriver',
firefox_binary='/mnt/sdb1/firefox/firefox-bin', capabilities=caps) as driver:
if len(sys.argv) == 2:
ali_search(url, sys.argv[1])
time.sleep(3600)
part_lookup()
--
https://mail.python.org/mailman/listinfo/python-list
Re: Applying winpdb_reborn
On Sat, 29 May 2021, Dennis Lee Bieber wrote: Except the copy of winpdb source I looked at has RPDBTERM = 'RPDBTERM' near the top. Then again, I don't know how up-to-date the OP's copy is -- I just Googled for a repository with the module. Let me check a Linux environment (Debian 10 VM) (Great -- I attempted to use the default password of a Beaglebone Black ) I've removed winpdb here as recommended by Philippe Fremy. I don't use IDEs other than emacs so I'll stick with pdb. What I find interesting is that every web page I find on 'using pdb' does no more than explain the available commands; they don't explain the debugging process. That's like showing someone what the word processor menus do; it doesn't teach the user how to be a writer. I knew the debugging process with Fortran and C, but haven't learned how to effectively use pdb to find bugs that don't issue a traceback or obvious wrong answer such as my module displaying an empty window with no widgets. Rich -- https://mail.python.org/mailman/listinfo/python-list
Re: Applying winpdb_reborn
On 29May2021 09:51, Rich Shepard wrote: >I've removed winpdb here as recommended by Philippe Fremy. I don't use >IDEs other than emacs so I'll stick with pdb. > >What I find interesting is that every web page I find on 'using pdb' does no >more than explain the available commands; they don't explain the debugging >process. That's like showing someone what the word processor menus do; it >doesn't teach the user how to be a writer. > >I knew the debugging process with Fortran and C, but haven't learned how to >effectively use pdb to find bugs that don't issue a traceback or obvious >wrong answer such as my module displaying an empty window with no >widgets. I've only just started with pdb. As of Python 3.7 there's a builtin function named breakpoint() which drops you into the debugger. I've never been a big debugger person, historicly using print() and equivalent. However, this makes it very easy to insert this call into a piece of code instead of having to invoke one's pogramme in a special way. I'd imagine debugging is much like it is in C. Wait for the breakpoint to trip, then inspect the programme variables. In your example above I'd blithely imagine checking that the list of widgets I expected were in fact constructed, etc. But I'd also be littering my window setup with progress print calls :-) Cheers, Cameron Simpson -- https://mail.python.org/mailman/listinfo/python-list
Re: Applying winpdb_reborn
On 30May2021 09:03, Cameron Simpson wrote: >>I knew the debugging process with Fortran and C, but haven't learned >>how to effectively use pdb to find bugs that don't issue a traceback or >>obvious >>wrong answer such as my module displaying an empty window with no >>widgets. Also, searching for "pdb tutorial" seems to find a bunch of links. Cheers, Cameron Simpson -- https://mail.python.org/mailman/listinfo/python-list
Re: Applying winpdb_reborn
On Sun, May 30, 2021 at 9:19 AM Cameron Simpson wrote: > > On 29May2021 09:51, Rich Shepard wrote: > >I knew the debugging process with Fortran and C, but haven't learned how to > >effectively use pdb to find bugs that don't issue a traceback or obvious > >wrong answer such as my module displaying an empty window with no > >widgets. > > I've only just started with pdb. As of Python 3.7 there's a builtin > function named breakpoint() which drops you into the debugger. I've > never been a big debugger person, historicly using print() and > equivalent. However, this makes it very easy to insert this call into a > piece of code instead of having to invoke one's pogramme in a special > way. > > I'd imagine debugging is much like it is in C. Wait for the breakpoint > to trip, then inspect the programme variables. > At its most fundamental level, debugging is the same regardless of the language: make a bug happen, then see as much as you possibly can about what caused it. Whether that's dumping print() calls through the code or using a debug harness, the normal way to figure out what code's doing is to see what your variables hold at different times. Using something like gdb, pdb, etc, etc, is just a shorthand for putting lots and LOTS of print calls into your code. So if you're not comfortable with those tools, there's no shame in using the more obvious/simple way of doing things! (Plus, there's not always an opportunity to use a debug harness. Sometimes you just have to put your prints into production and let it run for two weeks in the hope that the bug will show itself.) ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Applying winpdb_reborn
On 29/05/2021 19:10, Dennis Lee Bieber wrote: > On Sat, 29 May 2021 09:51:04 -0700 (PDT), Rich Shepard > declaimed the following: >> What I find interesting is that every web page I find on 'using pdb' does no >> more than explain the available commands; they don't explain the debugging >> process. That's like showing someone what the word processor menus do; it >> doesn't teach the user how to be a writer. I would point out that my book on learning to program included a chapter on using debugging, half of which was on pdb and how to use it. But that book is now over 20 years old and based on Python 1.5.3! but interestingly nearly all the reviewers specifically praised my inclusion of a chapter on debugging. >> I knew the debugging process with Fortran and C, but haven't learned how to >> effectively use pdb to find bugs that don't issue a traceback or obvious >> wrong answer such as my module displaying an empty window with no widgets. Like most debuggers, set breakpoints, watchpoints and variable traces. > The only debugger I used to be familiar with was that of (Open)VMS. My first job after graduating was writing white-box test scripts for a PABX control system. It was done by writing VMS Debug scripts and then running those scripts with various input files to provide the different initialization settings. I wrote around 100K lines of debug scripts to test around 60K lines of C. It was an amazingly powerful debugger. > I don't even want to think of GDB... I used gdb a few years after VMS DBG and thought it a dreadfully primitive beast by comparison. The scripting and automation features in particular were arcane by comparison. But they are all more similar to pdb than to winpdb. Maybe the OP could consider IDLE, it would at least be closer to an IDE debugger than pdb! And in its latest incarnations Idle is becoming a fairly useful tool once more. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos -- https://mail.python.org/mailman/listinfo/python-list
Re: Applying winpdb_reborn
On Sun, 30 May 2021, Cameron Simpson wrote: Also, searching for "pdb tutorial" seems to find a bunch of links. Cameron, That's true. All the ones I've read list the various pdb commands. Knowing the commands is different from knowing when and how to apply them. I'll work back to remembering how I debugged code years ago with stubs and print() statements and use pdb with them. Thanks, Rich -- https://mail.python.org/mailman/listinfo/python-list
Re: Applying winpdb_reborn
On Sun, 30 May 2021, Chris Angelico wrote: (Plus, there's not always an opportunity to use a debug harness. Sometimes you just have to put your prints into production and let it run for two weeks in the hope that the bug will show itself.) ChrisA, Please excuse my long-winded description of a FORTRAN IV bug in an lake ecosystem energy model I wrote at the University of Illinois in the early 1970s. It is an example of what you wrote above. The program fill two boxes of 80-column Hollerith cards punched on an IBM 29 (if I correctly recall the model) keypuch machine. The output was written on green-barred wide paper on a line printer. Each time I ran the model it would produce one of four different, but wrong, answers for one variable. I closely examined the 1.5" thick fan-folded line printer output many times. I never saw the error, even with diagnostic stubs added. When I took the output to the computer center they looked at it for a week and couldn't find the error, either. Shortly after that, when I again examined the code, line-by-line, I saw the error: a line that should have read FOR I=1 to N actually read FOR I=I to N All of us saw the uppercase I as a 1 because a) that's what we expected to see and b) line printer output on green-barred paper was not that clear, even with fresh ribbons. That lesson has stuck with me ever since. Regards, Rich -- https://mail.python.org/mailman/listinfo/python-list
Re: Applying winpdb_reborn
On Sun, May 30, 2021 at 9:53 AM Rich Shepard wrote: > > On Sun, 30 May 2021, Chris Angelico wrote: > > > (Plus, there's not always an opportunity to use a debug harness. Sometimes > > you just have to put your prints into production and let it run for two > > weeks in the hope that the bug will show itself.) > > ChrisA, > > Please excuse my long-winded description of a FORTRAN IV bug in an lake > ecosystem energy model I wrote at the University of Illinois in the early > 1970s. It is an example of what you wrote above. > > The program fill two boxes of 80-column Hollerith cards punched on an IBM > 29 (if I correctly recall the model) keypuch machine. The output was written > on green-barred wide paper on a line printer. > > Each time I ran the model it would produce one of four different, but wrong, > answers for one variable. I closely examined the 1.5" thick fan-folded line > printer output many times. I never saw the error, even with diagnostic stubs > added. When I took the output to the computer center they looked at it for a > week and couldn't find the error, either. > > Shortly after that, when I again examined the code, line-by-line, I saw the > error: a line that should have read > FOR I=1 to N > actually read > FOR I=I to N > > All of us saw the uppercase I as a 1 because a) that's what we expected to > see and b) line printer output on green-barred paper was not that clear, > even with fresh ribbons. > > That lesson has stuck with me ever since. > Good lesson. And it's one of the things we really enjoy with modern editing systems (IDEs, or just smart text editors) - we can use colour to distinguish variable names from numeric literals! Although that can sometimes get disrupted. The editor I use (SciTE) will nicely syntax-highlight JavaScript code if it's in a .js file, but if you're embedding a small amount of code inside a
Re: Applying winpdb_reborn
On 30/05/2021 11.52, Rich Shepard wrote:
> On Sun, 30 May 2021, Chris Angelico wrote:
> Please excuse my long-winded description of a FORTRAN IV bug in an lake
> ecosystem energy model I wrote at the University of Illinois in the early
> 1970s. It is an example of what you wrote above.
>
> The program fill two boxes of 80-column Hollerith cards punched on an IBM
> 29 (if I correctly recall the model) keypuch machine. The output was
> written
> on green-barred wide paper on a line printer.
We always referred to it as an "oh-two-nine" ("029"). Quite why the more
pedantic didn't demand it be "zero-two-nine" is anybody's guess.
Herewith some history (aka 'blast from the past') together with an
ironic photo showing an 029 alongside a lab of (DEC) VT-100 terminals -
several generations of input-device apart:
http://www.columbia.edu/cu/computinghistory/029.html
Those were the days...
--
Regards,
=dn
--
https://mail.python.org/mailman/listinfo/python-list
(OT) Re: Applying winpdb_reborn
On 30/05/21 1:46 pm, dn wrote:
We always referred to it as an "oh-two-nine" ("029").
I had the privilege of helping to dismantle a couple of those
when they were decommissioned at the University of Canterbury.
Amazing pieces of technology -- purely electromechanical, no
electronics in sight. Even the little dot-matrix printer that
wrote human-readable characters along the top of the card was
all mechanical!
--
Greg
--
https://mail.python.org/mailman/listinfo/python-list
Re: Selenium script - stuck - could someone take a look?
On 2021-05-29, Dennis Lee Bieber wrote:
> On Sat, 29 May 2021 09:40:35 - (UTC), Veek M declaimed
> the following:
>
ah, yeah - man that took me a while to do (save to local file and use
file:///). It's working now, basically xpath mistake because I've
forgotten stuff.. but the script is almost complete and cleaned up..
Basically forgot that div.xpath('// doesn't use div as the root window -
thing starts the lookup from the very beginning. You got to .//
My "Next Page" is creating problems - asked here as well:
https://www.reddit.com/r/selenium/comments/nnrxuu/
what_is_overlay_backdrop_how_does_it_block_a/
def page_next():
tmp = driver.find_element_by_xpath('//button[contains(@class,
" next-next")]')
tmp = driver.find_element_by_xpath('.//div[@class=
"next-overlay-backdrop")]')
#e = WebDriverWait(driver, 200).until(EC.element_to_be_clickable(
#(By.XPATH, "//button[contains(@class, ' next-next']"))).click()
#e.click()
if tmp: tmp.click()
else: raise SystemExit('no next')
This is the error I get
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.ElementClickInterceptedException: Message:
Element is not clickable at
point (699,649) because another element
obscures it
--
https://mail.python.org/mailman/listinfo/python-list
Re: imaplib: is this really so unwieldy?
On 5/28/21 2:36 AM, boB Stepp wrote: On Thu, May 27, 2021 at 6:22 PM Cameron Simpson wrote: On 27May2021 18:42, hw wrote: So it seems that IMAP support through python is virtually non-existent. This still sureprises me, but I've not tried to use IMAP seriously. I read email locally, and collect it with POP instead. With a tool I wrote myself in Python, as it happens. I am out of my league here, but what I found in one of my books might be helpful. Al Sweigart wrote a useful book, "Automate the Boring Stuff in Python". In chapter 16 he considers email. In the "IMAP" section he states: Just as SMTP is the protocol for sending email, the Internet Message Access Protocol (IMAP) specifies how to communicate with an email provider’s server to retrieve emails sent to your email address. Python comes with an imaplib module, but in fact the third-party imapclient module is easier to use. This chapter provides an introduction to using IMAPClient; the full documentation is at http://imapclient.readthedocs.org/. The imapclient module downloads emails from an IMAP server in a rather complicated format. Most likely, you’ll want to convert them from this format into simple string values. The pyzmail module does the hard job of parsing these email messages for you. You can find the complete documentation for PyzMail at http://www.magiksys.net/pyzmail/. Install imapclient and pyzmail from a Terminal window. Appendix A has steps on how to install third-party modules. In the next little section he shows how to retrieve and delete emails with IMAP using the two third-party tools mentioned above. And of course there is more. Apparently this book is now in its second edition. The first edition is available online for free. The link to chapter 16 which discusses email is: https://automatetheboringstuff.com/chapter16/ Hopefully this will prove helpful to the OP. Thanks for the pointer! I don't know which imaplib the author uses; the imaplib I found definitely doesn't give uids of the messages, contrary to the example he's giving. -- https://mail.python.org/mailman/listinfo/python-list
Re: name for new Enum decorator
On 5/28/21 5:23 AM, Ethan Furman wrote: Greetings! The Flag type in the enum module has had some improvements, but I find it necessary to move one of those improvements into a decorator instead, and I'm having a hard time thinking up a name. What is the behavior? Well, a name in a flag type can be either canonical (it represents one thing), or aliased (it represents two or more things). To use Color as an example: class Color(Flag): RED = 1 # 0001 GREEN = 2 # 0010 BLUE = 4 # 0100 PURPLE = RED | BLUE # 0101 WHITE = RED | GREEN | BLUE # 0111 The flags RED, GREEN, and BLUE are all canonical, while PURPLE and WHITE are aliases for certain flag combinations. But what if we have something like: class Color(Flag): RED = 1 # 0001 BLUE = 4 # 0100 WHITE = 7 # 0111 As you see, WHITE is an "alias" for a value that does not exist in the Flag (0010, or 2). That seems like it's probably an error. But what about this? class FlagWithMasks(IntFlag): DEFAULT = 0x0 FIRST_MASK = 0xF FIRST_ROUND = 0x0 FIRST_CEIL = 0x1 FIRST_TRUNC = 0x2 SECOND_MASK = 0xF0 SECOND_RECALC = 0x00 SECOND_NO_RECALC = 0x10 THIRD_MASK = 0xF00 THIRD_DISCARD = 0x000 THIRD_KEEP = 0x100 Here we have three flags (FIRST_MASK, SECOND_MASK, THIRD_MASK) that are aliasing values that don't exist, but it seems intentional and not an error. So, like the enum.unique decorator that can be used when duplicate names should be an error, I'm adding a new decorator to verify that a Flag has no missing aliased values that can be used when the programmer thinks it's appropriate... but I have no idea what to call it. Any nominations? Why don't you just use the colour names from rgb.txt and their values? That's plain, simple and pretty standard. -- https://mail.python.org/mailman/listinfo/python-list
