Re: Out of memory while reading excel file

2017-05-10 Thread Peter Otten
Mahmood Naderan via Python-list wrote: >>a = numpy.zeros((ws.max_row, ws.max_column), dtype=float) >>for y, row in enumerate(ws.rows): >> a[y] = [cell.value for cell in row] > > > > Peter, > > As I used this code, it gave me an error that cannot convert string to > float for the first cell.

Re: Out of memory while reading excel file

2017-05-10 Thread Mahmood Naderan via Python-list
Hi, I used the old fashion coding style to create a matrix and read/add the cells. W = load_workbook(fname, read_only = True) p = W.worksheets[0] m = p.max_row n = p.max_column arr = np.empty((m, n), dtype=object) for r in range(1, m): for c in range(1, n): d = p.cell(row=r, colu

Re: Out of memory while reading excel file

2017-05-10 Thread Mahmood Naderan via Python-list
>a = numpy.zeros((ws.max_row, ws.max_column), dtype=float) >for y, row in enumerate(ws.rows): > a[y] = [cell.value for cell in row] Peter, As I used this code, it gave me an error that cannot convert string to float for the first cell. All cells are strings. Regards, Mahmood -- https://

Re: Out of memory while reading excel file

2017-05-10 Thread Mahmood Naderan via Python-list
Hi, I am confused with that. If you say that numpy is not suitable for my case and may have large overhead, what is the alternative then? Do you mean that numpy is a good choice here while we can reduce its overhead? Regards, Mahmood -- https://mail.python.org/mailman/listinfo/python-list

Re: Low level I/O: because I could

2017-05-10 Thread eryk sun
On Wed, May 10, 2017 at 10:30 PM, Rob Gaddi wrote: > Sorry, but I'm just too proud of this. > > Given that you have: > > class RegisterLayout(ctypes.Structure): > ...yadayadayada... > > You can then: > > fh = os.open('/dev/devicethingy', os.O_RDWR) > mm = mmap.mmap(fh, ctypes.sizeof(Regi

Repeatedly crawl website every 1 min

2017-05-10 Thread liyucun2012
Hi Everyone, Thanks for stoping by. I am working on a feature to crawl website content every 1 min. I am curious to know if there any good open source project for this specific scenario. Specifically, I have many urls, and I want to maintain a thread pool so that each thread will repeatedly cr

Re: Iterating through a list. Upon condition I want to move on to next item in list

2017-05-10 Thread breamoreboy
On Wednesday, May 10, 2017 at 8:25:25 PM UTC+1, aaron.m@gmail.com wrote: You've all ready had some answers, but are you after something like this? for elem in mylist: if someThing(elem) is True: continue. Kindest regards. Mark Lawrence. -- https://mail.python.org/mailman/listin

Re: Iterating through a list. Upon condition I want to move on to next item in list

2017-05-10 Thread Chris Angelico
On Thu, May 11, 2017 at 8:41 AM, Dan Stromberg wrote: > On Wed, May 10, 2017 at 1:46 PM, MRAB wrote: > >> NEVER use a 'bare except' to suppress exceptions! It'll catch _all_ >> exceptions, even NameError (if you've misspelled a name, it'll catch that >> too). Catch only those exceptions that you'

Re: Iterating through a list. Upon condition I want to move on to next item in list

2017-05-10 Thread Erik
On 10/05/17 23:41, Dan Stromberg wrote: On Wed, May 10, 2017 at 1:46 PM, MRAB wrote: NEVER use a 'bare except' to suppress exceptions! It'll catch _all_ exceptions, even NameError (if you've misspelled a name, it'll catch that too). Catch only those exceptions that you're prepared to deal with

Re: Iterating through a list. Upon condition I want to move on to next item in list

2017-05-10 Thread Dan Stromberg
On Wed, May 10, 2017 at 1:46 PM, MRAB wrote: > NEVER use a 'bare except' to suppress exceptions! It'll catch _all_ > exceptions, even NameError (if you've misspelled a name, it'll catch that > too). Catch only those exceptions that you're prepared to deal with. When writing many kinds of applica

Low level I/O: because I could

2017-05-10 Thread Rob Gaddi
Sorry, but I'm just too proud of this. Given that you have: class RegisterLayout(ctypes.Structure): ...yadayadayada... You can then: fh = os.open('/dev/devicethingy', os.O_RDWR) mm = mmap.mmap(fh, ctypes.sizeof(RegisterLayout)) registers = RegisterLayout.from_buffer(mm) And it jus

Re: Iterating through a list. Upon condition I want to move on to next item in list

2017-05-10 Thread MRAB
On 2017-05-10 20:25, aaron.m.weisb...@gmail.com wrote: Good afternoon, I have a list that I'm iterating thorough in Python. Each item in the list will have between 1-200 urls associated with it. My goal is to move on to the next. I have a variable "associationsCount" that is counting the nu

Re: Iterating through a list. Upon condition I want to move on to next item in list

2017-05-10 Thread Erik
On 10/05/17 20:25, aaron.m.weisb...@gmail.com wrote: As my code stands right now, it counts through the stateList[0] and then when associationsCount gets to 0 it just stops. I know I need to add another couple lines of code to get it back going again It's very difficult to see what you're tryi

Re: Out of memory while reading excel file

2017-05-10 Thread Mahmood Naderan via Python-list
Hi I will try your code... meanwhile I have to say, as you pointed earlier and as stated in the documents, numpy is designed to handle large arrays and that is the reason I chose that. If there is a better option, please let me know. Regards, Mahmood -

Iterating through a list. Upon condition I want to move on to next item in list

2017-05-10 Thread aaron . m . weisberg
Good afternoon, I have a list that I'm iterating thorough in Python. Each item in the list will have between 1-200 urls associated with it. My goal is to move on to the next. I have a variable "associationsCount" that is counting the number of urls and once it gets to 0 i want to move on to

Re: Out of memory while reading excel file

2017-05-10 Thread Peter Otten
Mahmood Naderan via Python-list wrote: > Well actually cells are treated as strings and not integer or float > numbers. May I ask why you are using numpy when you are dealing with strings? If you provide a few details about what you are trying to achieve someone may be able to suggest a workabl

Re: Out of memory while reading excel file

2017-05-10 Thread Irmen de Jong
On 10-5-2017 17:12, Mahmood Naderan wrote: > So, I think numpy is unable to manage the memory. That assumption is very likely to be incorrect. >> np.array([[i.value for i in j] for j in p.rows]) I think the problem is in the way you feed your excel data into the numpy array constructor. The co

Re: Out of memory while reading excel file

2017-05-10 Thread Mahmood Naderan via Python-list
Well actually cells are treated as strings and not integer or float numbers. One way to overcome is to get the number of rows and then split it to 4 or 5 arrays and then process them. However, i was looking for a better solution. I read in pages that large excels are in the order of milion rows

Re: import docx error

2017-05-10 Thread Grant Edwards
On 2017-05-10, RRS1 via Python-list wrote: > I am very new to Python, have only done simple things >>>print("hello > world") type things. I've really been looking forward to using Python. I > bought Two books, downloaded Python 3.6.1 (32 & 64) and each time I try this: > > import doc

Re: Practice Python

2017-05-10 Thread Ian Kelly
On Mon, May 8, 2017 at 3:50 AM, Lutz Horn wrote: > A strange way to publish code. Not if your goal is to drive traffic toward your YouTube channel. -- https://mail.python.org/mailman/listinfo/python-list

import docx error

2017-05-10 Thread RRS1 via Python-list
Hi, I am very new to Python, have only done simple things >>>print("hello world") type things. I've really been looking forward to using Python. I bought Two books, downloaded Python 3.6.1 (32 & 64) and each time I try this: >>> import docx I get errors. Traceback (most recent call

Re: Practice Python

2017-05-10 Thread Andre Müller
Am 10.05.2017 um 14:18 schrieb Chris Angelico: > On Wed, May 10, 2017 at 10:11 PM, Andre Müller wrote: >> 1.) a short example for Python 3, but not exactly what they want. >> >> def square(numbers): >> yield from sorted(n**2 for n in numbers) >> >> numberlist = [99, 4, 3, 5, 6, 7, 0] >> resu

Re: Out of memory while reading excel file

2017-05-10 Thread Peter Otten
Mahmood Naderan via Python-list wrote: > Thanks for your reply. The openpyxl part (reading the workbook) works > fine. I printed some debug information and found that when it reaches the > np.array, after some 10 seconds, the memory usage goes high. > > > So, I think numpy is unable to manage th

Re: Out of memory while reading excel file

2017-05-10 Thread Mahmood Naderan via Python-list
Thanks for your reply. The openpyxl part (reading the workbook) works fine. I printed some debug information and found that when it reaches the np.array, after some 10 seconds, the memory usage goes high. So, I think numpy is unable to manage the memory. Regards, Mahmood On Wednesday, Ma

Re: Out of memory while reading excel file

2017-05-10 Thread Peter Otten
Mahmood Naderan via Python-list wrote: > Hello, > > The following code which uses openpyxl and numpy, fails to read large > Excel (xlsx) files. The file si 20Mb which contains 100K rows and 50 > columns. > > > > W = load_workbook(fname, read_only = True) > > p = W.worksheets[0] > > a=[] > >

Out of memory while reading excel file

2017-05-10 Thread Mahmood Naderan via Python-list
Hello, The following code which uses openpyxl and numpy, fails to read large Excel (xlsx) files. The file si 20Mb which contains 100K rows and 50 columns. W = load_workbook(fname, read_only = True) p = W.worksheets[0] a=[] m = p.max_row n = p.max_column np.array([[i.value for i in j] for

ANN: Wing Python IDE 6.0.5 released

2017-05-10 Thread Wingware
Hi, We've just released Wing 6.0.5 which simplifies remote externally launched debugging, improves remote development documentation, adds a How-To for remote web development, documents how to debug extension scripts written for the IDE, adds syntax highlighting for Markdown, solves some probl

Re: Practice Python

2017-05-10 Thread Chris Angelico
On Wed, May 10, 2017 at 10:11 PM, Andre Müller wrote: > 1.) a short example for Python 3, but not exactly what they want. > > def square(numbers): > yield from sorted(n**2 for n in numbers) > > numberlist = [99, 4, 3, 5, 6, 7, 0] > result = list(square(numberlist)) If you're going to use sort

Re: Practice Python

2017-05-10 Thread Andre Müller
Hello, 1.) a short example for Python 3, but not exactly what they want. def square(numbers): yield from sorted(n**2 for n in numbers) numberlist = [99, 4, 3, 5, 6, 7, 0] result = list(square(numberlist)) To solve this tutorial, you need a different way. I'm just showing how sexy Python 3 i

EuroPython 2017: Talk voting results

2017-05-10 Thread M.-A. Lemburg
Thank you all for participating in last week’s talk voting: https://ep2017.europython.eu/en/speakers/talk-voting/ We have again broken a record, with more than 13,353 cast votes, a 22% increase compared to last year. We had almost 400 submissions to vote on. Users voted on 47 sessions on ave