Re: Multiprocessing performance question

2019-02-21 Thread DL Neil
George: apologies for mis-identifying yourself as OP. Israel: On 22/02/19 6:04 AM, Israel Brewster wrote: Actually not a ’toy example’ at all. It is simply the first step in gridding some data I am working with - a problem that is solved by tools like SatPy, but unfortunately I can’t use SatPy

Re: Multiprocessing performance question

2019-02-21 Thread Israel Brewster
Actually not a ’toy example’ at all. It is simply the first step in gridding some data I am working with - a problem that is solved by tools like SatPy, but unfortunately I can’t use SatPy because it doesn’t recognize my file format, and you can’t load data directly. Writing a custom file import

Re: Multiprocessing performance question

2019-02-20 Thread george trojan
I don't know whether this is a toy example, having grid of this size is not uncommon. True, it would make more sense to do distribute more work on each box, if there was any. One has to find a proper balance, as with many other things in life. I simply responded to a question by the OP. George O

Re: Multiprocessing performance question

2019-02-20 Thread DL Neil
George On 21/02/19 1:15 PM, george trojan wrote: def create_box(x_y): return geometry.box(x_y[0] - 1, x_y[1], x_y[0], x_y[1] - 1) x_range = range(1, 1001) y_range = range(1, 801) x_y_range = list(itertools.product(x_range, y_range)) grid = list(map(create_box, x_y_range)) Which creates

Re: Multiprocessing performance question

2019-02-20 Thread george trojan
def create_box(x_y): return geometry.box(x_y[0] - 1, x_y[1], x_y[0], x_y[1] - 1) x_range = range(1, 1001) y_range = range(1, 801) x_y_range = list(itertools.product(x_range, y_range)) grid = list(map(create_box, x_y_range)) Which creates and populates an 800x1000 “grid” (represented as a fl

Re: Multiprocessing performance question

2019-02-18 Thread Israel Brewster
> On Feb 18, 2019, at 6:37 PM, Ben Finney wrote: > > I don't have anything to add regarding your experiments with > multiprocessing, but: > > Israel Brewster writes: > >> Which creates and populates an 800x1000 “grid” (represented as a flat >> list at this point) of “boxes”, where a box is a

Re: Multiprocessing performance question

2019-02-18 Thread Ben Finney
I don't have anything to add regarding your experiments with multiprocessing, but: Israel Brewster writes: > Which creates and populates an 800x1000 “grid” (represented as a flat > list at this point) of “boxes”, where a box is a > shapely.geometry.box(). This takes about 10 seconds to run. Thi

Multiprocessing performance question

2019-02-18 Thread Israel Brewster
I have the following code running in python 3.7: def create_box(x_y): return geometry.box(x_y[0] - 1, x_y[1], x_y[0], x_y[1] - 1) x_range = range(1, 1001) y_range = range(1, 801) x_y_range = list(itertools.product(x_range, y_range)) grid = list(map(create_box, x_y_range)) Which creates and