Hi, I need (or at least I think I do) to do a cross operation between two huge datasets. One dataset is a list of points. The other one is a list of shapes (areas).
I want to know, for each point, the areas (they might overlap so a point can be in multiple areas) it belongs to so I thought I’d “cross” my points and areas since I need to test each point against each area. I tried it and my job stucks seems to work for some seconds then, at some point, it stucks. I’m wondering if Flink, for cross operations, tries to load one of the two datasets into RAM or if it’s able to split the job in multiple iterations (even if it means reading one of the two datasets multiple times). Or maybe I’m going at it the wrong way, or missing some parameters, feel free to correct me ☺ I’m using flink 1.0.1. Thanks in advance Gwen’
