Re: Python API iteration issues

2016-10-25 Thread Chesnay Schepler
Hey Geoffrey, that's great to hear! Random.nextInt() should suffice; but this is only done twice per operator anyway so i wouldn't worry too much about it. Regards, Chesnay On 25.10.2016 17:52, Geoffrey Mon wrote: Hello Chesnay, Thank you, this solves the last problem that I had with my pr

Re: Python API iteration issues

2016-10-25 Thread Geoffrey Mon
Hello Chesnay, Thank you, this solves the last problem that I had with my project! I added code to append a random UUID to each data channel. (see https://github.com/GEOFBOT/flink/commit/b94b1249c4a014d8d1e412f7647ff192f3621b10) However, is a UUID overkill? We only need the channel to be unique f

Re: Python API iteration issues

2016-10-23 Thread Chesnay Schepler
Hey Geoffrey, I believe what we got here is that multiple operators are sending data over the same channel. The data transfer between the java and python processes uses memory-mapped files. The file name is based on the ID of the set and the subtask index of the operator and thus determinist

Python API iteration issues

2016-10-22 Thread Geoffrey Mon
Hello all, Thanks to Chesnay for the assistance with my other issues. I have one final issue that I can't figure out how to solve that should hopefully be the last one. I have been working on implementing bulk iterations in the Python API and using said iterations in a research project. At the mo