Re: Staged PIP package mysteriously ungzipped, non-installable inside the worker

2020-08-11 Thread Maximilian Michels
Looks like you ran into a bug. You could just run your program without specifying any arguments, since running with Python's FnApiRunner should be enough. Alternatively, how about trying to run the same pipeline with the FlinkRunner? Use: --runner=FlinkRunner and do not specify an endpoint.

Study On Rejected Refactorings

2020-08-11 Thread Jevgenija Pantiuchina
Dear contributors, As part of a research team from Università della Svizzera italiana (Switzerland) and University of Sannio (Italy), we have analyzed refactoring pull requests in apache/beam repository and are looking for developers for a short 5-10 min survey (https://usi.eu.qualtrics.com/jf

Re: Staged PIP package mysteriously ungzipped, non-installable inside the worker

2020-08-11 Thread Eugene Kirpichov
Hi Maximilian, Thank you - it works fine with the embedded Flink runner (per below, seems like it's not using Docker for running Python code? What is it using then?). However, the original bug appears to be wider than I thought - it is also present if I run --runner=FlinkRunner --environment_type

Re: Beam flink runner job not keeping up with input rate after downscaling

2020-08-11 Thread Kathula, Sandeep
Hi Eleanore, We are using atleast once semantics when writing to Kafka. We are Ok with duplicate messages. Thanks Sandeep Kathula From: Eleanore Jin Date: Monday, August 10, 2020 at 11:32 AM To: "Kathula, Sandeep" Cc: "user@beam.apache.org" , "Vora, Jainik" , "Benenson, Mikhail" , "Deshpande,

Re: Staged PIP package mysteriously ungzipped, non-installable inside the worker

2020-08-11 Thread Eugene Kirpichov
Ok I found the bug, and now I don't understand how it could have possibly ever worked. And if this was never tested, then I don't understand why it works after fixing this one bug :) Basically the Python ArtifactStaging/RetrievalService uses FileSystems.open() to read the artifacts to be staged, a

Re: Staged PIP package mysteriously ungzipped, non-installable inside the worker

2020-08-11 Thread Eugene Kirpichov
(FYI Sam +sbrot...@gmail.com ) On Tue, Aug 11, 2020 at 5:00 PM Eugene Kirpichov wrote: > Ok I found the bug, and now I don't understand how it could have possibly > ever worked. And if this was never tested, then I don't understand why it > works after fixing this one bug :) > > Basically the Py