Good points. Distilling one single item: can I, today, run the Java SDK's suite of ValidatesRunner command against the Python ULR + Java SDK Harness, in a single Gradle command?
Kenn On Fri, Apr 26, 2019 at 9:54 AM Anton Kedin <ke...@google.com> wrote: > If there is no plans to invest in ULR then it makes sense to remove it. > > Going forward, however, I think we should try to document the higher level > approach we're taking with runners (and portability) now that we have > something working and can reflect on it. For example, couple of things that > are not 100% clear to me: > - if the focus is on python runner for portability efforts, how does java > SDK (and other languages) tie into this? E.g. how do we run, test, measure, > and develop things (pipelines, aspects of the SDK, runner); > - what's our approach to developing new features, should we make sure > python runner supports them as early as possible (e.g. schemas and SQL)? > - java DirectRunner is still there: > - it is still the primary tool for java SDK development purposes, and > as Kenn mentioned in the linked threads it adds value by making sure users > don't rely on implementation details of specific runners. Do we have a > similar story for portable scenarios? > - I assume that extra validations in the DirectRunner have impact on > performance in various ways (potentially non-deterministic). While this > doesn't matter in some cases, it might do in others. Having a local runner > that is (better) optimized for execution would probably make more sense for > perf measurements, integration tests, and maybe even local production jobs. > Is this something potentially worth looking into? > > Regards, > Anton > > > On Fri, Apr 26, 2019 at 4:41 AM Maximilian Michels <m...@apache.org> wrote: > >> Thanks for following up with this. I have mixed feelings to see the >> portable Java DirectRunner go, but I'm in favor of this change because >> it removes a lot of code that we do not really make use of. >> >> -Max >> >> On 26.04.19 02:58, Kenneth Knowles wrote: >> > Thanks for providing all this background on the PR. It is very easy to >> > see where it came from. Definitely nice to have less code and fewer >> > things that can break. Perhaps lazy consensus is enough. >> > >> > Kenn >> > >> > On Thu, Apr 25, 2019 at 4:01 PM Daniel Oliveira <danolive...@google.com >> > <mailto:danolive...@google.com>> wrote: >> > >> > Hey everyone, >> > >> > I made a preliminary PR for removing all the Java Reference Runner >> > code (PR-8380 <https://github.com/apache/beam/pull/8380>) since I >> > wanted to see if it could be done easily. It seems to be working >> > fine, so I wanted to open up this discussion to make sure people are >> > still in agreement on getting rid of this code and that people don't >> > have any concerns. >> > >> > For those who need additional context about this, this previous >> > thread >> > < >> https://lists.apache.org/thread.html/b235f8ee55a737ea399756edd80b1218ed34d3439f7b0ed59bfa8e40@%3Cdev.beam.apache.org%3E >> > >> > is where we discussed deprecating the Java Reference Runner (in some >> > places it's called the ULR or Universal Local Runner, but it's the >> > same thing). Then there's this thread >> > < >> https://lists.apache.org/thread.html/0b68efce9b7f2c5297b32d09e5d903e9b354199fe2ce446fbcd240bc@%3Cdev.beam.apache.org%3E >> > >> > where we discussed removing the code from the repo since it's been >> > deprecated. >> > >> > If no one has any objections to trying to remove the code I'll have >> > someone review the PR I wrote and start a vote to have it merged. >> > >> > Thanks, >> > Daniel Oliveira >> > >> >
