Hi, +1 to make flink-streaming-java an API only module and solve it sooner rather than later. It would be more clear to only expose an SDK module for writing jobs.
Another benefit I can see is: the flink-streaming-java would be scala-free if we reverse the dependencies and this would be really nice for the Java API module. As for the issue of dependencies setup of users, I agree with Stephan that it's ok to do so if we add corresponding document and runtime error messages about the changes. Best, Hequn On Fri, Mar 6, 2020 at 3:03 AM Kostas Kloudas <kklou...@gmail.com> wrote: > Big +1 also from my side. > > This will eliminate some work-arounds used so far to bypass the module > structure (like code using reflection to extract a JobGraph from a > Pipeline). > > I agree with Stephan that with proper documentation, release notes and > tooling update, it will hopefully not be a big hassle for users to > migrate. > Also I think it should be done as early in the release as possible, so > that we can give it enough exposure and testing. In the past, such > deep changes late in the release have led to longer release-testing > periods and, eventually, longer release cycles. > > Cheers, > Kostas > > On Thu, Mar 5, 2020 at 3:35 PM Stephan Ewen <se...@apache.org> wrote: > > > > +1 to this fix, in general. > > > > If the main issue is that users have to now add "flink-clients" > explicitly, > > then I think this is okay, if we spell it out prominently in the release > > notes, and make sure quickstarts / etc are updated, and have a good error > > message when client/runtime classes are not found. > > > > On Thu, Mar 5, 2020 at 2:56 PM Aljoscha Krettek <aljos...@apache.org> > wrote: > > > > > Hi, > > > > > > thanks for starting the discussion, Tison! > > > > > > I'd like to fix this dependency mess rather sooner than later, but we > do > > > have to consider the fact that we are breaking the dependency setup of > > > users. If they they only had a dependency on flink-streaming-java > before > > > but used classes from flink-clients they would have to explicitly add > > > this dependency now. > > > > > > Let's see what others think. > > > > > > Best, > > > Aljoscha > > > > > > On 05.03.20 02:53, tison wrote: > > > > Hi devs, > > > > > > > > Here is a proposal to reverse the dependency from > flink-streaming-java to > > > > flink-client, for a proper > > > > module dependency graph. Since it changes current structure, it > should be > > > > discussed publicly. > > > > > > > > The original idea comes from that flink-streaming-java acts as an API > > > only > > > > module just as what > > > > we do in its batch companion flink-java. If a Flink user want to > write a > > > > minimum DataStream > > > > program, the only dependency should be flink-streaming java. > > > > > > > > However, currently as it is implemented, flink-client and even > > > > flink-runtime are transitively polluted > > > > in when user depends on flink-streaming-java. These dependencies > polluted > > > > in as > > > > > > > > flink-client: > > > > - previously, ClusterClient, which is removed by FLIP-73 Executors > > > > - accidentally, ProgramInvocationException, we just throw in > place as > > > it > > > > is accessible. > > > > - transitively, flink-optimizer, for one utility. > > > > - transitively, flink-java, for several utilities. > > > > flink-runtime: > > > > - mainly for JobGraph generating. > > > > > > > > With a previous discussion with @Aljoscha Krettek < > aljos...@apache.org> > > > our > > > > goal is briefly making flink-streaming-java > > > > an API only module. As a first step we can break the dependency from > > > > flink-streaming-java to > > > > flink-client[1][2]. > > > > > > > > With this first step, continuously we factor out common utilities in > > > > flink-java to > > > > flink-core and eventually eliminate dependencies from streaming to > batch; > > > > while > > > > orthogonally, we factor out job compilation logic into > > > > flink-streaming-compiler module and > > > > break the dependency to flink-runtime. The final dependency graph > will > > > be: > > > > > > > > > > > > flink-client -> flink-streaming-compiler -> flink-runtime > > > > \-> > > > > flink-streaming-java > > > > > > > > Looking forward to your feedback. Basically whether or not it is in a > > > right > > > > direction, and if so, > > > > how the community integrates this proposal. > > > > > > > > Best, > > > > tison. > > > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-15090 > > > > [2] https://issues.apache.org/jira/browse/FLINK-16427 > > > > > > > >
