+1 to this fix, in general. If the main issue is that users have to now add "flink-clients" explicitly, then I think this is okay, if we spell it out prominently in the release notes, and make sure quickstarts / etc are updated, and have a good error message when client/runtime classes are not found.
On Thu, Mar 5, 2020 at 2:56 PM Aljoscha Krettek <aljos...@apache.org> wrote: > Hi, > > thanks for starting the discussion, Tison! > > I'd like to fix this dependency mess rather sooner than later, but we do > have to consider the fact that we are breaking the dependency setup of > users. If they they only had a dependency on flink-streaming-java before > but used classes from flink-clients they would have to explicitly add > this dependency now. > > Let's see what others think. > > Best, > Aljoscha > > On 05.03.20 02:53, tison wrote: > > Hi devs, > > > > Here is a proposal to reverse the dependency from flink-streaming-java to > > flink-client, for a proper > > module dependency graph. Since it changes current structure, it should be > > discussed publicly. > > > > The original idea comes from that flink-streaming-java acts as an API > only > > module just as what > > we do in its batch companion flink-java. If a Flink user want to write a > > minimum DataStream > > program, the only dependency should be flink-streaming java. > > > > However, currently as it is implemented, flink-client and even > > flink-runtime are transitively polluted > > in when user depends on flink-streaming-java. These dependencies polluted > > in as > > > > flink-client: > > - previously, ClusterClient, which is removed by FLIP-73 Executors > > - accidentally, ProgramInvocationException, we just throw in place as > it > > is accessible. > > - transitively, flink-optimizer, for one utility. > > - transitively, flink-java, for several utilities. > > flink-runtime: > > - mainly for JobGraph generating. > > > > With a previous discussion with @Aljoscha Krettek <aljos...@apache.org> > our > > goal is briefly making flink-streaming-java > > an API only module. As a first step we can break the dependency from > > flink-streaming-java to > > flink-client[1][2]. > > > > With this first step, continuously we factor out common utilities in > > flink-java to > > flink-core and eventually eliminate dependencies from streaming to batch; > > while > > orthogonally, we factor out job compilation logic into > > flink-streaming-compiler module and > > break the dependency to flink-runtime. The final dependency graph will > be: > > > > > > flink-client -> flink-streaming-compiler -> flink-runtime > > \-> > > flink-streaming-java > > > > Looking forward to your feedback. Basically whether or not it is in a > right > > direction, and if so, > > how the community integrates this proposal. > > > > Best, > > tison. > > > > [1] https://issues.apache.org/jira/browse/FLINK-15090 > > [2] https://issues.apache.org/jira/browse/FLINK-16427 > > >
