I opened the PR to backport the changes: https://github.com/apache/flink/pull/15840
Without these fixes the new KafkaSource in 1.12 is near unusable. The most obvious problem I ran into during testing was that checkpoints fail when consumption has not started for a split (easily reproduced with a topic partition that does not have data). Thanks, Thomas On Tue, Apr 13, 2021 at 11:33 AM Stephan Ewen <se...@apache.org> wrote: > Hi all! > > Generally, avoiding API changes in Bug fix versions is the right thing, in > my opinion. > > But this case is a bit special, because we are changing something that > never worked properly in the first place. > So we are not breaking a "running thing" here, but making it usable. > > So +1 from my side to backport these changes, I think we make more users > happy than angry with this. > > Best, > Stephan > > > On Thu, Apr 8, 2021 at 11:35 AM Becket Qin <becket....@gmail.com> wrote: > > > Hi Arvid, > > > > There are interface changes to the Kafka source, and there is a backwards > > compatible change in the base source implementation. Therefore > technically > > speaking, users might be able to run the Kafka source in 1.13 with a 1.12 > > Flink job. However, it could be tricky because there might be some > > dependent jar conflicts between 1.12 and 1.13. So this solution seems a > > little fragile. > > > > I'd second Till's question if there is an issue for users that start with > > > the current Kafka source (+bugfixes) to later upgrade to 1.13 Kafka > > source > > > with API changes. > > > > > > Just to clarify, the bug fixes themselves include API changes, they are > not > > separable. So we basically have three options here: > > > > 1. Do not backport fixes in 1.12. So users have to upgrade to 1.13 in > order > > to use the new Kafka source. > > 2. Write some completely different fixes for release 1.12 and ask users > to > > migrate to the new API when they upgrade to 1.13 > > 3. Backport the fix with API changes to 1.12. So users don't need to > handle > > interface change when they upgrade to 1.13+. > > > > Personally I think option 3 here is better because it does not really > > introduce any trouble to the users. The downside is that we do need to > > change the API of Kafka source in 1.12. Given that the changed API won't > be > > useful without these bug fixes, changing the API seems to be doing more > > good than bad here. > > > > Thanks, > > > > Jiangjie (Becket) Qin > > > > > > > > On Thu, Apr 8, 2021 at 2:39 PM Arvid Heise <ar...@apache.org> wrote: > > > > > Hi Becket, > > > > > > did you need to change anything to the source interface itself? > Wouldn't > > it > > > be possible for users to simply use the 1.13 connector with their Flink > > > 1.12 deployment? > > > > > > I think the late-upgrade argument can be made for any feature, but I > also > > > see that the Kafka connector is of high interest. > > > > > > I'd second Till's question if there is an issue for users that start > with > > > the current Kafka source (+bugfixes) to later upgrade to 1.13 Kafka > > source > > > with API changes. > > > > > > Best, > > > > > > Arvid > > > > > > On Thu, Apr 8, 2021 at 2:54 AM Becket Qin <becket....@gmail.com> > wrote: > > > > > > > Thanks for the comment, Till and Thomas. > > > > > > > > As far as I know there are some users who have just upgraded their > > Flink > > > > version from 1.8 / 1.9 to Flink 1.12 and might not upgrade the > version > > > in 6 > > > > months or more. There are also some organizations that have the > > strategy > > > of > > > > not running the latest version of a project, but only the second > latest > > > > version with bug fixes. So those users may still benefit from the > > > backport. > > > > However, arguably the old Kafka source is there anyways in 1.12, so > > they > > > > should not be blocked on having the new source. > > > > > > > > I am leaning towards backporting the fixes mainly because this way we > > > might > > > > have more users migrating to the new Source and provide feedback. It > > will > > > > take some time for the users to pick up 1.13, especially for the > users > > > > running Flink at large scale. So backporting the fixes to 1.12 would > > help > > > > get the new source to be used sooner. > > > > > > > > Thanks, > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > On Thu, Apr 8, 2021 at 12:40 AM Thomas Weise <t...@apache.org> wrote: > > > > > > > > > Hi, > > > > > > > > > > Thanks for fixing the new KafkaSource issues. > > > > > > > > > > I'm interested in using these fixes with 1.12 for experimental > > > purposes. > > > > > > > > > > +1 for backporting. 1.12 is the current stable release and users > who > > > > would > > > > > like to try the FLIP-27 sources are likely to use that release. > > > > > > > > > > Thomas > > > > > > > > > > On Wed, Apr 7, 2021 at 2:50 AM Till Rohrmann <trohrm...@apache.org > > > > > > wrote: > > > > > > > > > > > Hi Becket, > > > > > > > > > > > > If I remember correctly, then we deliberately not documented the > > > Kafka > > > > > > connector in the 1.12 release. Hence, from this point there > should > > be > > > > no > > > > > > need to backport any fixes because users are not aware of this > > > feature. > > > > > > > > > > > > On the other hand this also means that we should be able to break > > > > > anything > > > > > > we want to. Consequently, backporting these fixes should be > > possible. > > > > > > > > > > > > The question would probably be whether we want to ship new > features > > > > with > > > > > a > > > > > > bug fix release. Do we know of any users who want to use the new > > > Kafka > > > > > > source, are using the 1.12 version and cannot upgrade to 1.13 > once > > it > > > > is > > > > > > released? If this is the case, then this could be an argument for > > > > > shipping > > > > > > this feature with a bug fix release. If not, then we could save > > some > > > > work > > > > > > by not backporting it. > > > > > > > > > > > > Cheers, > > > > > > Till > > > > > > > > > > > > On Wed, Apr 7, 2021 at 10:43 AM Becket Qin <becket....@gmail.com > > > > > > wrote: > > > > > > > > > > > > > Hi folks, > > > > > > > > > > > > > > I'd like to start a discussion thread about backporting some > > > FLIP-27 > > > > > > Kafka > > > > > > > source connector fixes to release-1.12. These fixes include > some > > > API > > > > > > > changes and thus needs a public discussion. > > > > > > > > > > > > > > The tickets in question are following: > > > > > > > https://issues.apache.org/jira/browse/FLINK-20379 > > > > > > > https://issues.apache.org/jira/browse/FLINK-20114 > > > > > > > https://issues.apache.org/jira/browse/FLINK-21817 > > > > > > > > > > > > > > Without these fixes, the FLIP-27 Kafka source in release-1.12 > is > > > not > > > > > > really > > > > > > > usable, and the API changes only affect the Kafka Source. So it > > > seems > > > > > > > breaking the API in this case is still worthwhile. > > > > > > > > > > > > > > It would be good to see what others think. > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > > > > > > > > >
