Hi Ayush, Your use-case should be supported. Sorry, we don't have a good way to support this in Flink 1.14.
I am going to propose a FLIP to fix it in Flink 1.15. Thanks, Dong On Thu, Dec 9, 2021 at 7:11 PM Ayush Chauhan <ayush.chau...@zomato.com> wrote: > My usecase is that as soon as the avro message version is changed, I want > to reload the job graph so that I can update the downstream iceberg table. > > Iceberg FlinkSink take table schema during the job start and cannot be > updated during runtime. So, I want to trigger graceful shutdown and restart > the job. > > Can I reload the job graph to achieve that? > > > > On Wed, Dec 8, 2021 at 8:11 PM Arvid Heise <ar...@apache.org> wrote: > >> Hi Ayush, >> >> DeserializationSchema.isEndOfStream was only ever supported by Kafka. For >> new Kafka source, the recommended way is to use the bounded mode like this >> >> KafkaSource<PartitionAndValue> source = >> KafkaSource.<PartitionAndValue>builder() >> ... >> .setStartingOffsets(OffsetsInitializer.earliest()) >> .setBounded(OffsetsInitializer.latest()) >> .build(); >> >> You can implement your own OffsetsInitializer or use a provided one. >> >> On Wed, Dec 8, 2021 at 9:19 AM Hang Ruan <ruanhang1...@gmail.com> wrote: >> >>> There is no way to end the kafka stream from the deserializer. >>> >>> When would you want to end the stream? Could you explain why you need to >>> end the kafka stream without using the offset? >>> >>> Ayush Chauhan <ayush.chau...@zomato.com> 于2021年12月8日周三 15:29写道： >>> >>>> >>>> https://github.com/apache/flink/blob/release-1.13.1/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/reader/deserializer/KafkaRecordDeserializationSchema.java#L69 >>>> >>>> >>>> >>>> On Wed, Dec 8, 2021 at 12:40 PM Robert Metzger <metrob...@gmail.com> >>>> wrote: >>>> >>>>> Hi Ayush, >>>>> >>>>> I couldn't find the documentation you've mentioned. Can you send me a >>>>> link to it? >>>>> >>>>> On Tue, Dec 7, 2021 at 10:40 AM Ayush Chauhan < >>>>> ayush.chau...@zomato.com> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> Can you please let me know the alternatives of isEndOfStream() as now >>>>>> according to docs this method will no longer be used to determine the end >>>>>> of the stream. >>>>>> >>>>>> -- >>>>>> Ayush Chauhan >>>>>> Data Platform >>>>>> [image: mobile-icon] +91 9990747111 >>>>>> >>>>>> >>>>>> This email is intended only for the person or the entity to whom it >>>>>> is addressed. If you are not the intended recipient, please delete this >>>>>> email and contact the sender. >>>>>> >>>>> >>>> >>>> -- >>>> Ayush Chauhan >>>> Data Platform >>>> [image: mobile-icon] +91 9990747111 >>>> >>>> >>>> This email is intended only for the person or the entity to whom it is >>>> addressed. If you are not the intended recipient, please delete this email >>>> and contact the sender. >>>> >>> > > -- > Ayush Chauhan > Data Platform > [image: mobile-icon] +91 9990747111 > > > This email is intended only for the person or the entity to whom it is > addressed. If you are not the intended recipient, please delete this email > and contact the sender. >
