Wouldn't it be better to ask the Iceberg maintainers to support dynamic schema change?
On Fri, Dec 17, 2021 at 3:03 AM Dong Lin <lindon...@gmail.com> wrote: > Hi Ayush, > > Your use-case should be supported. Sorry, we don't have a good way to > support this in Flink 1.14. > > I am going to propose a FLIP to fix it in Flink 1.15. > > Thanks, > Dong > > > On Thu, Dec 9, 2021 at 7:11 PM Ayush Chauhan <ayush.chau...@zomato.com> > wrote: > >> My usecase is that as soon as the avro message version is changed, I want >> to reload the job graph so that I can update the downstream iceberg table. >> >> Iceberg FlinkSink take table schema during the job start and cannot be >> updated during runtime. So, I want to trigger graceful shutdown and restart >> the job. >> >> Can I reload the job graph to achieve that? >> >> >> >> On Wed, Dec 8, 2021 at 8:11 PM Arvid Heise <ar...@apache.org> wrote: >> >>> Hi Ayush, >>> >>> DeserializationSchema.isEndOfStream was only ever supported by Kafka. >>> For new Kafka source, the recommended way is to use the bounded mode like >>> this >>> >>> KafkaSource<PartitionAndValue> source = >>> KafkaSource.<PartitionAndValue>builder() >>> ... >>> .setStartingOffsets(OffsetsInitializer.earliest()) >>> .setBounded(OffsetsInitializer.latest()) >>> .build(); >>> >>> You can implement your own OffsetsInitializer or use a provided one. >>> >>> On Wed, Dec 8, 2021 at 9:19 AM Hang Ruan <ruanhang1...@gmail.com> wrote: >>> >>>> There is no way to end the kafka stream from the deserializer. >>>> >>>> When would you want to end the stream? Could you explain why you need >>>> to end the kafka stream without using the offset? >>>> >>>> Ayush Chauhan <ayush.chau...@zomato.com> 于2021年12月8日周三 15:29写道： >>>> >>>>> >>>>> https://github.com/apache/flink/blob/release-1.13.1/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/reader/deserializer/KafkaRecordDeserializationSchema.java#L69 >>>>> >>>>> >>>>> >>>>> On Wed, Dec 8, 2021 at 12:40 PM Robert Metzger <metrob...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi Ayush, >>>>>> >>>>>> I couldn't find the documentation you've mentioned. Can you send me a >>>>>> link to it? >>>>>> >>>>>> On Tue, Dec 7, 2021 at 10:40 AM Ayush Chauhan < >>>>>> ayush.chau...@zomato.com> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Can you please let me know the alternatives of isEndOfStream() as >>>>>>> now according to docs this method will no longer be used to determine >>>>>>> the >>>>>>> end of the stream. >>>>>>> >>>>>>> -- >>>>>>> Ayush Chauhan >>>>>>> Data Platform >>>>>>> [image: mobile-icon] +91 9990747111 >>>>>>> >>>>>>> >>>>>>> This email is intended only for the person or the entity to whom it >>>>>>> is addressed. If you are not the intended recipient, please delete this >>>>>>> email and contact the sender. >>>>>>> >>>>>> >>>>> >>>>> -- >>>>> Ayush Chauhan >>>>> Data Platform >>>>> [image: mobile-icon] +91 9990747111 >>>>> >>>>> >>>>> This email is intended only for the person or the entity to whom it is >>>>> addressed. If you are not the intended recipient, please delete this email >>>>> and contact the sender. >>>>> >>>> >> >> -- >> Ayush Chauhan >> Data Platform >> [image: mobile-icon] +91 9990747111 >> >> >> This email is intended only for the person or the entity to whom it is >> addressed. If you are not the intended recipient, please delete this email >> and contact the sender. >> >
