Mainly bugfixes and no breaking AFAIK. As a side note there were intentions to close DStreams and discontinue as-is. It's not yet happened but it's on the road so I strongly recommend to migrate to Structured Streaming... We simply can't support 2 streaming engines for huge amount of time.
G On Fri, Mar 12, 2021 at 3:02 PM Renu Yadav <yren...@gmail.com> wrote: > Hi Gabor, > > It seems like it is better to upgrade my spark version . > > Are there major changes in terms of streaming from spark 2.2 to spark 2.4? > > PS: I am using KafkaUtils api to create steam > > Thanks & Regards, > Renu yadav > > On Fri, Mar 12, 2021 at 7:25 PM Renu Yadav <yren...@gmail.com> wrote: > >> Thanks Gabor, >> This is very useful. >> >> Regards, >> Renu Yadav >> >> On Fri, Mar 12, 2021 at 5:36 PM Gabor Somogyi <gabor.g.somo...@gmail.com> >> wrote: >> >>> Kafka client upgrade is not a trivial change which may or may not work >>> since new versions can contain incompatible API and/or behavior changes. >>> I've collected how Spark evolved in terms of Kafka client and there I've >>> gathered the breaking changes to make our life easier. >>> Have a look and based on that you can make your choice: >>> https://gist.github.com/gaborgsomogyi/3476c32d69ff2087ed5d7d031653c7a9 >>> >>> As a general suggestion it would be best to upgrade Spark as-is because >>> we've added many fixes which one can face... >>> >>> Hope this helps! >>> >>> G >>> >>> >>> On Fri, Mar 12, 2021 at 9:45 AM Renu Yadav <yren...@gmail.com> wrote: >>> >>>> Hi Team, >>>> I am using spark -2.2 and spark_streamin_kafka 2.2 , which is >>>> pointing to kafka-client 0.10 . How can I upgrade a kafka client to kafka >>>> 2.2.0 ? >>>> >>>> Thanks & Regards, >>>> Renu Yadav >>>> >>>
