Hi, Can you use console sink and make sure that the pipeline shows some progress?
Pozdrawiam, Jacek Laskowski ---- https://about.me/JacekLaskowski "The Internals Of" Online Books <https://books.japila.pl/> Follow me on https://twitter.com/jaceklaskowski <https://twitter.com/jaceklaskowski> On Wed, Jan 20, 2021 at 10:44 AM rajat kumar <kumar.rajat20...@gmail.com> wrote: > Hi, > > I want to apply custom logic for each row of data I am getting through > kafka and want to do it with microbatch. > When I am running it , it is not progressing. > > > kafka_stream_df \ > .writeStream \ > .foreach(process_records) \ > .outputMode("append") \ > .option("checkpointLocation", "checkpt") \ > .trigger(continuous="5 seconds").start() > > Regards > > Rajat > > >
