I am beginner with spark may , also know how to connect MySQL database with spark streaming
Thanks and regards Akash P On Wed, 17 Aug, 2022, 8:28 pm Saurabh Gulati, <saurabh.gul...@fedex.com> wrote: > Another take: > > - Debezium > <https://debezium.io/documentation/reference/stable/connectors/mysql.html> > to read Write Ahead logs(WAL) and send to Kafka > - Kafka connect to write to cloud storage -> Hive > - OR > > > - Spark streaming to parse WAL -> Storage -> Hive > > Regards > ------------------------------ > *From:* Gibson <gwasuk...@gmail.com> > *Sent:* 17 August 2022 16:53 > *To:* Akash Vellukai <akashvellukai...@gmail.com> > *Cc:* user@spark.apache.org <user@spark.apache.org> > *Subject:* [EXTERNAL] Re: Spark streaming - Data Ingestion > > *Caution! This email originated outside of FedEx. Please do not open > attachments or click links from an unknown or suspicious origin*. > If you have space for a message log like, then you should try: > > MySQL -> Kafka (via CDC) -> Spark (Structured Streaming) -> HDFS/S3/ADLS > -> Hive > > On Wed, Aug 17, 2022 at 5:40 PM Akash Vellukai <akashvellukai...@gmail.com> > wrote: > > Dear sir > > I have tried a lot on this could you help me with this? > > Data ingestion from MySql to Hive with spark- streaming? > > Could you give me an overview. > > > Thanks and regards > Akash P > >
