Hi folks, Thought I would ask here because it's somewhat confusing. I'm using Spark 2.4.5 on EMR 5.30.1 with Amazon MSK.
The version of Scala used is 2.11.12. I'm using this version of the libraries spark-streaming-kafka-0-8_2.11-2.4.5.jar Now I'm wanting to read from Kafka topics using Python (I need to stick to Python specifically). What seems confusing is that 0.8 has Python support, but 0.10 does not. Then 0.8 seems to have been deprecated as of Spark 2.3.0, so if I'm using 2.4.5 then clearly I'm going to hit a roadblock here. Can someone clarify these things for me? Have I got this right? Thanks in advance, Hamish
