Hi, I have released the first version of a new Kafka integration with Spark that we use in the company I work for: open sourced and named Maelstrom.
It is unique compared to other solutions out there as it reuses the Kafka Consumer connection to achieve sub-milliseconds latency. This library has been running stable in production environment and has been proven to be resilient to numerous production issues. Please check out the project's page in github: https://github.com/jeoffreylim/maelstrom Contributors welcome! Cheers! Jeoffrey Lim P.S. I am also looking for a job opportunity, please look me up at Linked In
