At the outset, this isn't about challenging the work that has been done, primarily by folks @ Confluence for wrapping kafka in a REST API. Clearly, there is a use case for a REST Service and they rose up to the challenge.
That said, I am trying to evaluate the limitations of a REST service around kafka, if any, specifically for a Spark Streaming app as a consumer? The reason I ask because I have been asked to move my Spark Streaming app, that's a native kafka consumer, to a REST service that provides security (auth/encryption). And, I am trying to figure out what requirements should I place on the REST service if I am give up my direct access to the kafka cluster? With Spark Streaming, what I have learned is that you need to maintain a certain ratio between Spark executors and kafka partitions to balance the processing load evenly on the app. The consumer group re-balancing in kafka works really well to transition a load off a dead Spark executor to an idle one. Not sure, how these features and other features that I use almost unknowingly in kafka translate to a REST service like the kafka-rest provides? More broadly, are there features in kafka that inherently run into limitations of the stateless/REST model? Thanks, Tim
