Reading the Spark Streaming Programming Guide I found a couple of interesting points. First of all, while talking about receivers, it says:
*"If the number of cores allocated to the application is less than or equal to the number of input DStreams / receivers, then the system will receive data, but not be able to process them."* Then, when talking about fault tolerance it says: *“However, if the worker node where a network receiver was running fails, then a tiny bit of data may be lost, that is, the data received by the system but not yet replicated to other node(s). The receiver will be started on a different node and it will continue to receive data.”* So I asked myself: what happens if I have 2 workers with 2 cores each and one worker dies? From what I've reported above my answer would be: the receiver of the dead worker will be moved to the other worker, so there will be 2 receivers on the same worker. But that worker has 2 cores, so it won't be able to process batches anymore. *Is this possible? * Well, I actually tried: I had 2 workers receiving from Kafka and processing RDDs properly. I killed one of the workers and observed the behaviour in the Spark web UI (port 4040). In the Streaming tab there still are 2 active receivers, both allocated to the only living worker. But the "Processed batches" number is stuck, as the evidence that no batches have been processed after the worker died. Also, given that the receivers are still active, they are updating Kafka offsets in Zookeeper, meaning that now those messages are lost, unless you replay them resetting the offsets properly (but where to start from?). Right, this was my test. I still hope I'm wrong, but does this mean that your number of workers needs to be decided at the beginning (based on the number of cores available) without a choice to scale the cluster if needed? I mean, I could use 2 workers with 3 cores each, but what if I want to add a new worker after a while? Looking forward to hear your feedback, I suppose this is a pretty important topic to get right. Thanks a lot, Max -- ------------------------------------------------ Massimiliano Tomassi ------------------------------------------------ web: http://about.me/maxtomassi e-mail: max.toma...@gmail.com ------------------------------------------------
