Greetings, This is a scenario that we need to come up with a comprehensive answers to fulfil please. If we have 6 spark VMs each running two executors via spark-submit. - we have two VMs failures at H/W level, rack failure - we lose 4 executors of spark out of 12 - Happening half way through the spark-submit job -
So my humble questions are: - Will there be any data lost from the final result due to missing nodes? - How will RDD lineage will handle this? - Will there be any delay in getting the final result? - How the driver will handle these two nodes failure - Will there be additional executors added to the existing nodes or the existing executors will handle the job of 4 failing executors. - If running in client mode and the node holding the driver dies? - If running in cluster mode happens Did search in Google no satisfactory answers gurus, hence turning to forum. Best A.K.
