Hi Team,
We are using spark java 3.5.3 and we have a requirement to run a batch of millions of transactions. i.e. I am running batch of 1M with 640 executors and each executor is having 8 cores and 16 GB memory running under kubernetes cluster. But our conversation is that one executor is done with completed task, it is not going away. All executors are released at the end of job with driver pod. And also, any performance suggestion which can help in reducing the timing. Currently we have 10M running with 640 executors and taking 60 plus mins to be finished. Let me know if need more information. Thanks, Shivang. -- *This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely for the use of the addressee(s). If you are not the intended recipient, please notify the sender by e-mail and delete the original message. Further, you are not to copy, disclose, or distribute this e-mail or its contents to any other person and any such actions maybe unlawful*. This e-mail may contain viruses. Provenir has taken every reasonable precaution to minimize this risk, but is not liable for any damage you may sustain as a result of any virus in this e-mail. You should carry out your own virus checks before opening the e-mail or attachment. Provenir reserves the right to monitor and review the content of all messages sent to or from this e-mail address. Messages sent to or from this e-mail address may be stored on the Provenir e-mail system. Think before printing - please consider the environment
