Hallo on 2020/1/8 11:31, RKandoji wrote:
I'm running my job on a EC2 instance with 32 cores and according to the documentation I tried to use as many task slots the number of cores, numOfTaskSlots=32 and parallelism=32. But I noticed that the performance is slightly degrading when I'm using 32 task slots. Performance seems better at 26 task slots than >26 task slots. So I was trying to understand if additional CPU cores are being utilized by checkpointing or any other async (or background operations, in the process I was trying to verify if the checkpointing is async.
How did you evaluate the performance? It's may due to busy IO or thread competition or something else similiar. You'd better dig into more details via logging. Regards.
