Hello. We have a very simple AWS Glue job running with Spark: get some events from Kafka stream, do minor transformations, and write to S3.
Recently, there was a change in Kafka topic which suddenly increased our data size * 10 and at the same time we were testing with different repartition values during df.repartition(n).write ... At the time when Kafka started sending an increased volume of data, we didn't actually have the repartition value set in our write. Suddenly, our Glue job (or save at NativeMethodAccessorImpl.java:0) jumped from 2h to 10h. Here are some details of the save stage from SparkUI: - Only 5 executors, which can run 16 tasks parallel each - 10500 tasks (job is still running...) with medians for duration=2,6min and GC time= 2s - Input size per executor is 9GB and output is 4,5GB - executor memory is 20GB My question is now that we're trying to find a proper value for repartition, what would be the optimal value here? Our data volume was not expected to go this high, but there are times when it might be. As this job is running in AWS Glue, should I also consider setting the executor amount, cores, and memory manually? I think Glue is actually setting those based on the Glue job configuration. Yes, this is not probably your concern but still, worth a shot :) Thank you!
