Hi Ed,
In the latest version of spark(>3.5), for both hash and range
distribution mode we can control the size of partition by spark property
"spark.sql.adaptive.advisoryPartitionSizeInBytes". This will control the
small files problem.
Regards,
Namratha
On Mon, Apr 7, 2025 at 8:44 AM Ed Mancebo
AQE in recent Spark versions should take care of any skew during writes.
Make sure it is enabled and configured correctly.
- Anton
пн, 14 квіт. 2025 р. о 13:50 namratha mk пише:
> Hi Ed,
>
> In the latest version of spark(>3.5), for both hash and range
> distribution mode we can control the siz
