Hi all, Been facing a weird issue lately. In our production code base , we have an explicit broadcast for a small table. It is just a look up table that is around 1gb in size in s3 and just had few million records and 5 columns.
The ETL was running fine , but with no change from the codebase nor the infrastructure, we are getting broadcast failures. Even weird fact is the older size of the data is 1.4gb while for the new run is just 900 MB Below is the error message Cannot broadcast table that is larger than 8 GB : 8GB. I find it extremely weird considering that the data size is very well under the thresholds. Are there any other ways to find what could be the issue and how we can rectify this issue? Could the data characteristics be an issue? Any help would be immensely appreciated. Thanks
