sandugood commented on PR #4982:
URL:
https://github.com/apache/datafusion-comet/pull/4982#issuecomment-5026073905
The error still persists. Here are the logs:
```
26/07/20 21:54:08 INFO CometIcebergNativeScan: Common pools
(unique/bytes): schema=1/516, partition_type=0/0, partition_spec=1/25,
name_mapping=0/0, project_field_ids=1/4, partition_data=0/0, delete_file=0/0,
delete_files_set=0/0, residual=1/108; total=653 bytes
26/07/20 21:54:08 INFO CometIcebergNativeScan: Per-partition messages: 1,
total=167 bytes, max=167 bytes
26/07/20 21:54:23 INFO CometIcebergNativeScan: IcebergScan: 1 tasks, 9 pools
(0.0% avg dedup)
26/07/20 21:54:23 INFO CometIcebergNativeScan: Common pools
(unique/bytes): schema=1/488, partition_type=0/0, partition_spec=1/25,
name_mapping=0/0, project_field_ids=1/5, partition_data=0/0, delete_file=0/0,
delete_files_set=0/0, residual=1/84; total=602 bytes
26/07/20 21:54:23 INFO CometIcebergNativeScan: Per-partition messages: 1,
total=175 bytes, max=175 bytes
26/07/20 21:58:51 INFO CometIcebergNativeScan: IcebergScan: 13695 tasks, 9
pools (81.6% avg dedup)
26/07/20 21:58:51 INFO CometIcebergNativeScan: Common pools
(unique/bytes): schema=1/1868, partition_type=1/97, partition_spec=1/106,
name_mapping=0/0, project_field_ids=1/9, partition_data=1395/12555,
delete_file=0/0, delete_files_set=0/0, residual=13695/3432661835;
total=3432676470 bytes
26/07/20 21:58:51 INFO CometIcebergNativeScan: Per-partition messages:
10352, total=3002592 bytes, max=1742 bytes
at
org.apache.comet.serde.operator.CometIcebergNativeScan$.serializePartitions(CometIcebergNativeScan.scala:1094)
```
So at one point the number of tasks just explodes and exceeds the 2GB
threshold.
Here is the error stacktrace:
```
Py4JJavaError: An error occurred while calling o12676.createOrReplace.
: java.lang.NegativeArraySizeException: -862232487
at
org.apache.comet.shaded.protobuf.AbstractMessageLite.toByteArray(AbstractMessageLite.java:46)
at
org.apache.comet.serde.operator.CometIcebergNativeScan$.serializePartitions(CometIcebergNativeScan.scala:1094)
at
org.apache.spark.sql.comet.CometIcebergNativeScanExec.serializedPartitionData$lzycompute(CometIcebergNativeScanExec.scala:111)
at
org.apache.spark.sql.comet.CometIcebergNativeScanExec.org$apache$spark$sql$comet$CometIcebergNativeScanExec$$serializedPartitionData(CometIcebergNativeScanExec.scala:86)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
