github-actions[bot] commented on code in PR #70675:
URL: https://github.com/apache/doris/pull/70675#discussion_r4597416579
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/ManifestCacheValue.java:
##########
@@ -43,10 +43,14 @@ public class ManifestCacheValue {
private final List<DataFile> dataFiles;
private final List<DeleteFile> deleteFiles;
private final MetaCacheSizeEstimate sizeEstimate;
+ private final boolean cacheable;
private ManifestCacheValue(List<DataFile> dataFiles, List<DeleteFile>
deleteFiles, boolean estimateWeight) {
this.dataFiles = dataFiles == null ? Collections.emptyList() :
dataFiles;
this.deleteFiles = deleteFiles == null ? Collections.emptyList() :
deleteFiles;
+ // SDK file-key metadata contains the plaintext key, even when the
enclosing manifest is plaintext.
+ this.cacheable = this.dataFiles.stream().noneMatch(file ->
file.keyMetadata() != null)
Review Comment:
[P2] Preserve lazy partition adaptation when checking retention
`forScan()` constructs this value over `Lists.transform(...,
adapter::adapt)` whenever partition types differ, for example after an identity
INT-to-LONG promotion or removal of its source column. These streams now
evaluate every transformed element, allocating a file view and two
`PartitionData` copies per file, then discard those objects; consuming the list
adapts the same files again. A warm streaming LIMIT scan therefore processes
the entire cached manifest before producing its first task, even if it only
needs one file. Carry the original `cacheable` flag into the query-local
projected value instead: partition adaptation delegates key metadata unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]