[
https://issues.apache.org/jira/browse/IMPALA-14349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18019310#comment-18019310
]
ASF subversion and git services commented on IMPALA-14349:
----------------------------------------------------------
Commit 711797e7fbda6f30fc49d91e30ad6ab31a4f4a69 in impala's branch
refs/heads/master from Zoltan Borok-Nagy
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=711797e7f ]
IMPALA-14349: Encode FileDescriptors in time in loading Iceberg Tables
With this patch we create Iceberg file descriptors from
LocatedFileStatus objects during IcebergFileMetadataLoader's
parallelListing(). This has the following benefits:
* We parallelize the creation of Iceberg file descriptor objects
* We don't need to maintain a large hash map with all the
LocatedFileStatus objects at once. Now we only need to keep a few
LocatedFileStatus objects per partition in memory while we are
converting them to Iceberg file descriptors. I.e., the GC is free to
destroy the LocatedFileStatus objects we don't use anymore.
This patch retires startup flag 'iceberg_reload_new_files_threshold'.
Since IMPALA-13254 we only list partitions that have new data files,
and we load them in parallel, i.e. efficient incremental table loading
is already covered. From that point the startup flag only added
unnecessary code complexity.
Measurements
I created two tables (from tpcds.store_sales) to measure table loading
times for large tables:
Table #1:
PARTITIONED BY SPEC(ss_item_sk, BUCKET(5, ss_sold_time_sk))
partitions: 107818
files: 754726
Table #2:
PARTITIONED BY SPEC(ss_item_sk)
partitions: 18000
files: 504224
Time taken in IcebergFileMetadataLoader.load() during full table reload:
+----------+-------+------+---------+
| | Base | New | Speedup |
+----------+-------+------+---------+
| Table #1 | 17.3s | 8.1s | 2.14 |
| Table #2 | 7.8s | 4.3s | 1.8 |
+----------+-------+------+---------+
I measured incremental table loading only for Table #2 (since there are
more files per partition this is the worse scenario for the new code, as
it only uses file listings, and each new file were created in a separate
partition)
Time taken in IcebergFileMetadataLoader.load() during incremental table
reload:
+------------+------+------+---------+
| #new files | Base | New | Speedup |
+------------+------+------+---------+
| 1 | 1.4s | 1.6s | 0.9 |
| 100 | 1.5s | 1.9s | 0.8 |
| 200 | 1.5s | 1.5s | 1 |
+------------+------+------+---------+
We lose a few tenths of a second, but I think the simplified code
justifies it.
Testing:
* some tests were updated because we we don't have
startup flag 'iceberg_reload_new_files_threshold' anymore
Change-Id: Ia1c2a7119d76db7ce7c43caec2ccb122a014851b
Reviewed-on: http://gerrit.cloudera.org:8080/23363
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Encode FileDescriptors in time in loading Iceberg Tables
> --------------------------------------------------------
>
> Key: IMPALA-14349
> URL: https://issues.apache.org/jira/browse/IMPALA-14349
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog
> Reporter: Quanlong Huang
> Assignee: Zoltán Borók-Nagy
> Priority: Major
> Labels: iceberg
>
> When loading file metadata of an IcebergTable in
> IcebergFileMetadataLoader#loadInternal() -> parallelListing(), we maintain a
> map from paths to FileStatus objects:
> [https://github.com/apache/impala/blob/50926b5d8e941c5cc10fd77d0b4556e3441c41e7/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java#L171]
> This map consumes lot of memory space since the loaded FileStatus objects are
> in HdfsLocatedFileStatus type and each of them consumes 6KB of the memory.
> E.g.
> {noformat}
> Class Name
> | Shallow Heap | Retained Heap
> ----------------------------------------------------------------------------------------------------------------------------------------
> org.apache.hadoop.hdfs.protocol.HdfsLocatedFileStatus @ 0x1008511620
> | 120 | 6,192
> |- <class> class org.apache.hadoop.hdfs.protocol.HdfsLocatedFileStatus @
> 0x1009e2a058 | 16 | 40
> |- isdir java.lang.Boolean @ 0x10056a7638 false
> | 16 | 16
> |- path org.apache.hadoop.fs.Path @ 0x1008511310
> | 16 | 784
> |- permission org.apache.hadoop.hdfs.protocol.FsPermissionExtension @
> 0x1008511698 | 32 | 32
> |- owner java.lang.String @ 0x10085116b8 id971832
> | 24 | 48
> |- group java.lang.String @ 0x10085116e8 hive
> | 24 | 48
> |- attr java.util.RegularEnumSet @ 0x1008511718
> | 32 | 32
> |- locations org.apache.hadoop.fs.BlockLocation[1] @ 0x1008511738
> | 24 | 192
> |- uPath byte[62] @ 0x1008511838
> 00668-28396-9dd59fc9-3ed9-40ca-8f39-e68bd2724c14-00040.parquet |
> 80 | 80
> |- hdfsloc org.apache.hadoop.hdfs.protocol.LocatedBlocks @ 0x1008511888
> | 40 | 5,576
> | |- <class> class org.apache.hadoop.hdfs.protocol.LocatedBlocks @
> 0x1009e20278 | 8 | 512
> | |- blocks java.util.ArrayList @ 0x10085118b0
> | 24 | 2,760
> | | |- <class> class java.util.ArrayList @ 0x100573da10 System Class
> | 32 | 240
> | | |- elementData java.lang.Object[1] @ 0x10085118c8
> | 24 | 2,736
> | | | |- class java.lang.Object[] @ 0x1005fc4650
> | 0 | 0
> | | | |- [0] org.apache.hadoop.hdfs.protocol.LocatedBlock @ 0x10085118e0
> | 48 | 2,712
> | | | | |- <class> class org.apache.hadoop.hdfs.protocol.LocatedBlock @
> 0x1009e26700 | 16 | 424
> | | | | |- storageIDs java.lang.String[3] @ 0x10085117f8
> | 32 | 32
> | | | | |- storageTypes org.apache.hadoop.fs.StorageType[3] @
> 0x1008511818 | 32 | 32
> | | | | |- b org.apache.hadoop.hdfs.protocol.ExtendedBlock @ 0x1008511910
> | 24 | 64
> | | | | |- locs
> org.apache.hadoop.hdfs.protocol.DatanodeInfoWithStorage[3] @ 0x1008511950
> | 32 | 2,456
> | | | | | |- class
> org.apache.hadoop.hdfs.protocol.DatanodeInfoWithStorage[] @ 0x102005b000
> | 0 | 0
> | | | | | |- [2] org.apache.hadoop.hdfs.protocol.DatanodeInfoWithStorage
> @ 0x1008511970 | 200 | 808
> | | | | | |- [1] org.apache.hadoop.hdfs.protocol.DatanodeInfoWithStorage
> @ 0x1008511c98 | 200 | 808
> | | | | | |- [0] org.apache.hadoop.hdfs.protocol.DatanodeInfoWithStorage
> @ 0x1008511fc0 | 200 | 808
> | | | | | '- Total: 4 entries
> | |
> | | | | |- blockToken org.apache.hadoop.security.token.Token @
> 0x10085122e8 | 32 | 144
> | | | | |- cachedLocs
> org.apache.hadoop.hdfs.protocol.DatanodeInfoWithStorage[0] @ 0x101b01f328
> | 16 | 16
> | | | | '- Total: 7 entries
> | |
> | | | '- Total: 2 entries
> | |
> | | '- Total: 2 entries
> | |
> | |- lastLocatedBlock org.apache.hadoop.hdfs.protocol.LocatedBlock @
> 0x1008512378 | 48 | 2,776
> | | |- <class> class org.apache.hadoop.hdfs.protocol.LocatedBlock @
> 0x1009e26700 | 16 | 424
> | | |- b org.apache.hadoop.hdfs.protocol.ExtendedBlock @ 0x10085123a8
> | 24 | 64
> | | |- locs org.apache.hadoop.hdfs.protocol.DatanodeInfoWithStorage[3] @
> 0x10085123e8 | 32 | 2,216
> | | | |- class org.apache.hadoop.hdfs.protocol.DatanodeInfoWithStorage[] @
> 0x102005b000 | 0 | 0
> | | | |- [2] org.apache.hadoop.hdfs.protocol.DatanodeInfoWithStorage @
> 0x1008512408 | 200 | 728
> | | | |- [1] org.apache.hadoop.hdfs.protocol.DatanodeInfoWithStorage @
> 0x1008512730 | 200 | 728
> | | | |- [0] org.apache.hadoop.hdfs.protocol.DatanodeInfoWithStorage @
> 0x1008512a58 | 200 | 728
> | | | | |- <class> class
> org.apache.hadoop.hdfs.protocol.DatanodeInfoWithStorage @ 0x102005aee8 |
> 8 | 104
> | | | | |- ipAddr java.lang.String @ 0x1008512b20 xxx.xxx.xxx.xxx
> | 24 | 56
> | | | | |- ipAddrBytes com.google.protobuf.LiteralByteString @
> 0x1008512b58 | 24 | 56
> | | | | |- hostName java.lang.String @ 0x1008512b90 www.abc.com
> | 24 | 56
> | | | | |- hostNameBytes com.google.protobuf.LiteralByteString @
> 0x1008512bc8 | 24 | 56
> | | | | |- xferAddr java.lang.String @ 0x1008512c00 xxx.xxx.xxx.xxx:9866
> | 24 | 64
> | | | | |- datanodeUuid java.lang.String @ 0x1008512c40
> 2f6e6e42-9347-4370-a318-79efdadcc3cf | 24 | 80
> | | | | |- datanodeUuidBytes com.google.protobuf.LiteralByteString @
> 0x1008512c90 | 24 | 80
> | | | | |- location java.lang.String @ 0x1008512ce0 /default
> | 24 | 48
> | | | | |- dependentHostNames java.util.LinkedList @ 0x1008512d10
> | 32 | 32
> | | | | |- storageID java.lang.String @ 0x1008512d30
> DS-f190d2ef-755b-4f73-bb3d-67b6e72805e2 | 24 | 80
> | | | | |- adminState
> org.apache.hadoop.hdfs.protocol.DatanodeInfo$AdminStates @ 0x101b01ef50
> NORMAL| 24 | 24
> | | | | |- storageType org.apache.hadoop.fs.StorageType @ 0x101b01f000
> DISK | 24 | 24
> | | | | '- Total: 13 entries
> | |
> | | | '- Total: 4 entries
> | |
> | | |- storageIDs java.lang.String[3] @ 0x1008512d80
> | 32 | 32
> | | |- storageTypes org.apache.hadoop.fs.StorageType[3] @ 0x1008512da0
> | 32 | 32
> | | |- blockToken org.apache.hadoop.security.token.Token @ 0x1008512dc0
> | 32 | 144
> | | |- cachedLocs
> org.apache.hadoop.hdfs.protocol.DatanodeInfoWithStorage[0] @ 0x101b01f328
> | 16 | 16
> | | '- Total: 7 entries
> | |
> | '- Total: 3 entries
> | |
> '- Total: 10 entries{noformat}
> There are some duplicate strings like storageIDs and hostnames. We can invoke
> String.intern() on them to save some space. But it'd be better to convert
> these FileStatus objects into IcebergFileDescriptor in time to reduce the
> space usage. Encoding IcebergFileDescriptor into bytes (which usually takes
> 200 bytes for each file) in time can further save more space.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
