Hello Riza Suminto, Jason Fehr, k.venureddy2...@gmail.com, Sai Hemanth
Gantasala, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22653
to look at the new patch set (#8).
Change subject: IMPALA-13884: Add more details in metadata loading logs
......................................................................
IMPALA-13884: Add more details in metadata loading logs
Adds the target of the event in HMS event processing logs. For
AlterPartitionEvent, the partition name will also be printed. Note that
Add/DropPartitionEvent could have multiple partitions so ignore them for
now. E.g.
EventId: 5271 EventType: ALTER_PARTITION Target: db1.tbl year=2009/month=1
EventId: 28781 EventType: ALTER_TABLE Target: tpcds.store_sales
In the reason of loading table metadata, adds the event id if the
request comes from the EventProcessor. E.g.
Reloading metadata for table definition and all partition(s) of db1.tbl (INSERT
event 15765)
Reloading partition metadata: db1.tbl part=1 (INSERT event 15685)
Reloading partition metadata: cachedb.cached_tbl_part j=4,j=1,j=2... and 2
others (batch of 5 ALTER_PARTITIONS events from 38063 to 38067)
Renamed event type strings "ALLOC_WRITE_ID_EVENT" to "ALLOC_WRITE_ID"
and "COMMIT_COMPACTION_EVENT" to "COMMIT_COMPACTION". So they are
consisent with other event types.
For logs of execDdl requests on AlterTable operations, add the
AlterType. E.g.
execDdl request: ALTER_TABLE db1.tbl UPDATE_STATS issued by user1
execDdl request: ALTER_TABLE db1.tbl RECOVER_PARTITIONS issued by user1
For the warning of waiting for table write lock too long in
CatalogServiceCatalog.tryLock(), also logs the stacktrace. E.g.
Write lock for table db1.tbl was acquired in 3329 msec. Caller stacktrace:
org.apache.impala.catalog.CatalogServiceCatalog.tryLock(CatalogServiceCatalog.java:635)
at
org.apache.impala.catalog.CatalogServiceCatalog.tryWriteLock(CatalogServiceCatalog.java:537)
at
org.apache.impala.catalog.CatalogServiceCatalog.evaluateSelfEvent(CatalogServiceCatalog.java:1250)
at
org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.isSelfEvent(MetastoreEvents.java:928)
at
org.apache.impala.catalog.events.MetastoreEvents$MetastoreTableEvent.isSelfEvent(MetastoreEvents.java:1259)
at
org.apache.impala.catalog.events.MetastoreEvents$BatchPartitionEvent.processTableEvent(MetastoreEvents.java:2819)
at
org.apache.impala.catalog.events.MetastoreEvents$MetastoreTableEvent.process(MetastoreEvents.java:1348)
at
org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.processIfEnabled(MetastoreEvents.java:703)
at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:1354)
at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:1097)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
For metadata loading that got ignored due to the table being updated,
e.g. invalidated concurrently, log the reason. E.g.
Not updating table db1.tbl since it has been modified. Current catalog version:
30229. Expected catalog version: 25877
Adds a log when start recovering partitions. E.g.
Recovering 9 partitions for db1.tbl
Also adds an overload of AcidUtils.isTransactionalTable() for FeTable to
simplify some codes.
Fix a wrong errorMessageTemplate of Preconditions.checkArgument() in
FeCatalogUtils.parsePartitionKeyValues(). Only "%s" can be used as
placeholders. Arguments will be converted to strings using
String.valueOf(Object) so don't need "%d".
Tests:
- Passed exhaustive tests
Change-Id: I204d5922e055fd8501b5573e3b913f8874d891d6
---
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/ConvertTableToIcebergStmt.java
M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java
M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/AcidUtils.java
M fe/src/main/java/org/apache/impala/util/CatalogOpUtil.java
M fe/src/test/java/org/apache/impala/util/CatalogOpUtilTest.java
14 files changed, 135 insertions(+), 99 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/22653/8
--
To view, visit http://gerrit.cloudera.org:8080/22653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I204d5922e055fd8501b5573e3b913f8874d891d6
Gerrit-Change-Number: 22653
Gerrit-PatchSet: 8
Gerrit-Owner: Quanlong Huang <huangquanl...@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k.venureddy2...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com>
Gerrit-Reviewer: Riza Suminto <riza.sumi...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <saihema...@cloudera.com>
