hidataplus created HIVE-30023:
---------------------------------
Summary: NPE Cannot invoke "String.trim()" because "identifier" is
null in ObjectStore.getMTable when running ANALYZE TABLE ... COMPUTE STATISTICS
FOR COLUMNS on an Iceberg table with a NULL partition value
Key: HIVE-30023
URL: https://issues.apache.org/jira/browse/HIVE-30023
Project: Hive
Issue Type: Bug
Environment: * Apache Hive *4.2.0* ({{{}rel/release-4.2.0{}}}); also
expected on master
* Bundled Iceberg *1.9.1* ({{{}iceberg/pom.xml{}}})
* Table: *Iceberg* table, partitioned by a *bigint* partition column
* The partition column contains a *NULL* value (Iceberg treats NULL as a
first-class partition value)
Reporter: hidataplus
Running {{ANALYZE TABLE <iceberg_table> COMPUTE STATISTICS FOR COLUMNS}} on an
Iceberg table whose *partition column contains a NULL value* fails with:
java.lang.NullPointerException: Cannot invoke "String.trim()" because
"identifier" is null
The NPE is thrown *server-side* in the metastore
({{{}ObjectStore.getMTable{}}}) while processing the {{alter_partitions}} RPC
that the column-stats path issues to write back partition parameters. It is
caused by {{ObjectStore.alterPartitionNoTxn}} trusting the incoming Thrift
{{Partition}} object's own {{dbName}} (which is null for the Iceberg null-value
partition) instead of the request-level database name.
h2. Reproduction steps
# Create an Iceberg table partitioned by a bigint column:
CREATE TABLE db.t (id INT) PARTITIONED BY (pt BIGINT) STORED BY ICEBERG;
# Insert a row with a NULL partition value:
INSERT INTO db.t VALUES (1, NULL);
# Run column-statistics analysis:
ANALYZE TABLE db.t COMPUTE STATISTICS FOR COLUMNS;
*Expected:* the statement succeeds (or the null partition is handled
gracefully).
*Actual:* {{{}NullPointerException: Cannot invoke "String.trim()" because
"identifier" is null{}}}.
Notes:
* Inserting the NULL value itself succeeds; only {{ANALYZE ... FOR COLUMNS}}
fails.
* {{ANALYZE TABLE ... COMPUTE STATISTICS}} (basic stats, no {{{}FOR
COLUMNS{}}}) does not hit this path.
* Tables without a NULL partition value are unaffected.
h2. Stack traces
Client side (the NPE is deserialized from the Thrift response):
org.apache.hadoop.hive.ql.metadata.Hive.alterPartitions(Hive.java:1218)
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_partitions_req_result$alter_partitions_req_resultStandardScheme.read(ThriftHiveMetastore.java)
Server side (metastore log), the actual throw site:
org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:2029)
h2. Root cause analysis (verified against rel/release-4.2.0)
# {{ANALYZE ... FOR COLUMNS}} computes column stats, then writes back
partition parameters via {{Hive.alterPartitions}}
({{{}ql/.../metadata/Hive.java{}}}, ~L1211-1218), issuing the
{{alter_partitions}} Thrift RPC.
# Server: {{HMSHandler.alter_partitions}} →
{{HiveAlterHandler.alterPartitions}} →
{{{}ObjectStore.alterPartitionNoTxn(catName, dbname, name, part_vals, newPart,
...){}}}.
# {{ObjectStore.alterPartitionNoTxn}} resolves the table using the {*}incoming
Partition object's own names{*}, not the request-level parameters
({{{}ObjectStore.java:4916{}}}):
MTable table = this.getMTable(newPart.getCatName(), newPart.getDbName(),
newPart.getTableName());
# {{ObjectStore.getMTable(catName, db, table, retrieveCD)}} normalizes {{db}}
*without a null guard* ({{{}ObjectStore.java:2028-2030{}}}):
catName =
normalizeIdentifier(Optional.ofNullable(catName).orElse(getDefaultCatalog(conf)));
// guarded
db = normalizeIdentifier(db); // <-- L2029: NPE when db == null
table = normalizeIdentifier(table);where {{normalizeIdentifier}} is
{{identifier.trim().toLowerCase()}}
({{{}metastore-common/.../StringUtils.java:93-95{}}}).
# For an Iceberg table, the partition whose partition-column value is NULL is
represented client-side by a Thrift {{Partition}} whose {{dbName}} is *not
populated* (the Iceberg storage handler synthesizes/maps this partition without
filling db/table names). When it reaches {{{}alterPartitionNoTxn{}}},
{{newPart.getDbName()}} is null → {{getMTable(..., null, ...)}} →
{{normalizeIdentifier(null)}} → NPE.
# Normal (non-null) partitions carry populated
{{{}dbName{}}}/{{{}tableName{}}}, so only the null-value partition triggers the
failure. The NPE is serialized into the Thrift response and re-thrown on the
client (hence the client frames above).
h2. Suggested fix
Preferred (semantically correct, server-side): in
{{{}ObjectStore.alterPartitionNoTxn{}}}, fall back to the request-level names
when the incoming {{{}Partition{}}}'s are unset:
// ObjectStore.java:4916, before
MTable table = this.getMTable(newPart.getCatName(), newPart.getDbName(),
newPart.getTableName());
// after
MTable table = this.getMTable(
newPart.isSetCatName() ? newPart.getCatName() : catName,
newPart.getDbName() != null ? newPart.getDbName() : dbname,
newPart.getTableName()!= null ? newPart.getTableName(): name);
Alternative / complementary:
* Add a defensive null guard for {{{}db{}}}/{{{}table{}}} in {{getMTable}}
mirroring the existing {{catName}} pattern (or fail fast with a clear
{{InvalidObjectException}} instead of an NPE).
* Ensure the Iceberg storage handler populates
{{{}dbName{}}}/{{{}tableName{}}}/{{{}catName{}}} on every {{Partition}} it
synthesizes, so clients never send partially-populated partition objects.
h2. Workaround (until fixed)
* Avoid NULL values in the partition column (use a sentinel such as
{{{}-1{}}}/{{{}0{}}}), or rewrite/drop the null partition.
* Analyze only non-null partitions: {{ANALYZE TABLE db.t
PARTITION(pt=<non-null>) COMPUTE STATISTICS FOR COLUMNS;}}
* Or use basic stats only: {{ANALYZE TABLE db.t COMPUTE STATISTICS;}}
h2. Additional context
* The exact message {{Cannot invoke "String.trim()" because "identifier" is
null}} has no prior report in apache/hive (searched), so this appears to be a
new issue.
* Related but distinct: this is independent of the Iceberg multi-spec
partition-evolution {{Cannot project non-struct}} family of bugs; it is purely
a metastore null-name handling defect triggered by Iceberg null partition
values.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
