[
https://issues.apache.org/jira/browse/IMPALA-12621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18062279#comment-18062279
]
ASF subversion and git services commented on IMPALA-12621:
----------------------------------------------------------
Commit 6a70307092b51447ce13ae6546e6f2a978066997 in impala's branch
refs/heads/master from Zoltan Borok-Nagy
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=6a7030709 ]
IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM threads
JVM threads are being reused across JNI invocations. This means
if the interrupt flag is being set for a thread when the JNI call
returns, the next JNI call might get the interrupt (if it checks for
it).
It can be reproduced manually by adding
Thread.currentThread().interrupt() calls at the end of our methods
(e.g. JniFrontend.convertTable(), JniCatalog.updateCatalog()).
Doing so we can trigger errors we can see in IMPALA-12621, IMPALA-10633,
IMPALA-10924, IMPALA-10540, IMPALA-12261. All these issues have
a stack trace similar to the following:
W20260221 07:53:31.855443 1324125 DataStreamer.java:832] DataStreamer Exception
Java exception follows:
java.nio.channels.ClosedByInterruptException
at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:477)
at
org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:63)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:141)
at
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159)
at
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117)
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:781)
I20260221 07:53:31.863165 1312138 jni-util.cc:321]
fa4015e85de1494e:e1586d1500000000]
org.apache.iceberg.exceptions.RuntimeIOException: Failed to write json to file:
hdfs://localhost:20500/test-warehouse/lineitem_sixblocks_iceberg/metadata/00000-93a5f622-d41e-4ccb-87a4-eb26d9bd7a5c.metadata.json
at
org.apache.iceberg.TableMetadataParser.internalWrite(TableMetadataParser.java:133)
at
org.apache.iceberg.TableMetadataParser.overwrite(TableMetadataParser.java:115)
at
org.apache.iceberg.BaseMetastoreTableOperations.writeNewMetadata(BaseMetastoreTableOperations.java:170)
at
org.apache.iceberg.BaseMetastoreTableOperations.writeNewMetadataIfRequired(BaseMetastoreTableOperations.java:160)
at
org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:173)
at
org.apache.iceberg.BaseMetastoreTableOperations.commit(BaseMetastoreTableOperations.java:135)
at
org.apache.iceberg.BaseMetastoreCatalog$BaseMetastoreCatalogTableBuilder.create(BaseMetastoreCatalog.java:201)
at org.apache.iceberg.catalog.Catalog.createTable(Catalog.java:75)
at
org.apache.impala.catalog.iceberg.IcebergHiveCatalog.createTable(IcebergHiveCatalog.java:74)
at
org.apache.impala.util.MigrateTableUtil.migrateToIcebergTable(MigrateTableUtil.java:99)
at org.apache.impala.service.Frontend.convertTable(Frontend.java:1004)
at
org.apache.impala.service.JniFrontend.convertTable(JniFrontend.java:243)
I.e., something interrupted the thread when it tried to create a new
JSON file for an Iceberg table. Looking at the HDFS logs, the file
was created successfully:
org.apache.hadoop.hdfs.StateChange: DIR* completeFile: ...metadata.json
There is also no thread interruption logic in the code path of
JniFrontend.convertTable() that could explain the interruption.
So probably the best explanation to the above is:
* JNI thread gets interrupted in a way its interrupt flag remains set
* New JNI call on thread runs Iceberg code that checks interruption
* Above exception being thrown
Testing:
* tested manually by adding Thread.currentThread().interrupt() calls
at the end of JniCatalog/JniFronted methods.
Generated-by: Gemini Pro
Generated-by: Claude Sonnet 4.5
Change-Id: Iaec6860433431064737e994999dd57a63f223a20
Reviewed-on: http://gerrit.cloudera.org:8080/24029
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Flaky tests due to "Failed to write json to file" in creating Iceberg tables
> ----------------------------------------------------------------------------
>
> Key: IMPALA-12621
> URL: https://issues.apache.org/jira/browse/IMPALA-12621
> Project: IMPALA
> Issue Type: Bug
> Reporter: Quanlong Huang
> Assignee: Zoltán Borók-Nagy
> Priority: Critical
>
> There are some flaky tests that could fail in creating Iceberg tables. E.g.
> saw TestIcebergTable.test_create_table_like_table failed in
> [https://jenkins.impala.io/job/ubuntu-20.04-dockerised-tests/930/]
> {code:java}
> query_test/test_iceberg.py:937: in test_create_table_like_table
> use_db=unique_database)
> common/impala_test_suite.py:720: in run_test_case
> result = exec_fn(query, user=test_section.get('USER', '').strip() or None)
> common/impala_test_suite.py:658: in __exec_in_impala
> result = self.__execute_query(target_impalad_client, query, user=user)
> common/impala_test_suite.py:994: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:214: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:191: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:369: in __execute_query
> self.wait_for_finished(handle)
> beeswax/impala_beeswax.py:390: in wait_for_finished
> raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> E Query aborted:ImpalaRuntimeException: Error making 'createTable' RPC to
> Hive Metastore:
> E CAUSED BY: RuntimeIOException: Failed to write json to file:
> hdfs://172.18.0.1:20500/test-warehouse/test_create_table_like_table_260cb6ec.db/ice_hadoop_tbl_no_part/metadata/3db9a772-baf3-4cd8-9e6b-273ed2839ebb.metadata.json
> E CAUSED BY: IOException: The stream is closed{code}
> The CreateTable command is
> {noformat}
> create table ice_hadoop_tbl_no_part(
> id INT comment "this is id",
> col_bool BOOLEAN,
> col_int INT,
> col_bigint BIGINT,
> col_float FLOAT,
> col_double DOUBLE,
> col_dc1 DECIMAL(9, 3),
> col_dc2 DECIMAL(18, 3),
> col_dc3 DECIMAL(38, 3),
> col_date DATE,
> col_timestamp TIMESTAMP,
> col_string STRING,
> col_struct STRUCT < f1: BIGINT,
> f2: BIGINT >,
> col_array ARRAY < STRING >,
> col_map MAP < STRING,
> array < STRING > >
> ) stored as iceberg tblproperties('iceberg.catalog' = 'hadoop.tables');
> {noformat}
> There are several JIRAs tracking the same failure, e.g. IMPALA-10540,
> IMPALA-12261, IMPALA-10924, IMPALA-10633.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
