> On April 20, 2020, 12:19 p.m., Peter Vary wrote: > > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java > > Lines 699 (patched) > > <https://reviews.apache.org/r/72388/diff/1/?file=2223825#file2223825line731> > > > > openTxns(dbConn, stmt, rqst) is used in replTableWriteIdState as well. > > Do we have to check there for timeout as well? > > Peter Varga wrote: > This is an alarming catch. What kind of queries run on a replication. I > put there the table lock, but it will be a rather long synchronized block. On > the other hand aren't the replication tasks somehow serialized? If there are > no select queries on replication the timeout is not needed, if there are, we > might have a problem, because we do a lot of think before the commit, where > we can measure. > And the most seriuos problem, if I see correctly, the replication will > use this newly opened transactionId-s to write into the TXN_TO_WRITE_ID, > wouldn't be a problem if this will be different than the original value?
As we discussed, the txn_id change is not a problem, and we don't need either the timeout check or the txnLock, because all of the opened transactions will be aborted in the same rdbms transaction > On April 20, 2020, 12:19 p.m., Peter Vary wrote: > > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java > > Lines 780 (patched) > > <https://reviews.apache.org/r/72388/diff/1/?file=2223825#file2223825line826> > > > > TXN_META_INFO? What is it used for before? Or is it new? Could we use a > > specific "state" for example? > > Denys Kuzmenko wrote: > My understanding was, it's kinda marker to flag the new txn entries and > get their ids. > > Peter Varga wrote: > It is a marker, I just used an existing field. It is updated to null > later. change it to txn_state > On April 20, 2020, 12:19 p.m., Peter Vary wrote: > > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java > > Line 641 (original), 783 (patched) > > <https://reviews.apache.org/r/72388/diff/1/?file=2223825#file2223825line829> > > > > Could we use a correct PreparedStatement, with the values set without > > the "hacky" quoting? > > > > Like: pstmt.setXX for state/user/host/type/metainfo? > > > > Or does this have a noticable performance gain? > > Peter Varga wrote: > Changed the metainfo select back and update to preparedstatements. The > original insert will entirely change in your batch insert patch if I know > correctly, so I will leave that. incorporated your batchinsert patch also - Peter ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72388/#review220369 ----------------------------------------------------------- On April 21, 2020, 12:51 p.m., Peter Varga wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72388/ > ----------------------------------------------------------- > > (Updated April 21, 2020, 12:51 p.m.) > > > Review request for hive and Peter Vary. > > > Repository: hive-git > > > Description > ------- > > * Use sequences for TXN_ID generation. > * Make it possible to open transactions in parallel > * drop Oracle 11g support > > > Diffs > ----- > > ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java > 37a5862791 > > ql/src/test/org/apache/hadoop/hive/metastore/txn/TestCompactionTxnHandler.java > 15fcfc0e35 > ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java > 1d211857bf > ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java 2c13e8dd03 > ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands3.java 51b0fa336f > ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommandsForMmTable.java > 6525ffc00a > ql/src/test/org/apache/hadoop/hive/ql/TxnCommandsBaseForTests.java > 1435269ed3 > > ql/src/test/org/apache/hadoop/hive/ql/lockmgr/DbTxnManagerEndToEndTestBase.java > PRE-CREATION > ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java > 73d3b91585 > > ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManagerIsolationProperties.java > PRE-CREATION > ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestInitiator.java > 1151466f8c > > standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java > a874121e12 > > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/SQLGenerator.java > 49b737ecf9 > > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java > 2344c2d5f6 > > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnDbUtil.java > bb29410e7d > > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java > e7910c1c5d > > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStore.java > 87130a519d > > standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql > 1ace9d3ef0 > > standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql > 8a3cd56658 > > standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql > 2e0177723d > > standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql > 9f3951575b > > standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql > 0512a45cad > > standalone-metastore/metastore-server/src/main/sql/mysql/upgrade-3.2.0-to-4.0.0.mysql.sql > 4b82e36ab4 > > standalone-metastore/metastore-server/src/main/sql/oracle/hive-schema-4.0.0.oracle.sql > db398e5f66 > > standalone-metastore/metastore-server/src/main/sql/oracle/upgrade-3.2.0-to-4.0.0.oracle.sql > 1be83fc4a9 > > standalone-metastore/metastore-server/src/main/sql/postgres/hive-schema-4.0.0.postgres.sql > e6e30160af > > standalone-metastore/metastore-server/src/main/sql/postgres/upgrade-3.2.0-to-4.0.0.postgres.sql > b90cecb173 > > standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/DbInstallBase.java > c1a1629548 > > standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/rules/DatabaseRule.java > a6d22d19ef > > standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/rules/Oracle.java > 0b070e19ac > > standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/txn/TestOpenTxn.java > PRE-CREATION > > > Diff: https://reviews.apache.org/r/72388/diff/3/ > > > Testing > ------- > > > Thanks, > > Peter Varga > >
