yuqi1129 opened a new pull request, #13792: URL: https://github.com/apache/gravitino/pull/13792
### What changes were proposed in this pull request? A write to the relational entity store that waits on a row lock longer than the backend's lock-wait timeout is now reported as `OptimisticLockException` (HTTP 409, code 1012) instead of an internal error. - `ExceptionUtils.isLockWaitTimeout` recognises the timeout once: MySQL error code 1205, H2 error code 50200 (SQLState HYT00), and PostgreSQL SQLState 55P03. The vendor codes are matched for MySQL and H2, because the MySQL driver reports 1205 as SQLState 40001 or HY000, and 40001 is also a deadlock. Deadlocks are not mapped. - `ExceptionUtils.checkSQLException` classifies the timeout before the backend converter, which would otherwise turn it into an `IOException`. This covers create, alter and the relation writes of every service. Tag associations and model versions name what was written via `throwIfLockWaitTimeout`. - `RelationalEntityStore` maps the timeout on every write (put, update, delete, deleteAndGet, the relation writes, batchPut, batchDelete), for paths such as drop that don't go through `checkSQLException`. - `StatisticMetaService` reuses the classification and keeps its own deadlock handling for batch statements, whose row lock order follows the database's plan. - The conflict reaches the caller instead of being wrapped into a `RuntimeException` (500) on the external schema, topic and view drops, on the schema, table, topic and view import paths, on the schema drops of a managed catalog force drop, and in Lance table create, drop and purge. A Lance create that conflicts drops the dataset it just created, as it already does when the table exists. - `JobManager.runJob` cancels the submitted execution when registering the job conflicts, so that a retry does not run the job twice. ### Why are the changes needed? With the tree lock removed (#13737), writers on one server queue on rows inside the database. For example, a table drop waits behind a statistic update or a tag association that holds the table row. If the holder takes longer than the lock-wait timeout, the waiting statement is aborted and the request returned 500. The store rolls the whole transaction back, so the operation can be retried, and a 409 tells the caller so. H2, the default backend, has a lock timeout of about 1 to 2 seconds; MySQL's default `innodb_lock_wait_timeout` is 50 seconds; PostgreSQL waits by default (`lock_timeout` = 0). Fix: #13776 ### Does this PR introduce _any_ user-facing change? Yes. A write that times out waiting for a row lock now fails with 409 (`OptimisticLockException`, "... could not be changed because a concurrent operation held its lock longer than the backend's lock wait timeout; retry the operation") instead of 500. No API or property changes. Known limits: - If an external table, view or schema was already dropped in the catalog, or an Iceberg REST change was already committed, and the store write then times out, a retry does not redo the store write: the stale Gravitino registration stays, as for any other store failure after the external change. - Loading an object that exists only in the external catalog imports it, so a load can return 409 if that import times out; the load can be retried. - For a rename, the message names the new name, because services report the entity they write. - PostgreSQL only reports 55P03 when `lock_timeout` is set; by default it waits. - When a job registration conflicts, the cancelled execution keeps its staging directory, because cancellation is asynchronous and the execution may still be writing to it. - The `EntityStore` interface still does not document `OptimisticLockException`; that is #13622. - Not covered: `JdbcPartitionStatisticStorage` (separate data source), and the Lance and Iceberg REST services, which still map the conflict to 500. For Iceberg REST that is deliberate, since the Iceberg change is already committed and a client retry is not safe; Lance REST can map it to its 409 in a follow-up. ### How was this patch tested? - `TestRelationalEntityStoreLockWaitTimeout` (new): real H2 lock-wait timeouts with another session holding the row `FOR UPDATE`, for a table drop, an alter, a create waiting on its schema row, and a tag association; the failed write is rolled back and a retry succeeds. Every store write is also checked with a failing backend, and deadlocks and other failures are checked to stay unchanged. - `TestExceptionUtils`: the codes of each backend, deadlocks not matched, `checkSQLException` reporting the timeout as a conflict. - `TestSchemaOperationDispatcher`, `TestTableOperationDispatcher`, `TestTopicOperationDispatcher`, `TestViewOperationDispatcher`: conflicts on drop and import reach the caller. - `TestLanceTableOperations`: create drops the dataset and drop/purge propagate the conflict. - `TestCatalogManager`: a managed catalog force drop propagates a schema conflict. `TestJobManager`: a conflicting job registration cancels the submitted execution. - Existing `TestStatisticMetaService`, `TestTagMetaService`, `TestModelVersionMetaService`, `TestTagManager` pass. - Mutation checks: removing each mapping makes the corresponding new tests fail. - All on H2. MySQL 1205 and PostgreSQL 55P03 are covered with synthetic `SQLException`s; the docker backends run in CI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
