yihua commented on code in PR #12005: URL: https://github.com/apache/hudi/pull/12005#discussion_r1794397591
########## rfc/rfc-82/rfc-82.md: ########## @@ -0,0 +1,290 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +# RFC-82: Concurrency control of schema evolution + +## Proposers + +- @jonvex +- @Davis-Zhang-Onehouse + +## Approvers +- @yihua +- @danny0405 + +## Status + +JIRA: https://issues.apache.org/jira/browse/HUDI-8221 + +## Abstract + +This RFC proposes to enhance Hudi's concurrency control mechanism to handle concurrent schema evolution scenarios. The current implementation does not adequately address conflicts that may arise when multiple transactions attempt to alter the **table schema** simultaneously. This enhancement aims to detect and prevent such conflicts, ensuring data consistency and integrity in multi-writer environments. + +## Background + +### Hudi Schema +In Apache Hudi, schema management is a critical component that ensures data consistency and facilitates efficient data processing, especially in environments where data structures may evolve over time. + +The schema of the input data represents the structure of incoming records being ingested into Hudi from various sources. During a write operation, the Hudi write client utilizes a **write schema**, which is typically derived from the input data schema or specified by a schema provider. This **write schema** is applied throughout the transaction to maintain consistency. + +Additionally, Hudi stores the **table schema** within the commit metadata on storage, capturing the authoritative schema of the hudi table table at the time of each commit. This stored schema is crucial for readers to correctly interpret the data and for managing schema evolution across different data versions. These schemas are determined through a reconciliation process that considers both the incoming data schema and the existing **table schema**, allowing Hudi to handle schema changes gracefully. Currently, Hudi uses this schema management approach to enable seamless read and write operations, support upserts and deletes, and manage schema evolution, ensuring that the system remains robust even as the underlying data structures change. + +### Concurrency control over concurrent schema evolution +Hudi supports concurrency control to handle concurrent write operations. However, the existing implementation does not specifically account for schema evolution conflicts. In a multi-writer environment, it's possible for different clients to attempt schema changes concurrently, which can lead to inconsistencies if not properly managed. + +Schema evolution is a critical operation that can significantly impact data compatibility and query results. Uncontrolled concurrent schema changes can result in data inconsistencies, failed reads, or incorrect query results. By extending the concurrency control mechanism to handle schema evolution, we can prevent these issues and ensure a more robust and reliable data management system. + +Depending on whether there are concurrent schema evolution transactions, an inflight transaction may see a different latest schema of the table when it checks at different time, as there can always be other transactions committed and potentially changed schema as a result of that in multi-writer scenarios. + +We use "txn" as a abbreviation of transaction. + +| Scenario | Table Schema when Txn Start | Table Schema when Txn Validates | Write Schema Used by curr Txn | Should Conflict? | New Table Schema in the Commit Metadata of curr Txn | Notes | +|----------|---------------------|--------------------------|-------------------|------------------|---------------------------------------|-------| +| 1 | Not exists | Not exists | S1 | No | S1 | Current txn is the first commit ever, conflict is impossible | +| 2 | Not exists | S1 | S1 | No | S1 | Second commit, no schema evolution | +| 3 | Not exists | S2 | S3 | Yes | N/A (throws exception) | No predefined schema, effectively concurrent schema definition. The design decision made here is to identify this as schema conflict for simplicity. It can also be broken down into cases where (1) S3 is evolvable from S2, (2) S3 is not evolvable from S2, for further optimization. | +| 4 | S1 | S1 | S1 | No | S1 | No schema evolution | +| 5 | S1 | S1 | S2 | No | S2 | Schema evolution in current transaction | +| 6 | S1 | S2 | S1 | No | S2 | Backwards compatibility handles it | +| 7 | S1 | S2 | S2 | No | S2 | Concurrent evolution to same schema | +| 8 | S1 | S2 | S3 | Yes | N/A (throws exception) | Concurrent evolution to different schemas. Same as case 2, for future optimization we should consider evolvibility from S2 to S3 | + +For timeline graph of each case please refer appendix. + +Notes: +- S1, S2, S3 represent different schemas. The proposed solution does not introduce any new assumptions on compatibility/evolvbility among those schemas. But if "Table Schema when Txn Start" is schema X and Table Schema when Txn Validates is Y, it naturally means Y is compatible and is evolved from X as it is guaranteed by today's implementation. Review Comment: ```suggestion - S1, S2, S3 represent different schemas. The proposed solution does not introduce any new assumptions on compatibility among those schemas. But if "Table Schema when Txn Start" is schema X and Table Schema when Txn Validates is Y, it naturally means Y is compatible and is evolved from X as it is guaranteed by today's implementation. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
