codope commented on a change in pull request #3957: URL: https://github.com/apache/hudi/pull/3957#discussion_r794432686
########## File path: rfc/rfc-40/rfc-40.md ########## @@ -0,0 +1,195 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# RFC-40: Hudi Connector for Trino + +## Proposers + +- @codope + +## Approvers + +- @bvaradar +- @vinothchandar + +## Status + +JIRA: https://issues.apache.org/jira/browse/HUDI-2687 + +> Please keep the status updated in `rfc/README.md`. + +## Abstract + +Today, Hudi supports snapshot queries on Copy-On-Write (COW) tables and read-optimized queries on Merge-On-Read (MOR) +tables with Trino, through the input format based integration in the Hive connector. This approach has known performance +limitations with very large tables. Moreover, as Hudi keeps getting better, a new plugin to provide access to Hudi data +and metadata will help in unlocking capabilities such as metadata-based listing, full schema evolution, etc. for the +Trino users. A separate Hudi connector would also allow its independent evolution without having to worry about +hacking/breaking the Hive connector. A separate connector also falls in line with our vision when we think of a +standalone timeline server or a lake cache to balance the tradeoff between writing and querying. + +## Background + +The current Trino integration relies on a custom annotation `@UseFileSplitsFromInputFormat`. Any input format that has +this annotation would fetch splits by invoking the corresponding input format’s `getSplits()` method instead of Trino's +Hive connector native split loading logic. For instance, realtime queries on Hudi tables queried via Trino, this would +be a simple call to `HoodieParquetRealtimeInputFormat.getSplits()`. This approach has known performance limitations +because of the redundant Hudi table metadata listing while loading splits. This issue has been fixed to some extent in +Presto and the work to upstream those changes to Trino is [in progress](https://github.com/trinodb/trino/pull/9641). + +A connector enables Trino to communicate with external data sources. The connector interface is composed of four parts: +the Metadata API, Data Location API, Data Source API, and Data Sink API. These APIs are designed to allow performant +implementations of connectors within the environment of Trino's distributed execution engine. For an overview of the +Trino architecture please see [Trino concepts](https://trino.io/docs/current/overview/concepts.html). + +### Trino query execution model + +When Trino executes a query, it does so by breaking up the execution into a hierarchy of **stages**. A single stage is +implemented as a series of **tasks** distributed over a network of Trino workers. Tasks operate on **splits**, which are +partitions of a larger data set. Tasks at the source stage produce data in the form of **pages**, which are a collection +of rows in columnar format. These pages flow to other intermediate downstream stages. + +## Implementation + +Trino provides a service provider interface (SPI), which is a type of API used to implement a connector. By implementing +the SPI in a connector, Trino can use standard operations internally to connect to any data source and perform +operations on any data source. The connector takes care of the details relevant to the specific data source. + +Hudi connector will implement three parts of the API: + +- Operations to fetch table/view/schema metadata. +- Operations to produce logical units of data partitioning, so that Trino can parallelize reads and writes. +- Data sources and sinks that convert the source data to/from the in-memory format expected by the query engine. + +Hudi connector will be registered as a plugin, which will be loaded by Trino server at startup. The entry point will +be `HudiPlugin`, an implementation of the `Plugin` interface. Instances of Hudi connector are created by a +ConnectorFactory instance which is created when Trino calls `getConnectorFactory()` on the plugin. +A class-diagrammatic view of the different components is shown below. + + +### Operations to fetch table/view/schema metadata + +The `ConnectorMetadata` interface provides important methods that are responsible for allowing Trino to look at lists of +schemas, lists of tables, lists of columns, and other metadata about a particular data source. The implementation of +this interface will create the `HoodieTableMetaClient` and pass it to the connector table handle through which Trino +can access metadata of a Hudi table. + + +### Operations to produce logical units of data partitioning + +We will need to implement the `ConnectorSplit` and `ConnectorSplitManager` interfaces. Hudi splits will be similar to +how Hive connector describes splits in the form of a path to a file with offset and length that indicate which part of +the file needs to be processed. + +```java +public class HudiSplit + implements ConnectorSplit { + private final String path; + private final long start; + private final long length; + private final long fileSize; + private final List<HostAddress> addresses; + private final TupleDomain<HiveColumnHandle> predicate; + private final List<HivePartitionKey> partitionKeys; +} +``` + +The split manager will partition the data for a table into the individual chunks that Trino will distribute to workers +for processing. This is where the partition loader logic will reside. While listing the files for each Hudi partition +the split manager will create one or more split per file. For non-partitioned table, the split mamager will simply Review comment: That's not correct anymore. We do have size-based splitting strategy. I will update the docs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
