FANNG1 commented on code in PR #5006: URL: https://github.com/apache/gravitino/pull/5006#discussion_r1776297765
########## docs/hive-catalog-with-s3.md: ########## @@ -0,0 +1,206 @@ +--- +title: "Hive catalog with s3" +slug: /hive-catalog +date: 2024-9-24 +keyword: Hive catalog cloud storage S3 +license: "This software is licensed under the Apache License version 2." +--- + + +## Introduction +Since Hive 2.x, Hive has supported S3 as a storage backend, enabling users to store and manage data in Amazon S3 directly through Hive. Gravitino enhances this capability by supporting the Hive catalog with S3, allowing users to efficiently manage the storage locations of files located in S3. This integration simplifies data operations and enables seamless access to S3 data from Hive queries. + +The following sections will guide you through the necessary steps to configure the Hive catalog to utilize S3 as a storage backend, including configuration details and examples for creating databases and tables. + +## Hive cluster configuration +To use the Hive catalog with S3, you must configure your Hive cluster to recognize S3 as a storage backend. The following example illustrates the required changes in the `hive-site.xml` configuration file: + +### Example Configuration Changes +Below are the essential properties to add or modify in the `hive-site.xml` file to support S3: + +```xml +<property> + <name>fs.s3a.access.key</name> + <value>S3_ACCESS_KEY_ID</value> +</property> + +<property> + <name>fs.s3a.secret.key</name> + <value>S3_SECRET_KEY_ID</value> +</property> + +<property> + <name>fs.s3a.endpoint</name> + <value>S3_ENDPOINT_ID</value> +</property> + +<property> + <name>fs.s3a.aws.credentials.provider</name> + <value>org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider,com.amazonaws.auth.EnvironmentVariableCredentialsProvider,org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider</value> +</property> +``` + +### Adding Required JARs + + +After updating the `hive-site.xml`, you need to ensure that the necessary S3-related JARs are included in the Hive classpath. You can do this by executing the following command: +```shell Review Comment: add a empty line? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
