Hi Somesh, But while trying so we are seeing following exception : > hadoop fs -ls s3a://somesh.qa.bucket/ -:
This has nothing to do with Hive as such, You have configured Hadoop S3 client wrong, you are missing configs, your hadoop ls command itself is failing, there is no Hive involved here. You need to setup the FileSystem correctly... This is a hadoop problem, maybe you can explore reading this doc in hadoop [1] & that might help, if you still face issues, you should bug the Hadoop mailing lists not hive -Ayush [1] https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html On Wed, 18 Sept 2024 at 11:12, Awasthi, Somesh <soawas...@informatica.com.invalid> wrote: > Hi Team, > > > > I want to setup hive4 standalone to store table on S3 and Adls gen2 as a > storage . > > > > Could you please help me as with proper steps and configurations required > for this. > > > > Because we are facing multiple issue on this please help me here ASPA. > > > > *What we tried.* > > > > I am trying to configure AWS S3 configuration with the Hadoop and Hive > setup. > > But while trying so we are seeing following exception : > > hadoop fs -ls s3a://somesh.qa.bucket/ -: > > Fatal internal error java.lang.RuntimeException: > java.lang.ClassNotFoundException: Class > org.apache.hadoop.fs.s3a.S3AFileSystem not found > > *To resolve this I have added hadoop-aws-3.3.6.jar and > aws-java-sdk-bundle-1.12.770.jar in Hadoop classpath.* > > *i.e is under : /usr/local/hadoop/share/hadoop/common/lib* > > *And S3 related configurations in the core-site.xml file: under > /usr/local/hadoop/etc/hadoop directory.* > > fs.default.name s3a://somesh.qa.bucket fs.s3a.impl > org.apache.hadoop.fs.s3a.S3AFileSystem fs.s3a.endpoint > s3.us-west-2.amazonaws.com fs.s3a.access.key {Access _Key_Value} > fs.s3a.secret.key {Secret_Key_Value} fs.s3a.path.style.access false > > Now when we try hadoop fs -ls s3a://somesh.qa.bucket/ > > We are observing following exception : > > 2024-08-22 13:50:11,294 INFO impl.MetricsConfig: Loaded properties from > hadoop-metrics2.properties > 2024-08-22 13:50:11,376 INFO impl.MetricsSystemImpl: Scheduled Metric > snapshot period at 10 second(s). > 2024-08-22 13:50:11,376 INFO impl.MetricsSystemImpl: s3a-file-system > metrics system started > 2024-08-22 13:50:11,434 WARN util.VersionInfoUtils: The AWS SDK for Java > 1.x entered maintenance mode starting July 31, 2024 and will reach end of > support on December 31, 2025. For more information, see > https://aws.amazon.com/blogs/developer/the-aws-sdk-for-java-1-x-is-in-maintenance-mode-effective-july-31-2024/ > You can print where on the file system the AWS SDK for Java 1.x core > runtime is located by setting the AWS_JAVA_V1_PRINT_LOCATION environment > variable or aws.java.v1.printLocation system property to 'true'. > This message can be disabled by setting the > AWS_JAVA_V1_DISABLE_DEPRECATION_ANNOUNCEMENT environment variable or > aws.java.v1.disableDeprecationAnnouncement system property to 'true'. > The AWS SDK for Java 1.x is being used here: > at java.lang.Thread.getStackTrace(Thread.java:1564) > at > com.amazonaws.util.VersionInfoUtils.printDeprecationAnnouncement(VersionInfoUtils.java:81) > at com.amazonaws.util.VersionInfoUtils.(VersionInfoUtils.java:59) > at com.amazonaws.internal.EC2ResourceFetcher.(EC2ResourceFetcher.java:44) > at > com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.(InstanceMetadataServiceCredentialsFetcher.java:38) > at > com.amazonaws.auth.InstanceProfileCredentialsProvider.(InstanceProfileCredentialsProvider.java:111) > at > com.amazonaws.auth.InstanceProfileCredentialsProvider.(InstanceProfileCredentialsProvider.java:91) > at > com.amazonaws.auth.InstanceProfileCredentialsProvider.(InstanceProfileCredentialsProvider.java:75) > at > com.amazonaws.auth.InstanceProfileCredentialsProvider.(InstanceProfileCredentialsProvider.java:58) > at > com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.initializeProvider(EC2ContainerCredentialsProviderWrapper.java:66) > at > com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.(EC2ContainerCredentialsProviderWrapper.java:55) > at > org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider.(IAMInstanceCredentialsProvider.java:53) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:727) > at > org.apache.hadoop.fs.s3a.S3AUtils.buildAWSProviderList(S3AUtils.java:659) > at > org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:585) > at > org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient(S3AFileSystem.java:959) > at > org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:586) > at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3611) > at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174) > at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3712) > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3663) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:557) > at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365) > at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:347) > at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:264) > at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:247) > at > org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:105) > at org.apache.hadoop.fs.shell.Command.run(Command.java:191) > at org.apache.hadoop.fs.FsShell.run(FsShell.java:327) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:82) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:97) > at org.apache.hadoop.fs.FsShell.main(FsShell.java:390) > ls: s3a://infa.qa.bucket/: > org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials > provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider > EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider : > com.amazonaws.SdkClientException: Unable to load AWS credentials from > environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and > AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)) > 2024-08-22 13:50:14,248 INFO impl.MetricsSystemImpl: Stopping > s3a-file-system metrics system... > 2024-08-22 13:50:14,248 INFO impl.MetricsSystemImpl: s3a-file-system > metrics system stopped. > 2024-08-22 13:50:14,248 INFO impl.MetricsSystemImpl: s3a-file-system > metrics syst > > > > > > *Could you please help us to resolve this issue as soon as possible* > > > > Thanks, > > Somesh > > >
