Talked with Lian on Slack, the user is using a hadoop 3.2.1 + hive
(postgres) + spark + minio docker installation. There might be some S3A
related dependencies missing on the Hive server side based on the stack
trace. Let's see if that fixes the issue.
-Jack
On Mon, Aug 16, 2021 at 7:32 PM Lian Jiang <jiangok2...@gmail.com> wrote:
> This is my full script launching spark-shell:
>
> # add Iceberg dependency
> export AWS_REGION=us-east-1
> export AWS_ACCESS_KEY_ID=minio
> export AWS_SECRET_ACCESS_KEY=minio123
>
> ICEBERG_VERSION=0.11.1
>
> DEPENDENCIES="org.apache.iceberg:iceberg-spark3-runtime:$ICEBERG_VERSION,org.apache.iceberg:iceberg-hive-runtime:$ICEBERG_VERSION,org.apache.hadoop:hadoop-aws:3.2.0"
>
> MINIOSERVER=192.168.176.5
>
>
> # add AWS dependnecy
> AWS_SDK_VERSION=2.15.40
> AWS_MAVEN_GROUP=software.amazon.awssdk
> AWS_PACKAGES=(
> "bundle"
> "url-connection-client"
> )
> for pkg in "${AWS_PACKAGES[@]}"; do
> DEPENDENCIES+=",$AWS_MAVEN_GROUP:$pkg:$AWS_SDK_VERSION"
> done
>
> # start Spark SQL client shell
> /spark/bin/spark-shell --packages $DEPENDENCIES \
> --conf
> spark.sql.catalog.hive_test=org.apache.iceberg.spark.SparkCatalog \
> --conf spark.sql.catalog.hive_test.type=hive \
> --conf
> spark.sql.catalog.hive_test.io-impl=org.apache.iceberg.hadoop.HadoopFileIO \
> --conf spark.sql.catalog.hive_test.warehouse=s3a://east/warehouse \
> --conf spark.hadoop.fs.s3a.endpoint=http://$MINIOSERVER:9000 \
> --conf spark.hadoop.fs.s3a.access.key=minio \
> --conf spark.hadoop.fs.s3a.secret.key=minio123 \
> --conf spark.hadoop.fs.s3a.path.style.access=true \
> --conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
>
>
> Let me know if anything is missing. Thanks.
>
> On Mon, Aug 16, 2021 at 7:29 PM Jack Ye <yezhao...@gmail.com> wrote:
>
>> Have you included the hadoop-aws jar?
>> https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws
>> -Jack
>>
>> On Mon, Aug 16, 2021 at 7:09 PM Lian Jiang <jiangok2...@gmail.com> wrote:
>>
>>> Jack,
>>>
>>> You are right. S3FileIO will not work on minio since minio does not
>>> support ACL:
>>> https://docs.min.io/docs/minio-server-limits-per-tenant.html
>>>
>>> To use iceberg, minio + s3a, I used below script to launch spark-shell:
>>>
>>> /spark/bin/spark-shell --packages $DEPENDENCIES \
>>> --conf
>>> spark.sql.catalog.hive_test=org.apache.iceberg.spark.SparkCatalog \
>>> --conf spark.sql.catalog.hive_test.type=hive \
>>> * --conf
>>> spark.sql.catalog.hive_test.io-impl=org.apache.iceberg.hadoop.HadoopFileIO
>>> \*
>>> --conf spark.sql.catalog.hive_test.warehouse=s3a://east/warehouse \
>>> --conf spark.hadoop.fs.s3a.endpoint=http://$MINIOSERVER:9000 \
>>> --conf spark.hadoop.fs.s3a.access.key=minio \
>>> --conf spark.hadoop.fs.s3a.secret.key=minio123 \
>>> --conf spark.hadoop.fs.s3a.path.style.access=true \
>>> --conf
>>> spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
>>>
>>>
>>>
>>> *The spark code:*
>>>
>>> import org.apache.spark.sql.SparkSession
>>> val values = List(1,2,3,4,5)
>>>
>>> val spark = SparkSession.builder().master("local").getOrCreate()
>>> import spark.implicits._
>>> val df = values.toDF()
>>>
>>> val core = "mytable"
>>> val table = s"hive_test.mydb.${core}"
>>> val s3IcePath = s"s3a://east/${core}.ice"
>>>
>>> df.writeTo(table)
>>> .tableProperty("write.format.default", "parquet")
>>> .tableProperty("location", s3IcePath)
>>> .createOrReplace()
>>>
>>>
>>> *Still the same error:*
>>> java.lang.ClassNotFoundException: Class
>>> org.apache.hadoop.fs.s3a.S3AFileSystem not found
>>>
>>>
>>> What else could be wrong? Thanks for any clue.
>>>
>>>
>>>
>>> On Mon, Aug 16, 2021 at 9:35 AM Jack Ye <yezhao...@gmail.com> wrote:
>>>
>>>> Sorry for the late reply, I thought I replied on Friday but the email
>>>> did not send successfully.
>>>>
>>>> As Daniel said, you don't need to setup S3A if you are using S3FileIO.
>>>>
>>>> Th S3FileIO by default reads the default credentials chain to check
>>>> credential setups one by one:
>>>> https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials.html#credentials-chain
>>>>
>>>> If you would like to use a specialized credential provider, you can
>>>> directly customize your S3 client:
>>>> https://iceberg.apache.org/aws/#aws-client-customization
>>>>
>>>> It looks like you are trying to use MinIO to mount S3A file system? If
>>>> you have to use MinIO then there is not a way to integrate with S3FileIO
>>>> right now. (maybe I am wrong on this, I don't know much about MinIO)
>>>>
>>>> To directly use S3FileIO with HiveCatalog, simply do:
>>>>
>>>> /spark/bin/spark-shell --packages $DEPENDENCIES \
>>>> --conf
>>>> spark.sql.catalog.hive_test=org.apache.iceberg.spark.SparkCatalog \
>>>> --conf spark.sql.catalog.hive_test.type=hive \
>>>> --conf
>>>> spark.sql.catalog.hive_test.io-impl=org.apache.iceberg.aws.s3.S3FileIO \
>>>> --conf spark.sql.catalog.hive_test.warehouse=s3://bucket
>>>>
>>>> Best,
>>>> Jack Ye
>>>>
>>>>
>>>>
>>>> On Sun, Aug 15, 2021 at 2:53 PM Lian Jiang <jiangok2...@gmail.com>
>>>> wrote:
>>>>
>>>>> Thanks. I prefer S3FileIO as it is recommended by iceberg. Do you have
>>>>> a sample using hive catalog, s3FileIO, spark API (as opposed to SQL), S3
>>>>> access.key and secret.key? It is hard to get all settings right for this
>>>>> combination without an example. Appreciate any help.
>>>>>
>>>>> On Fri, Aug 13, 2021 at 6:01 PM Daniel Weeks <daniel.c.we...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> So, if I recall correctly, the hive server does need access to check
>>>>>> and create paths for table locations.
>>>>>>
>>>>>> There may be an option to disable this behavior, but otherwise the fs
>>>>>> implementation probably needs to be available to the hive metastore.
>>>>>>
>>>>>> -Dan
>>>>>>
>>>>>> On Fri, Aug 13, 2021, 4:48 PM Lian Jiang <jiangok2...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks Daniel.
>>>>>>>
>>>>>>> After modifying the script to,
>>>>>>>
>>>>>>> export AWS_REGION=us-east-1
>>>>>>> export AWS_ACCESS_KEY_ID=minio
>>>>>>> export AWS_SECRET_ACCESS_KEY=minio123
>>>>>>>
>>>>>>> ICEBERG_VERSION=0.11.1
>>>>>>>
>>>>>>> DEPENDENCIES="org.apache.iceberg:iceberg-spark3-runtime:$ICEBERG_VERSION,org.apache.iceberg:iceberg-hive-runtime:$ICEBERG_VERSION,org.apache.hadoop:hadoop-aws:3.2.0"
>>>>>>>
>>>>>>> MINIOSERVER=192.168.160.5
>>>>>>>
>>>>>>>
>>>>>>> # add AWS dependnecy
>>>>>>> AWS_SDK_VERSION=2.15.40
>>>>>>> AWS_MAVEN_GROUP=software.amazon.awssdk
>>>>>>> AWS_PACKAGES=(
>>>>>>> "bundle"
>>>>>>> "url-connection-client"
>>>>>>> )
>>>>>>> for pkg in "${AWS_PACKAGES[@]}"; do
>>>>>>> DEPENDENCIES+=",$AWS_MAVEN_GROUP:$pkg:$AWS_SDK_VERSION"
>>>>>>> done
>>>>>>>
>>>>>>> # start Spark SQL client shell
>>>>>>> /spark/bin/spark-shell --packages $DEPENDENCIES \
>>>>>>> --conf
>>>>>>> spark.sql.catalog.hive_test=org.apache.iceberg.spark.SparkCatalog \
>>>>>>> --conf spark.sql.catalog.hive_test.type=hive \
>>>>>>> --conf spark.hadoop.fs.s3a.endpoint=http://$MINIOSERVER:9000 \
>>>>>>> --conf spark.hadoop.fs.s3a.access.key=minio \
>>>>>>> --conf spark.hadoop.fs.s3a.secret.key=minio123 \
>>>>>>> --conf spark.hadoop.fs.s3a.path.style.access=true \
>>>>>>> --conf
>>>>>>> spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
>>>>>>>
>>>>>>> I got: MetaException: java.lang.RuntimeException:
>>>>>>> java.lang.ClassNotFoundException: Class
>>>>>>> org.apache.hadoop.fs.s3a.S3AFileSystem not found. My hive server is not
>>>>>>> using s3 and should not cause this error. Any ideas? Thanks.
>>>>>>>
>>>>>>>
>>>>>>> I got "ClassNotFoundException: Class
>>>>>>> org.apache.hadoop.fs.s3a.S3AFileSystem not found". Any idea what
>>>>>>> dependency
>>>>>>> could I miss?
>>>>>>>
>>>>>>> On Fri, Aug 13, 2021 at 4:03 PM Daniel Weeks <
>>>>>>> daniel.c.we...@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hey Lian,
>>>>>>>>
>>>>>>>> At a cursory glance, it appears that you might be mixing two
>>>>>>>> different FileIO implementations, which may be why you are not getting
>>>>>>>> the
>>>>>>>> expected result.
>>>>>>>>
>>>>>>>> When you set: --conf
>>>>>>>> spark.sql.catalog.hive_test.io-impl=org.apache.iceberg.aws.s3.S3FileIO
>>>>>>>> you're
>>>>>>>> actually switching over to the native S3 implementation within Iceberg
>>>>>>>> (as
>>>>>>>> opposed to S3AFileSystem via HadoopFileIO). However, all of the
>>>>>>>> following
>>>>>>>> settings to setup access are then set for the S3AFileSystem (which
>>>>>>>> would
>>>>>>>> not be used with S3FileIO).
>>>>>>>>
>>>>>>>> You might try just removing that line since it should use the
>>>>>>>> HadoopFileIO at that point and may work.
>>>>>>>>
>>>>>>>> Hope that's helpful,
>>>>>>>> -Dan
>>>>>>>>
>>>>>>>> On Fri, Aug 13, 2021 at 3:50 PM Lian Jiang <jiangok2...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I try to create an iceberg table on minio s3 and hive.
>>>>>>>>>
>>>>>>>>> *This is how I launch spark-shell:*
>>>>>>>>>
>>>>>>>>> # add Iceberg dependency
>>>>>>>>> export AWS_REGION=us-east-1
>>>>>>>>> export AWS_ACCESS_KEY_ID=minio
>>>>>>>>> export AWS_SECRET_ACCESS_KEY=minio123
>>>>>>>>>
>>>>>>>>> ICEBERG_VERSION=0.11.1
>>>>>>>>>
>>>>>>>>> DEPENDENCIES="org.apache.iceberg:iceberg-spark3-runtime:$ICEBERG_VERSION"
>>>>>>>>>
>>>>>>>>> MINIOSERVER=192.168.160.5
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> # add AWS dependnecy
>>>>>>>>> AWS_SDK_VERSION=2.15.40
>>>>>>>>> AWS_MAVEN_GROUP=software.amazon.awssdk
>>>>>>>>> AWS_PACKAGES=(
>>>>>>>>> "bundle"
>>>>>>>>> "url-connection-client"
>>>>>>>>> )
>>>>>>>>> for pkg in "${AWS_PACKAGES[@]}"; do
>>>>>>>>> DEPENDENCIES+=",$AWS_MAVEN_GROUP:$pkg:$AWS_SDK_VERSION"
>>>>>>>>> done
>>>>>>>>>
>>>>>>>>> # start Spark SQL client shell
>>>>>>>>> /spark/bin/spark-shell --packages $DEPENDENCIES \
>>>>>>>>> --conf
>>>>>>>>> spark.sql.catalog.hive_test=org.apache.iceberg.spark.SparkCatalog \
>>>>>>>>> --conf spark.sql.catalog.hive_test.warehouse=s3a://east/prefix
>>>>>>>>> \
>>>>>>>>> --conf spark.sql.catalog.hive_test.type=hive \
>>>>>>>>> --conf
>>>>>>>>> spark.sql.catalog.hive_test.io-impl=org.apache.iceberg.aws.s3.S3FileIO
>>>>>>>>> \
>>>>>>>>> --conf spark.hadoop.fs.s3a.endpoint=http://$MINIOSERVER:9000 \
>>>>>>>>> --conf spark.hadoop.fs.s3a.access.key=minio \
>>>>>>>>> --conf spark.hadoop.fs.s3a.secret.key=minio123 \
>>>>>>>>> --conf spark.hadoop.fs.s3a.path.style.access=true \
>>>>>>>>> --conf
>>>>>>>>> spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
>>>>>>>>>
>>>>>>>>> *Here is the spark code to create the iceberg table:*
>>>>>>>>>
>>>>>>>>> import org.apache.spark.sql.SparkSession
>>>>>>>>> val values = List(1,2,3,4,5)
>>>>>>>>>
>>>>>>>>> val spark = SparkSession.builder().master("local").getOrCreate()
>>>>>>>>> import spark.implicits._
>>>>>>>>> val df = values.toDF()
>>>>>>>>>
>>>>>>>>> val core = "mytable8"
>>>>>>>>> val table = s"hive_test.mydb.${core}"
>>>>>>>>> val s3IcePath = s"s3a://spark-test/${core}.ice"
>>>>>>>>>
>>>>>>>>> df.writeTo(table)
>>>>>>>>> .tableProperty("write.format.default", "parquet")
>>>>>>>>> .tableProperty("location", s3IcePath)
>>>>>>>>> .createOrReplace()
>>>>>>>>>
>>>>>>>>> I got an error "The AWS Access Key Id you provided does not exist
>>>>>>>>> in our records.".
>>>>>>>>>
>>>>>>>>> I have verified that I can login minio UI using the same username
>>>>>>>>> and password that I passed to spark-shell via AWS_ACCESS_KEY_ID and
>>>>>>>>> AWS_SECRET_ACCESS_KEY env variables.
>>>>>>>>> https://github.com/apache/iceberg/issues/2168 is related but does
>>>>>>>>> not help me. Not sure why the credential does not work for iceberg +
>>>>>>>>> AWS.
>>>>>>>>> Any idea or an example of writing an iceberg table to S3 using hive
>>>>>>>>> catalog
>>>>>>>>> will be highly appreciated! Thanks.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Create your own email signature
>>>>>>> <https://www.wisestamp.com/signature-in-email/?utm_source=promotion&utm_medium=signature&utm_campaign=create_your_own&srcid=5234462839406592>
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Create your own email signature
>>>>> <https://www.wisestamp.com/signature-in-email/?utm_source=promotion&utm_medium=signature&utm_campaign=create_your_own&srcid=5234462839406592>
>>>>>
>>>>
>>>
>>> --
>>>
>>> Create your own email signature
>>> <https://www.wisestamp.com/signature-in-email/?utm_source=promotion&utm_medium=signature&utm_campaign=create_your_own&srcid=5234462839406592>
>>>
>>
>
> --
>
> Create your own email signature
> <https://www.wisestamp.com/signature-in-email/?utm_source=promotion&utm_medium=signature&utm_campaign=create_your_own&srcid=5234462839406592>
>
