Re: Exploring Nessie Integration for Iceberg Tables: Insights Needed

2024-05-06 Thread Pani Dhakshnamurthy
=na&utm_content=email-signature&utm_campaign=email-signature>*/ > **Follow Us on LinkedIn!* <https://www.linkedin.com/company/dremio> > *Resources for Getting Hands-on with Apache Iceberg/Dremio* > <https://medium.com/data-engineering-with-dremio/a-deep-intro-to

Exploring Nessie Integration for Iceberg Tables: Insights Needed

2024-05-06 Thread Pani Dhakshnamurthy
Hello Everyone, I'm currently planning to build a Lakehouse solution, leveraging Apache Spark and Hive Metastore in my Hadoop workflows. I'm also exploring the potential benefits of integrating Nessie as my catalog solution. With this setup, we'll have two catalogs, and we anticipate working with

Re: How to Set S3 Credentials at bucket level in Iceberg Spark Session

2024-04-22 Thread Pani Dhakshnamurthy
Hi Awashi, S3A supports setting credentials at the S3 bucket level - ref: https://docs.cloudera.com/runtime/7.2.0/cloud-data-access/topics/cr-cda-configuring-per-bucket-settings.html . I am not sure if S3FileIO supports this feature. Thanks Pani On Mon, Apr 22, 2024 at 2:01 PM Yufei Gu wro

Re: Help needed with Iceberg on Databricks platform

2023-04-28 Thread Pani Dhakshnamurthy
`org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions` to the >comma separated list >3. Submit a new job with `spark.sql.extensions` set to the modified >value > > > On Fri, Apr 28, 2023 at 12:28 PM Pani Dhakshnamurthy > wrote: > >> Hi all, &g

Help needed with Iceberg on Databricks platform

2023-04-28 Thread Pani Dhakshnamurthy
Hi all, I am currently attempting to make Iceberg work on the Databricks platform. Without adding IcebergSparkSessionExtensions, I am able to perform both data and metadata reads. However, when I add IcebergSparkSessionExtensions with spark.sql.extensions, Databricks throws the following error: C

Support relative paths in Table Metadata

2020-11-10 Thread Pani Dhakshnamurthy
Hi, We have raised an issue for relative paths in manifest files to address replication and back use cases. Am looking for thought and feedback on this. Additionally, we want to know how everyone is addressing these use cases. https://github.com/apache/iceberg/issues/1617 Thanks Pani