Re: Followup from iceberg newbie questions

I'm not sure what you're asking in question 1, but the generally correct way to write to an Iceberg table is to use the integration. Writing to pre-existing paths sounds like an unreliable idea to me. For question 2, my answer is also to use the existing integration. If you write your own files an

Re: Followup from iceberg newbie questions

Ryan, I agree with you about that part, could you please clarify (1) and (2) Sent from my iPhone > On Feb 10, 2021, at 8:55 AM, Ryan Blue wrote: > >  > You should always write to Iceberg using the supplied integration. That way > your data has metrics to make your queries faster and schemas

Re: Followup from iceberg newbie questions

You should always write to Iceberg using the supplied integration. That way your data has metrics to make your queries faster and schemas are written correctly. On Tue, Feb 9, 2021 at 6:24 PM kkishore iiith wrote: > Ryan, > > It would be nice to include that in iceberg website as the feature see

Re: Followup from iceberg newbie questions

Ryan, It would be nice to include that in iceberg website as the feature seems like a common ask. Our spark job needs to return the gcs filenames as the downstream service would load these gcs files into bigquery. So, we have two options here, could you please clarify for both (1) We write to ic

Re: Followup from iceberg newbie questions

Sorry, I was mistaken about this. We have exposed the incremental read functionality using DataFrame options . You should be able

Re: Followup from iceberg newbie questions

Replies inline. On Tue, Feb 9, 2021 at 5:36 PM kkishore iiith wrote: > *If a file system does not support atomic renames, then you should use a > metastore to track tables. You can use Hive, Nessie, or Glue. We also are > working on a JDBC catalog.* > > 1. What would go wrong if I write directly