Correct. You should just need to set the warehouse path so that the new table is created in S3. You can also create a table with a specific path.
On Wed, Jun 24, 2020 at 11:39 AM Chen Song <chen.song...@gmail.com> wrote: > Thanks Ryan/Edgar. > > I understand that I need to create a table via HiveCatalog. I already > successfully set up a hive megastore for this and successfully created an > Iceberg with that. I meant to create an Iceberg backed by data files > created on S3. Sorry if my question was unclear. It seems like that these > are already supported out of the box? > Is it just a matter to change hive warehouse to a path like `s3://...`? > > Chen > > On Wed, Jun 24, 2020 at 2:00 PM Ryan Blue <rb...@netflix.com.invalid> > wrote: > >> Chen, >> >> We have a getting started page that covers creating a table using the >> HiveCatalog: https://iceberg.apache.org/getting-started/#creating-a-table >> >> I agree with Edgar that you probably want to start with a HiveCatalog >> table. By default, tables are created under your warehouse path. If that's >> an S3 location, then it will use the S3 file system that is available. You >> can set the normal Hive warehouse config property to change the path. You >> can also specify the path directly by setting "location". >> >> One more thing to note about using the Hive catalog vs a table location: >> you _can_ use a table location without a metastore with S3. The trade-off >> is that concurrent writes will not be atomic. If you want to just try it >> out and don't need concurrent writes, then just using an S3 path is a >> simple way to get started. >> >> rb >> >> On Wed, Jun 24, 2020 at 10:36 AM Edgar Rodriguez >> <edgar.rodrig...@airbnb.com.invalid> wrote: >> >>> Hi Chen, >>> >>> Since S3 does not have atomic rename operation, for create/write/read of >>> tables in S3 currently the only way to do it is via the HiveCatalog >>> implementation which requires a Hive metastore with Lock support to provide >>> the atomic commit required by Iceberg. You can alternatively write your >>> custom Catalog implementation in which you set up your custom atomic commit >>> mechanism as shown in http://iceberg.apache.org/custom-catalog/. >>> >>> Cheers, >>> >>> On Wed, Jun 24, 2020 at 6:12 AM Chen Song <chen.song...@gmail.com> >>> wrote: >>> >>>> Hi >>>> >>>> Are there any Java examples to create/write/read tables backed by S3? I >>>> tried to search in the documentation and github but did not find anything. >>>> >>>> Thanks >>>> Chen >>>> >>>> >>> >>> -- >>> Edgar R >>> >> >> >> -- >> Ryan Blue >> Software Engineer >> Netflix >> > > > -- > Chen Song > > -- Ryan Blue Software Engineer Netflix
