I am using Glue catalog. I figured out the issue here. When i used rowDelta
transaction and FanoutWriters the issue was resolved and data was available
in Athena.
Thanks for the input Ryan
On Thu, 30 Jun, 2022, 4:29 am Ryan Blue, <b...@tabular.io> wrote:
> Taher, what catalog are you using? You would need to use the same catalog
> that Athena is talking to.
>
> On Mon, Jun 27, 2022 at 10:55 PM Taher Koitawala <taher...@gmail.com>
> wrote:
>
>> Hi All,
>> I am trying to append data files to an existing iceberg table
>> in Athena. I wrote a Java app that uses the Iceberg apis to do this.
>> Attached below is a sample source code. I do see parquet files being placed
>> on s3 however I cannot read those files in Athena. Am I missing something?
>>
>> ```
>> //Use glueCatalog and loaded table
>> Table table =
>> catalog.loadTable(TableIdentifier.of(Namespace.of("tpc_ds"),"store_sales"));
>>
>> //created a file appender
>> FileAppender<Record> appender
>> =factory.newAppender(S3OutputFile.fromLocation(path,s3Client),
>> FileFormat.PARQUET);
>>
>> //Added records to appender
>> appender.add(record);
>>
>> //get len and close appender
>> long fileLen = appender.length();
>> appender.close();
>>
>> //get metrics
>> appender.metrics();
>>
>> //register datafiles with table
>> Transaction transaction = table.newTransaction();
>>
>> //Append data file
>> AppendFiles appendFiles = transaction.newAppend();
>> appendFiles.appendFile(DataFiles.builder(spec)
>> .withMetrics(metrics)
>> .withPath(location)
>> .withFormat(FileFormat.PARQUET)
>> .withRecordCount(metrics.recordCount())
>> .withFileSizeInBytes(fileLen).
>> build());
>>
>> //commit append
>> appendFiles.commit();
>>
>> //commit transaction
>> transaction.commitTransaction();
>>
>> --program end
>>
>> ```
>>
>> Regards,
>> Taher Koitawala
>>
>
>
> --
> Ryan Blue
> Tabular
>
