Thanks Ryan for checking this out!
IcebergWritable wraps a Record to a Container, and a Writable, so that is
why I try to create a Record here.
The problem is that the metadata table scan returns a StructLike and I have
to match that with the metadata schema and then with the read schema.
I have seen in the doc that metadata table query is already working for
Spark and I was hoping to avoid the duplicated work, but could not find the
relevant part of the code yet.
Thanks Peter
On Tue, 20 Jul 2021, 02:05 Ryan Blue, <b...@tabular.io> wrote:
> Peter,
>
> The "data" tasks produce records using Iceberg's Record class and the
> internal representations. I believe that's what the existing Iceberg object
> inspectors use. Couldn't you just wrap this with an IcebergWritable and use
> the regular object inspectors?
>
> On Thu, Jul 15, 2021 at 8:53 AM Peter Vary <pv...@cloudera.com.invalid>
> wrote:
>
>> I have put together a somewhat working solution:
>>
>> case METADATA:
>> return (CloseableIterable) CloseableIterable.transform(((DataTask)
>> currentTask).rows(), row -> {
>> Record record = GenericRecord.create(readSchema);
>> List<Types.NestedField> tableFields = tableSchema.asStruct().fields();
>> for (int i = 0; i < row.size(); i++) {
>> Types.NestedField tableField = tableFields.get(i);
>> if (readSchema.findField(tableField.name()) != null) {
>> record.setField(tableField.name(), row.get(i,
>> tableField.type().typeId().javaClass()));
>> }
>> }
>> return record;
>> });
>>
>> Which is working only for int/long/string etc types and it has problems
>> with Long->OffsetDateTime conversion and friends.
>> I am almost sure that this should have an existing and better solution
>> already somewhere :)
>>
>> On Jul 15, 2021, at 15:57, Peter Vary <pv...@cloudera.com> wrote:
>>
>> Hi Team,
>>
>> I am working to enable running queries above metadata tables through Hive.
>> I was able to load the correct metadata table though the Catalogs, and I
>> created the TableScan, but I am stuck there ATM.
>>
>> What is the recommended way to get the Record-s for the Schema defined by
>> the MetadataTable using the Java API?
>> For data files we create our own readers, but I guess we already has some
>> better way to do that for metadata.
>>
>> Any pointers would be welcome.
>>
>> Thanks,
>> Peter
>>
>>
>>
>
> --
> Ryan Blue
> Tabular
>
