Hello,
I'm repeating slack messages here.
I'm experimenting with ingesting JDBC into Paquet. ie repeating
spotify/dbeam with
- JdbcIO.readRows()
- AvroUtils.getAvroSchema(beamRows.getSchema()).
- AvroUtils.schemaCoder(avroSchema)
- AvroUtils.getRowToGenericRecordFunction(avroSchema)
Here's the observed issues:
- DECIMAL(21,2) can't be handled due to loosing scale param (2).
org.apache.avro.Conversions.DecimalConversion.validate()
AvroTypeException("Cannot
encode decimal with scale 2 as scale 0 without rounding")
- it can be fixing Beam Row schema by FieldType.logicalType(
FixedPrecisionNumeric.of(Integer.MAX_VALUE, 2)) and then it should be
passed to AvroSchema as LogicalTypes.decimal(Integer.MAX_VALUE, ((
RowWithStorage)
(field.getType().getLogicalType()).getArgument()).getValue("scale"
)).addToSchema(Schema.create(Schema.Type.BYTES)) (it might not be the
best approach, you know) I noticed
https://github.com/apache/beam/issues/21226
https://github.com/apache/beam/issues/20978 which might be related.
- INT16 represented in beam schema as-is, but its 32-bit INT in avro
and java Short in runtime that causes
ClassCastException: class java.lang.Short cannot be cast to class
java.lang.Integer (java.lang.Short and java.lang.Integer are in module
java.base of loader 'bootstrap')
at
org.apache.beam.sdk.extensions.avro.schemas.utils.AvroUtils.convertAvroFieldStrict(AvroUtils.java:1299)
I suppose this method can accept Number and then call intValue() wdyt?
--
Sincerely yours
Mikhail Khludnev
