*Historically, the stringable represention of an BigDecimal was used*
Currently, if i use decimal logical type with json encoder; with this java
code :
String schemaStr =
"{\"type\":\"record\",\"name\":\"my_record\",\"doc\":\"doc\","
+ "\"fields\":[{\"name\":\"f\", "
+
"\"type\":{\"type\":\"bytes\",\"logicalType\":\"decimal\",\"precision\":9,\"scale\":2}},"
+ "{\"name\":\"f2\", "
+ "\"type\":{\"type\":\"fixed\", \"name\":\"FIX1\", \"size\": 10,
\"logicalType\":\"decimal\",\"precision\":9,\"scale\":2}}"
+ "]}";
Schema schema = new Schema.Parser().parse(schemaStr);
GenericData.get().addLogicalTypeConversion(new Conversions.DecimalConversion());
GenericData.Record record = new GenericData.Record(schema);
record.put(0, new BigDecimal("117230.150"));
record.put(1, new BigDecimal("-230.150"));
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
JsonEncoder encoder = EncoderFactory.get().jsonEncoder(schema,
outputStream, true);
GenericDatumWriter<Object> writer = new GenericDatumWriter<>(schema);
writer.write(record, encoder);
encoder.flush();
System.out.println(outputStream.toString());
I get this result
{
"f" : "\u0000²á\u0007",
"f2" : "ÿÿÿÿÿÿÿÿ¦\u0019"
}
But not a stringable as "f" : "117230.15"
Unit test TestBigDecimalConversion
<https://github.com/apache/avro/blob/4289c51ab95af84d7fdcccca1fa44d7a1e54589c/lang/java/avro/src/test/java/org/apache/avro/TestBigDecimalConversion.java>
shows that it can convert a widely range of BigDecimal without error; same
test* fail with current decimal converter*, because once you chose scale
and precision, it can't be adapted to every number. So, for use case, when
you know roughly the range of values the field will take, it's ok with
current logical type as you can anticipate scale & precision. So, the new
type is adapted when you strictly have no idea of what would be scale and
precision.
For naming (currently "big-decimal"), i'm open to change. It's a Java &
Rust name as SDK of both language contains this type; C# has BigRational,
but with another logic, avro C# module contains "AvroDecimal" class with
error message that contains BigDecimal terms. May be we could still use
"decimal" name in schema definition and use one or the other if precision
or scale parameters are defined or not; but it could also be more confused.
Regards,
Christophe.
Le mer. 20 sept. 2023 à 19:10, Ryan Skraba <r...@skraba.com> a écrit :
> Hello! There's a couple things going on, but I think this new type
> needs to be better specified in in the documentation before merging!
>
> Specifically, how are the bytes constructed and why was this
> representation chosen? Is there a better, more neutral represention
> where the underlying type could still be useful in languages that
> don't support this logical type? I'd also be interested in finding a
> less "Java" name for arbitrary precision values than BigDecimal if
> possible!
>
> Historically, the stringable represention of an BigDecimal was used
> for these value. I think we'd have to make an argument that the
> binary representation is more compact and/or efficient than this.
>
> In my experience, there aren't many use cases that require an
> arbitrary precision per value (as opposed to decimal, which sets it
> for the schema), and it's usually because "we don't know the precision
> of our data yet". As an example, Google Cloud Spanner recommends
> using Strings when a predefined precision and scale is unsatisfactory
> for a column[1]. Is there a better way for us to help the use when
> they don't know the precision of data they expect to work with?
>
> If there's other use cases for arbitrary sized data, maybe we can
> align on a solution!
>
> All my best, Ryan
>
> [1]:
> https://cloud.google.com/spanner/docs/storing-numeric-data#recommendation_store_arbitrary_precision_numbers_as_strings
>
> On Wed, Sep 20, 2023 at 6:19 PM Oscar Westra van Holthe - Kind
> <os...@westravanholthe.nl> wrote:
> >
> > On Wed, 20 Sept 2023 at 15:53, Martin Grigorov <mgrigo...@apache.org>
> wrote:
> >
> > > On Mon, Sep 18, 2023 at 11:21 AM Christophe Le Saëc <
> chles...@gmail.com>
> > > wrote:
> > > > This JIRA ticket <https://issues.apache.org/jira/browse/AVRO-3779>
> would
> > > > introduce a new Big-Decimal logical type [...]
> > > >
> > > > For the moment, there is only one PR for Java and one for Rust.
> > > > So, can we introduce this new feature knowing there is no
> implementation
> > > > for other module (at least for the moment) ?
> > >
> > > I am OK with this approach !
> > > Do you think it would be a good idea to mark this new logical type as
> > > experimental in the specification ?
> > >
> >
> > Yes, I do.
> >
> > There are similarities with the "duration" logical type: that is
> available
> > as logical type in most (all?) implementations, but using actual values
> of
> > that type is (AFAIK) only possible in Rust. The Java codebase, for
> example,
> > cannot convert the Avro value.
> >
> > So in all honesty, I think marking it as experimental is not needed. But
> I
> > think we should, because marking new logical types as experimental allows
> > the option to abandon it later on.
> >
> >
> > Kind regards,
> > Oscar
> >
> > --
> >
> > ✉️ Oscar Westra van Holthe - Kind <os...@westravanholthe.nl>
>
