Re: [DISCUSS] Support decimals with negative scale in decimal operation

Jörn, may you explain a bit more your proposal, please? We are not modifying the existing decimal datatype. This is how it works now. If you check the PR, the only difference is how we compute the result for the divsion operation. The discussion about precision and scale is about: shall we limit th

Re: [DISCUSS] Support decimals with negative scale in decimal operation

Maybe it is better to introduce a new datatype that supports negative scale, otherwise the migration and testing efforts for organizations running Spark application becomes too large. Of course the current decimal will be kept as it is. > Am 07.01.2019 um 15:08 schrieb Marco Gaido : > > In gen

Re: [DISCUSS] Support decimals with negative scale in decimal operation

Oracle does the same: "The *scale* must be less than or equal to the precision." (see https://docs.oracle.com/javadb/10.6.2.1/ref/rrefsqlj15260.html). Il giorno mer 9 gen 2019 alle ore 05:31 Wenchen Fan ha scritto: > Some more thoughts. If we support unlimited negative scale, why can't we > supp

Re: [DISCUSS] Support decimals with negative scale in decimal operation

Some more thoughts. If we support unlimited negative scale, why can't we support unlimited positive scale? e.g. 0.0001 can be decimal(1, 4) instead of (4, 4). I think we need more references here: how other databases deal with decimal type and parse decimal literals? On Mon, Jan 7, 2019 at 10:36 P

Re: [DISCUSS] Support decimals with negative scale in decimal operation

I'm OK with it, i.e. fail the write if there are negative-scale decimals (we need to document it though). We can improve it later in data source v2. On Mon, Jan 7, 2019 at 10:09 PM Marco Gaido wrote: > In general we can say that some datasources allow them, others fail. At > the moment, we are d

Re: [DISCUSS] Support decimals with negative scale in decimal operation

In general we can say that some datasources allow them, others fail. At the moment, we are doing no casting before writing (so we can state so in the doc). But since there is ongoing discussion for DSv2, we can maybe add a flag/interface there for "negative scale intollerant" DS and try and cast be

Re: [DISCUSS] Support decimals with negative scale in decimal operation

AFAIK parquet spec says decimal scale can't be negative. If we want to officially support negative-scale decimal, we should clearly define the behavior when writing negative-scale decimals to parquet and other data sources. The most straightforward way is to fail for this case, but maybe we can do

Re: [DISCUSS] Support decimals with negative scale in decimal operation

Hi Wenchen, thanks for your email. I agree adding doc for decimal type, but I am not sure what you mean speaking of the behavior when writing: we are not performing any automatic casting before writing; if we want to do that, we need a design about it I think. I am not sure if it makes sense to s

Re: [DISCUSS] Support decimals with negative scale in decimal operation

I think we need to do this for backward compatibility, and according to the discussion in the doc, SQL standard allows negative scale. To do this, I think the PR should also include a doc for the decimal type, like the definition of precision and scale(this one

[DISCUSS] Support decimals with negative scale in decimal operation

Hi all, a bit more than one month ago, I sent a proposal for handling properly decimals with negative scales in our operations. This is a long standing problem in our codebase as we derived our rules from Hive and SQLServer where negative scales are forbidden, while in Spark they are not. The dis