Looking at SparkV2Filters, I think that Spark’s Decimal just isn’t handled
when converting a literal. This should fix it:
diff --git
a/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java
b/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java
index 06cb1335aa..57b9d61e38 100644
---
a/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java
+++
b/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java
@@ -64,6 +64,7 @@ import org.apache.spark.sql.connector.expressions.filter.And;
import org.apache.spark.sql.connector.expressions.filter.Not;
import org.apache.spark.sql.connector.expressions.filter.Or;
import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.sql.types.Decimal;
import org.apache.spark.unsafe.types.UTF8String;
public class SparkV2Filters {
@@ -378,6 +379,8 @@ public class SparkV2Filters {
private static Object convertLiteral(Literal<?> literal) {
if (literal.value() instanceof UTF8String) {
return ((UTF8String) literal.value()).toString();
+ } else if (literal.value() instanceof Decimal) {
+ return ((Decimal) literal.value()).toJavaBigDecimal();
}
return literal.value();
}
I don’t see decimal support in SparkFilters either, so this may not be a
regression, just something that we’ve never supported.
On Thu, Sep 28, 2023 at 3:02 PM Anton Okolnychyi <aokolnyc...@apache.org>
wrote:
> Is there a local snippet to reproduce this log message? I'll check our
> unit tests in a bit.
>
> On 2023/09/28 21:38:03 Bryan Keller wrote:
> > One note, when running TPC-DS I noticed a log message about decimal
> pushdown that wasn't present in 1.3 and earlier, e.g. "Failed to check if
> i_current_price >= 0.99 can be pushed down: Cannot create expression
> literal from org.apache.spark.sql.types.Decimal: 0.99", though not sure if
> that is a regression or just a new warning.
> >
> > > On Sep 28, 2023, at 1:46 PM, Bryan Keller <brya...@gmail.com> wrote:
> > >
> > > +1 (non-binding)
> > >
> > > I ran the TPC-DS 3 TB benchmark with AWS EMR 6.13/Spark 3.4.1, Iceberg
> 1.4.0 RC1, and a REST catalog. The results were slightly slower than 1.3.1
> but within the margin of error.
> > >
> > > 1.3.1 load: 1671 sec
> > > 1.4.0 load: 1820 sec
> > > 1.3.1 query: 4961 sec
> > > 1.4.0 query: 5210 sec
> > >
> > > On Thu, Sep 28, 2023 at 10:09 AM Anton Okolnychyi <
> aokolnyc...@apache.org <mailto:aokolnyc...@apache.org>> wrote:
> > >> I'll start with +1 (binding).
> > >>
> > >> Signature: OK
> > >> Checksum: OK
> > >> RAT checks: OK
> > >>
> > >> I also tested our new write defaults in Spark 3.5 on a cluster. No
> local sort, no ridiculously tiny files. We still prefer parallelism by
> default so folks would have to manually tune the advisory partition size
> for the final write to get closer to the target file size (if that's the
> goal). We will offer some knobs to do that automatically in the future.
> > >>
> > >> - Anton
> > >>
> > >> On 2023/09/28 02:02:50 Anton Okolnychyi wrote:
> > >> > Hi Everyone,
> > >> >
> > >> > I propose that we release the following RC as the official Apache
> Iceberg 1.4.0 release.
> > >> >
> > >> > The commit ID is 8f37faa6a21e863551b17992370edc0f8706465d
> > >> > * This corresponds to the tag: apache-iceberg-1.4.0-rc1
> > >> > *
> https://github.com/apache/iceberg/commits/apache-iceberg-1.4.0-rc1
> > >> > *
> https://github.com/apache/iceberg/tree/8f37faa6a21e863551b17992370edc0f8706465d
> > >> >
> > >> > The release tarball, signature, and checksums are here:
> > >> > *
> https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-1.4.0-rc1
> > >> >
> > >> > You can find the KEYS file here:
> > >> > * https://dist.apache.org/repos/dist/dev/iceberg/KEYS
> > >> >
> > >> > Convenience binary artifacts are staged on Nexus. The Maven
> repository URL is:
> > >> > *
> https://repository.apache.org/content/repositories/orgapacheiceberg-1145/
> > >> >
> > >> > Please download, verify, and test.
> > >> >
> > >> > Please vote in the next 72 hours. (Weekends excluded)
> > >> >
> > >> > [ ] +1 Release this as Apache Iceberg 1.4.0
> > >> > [ ] +0
> > >> > [ ] -1 Do not release this because...
> > >> >
> > >> > Only PMC members have binding votes, but other community members
> are encouraged to cast non-binding votes. This vote will pass if there are
> 3 binding +1 votes and more binding +1 votes than -1 votes.
> > >> >
> > >> > - Anton
> >
> >
>
--
Ryan Blue
Tabular
