Hi, I received a response from AWS, this is an issue with EMR, and they are working on resolving the issue I believe.
Thanks and Regards, Gourav Sengupta On Mon, Aug 23, 2021 at 1:35 PM Gourav Sengupta < gourav.sengupta.develo...@gmail.com> wrote: > Hi, > > the query still gives the same error if we write "SELECT * FROM table_name > WHERE data_partition > CURRENT_DATE() - INTERVAL 10 DAYS". > > Also the queries work fine in SPARK 3.0.x, or in EMR 6.2.0. > > > Thanks and Regards, > Gourav Sengupta > > On Mon, Aug 23, 2021 at 1:16 PM Sean Owen <sro...@gmail.com> wrote: > >> Date handling was tightened up in Spark 3. I think you need to compare to >> a date literal, not a string literal. >> >> On Mon, Aug 23, 2021 at 5:12 AM Gourav Sengupta < >> gourav.sengupta.develo...@gmail.com> wrote: >> >>> Hi, >>> >>> while I am running in EMR 6.3.0 (SPARK 3.1.1) a simple query as "SELECT >>> * FROM <table_name> WHERE <date parition field> > '2021-03-01'" the query >>> is failing with error: >>> >>> --------------------------------------------------------------------------- >>> pyspark.sql.utils.AnalysisException: >>> org.apache.hadoop.hive.metastore.api.InvalidObjectException: Unsupported >>> expression '2021 - 03 - 01' (Service: AWSGlue; Status Code: 400; Error >>> Code: InvalidInputException; Request ID: >>> dd3549c2-2eeb-4616-8dc5-5887ba43dd22; Proxy: null) >>> >>> --------------------------------------------------------------------------- >>> >>> The above query works fine in all previous versions of SPARK. >>> >>> Is this the expected behaviour in SPARK 3.1.1? If so can someone please >>> let me know how to write this query. >>> >>> Also if this is the expected behaviour I think that a lot of users will >>> have to make these changes in their existing code making transition to >>> SPARK 3.1.1 expensive I think. >>> >>> Regards, >>> Gourav Sengupta >>> >>
