Hi Xiao,
Just report this with JIRA SPARK-28103.
https://issues.apache.org/jira/browse/SPARK-28103
Thanks and Regards,
William
On Wed, 19 Jun 2019 at 1:35 AM, Xiao Li wrote:
> Hi, William,
>
> Thanks for reporting it. Could you open a JIRA?
>
> Cheers,
>
> Xiao
>
>
BTW, I noticed a workaround is creating a custom rule to remove 'empty
local relation' from a union table. However, I am not 100% sure if it is
the right approach.
On Tue, Jun 18, 2019 at 11:53 PM William Wong wrote:
> Dear all,
>
> I am not sure if it is something expected o
l("SELECT * FROM partitioned_table_1 t1, table2 t2 WHERE
t1.id = t2.id AND t1.id IN
('a','b','c','d')").queryExecution.optimizedPlan.children(0).constraints
res44: org.apache.spark.sql.catalyst.expressions.ExpressionSet =
Set(isnotnull(id#0), id#0 IN (
al#5] Batched: true,
Format: Parquet, Location:
InMemoryFileIndex[file:/Users/williamwong/spark-warehouse/table2],
PartitionFilters: [], *PushedFilters: [EqualTo(id,a), IsNotNull(id)],*
ReadSchema: struct
scala>
Thanks and regards,
William
On Sat, Jun 15, 2019 at 12:13 AM William Wong wrote:
Dear all,
I created two tables.
scala> spark.sql("CREATE TABLE IF NOT EXISTS table1(id string, val string)
USING PARQUET");
19/06/14 23:49:10 WARN ObjectStore: Version information not found in
metastore. hive.metastore.schema.verification is not enabled so recording
the schema version 1.2.0
19/06
