Shiva Prashanth Vallabhaneni would like to recall the message, "spark sql
in-clause problem".
Any comments or statements made in this email are not necessarily those of
Tavant Technologies. The information transmitted is intended only for the
person
uery;
Table coordinates (
Integer X,
Integer Y
)
sparkSqlContext.sql(select * from mytable where key = 1 and (X,Y) IN (select X,
Y from coordinates))
From: onmstester onmstester
Sent: Wednesday, May 23, 2018 10:33 AM
To: user
Subject: spark sql in-clause problem
I'm reading from this
I'm reading from this table in cassandra:
Table mytable (
Integer Key,
Integer X,
Interger Y
Using:
sparkSqlContext.sql(select * from mytable where key = 1 and (X,Y) in
((1,2),(3,4)))
Encountered error:
StructType(StructField((X,IntegerType,true),StructField((Y,IntegerType,true))
!=
Thanks for the pointer, Xiao.
I found that leftanti join type is no longer
in
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/joinTypes.scala
FYI
On Fri, Dec 4, 2015 at 12:04 PM, Xiao Li wrote:
> https://github.com/apache/spark/pull/9055
>
> This JIRA explains how to convert I
https://github.com/apache/spark/pull/9055
This JIRA explains how to convert IN to Joins.
Thanks,
Xiao Li
2015-12-04 11:27 GMT-08:00 Michael Armbrust :
> The best way to run this today is probably to manually convert the query
> into a join. I.e. create a dataframe that has all the numbers i
The best way to run this today is probably to manually convert the query
into a join. I.e. create a dataframe that has all the numbers in it, and
join/outer join it with the other table. This way you avoid parsing a
gigantic string.
On Fri, Dec 4, 2015 at 10:36 AM, Ted Yu wrote:
> Have you see
Have you seen this JIRA ?
[SPARK-8077] [SQL] Optimization for TreeNodes with large numbers of children
>From the numbers Michael published, 1 million numbers would still need 250
seconds to parse.
On Fri, Dec 4, 2015 at 10:14 AM, Madabhattula Rajesh Kumar <
mrajaf...@gmail.com> wrote:
> Hi,
>
>
Hi,
How to use/best practices "IN" clause in Spark SQL.
Use Case :- Read the table based on number. I have a List of numbers. For
example, 1million.
Regards,
Rajesh
