unsubscribe
- dev + user
Can you give more info about the query? Maybe a full explain()? Are you
using a datasource like JDBC? The API does not currently push down limits,
but the documentation talks about how you can use a query instead of a
table if that is what you are looking to do.
On Mon, Oct 24, 20
The rule SpecialLimits converted GlobalLimit / LocalLimit
to CollectLimitExec.
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala#L74-L75
Spark will not scan all the records based on your plan. CollectLimitExec
should behave as
Hi all,
Let me clarify the problem:
Suppose we have a simple table `A` with 100 000 000 records
Problem:
When we execute sql query ‘select * from A Limit 500`,
It scan through all 100 000 000 records.
Normal behaviour should be that once 500 records is found, engine stop scanning.
Detailed ob
Hi,
While reviewing SparkUI I found two artifacts -- appUIAddress +
appName (with the entire SparkUITab) -- that I believe are not needed
at all as they seem to introduce nothing.
Please have a look at https://github.com/apache/spark/pull/15603 and
let me know your thoughts.
I'd appreciate your
Hi, Liz,
CollectLimit means `Take the first `limit` elements and collect them to a
single partition.`
Thanks,
Xiao
2016-10-23 5:21 GMT-07:00 Ran Bai :
> Hi all,
>
> I found the runtime for query with or without “LIMIT” keyword is the same.
> We looked into it and found actually there is “Globa
Hi all,
I found the runtime for query with or without “LIMIT” keyword is the same. We
looked into it and found actually there is “GlobalLimit / LocalLimit” in
logical plan, however no relevant physical plan there. Is this a bug or
something else? Attached are the logical and physical plans when
