[
https://issues.apache.org/jira/browse/SPARK-19553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15864326#comment-15864326
]
Michael Armbrust commented on SPARK-19553:
------------------------------------------
It seems like there are a couple of distinct feature requests here:
1) A different implementation of countApprox that that is based on scheduling
timeouts instead of sketching to find distinct items.
2) A short hand on GroupedData that could be used to call the aforementioned
function.
3) Better docs for the existing approx count distinct function. [Perhaps
copied from
here|https://github.com/apache/spark/blob/7a7ce272fe9a703f58b0180a9d2001ecb5c4b8db/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L235]
I'm skeptical of the utility of #1, but could be convinced with performance for
real work tasks (you can call {{df.rdd.countApprox()}} today). #3 seems like
we should obviously do it.
> Add GroupedData.countApprox()
> -----------------------------
>
> Key: SPARK-19553
> URL: https://issues.apache.org/jira/browse/SPARK-19553
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.1.0
> Reporter: Nicholas Chammas
> Priority: Minor
>
> We already have a
> [{{pyspark.sql.functions.approx_count_distinct()}}|http://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.functions.approx_count_distinct]
> that can be applied to grouped data, but it seems odd that you can't just
> get regular approximate count for grouped data.
> I imagine the API would mirror that for
> [{{RDD.countApprox()}}|http://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.RDD.countApprox],
> but I'm not sure:
> {code}
> (df
> .groupBy('col1')
> .countApprox(timeout=300, confidence=0.95)
> .show())
> {code}
> Or, if we want to mirror the {{approx_count_distinct()}} function, we can do
> that too. I'd want to understand why that function doesn't take a timeout or
> confidence parameter, though. Also, what does {{rsd}} mean? It's not
> documented.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
