[
https://issues.apache.org/jira/browse/SPARK-20294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15965579#comment-15965579
]
Sean Owen commented on SPARK-20294:
-----------------------------------
CC [~andrewor14] who might be able to comment better on this.
If there's no sampling, it tries the first element, then first 100, then fails
and asks you to try sampling.
If there's sampling, it just tries to operate on the sample.
Would it be more sensible to:
- Try the first element
- Try the first hundred
- Try a sample
> _inferSchema for RDDs fails if sample returns empty RDD
> -------------------------------------------------------
>
> Key: SPARK-20294
> URL: https://issues.apache.org/jira/browse/SPARK-20294
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 2.1.0
> Reporter: João Pedro Jericó
> Priority: Minor
>
> Currently the _inferSchema function on
> [session.py](https://github.com/apache/spark/blob/master/python/pyspark/sql/session.py#L354)
> line 354 fails if applied to an RDD for which the sample call returns an
> empty RDD. This is possible for example if one has a small RDD but that needs
> the schema to be inferred by more than one Row. For example:
> {code}
> small_rdd = sc.parallelize([(1, 2), (2, 'foo')])
> small_rdd.toDF(samplingRatio=0.01).show()
> {code}
> This will fail with high probability because when sampling the small_rdd with
> the .sample method it will return an empty RDD most of the time. However,
> this is not the desired result because we are able to sample at least 1% of
> the RDD.
> This is probably a problem with the other Spark APIs however I don't have the
> knowledge to look at the source code for other languages.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
