[
https://issues.apache.org/jira/browse/SPARK-20408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15976505#comment-15976505
]
Apache Spark commented on SPARK-20408:
--------------------------------------
User 'xuanyuanking' has created a pull request for this issue:
https://github.com/apache/spark/pull/17702
> Get glob path in parallel to reduce resolve relation time
> ---------------------------------------------------------
>
> Key: SPARK-20408
> URL: https://issues.apache.org/jira/browse/SPARK-20408
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.1.0
> Reporter: Li Yuanjian
>
> The datasource read from a path with wildcard like below will cause a long
> time waiting(each star may represent 100~1000 file or dir), especially in a
> cross region env, driver and hdfs in different region, the drawback will
> enlarge.
> bq. spark.read.text("/log/product/201704/\*/\*/\*/\*")
> Optimize strategy is same with bulkListLeafFiles in InMemoryFileIndex, get
> the wildcard path in parallel.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
