[
https://issues.apache.org/jira/browse/SPARK-20408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Yuanjian updated SPARK-20408:
--------------------------------
Summary: Get glob path in parallel to reduce resolve relation time (was:
Get glob path in parallel to boost resolve relation time)
> Get glob path in parallel to reduce resolve relation time
> ---------------------------------------------------------
>
> Key: SPARK-20408
> URL: https://issues.apache.org/jira/browse/SPARK-20408
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.1.0
> Reporter: Li Yuanjian
> Fix For: 2.2.0
>
>
> The datasource read from a path with wildcard like below will cause a long
> time waiting(each star may represent 100~1000 file or dir), especially in a
> cross region env, driver and hdfs in different region, the drawback will
> enlarge.
> bq. spark.read.text("/log/product/201704/\*/\*/\*/\*")
> Optimize strategy is same with bulkListLeafFiles in InMemoryFileIndex, get
> the wildcard path in parallel.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
