Li Yuanjian created SPARK-20408:
-----------------------------------
Summary: Get glob path in parallel to boost resolve relation time
Key: SPARK-20408
URL: https://issues.apache.org/jira/browse/SPARK-20408
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 2.1.0
Reporter: Li Yuanjian
Fix For: 2.2.0
The datasource read from a path with wildcard like below will cause a long time
waiting(each star may represent 100~1000 file or dir), especially in a cross
region env, driver and hdfs in different region, the drawback will enlarge.
bq. spark.read.text("/log/product/201704/\*/\*/\*/\*")
Optimize strategy is same with bulkListLeafFiles in InMemoryFileIndex, get the
wildcard path in parallel.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
