GitHub user Leolh opened a pull request:
https://github.com/apache/spark/pull/2132
[SPARK-3228][Streaming]
When I use DStream to save files to hdfs, it will create a directory and a
empty file named "_SUCCESS" for each job which made in the batch duration.
But if there are no data from source for a long time , and the duration is
very short(e.g. 10s), it will create so many directory and empty files in hdfs.
I don't think it is necessary. So I want to modify class DStream's method
saveAsObjectFiles and saveAsTextFiles , it creates directory and files just
when the RDD's partitions size > 0 .
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Leolh/spark spark-streaming
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2132.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2132
----
commit 7eca5c7b323f4ba0e83355e22d0508cfb9381880
Author: leo <[email protected]>
Date: 2014-08-26T07:14:13Z
When DStream save RDD to hdfs , don't create directory and empty file if
there are no data received from source in the batch duration .
commit 35678d22a97c059e319b2fe53be69c989a855674
Author: leo <[email protected]>
Date: 2014-08-26T07:23:17Z
modify the code format
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
