Bump + adding one more issue I fixed (and by chance there's relevant report
in user mailing list recently)
* [SPARK-30462][SS] Streamline the logic on file stream source and sink to
avoid memory issue [1]
The patch stabilizes the driver's memory usage on utilizing a huge metadata
log, which was t
Bump again - hope to get some traction because these issues are either
long-standing problems or noticeable improvements (each PR has numbers/UI
graph to show the improvement).
Fixed long-standing problems:
* [SPARK-17604][SS] FileStreamSource: provide a new option to have
retention on input file
Worth noting that I got similar question around local community as well.
These reporters didn't encounter the edge-case, they're encountered the
critical issue in the normal running of streaming query.
On Fri, May 8, 2020 at 4:49 PM Jungtaek Lim
wrote:
> (bump to expose the discussion to more re
(bump to expose the discussion to more readers)
On Mon, May 4, 2020 at 5:45 PM Jungtaek Lim
wrote:
> Hi devs,
>
> I'm seeing more and more structured streaming end users encountered the
> metadata issues on file stream source and sink. They have been known-issues
> and there're even long-standin
Hi devs,
I'm seeing more and more structured streaming end users encountered the
metadata issues on file stream source and sink. They have been known-issues
and there're even long-standing JIRA issues reported before, end users
report them again in user@ mailing list in April.
* Spark Structure S
