I'm not 100% sure I understand the question. Assuming you're referring
"both" as SPARK-26283 [1] and SPARK-29322 [2], if you ask about the fix
then yes, only master branch as fix for SPARK-26283 is not ported back to
branch-2.4. If you ask about the issue (problem) then maybe no, according
to the a
Thank you for the investigation and making a fix.
So, both issues are on only master (3.0.0) branch?
Bests,
Dongjoon.
On Wed, Oct 2, 2019 at 00:06 Jungtaek Lim
wrote:
> FYI: patch submitted - https://github.com/apache/spark/pull/25996
>
> On Wed, Oct 2, 2019 at 3:25 PM Jungtaek Lim
> wrote:
FYI: patch submitted - https://github.com/apache/spark/pull/25996
On Wed, Oct 2, 2019 at 3:25 PM Jungtaek Lim
wrote:
> I need to do full manual test to make sure, but according to experiment
> (small UT) "closeFrameOnFlush" seems to work.
>
> There was relevant change on master branch SPARK-2628
I need to do full manual test to make sure, but according to experiment
(small UT) "closeFrameOnFlush" seems to work.
There was relevant change on master branch SPARK-26283 [1], and it changed
the way to read the zstd event log file to "continuous", which seems to
read open frame. With "closeFrame
The change log for zstd v1.4.3 feels me that the changes don't seem to be
related.
https://github.com/facebook/zstd/blob/dev/CHANGELOG#L1-L5
v1.4.3
bug: Fix Dictionary Compression Ratio Regression by @cyan4973 (#1709)
bug: Fix Buffer Overflow in v0.3 Decompression by @felixhandte (#1722)
build: A
Thank you for reporting, Jungtaek.
Can we try to upgrade it to the newer version first?
Since we are at 1.4.2, the newer version is 1.4.3.
Bests,
Dongjoon.
On Tue, Oct 1, 2019 at 9:18 PM Mridul Muralidharan wrote:
> Makes more sense to drop support for zstd assuming the fix is not
> somethi
Makes more sense to drop support for zstd assuming the fix is not
something at spark end (configuration, etc).
Does not make sense to try to detect deadlock in codec.
Regards,
Mridul
On Tue, Oct 1, 2019 at 8:39 PM Jungtaek Lim
wrote:
>
> Hi devs,
>
> I've discovered an issue with event logger, s
Hi devs,
I've discovered an issue with event logger, specifically reading incomplete
event log file which is compressed with 'zstd' - the reader thread got
stuck on reading that file.
This is very easy to reproduce: setting configuration as below
- spark.eventLog.enabled=true
- spark.eventLog.co
