We haven't seen many of these, but we have seen it a couple of times --
there is ongoing work under SPARK-26089 to address the issue we know about,
namely that we don't detect corruption in large shuffle blocks.
Do you believe the cases you have match that -- does it appear to be
corruption in lar
I/We have seen this error before on 1.6 but ever since we upgraded to 2.1
two years ago we haven't seen it
On Tue, Mar 12, 2019 at 2:19 AM wangfei wrote:
> Hi all,
> Non-deterministic FAILED_TO_UNCOMPRESS(5) or ’Stream is corrupted’
> errors
> may occur during shuffle read, described as t
Hi all,
Non-deterministic FAILED_TO_UNCOMPRESS(5) or ’Stream is corrupted’ errors
may occur during shuffle read, described as this
JIRA(https://issues.apache.org/jira/browse/SPARK-4105).
There is not new comment for a long time in this JIRA. So, Is there
anyone seen these errors in
I am seeing the same issue with Spark 1.3.1.
I see this issue when reading sequence file stored in Sequence File format
(SEQorg.apache.hadoop.io.Textorg.apache.hadoop.io.Text'org.apache.hadoop.io.compress.GzipCodec?v?
)
All i do is
sc.sequenceFile(dwTable, classOf[Text], classOf[Text]).partitionB
I'm using the default settings.
Jianshi
On Wed, May 6, 2015 at 7:05 PM, twinkle sachdeva wrote:
> Hi,
>
> Can you please share your compression etc settings, which you are using.
>
> Thanks,
> Twinkle
>
> On Wed, May 6, 2015 at 4:15 PM, Jianshi Huang
> wrote:
>
>> I'm facing this error in Spar
Hi,
Can you please share your compression etc settings, which you are using.
Thanks,
Twinkle
On Wed, May 6, 2015 at 4:15 PM, Jianshi Huang
wrote:
> I'm facing this error in Spark 1.3.1
>
> https://issues.apache.org/jira/browse/SPARK-4105
>
> Anyone knows what's the workaround? Change the com
I'm facing this error in Spark 1.3.1
https://issues.apache.org/jira/browse/SPARK-4105
Anyone knows what's the workaround? Change the compression codec for
shuffle output?
--
Jianshi Huang
LinkedIn: jianshi
Twitter: @jshuang
Github & Blog: http://huangjs.github.com/
