Hi Smokeriu,
Please verify if there are conflicting netty jars by adding the below
configurations at the cluster level to print more logs to identify the
jars from which the class is loaded.
spark.executor.extraJavaOptions=-verbose:class
spark.driver.extraJavaOptions=-verbose:class
regards,
Guru
On Thu, Oct 31, 2024 at 1:57 PM Smokeriu <smoke...@gmail.com> wrote:
>
> I am trying to use Spark 3.5.3 in an old CDH cluster. The previous version on
> this CDH cluster of Spark was 3.3.2.
>
> When I submit a task to Yarn, netty prompts a conversion error:
>
>> Caused by: java.lang.ClassCastException:
>> io.netty.util.Recycler$DefaultHandle cannot be cast to
>> io.netty.util.Recycler$EnhancedHandle
>> at
>> io.netty.channel.ChannelOutboundBuffer$Entry.(ChannelOutboundBuffer.java:827)
>> at
>> io.netty.channel.ChannelOutboundBuffer$Entry.(ChannelOutboundBuffer.java:806)
>> at
>> io.netty.channel.ChannelOutboundBuffer$Entry$1.newObject(ChannelOutboundBuffer.java:810)
>> at
>> io.netty.channel.ChannelOutboundBuffer$Entry$1.newObject(ChannelOutboundBuffer.java:807)
>> at
>> io.netty.util.internal.ObjectPool$RecyclerObjectPool$1.newObject(ObjectPool.java:77)
>
>
> I checked the Netty version for Spark3.5.3 and it is 4.1.96, while my old CDH
> cluster is using version 4.1.17.
>
> But I checked the error reporting location code for both versions, and I
> found that the code in 4.1.17 converts DefaultHandle to EnhancedHandle as
> well.
>
> The code in netty io.netty.channel.ChannelOutboundBuffer.Entry is same:
>
>> private Entry(Handle<Entry> handle) {
>> this.handle = (EnhancedHandle<Entry>) handle;
>> }
>
>
> Any ideas on how to solve this problem please.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
