Hi.
Just an update, it is working when I use the Default HiveServer JDBC URL.
The error occurs when I use LLAP.
Regards,
Bernard
On Fri, Jul 5, 2019 at 10:40 AM Bernard Quizon <
bernard.qui...@cheetahdigital.com> wrote:
> Hi.
>
> So I created a GenericUDF that returns a map, it works fine on simple
> SELECT statements.
> *For example:*
> SELECT member_id, map_merge(src_map, dest_map, array('key1')) from
> test_table limit 100;
>
> But returns an error when I use it on JOINs, for example:
>
> SELECT
> cust100.map_merge(e.map_1, t.map_1, array('key1'))
> FROM test_table t
> INNER JOIN ext_test_table e
> ON t.id = e.id
>
> Please see stack trace below:
>
> Serialization trace:
>
> genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
>
> colExprMap (org.apache.hadoop.hive.ql.plan.SelectDesc)
>
> conf (org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator)
>
> childOperators
> (org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerStringOperator)
>
> childOperators (org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator)
>
> childOperators (org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator)
>
> childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
>
> aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
>
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:156)
>
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:133)
>
> at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670)
>
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClass(SerializationUtilities.java:185)
>
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:118)
>
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
>
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
>
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:180)
>
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:161)
>
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:39)
>
> at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
>
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:218)
>
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
>
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
>
> at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
>
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:218)
>
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
>
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
>
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
>
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:180)
>
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
>
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
>
> at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
>
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:218)
>
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
>
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
>
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
>
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:180)
>
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
>
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
>
> at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
>
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:218)
>
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
>
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
>
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
>
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:180)
>
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
>
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
>
> at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
>
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:218)
>
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
>
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
>
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
>
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:180)
>
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
>
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
>
> at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
>
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:218)
>
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
>
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
>
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
>
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:180)
>
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:161)
>
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:39)
>
> at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
>
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:218)
>
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
>
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
>
> at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:686)
>
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:210)
>
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities.deserializeObjectByKryo(SerializationUtilities.java:707)
>
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities.deserializePlan(SerializationUtilities.java:613)
>
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities.deserializePlan(SerializationUtilities.java:590)
>
> at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:468)
>
> ... 21 more
>
> Caused by: java.lang.ClassNotFoundException: com.test.hiveudf.MapMergeUdf
>
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>
> at java.lang.Class.forName0(Native Method)
>
> at java.lang.Class.forName(Class.java:348)
>
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:154)
>
> ... 84 more
>
> I also applied the suggestions from HIVE-7711, I added a
> `DoNothingSerializer` and used the annotation `DefaultSerializer` to set it
> in my MapMergeUdf.
> Something like:
>
> @DefaultSerializer(value = classOf[DoNothingSerializer])
> class MapMergeUdf extends GenericUDF {
>
> Would you guys know how to resolve this issue?
>
> Thanks,
> Bernard
>
