Nick Allen created METRON-1551:
----------------------------------
Summary: Profiler Should Not Use Java Serialization
Key: METRON-1551
URL: https://issues.apache.org/jira/browse/METRON-1551
Project: Metron
Issue Type: Bug
Reporter: Nick Allen
When running the Profiler in a topology where serialization occurs, the
following error happens. This can occur when the number of workers is greater
than 1.
The topology should not be using Java serialization for serializing tuple
values as this will negatively impact performance.
{code}
2018-05-09 10:48:35.136 o.a.s.d.executor [ERROR]
java.lang.RuntimeException: java.lang.RuntimeException:
java.io.NotSerializableException:
org.apache.metron.common.configuration.profiler.ProfileResult
at
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:485)
~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at
org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at
org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at
org.apache.storm.disruptor$consume_loop_STAR_$fn__7183.invoke(disruptor.clj:83)
~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484)
[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: java.lang.RuntimeException: java.io.NotSerializableException:
org.apache.metron.common.configuration.profiler.ProfileResult
at
org.apache.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:41)
~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628)
~[kryo-3.0.3.jar:?]
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100)
~[kryo-3.0.3.jar:?]
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40)
~[kryo-3.0.3.jar:?]
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:534)
~[kryo-3.0.3.jar:?]
at
org.apache.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:44)
~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at
org.apache.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:44)
~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at
org.apache.storm.daemon.worker$mk_transfer_fn$transfer_fn__7805.invoke(worker.clj:193)
~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at
org.apache.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__7430.invoke(executor.clj:309)
~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at
org.apache.storm.disruptor$clojure_handler$reify__7166.onEvent(disruptor.clj:40)
~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
... 6 more
Caused by: java.io.NotSerializableException:
org.apache.metron.common.configuration.profiler.ProfileResult
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
~[?:1.8.0_162]
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
~[?:1.8.0_162]
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
~[?:1.8.0_162]
at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
~[?:1.8.0_162]
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
~[?:1.8.0_162]
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
~[?:1.8.0_162]
at
org.apache.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:38)
~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628)
~[kryo-3.0.3.jar:?]
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100)
~[kryo-3.0.3.jar:?]
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40)
~[kryo-3.0.3.jar:?]
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:534)
~[kryo-3.0.3.jar:?]
at
org.apache.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:44)
~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at
org.apache.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:44)
~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at
org.apache.storm.daemon.worker$mk_transfer_fn$transfer_fn__7805.invoke(worker.clj:193)
~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at
org.apache.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__7430.invoke(executor.clj:309)
~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at
org.apache.storm.disruptor$clojure_handler$reify__7166.onEvent(disruptor.clj:40)
~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
... 6 more
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
