Hi Sung Gon, Have you ever registered protobuf classes with kryo[1]?
[1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/custom_serializers.html Best, Yun Tang ________________________________ From: Sung Gon Yi <skonmem...@mac.com> Sent: Thursday, July 16, 2020 23:00 To: user@flink.apache.org <user@flink.apache.org> Subject: UnsupportedOperatorException with TensorFlow on checkpointing Hi, Following codes have a UnsupportedOperatorException on checkpointing (every time). Could you suggest any solution? Example code: A.java ---------------------------------------------------------- public class A extends RichWindowFunction<Double, Double, String, GlobalWindow> { private transient MapState<String, B> state; @Override public void apply(String key, GlobalWindow window, Iterable<Double> elements, Collector<Double> out) { String k; ... B b = state.get(k); // error reported point (eg. 58th line) ... } … } B.java ----------------- ... import org.tensorflow.SavedModelBundle; import org.tensorflow.framework.*; public class B { ... private SavedModelBundle savedModel; private MetaGraph metaGraph; public void initialize(String path, ConfigProto config, String... tags) throws Exception { ... this.savedModel = SavedModelBundle.loader(path).withTags(tags).withConfigProto(config.toByteArray()).load(); this.metaGraph = MetaGraph.parseFrom(this.saveModel.metaGraphDef()); ... } } Error logs: ---------------------------------------------------------- 2020-07-07 15:55:30,144 INFO org.apache.flink.runtime.taskmanager.Task - Window(GlobalWindows(), CountTrigger, TimeCountEvictor, SequentialOperator) -> asset_score-evaluate -> Sink: Unnamed (1/1) (96fcb5ceb56bc95177b27697cc264edf) switched from RUNNING to FAILED. om.esotericsoftware.kryo.KryoException: java.lang.UnsupportedOperationException Serialization trace: field_ (com.google.protobuf.DescriptorProtos$DescriptorProto) proto (com.google.protobuf.Descriptors$Descriptor) messageType (com.google.protobuf.Descriptors$FieldDescriptor) fields (com.google.protobuf.Descriptors$Descriptor) messageTypes (com.google.protobuf.Descriptors$FileDescriptor) dependencies (com.google.protobuf.Descriptors$FileDescriptor) dependencies (com.google.protobuf.Descriptors$FileDescriptor) file (com.google.protobuf.Descriptors$FieldDescriptor) fields (com.google.protobuf.Descriptors$Descriptor) containingType (com.google.protobuf.Descriptors$Descriptor) descriptor (com.google.protobuf.MapEntry$Metadata) metadata (com.google.protobuf.MapEntry) defaultEntry (com.google.protobuf.MapField$ImmutableMessageConverter) converter (com.google.protobuf.MapField) deviceCount_ (org.tensorflow.framework.ConfigProto) … metaGraph_(org.tensorflow.framework.MetaGraphDef) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528) at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528) at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528) at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730) at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:368) at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:289) at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528) at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730) at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:368) at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:289) at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528) at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730) at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:368) at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:289) at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528) at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730) at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:368) at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:289) at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528) at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528) at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730) at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:368) at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:289) at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528) at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528) at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528) at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528) at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528) at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:657) at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.copy(KryoSerializer.java:262) at org.apache.flink.api.java.typeutils.runtime.TupleSerializer.copy(TupleSerializer.java:115) at org.apache.flink.api.java.typeutils.runtime.TupleSerializer.copy(TupleSerializer.java:37) at org.apache.flink.api.common.typeutils.base.MapSerializer.copy(MapSerializer.java:107) at org.apache.flink.api.common.typeutils.base.MapSerializer.copy(MapSerializer.java:43) at org.apache.flink.runtime.state.heap.CopyOnWriteStateMap.get(CopyOnWriteStateMap.java:297) at org.apache.flink.runtime.state.heap.StateTable.get(StateTable.java:244) at org.apache.flink.runtime.state.heap.StateTable.get(StateTable.java:138) at org.apache.flink.runtime.state.heap.HeapMapState.get(HeapMapState.java:85) at org.apache.flink.runtime.state.UserFacingMapState.get(UserFacingMapState.java:47) at A.apply (A.java:58) ... Sincerely, Sung Gon
