Hi, Stanislav, Could it be related to SAMZA-727? Your exception stack trace is different though. Besides that, Samza 0.10 requires minimum YARN 2.6.1 now, due to a token expiration issue in YARN 2.6.0 client lib.
-Yi On Wed, Jan 6, 2016 at 7:52 AM, Stanislav Los <stanis...@magnetic.com> wrote: > Hi, > > Any ideas why Samza can't start a container while running hello-samza on > CDH5.4? > > It runs fine on local grid. But when running on cluster I can see it talks > fine to YARN's ResourceManager and Kafka, get's resources, but fails to > start container. At first, I thought it's because CDH5.4 uses YARN 2.6.0, > while Samza uses 2.6.1, so I recompiled Samza with YARN 2.6.0 and > re-assambled hello-samza. No luck so far. Also, I thought it's because of > different protobuf version, but I checked, it's the same in CDH and Samza. > > Followed > http://samza.apache.org/learn/tutorials/0.10/deploy-samza-to-CDH.html to > deploy. > > See full logs: > Samza Deploy Script output http://pastebin.com/0K8HVMzt > YARN logs http://pastebin.com/UvNp6240 > > An error is: > > java version "1.7.0_80" > > Java(TM) SE Runtime Environment (build 1.7.0_80-b15) > > Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode) > > com.google.protobuf.InvalidProtocolBufferException: Protocol message > contained an invalid tag (zero). > > at > > com.google.protobuf.InvalidProtocolBufferException.invalidTag(InvalidProtocolBufferException.java:89) > > at com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:108) > > at > > org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto.<init>(YarnSecurityTokenProtos.java:1078) > > at > > org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto.<init>(YarnSecurityTokenProtos.java:1042) > > at > > org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto$1.parsePartialFrom(YarnSecurityTokenProtos.java:1138) > > at > > org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto$1.parsePartialFrom(YarnSecurityTokenProtos.java:1133) > > at > > com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200) > > at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217) > > at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223) > > at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49) > > at > > org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto.parseFrom(YarnSecurityTokenProtos.java:1307) > > at > > org.apache.hadoop.yarn.security.AMRMTokenIdentifier.readFields(AMRMTokenIdentifier.java:81) > > at org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:142) > > at > > org.apache.samza.job.yarn.ContainerUtil.startContainer(ContainerUtil.java:181) > > at > > org.apache.samza.job.yarn.ContainerUtil.runContainer(ContainerUtil.java:119) > > at > > org.apache.samza.job.yarn.ContainerAllocator.run(ContainerAllocator.java:65) > > at java.lang.Thread.run(Thread.java:745) > > Exception in thread "Container Allocator Thread" > org.apache.samza.SamzaException: IO Exception when writing credentials to > output buffer > > at > > org.apache.samza.job.yarn.ContainerUtil.startContainer(ContainerUtil.java:190) > > at > > org.apache.samza.job.yarn.ContainerUtil.runContainer(ContainerUtil.java:119) > > at > > org.apache.samza.job.yarn.ContainerAllocator.run(ContainerAllocator.java:65) > > at java.lang.Thread.run(Thread.java:745) > > > Regards, > Stan >
