---------- Forwarded message --------- From: Neeraj Koul <neeraj.k...@go-mmt.com> Date: Fri, May 3, 2019 at 3:45 PM Subject: Protobuf message parsing To: <druid-u...@googlegroups.com>
Our ingest task tries parse protobuf messages from a kafka topic. We couldn't get it done. Followed the steps as mentioned in " http://druid.io/docs/latest/development/extensions-core/protobuf.html", but that is not working out. Infact, we could read and parse those protobuf messages in our sample code on the same druid machine. but the ingestion task doesn't seem to get work. We are putting .desc file in the classpath and also parseSpec formagt is json, infact we have followed whatever is there in the url. Our sample protobuf message has just a couple of string fields, name and company, and we use proto3. The trace we get is like 19-05-02T12:33:54,751 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner - Encountered parse exception on row from partition[0] offset[0] org.apache.druid.java.util.common.parsers.ParseException: Unable to parse row [ Sam Google] at org.apache.druid.java.util.common.parsers.JSONPathParser.parseToMap(JSONPathParser.java:74) ~[java-util-0.13.0-incubating-iap11.jar:0.13.0-incubating-iap11] at org.apache.druid.data.input.impl.StringInputRowParser.parseString(StringInputRowParser.java:155) ~[druid-api-0.13.0-incubating-iap11.jar:0.13.0-incubating-iap11] at org.apache.druid.data.input.impl.StringInputRowParser.buildStringKeyMap(StringInputRowParser.java:119) ~[druid-api-0.13.0-incubating-iap11.jar:0.13.0-incubating-iap11] at org.apache.druid.data.input.impl.StringInputRowParser.parseBatch(StringInputRowParser.java:80) ~[druid-api-0.13.0-incubating-iap11.jar:0.13.0-incubating-iap11] at org.apache.druid.segment.transform.TransformingStringInputRowParser.parseBatch(TransformingStringInputRowParser.java:50) ~[druid-processing-0.13.0-incubating-iap11.jar:0.13.0-incubating-iap11] at org.apache.druid.segment.transform.TransformingStringInputRowParser.parseBatch(TransformingStringInputRowParser.java:31) ~[druid-processing-0.13.0-incubating-iap11.jar:0.13.0-incubating-iap11] at org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner.runInternal(IncrementalPublishingKafkaIndexTaskRunner.java:493) [druid-kafka-indexing-service-0.13.0-incubating-iap11.jar:0.13.0-incubating-iap11] at org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner.run(IncrementalPublishingKafkaIndexTaskRunner.java:232) [druid-kafka-indexing-service-0.13.0-incubating-iap11.jar:0.13.0-incubating-iap11] at org.apache.druid.indexing.kafka.KafkaIndexTask.run(KafkaIndexTask.java:210) [druid-kafka-indexing-service-0.13.0-incubating-iap11.jar:0.13.0-incubating-iap11] at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:422) [druid-indexing-service-0.13.0-incubating-iap11.jar:0.13.0-incubating-iap11] at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:394) [druid-indexing-service-0.13.0-incubating-iap11.jar:0.13.0-incubating-iap11] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_201] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_201] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_201] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201] Caused by: com.fasterxml.jackson.core.JsonParseException: Illegal character ((CTRL-CHAR, code 6)): only regular white space (\r, \n, \t) is allowed between tokens at [Source: Sam Google; line: 2, column: 2] at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1581) ~[jackson-core-2.6.7.jar:2.6.7] at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:533) ~[jackson-core-2.6.7.jar:2.6.7] at com.fasterxml.jackson.core.base.ParserMinimalBase._throwInvalidSpace(ParserMinimalBase.java:484) ~[jackson-core-2.6.7.jar:2.6.7] at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._skipWSOrEnd(ReaderBasedJsonParser.java:2056) ~[jackson-core-2.6.7.jar:2.6.7] at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:577) ~[jackson-core-2.6.7.jar:2.6.7] at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3776) ~[jackson-databind-2.6.7.jar:2.6.7] at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3721) ~[jackson-databind-2.6.7.jar:2.6.7] at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2726) ~[jackson-databind-2.6.7.jar:2.6.7] at org.apache.druid.java.util.common.parsers.JSONPathParser.parseToMap(JSONPathParser.java:70) ~[java-util-0.13.0-incubating-iap11.jar:0.13.0-incubating-iap11] ... 14 more What it seems like is that it is trying to parse proto as json.Would be great if you could point us to the problem that we have in our setup. Regards, Neeraj -- ::DISCLAIMER:: ---------------------------------------------------------------------------------------------------------------------------------------------------- This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from your system.