2020-03-11 09:57:39 UTC - Konstantinos Papalias: I'm coming into Pulsar from a Kafka / Kafka Streams background, and have been trying to adopt Pulsar / Functions for similar tasks, the main hurdles I see for Pulsar adoption, are the following based on my limited exposure: 1. The benefit of Kafka is not age, is the wide adoption across many companies and industries which brings a huge community around it in regards to forums / presentations / discussions / tooling and documentation. 2. Kafka Streams is a really powerful tool to have in your arsenal that allows for advanced computations such as joins, aggregations, chaining functions, fan out, etc, while still running it as a library, no need to maintain a separate cluster (Spark / Flink) for simple enough computations. Whereas Pulsar Functions are still too limited in scope to be realistically practical. Or can be useful, but in a more limited scope. +1 : Eugen ---- 2020-03-11 12:17:04 UTC - Konstantinos Papalias: Hello, how can someone enforce Pulsar Functions to use an Avro Schema for the output topic ? As far as I can see the default behaviour is still suffering from this bug <https://github.com/apache/pulsar/issues/3741> , how can I configure Avro Functions with `allowNull = false` ? ---- 2020-03-11 14:14:51 UTC - Michael Kaufman: @Michael Kaufman has joined the channel ---- 2020-03-11 16:26:24 UTC - Sijie Guo: Is your POJO generated from AVRO? ---- 2020-03-11 16:27:06 UTC - Konstantinos Papalias: It's not I'm afraid, it's defined by hand ---- 2020-03-11 16:32:38 UTC - Konstantinos Papalias: So if I deploy the function where the output topic has not defined a Schema, the Auto-generated Schema would be Nullable, if I predefine the Schema for the Topic, it throws an exception with incompatible Schema: ```ERROR o.a.pulsar.functions.sink.PulsarSink - Failed to create Producer while doing user publish org.apache.pulsar.client.api.PulsarClientException$IncompatibleSchemaException: org.apache.pulsar.broker.service.schema.exceptions.IncompatibleSchemaException: Incompatible schema used at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:252) at org.apache.pulsar.client.impl.ProducerBuilderImpl.create(ProducerBuilderImpl.java:88) at org.apache.pulsar.functions.sink.PulsarSink$PulsarSinkProcessorBase.createProducer(PulsarSink.java:104) at org.apache.pulsar.functions.sink.PulsarSink$PulsarSinkAtMostOnceProcessor.<init>(PulsarSink.java:172) at org.apache.pulsar.functions.sink.PulsarSink$PulsarSinkAtLeastOnceProcessor.<init>(PulsarSink.java:194) at org.apache.pulsar.functions.sink.PulsarSink.open(PulsarSink.java:267) at org.apache.pulsar.functions.instance.JavaInstanceRunnable.setupOutput(JavaInstanceRunnable.java:788) at org.apache.pulsar.functions.instance.JavaInstanceRunnable.setupJavaInstance(JavaInstanceRunnable.java:213) at org.apache.pulsar.functions.instance.JavaInstanceRunnable.run(JavaInstanceRunnable.java:244) at java.base/java.lang.Thread.run(Thread.java:834)``` ---- 2020-03-11 16:49:20 UTC - Sijie Guo: I see. If it is hand-wrote POJO, I am afraid that we cannot disable `allowNull` at this moment. Can you create a github issue about this? ok_hand : Konstantinos Papalias ---- 2020-03-11 16:50:19 UTC - Konstantinos Papalias: Sure np, I was afraid this might have been the case, but do you believe it would have worked if the POJO was generated from Schema ? ----
