thank you ! which java-client api can has same effect about kafka-consumer-groups.sh command ?

2020-04-21 Thread ????????
./kafka-consumer-groups.sh --bootstrap-server localhost:9081 --describe --group test use describeConsumerGroups method ?? private static void print() throws InterruptedException, ExecutionException { Properties props = new Properties(); props.setPropert

Re: Unknown topic at org.apache.kafka.streams.TopologyTestDriver.pipeRecord

2020-04-21 Thread Murilo Tavares
If topic is "users", do you have any idea where the name on the exception comes from? The exception says "Unknown topic: identity_users_v1". On Tue, 21 Apr 2020 at 06:08, Dumitru-Nicolae Marasoui < nicolae.maras...@ovoenergy.com> wrote: > Hi Liam, > It is "users". > Thanks > > On Tue, 21 Apr 202

Re: Retrieve Kafka Principal from Client

2020-04-21 Thread Newton, Brandt (CAI - Burlington)
Hi Dave, Thank you for the quick reply! I do set the sasl.jaas.config property to `org.apache.kafka.common.security.plain.PlainLoginModule required username="{{ CLUSTER_API_KEY }}" password="{{ CLUSTER_API_SECRET }}";` Perhaps I'm not using the correct terms here but I know CLUSTER_API_KEY

Re: Deserialization Exception when performing state store operations

2020-04-21 Thread Carl Graving
Yeah, avro was an example of how it could happen. More accurate would be a schema accidentally removed somehow. Basically the serializers worked to write to state store, deserialize throws exception. Of course it is a corner case, was just trying to see if there was a better way to handle the bad m

Re: Deserialization Exception when performing state store operations

2020-04-21 Thread Carl Graving
I have handled it everywhere, just is not seeming possible from the state store get. I can wrap in a try catch, but with get all it is seeming hard to skip these or remove them. Thanks On Tue, Apr 21, 2020, 01:38 Ezequiel Puig wrote: > Hello Carl, > > I think that the problem you are facing is

Re: Unknown topic at org.apache.kafka.streams.TopologyTestDriver.pipeRecord

2020-04-21 Thread Dumitru-Nicolae Marasoui
Hi Liam, It is "users". Thanks On Tue, 21 Apr 2020 at 10:23, Liam Clarke-Hutchinson < liam.cla...@adscale.co.nz> wrote: > Hi Nicu, > > I'd need to see more context to help - for example, what is the value of > `topicName`? I've just finished writing Streams tests using the test > driver, so can h

Re: Unknown topic at org.apache.kafka.streams.TopologyTestDriver.pipeRecord

2020-04-21 Thread Liam Clarke-Hutchinson
Hi Nicu, I'd need to see more context to help - for example, what is the value of `topicName`? I've just finished writing Streams tests using the test driver, so can hopefully help with more code :) Cheers, Liam Clarke-Hutchinson On Tue, Apr 21, 2020 at 8:40 PM Dumitru-Nicolae Marasoui < nicola

Re: Help with setting up Kafka Node

2020-04-21 Thread Liam Clarke-Hutchinson
Hi Sebastian, Killing implies that something external is killing the process. Kafka's default min heap size is 1G, you may be running under an environment that limits processes to less than this and thus invoking the Linux OOMKiller or similar, what is the output of `free -m` and `ulimit -a` on th

Re: Unknown topic at org.apache.kafka.streams.TopologyTestDriver.pipeRecord

2020-04-21 Thread Dumitru-Nicolae Marasoui
Hi Murilo & community, Thanks for your answer, I see that in the code this is done just before: val topicInput = inner.createInputTopic(topicName, serdeKey.serializer(), serdeValue.serializer()) val input = records.map { case (k, v) => new TestRecord(k, v) } topicInput.pipeRecordList(input.asJava)

Re: Kafka Streams - issues with windowing and suppress

2020-04-21 Thread Liam Clarke-Hutchinson
Hi John, Yep, I saw there was a few issues filed around default grace periods, and I have a few ideas about sensible defaults and possible APIs. I'll sign up for a Jira account in the next few days to join the discussion :) Kind regards, Liam Clarke-Hutchinson On Tue, Apr 21, 2020 at 9:09 AM Jo

Re: kafka-consumer-groups.sh CURRENT-OFFSET column show "-" , what mean about it ! thank you !

2020-04-21 Thread Liam Clarke-Hutchinson
Hi, It means the consumer group exists, and Kafka's aware of it's topic subscriptions, but either the consumers haven't consumed yet (if using auto offset commit) or the consumers haven't committed any offsets, if using manual offset committing. Does that make sense? Kind regards, Liam Clarke-

Re: kafka-consumer-groups.sh CURRENT-OFFSET column show "-" , what mean about it ! thank you !

2020-04-21 Thread Karolis Pocius
It means no consumer has consumed anything from that partition. Likely because there's no data in that partition yet. On Tue, Apr 21, 2020 at 8:12 PM 一直以来 <279377...@qq.com> wrote: > ghy@ghy-VirtualBox:~/T/k/bin$ ./kafka-consumer-groups.sh > --bootstrap-server localhost:9081 --describe --group te

kafka-consumer-groups.sh CURRENT-OFFSET column show "-" , what mean about it ! thank you !

2020-04-21 Thread ????????
ghy@ghy-VirtualBox:~/T/k/bin$ ./kafka-consumer-groups.sh --bootstrap-server localhost:9081 --describe --group test GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                          HOST            CLIENT-ID test 

AW: Help with setting up Kafka Node

2020-04-21 Thread info
Zookepper: /home/kafka/kafka/bin/zookeeper-server-start.sh /home/kafka/kafka/config/zookeeper.properties Kafka: /home/kafka/kafka/bin/kafka-server-start.sh /home/kafka/kafka/config/server.properties As described here: https://www.digitalocean.com/community/tutorials/how-to-install-apache-kafka-