Re: Only use SSL to encrypt the authentication messages

2019-09-29 Thread 张祥
So, is this a no ? | | | | | 签名由网易邮箱大师定制 On 09/29/2019 17:38,Pere Urbón Bayes wrote: Hi, if you're worried about the performance impact i would suggest : * first benchmark the impacts, by experience i would see around 30 to 40% performance degradation. * use java 11, and you will see a lo

Re: Broker regularly facing memory issues

2019-09-29 Thread Arpit Gogia
Thanks for the input. I do plan on reducing the heap for the next restart. Initially I'd thought that less heap memory could also be the reason but since then I've learned that cannot be the issue. I also found the number 262144 online but I believe those results were related to Elastic Search. O

Re: Idempotent Producers and Exactly Once Consumers

2019-09-29 Thread Matthias J. Sax
If you only enable "read_committed", yes. If you set `processing.guarantees="exactly_once"` the guarantee is stronger, because using a read-process-write pattern allows to couple writing result data and committing offsets into a single transaction (in contrast to a consumer-only scenario, for whic

Re: Idempotent Producers and Exactly Once Consumers

2019-09-29 Thread Jörn Franke
This is not only a configuration. Like in all messaging solutions you need to make sure that all involved applications ensure once and only once delivery from producer(s) over Kafka to consumer(s). > Am 30.08.2019 um 22:00 schrieb Peter Groesbeck : > > For a producer that emits messages to a

Re: One Partition missing a node in ISR

2019-09-29 Thread Sebastian Schmitz
Hello again, after like 15 minutes I have now this result: root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin# ./kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --zookeeper node1:2181 --reassignment-json-file move2.json --verify Status of partition reassignment: Reassignment of partit

Re: One Partition missing a node in ISR

2019-09-29 Thread Sebastian Schmitz
Hello, I just ran the kafka-reassign-partitions with --generate to create the json and then with --execute to run it. Now when checking with --verify I can see that the 4 partitions (it now changed from only one partitions not having all in ISR to 12 not being all in ISR) are successful, but t

Re: Idempotent Producers and Exactly Once Consumers

2019-09-29 Thread christopher palm
Does a Kafka Streams consumer also have that same limitation of possible duplicates? Thanks, Chris On Fri, Sep 27, 2019 at 11:56 AM Matthias J. Sax wrote: > Enabling "read_committed" only ensures that a consumer does not return > uncommitted data. > > However, on failure, a consumer might still

Re: Kafka Broker - High CPU

2019-09-29 Thread M. Manna
1.0.1 is very old version of Kafka. Is there any chance you would consider a rolling upgrade ? Regards, On Sun, 29 Sep 2019 at 22:43, Jamie wrote: > Have you got any producers using an older version of Kafka? Does the > broker with high CPU usage contain the leader of any topics which don't > h

Re: Kafka Broker - High CPU

2019-09-29 Thread Jamie
Have you got any producers using an older version of Kafka? Does the broker with high CPU usage contain the leader of any topics which don't have leaders on other brokers? Is the server.properties file the same across all brokers (minus broker.id etc)?  Many Thanks,  Jamie -Original Messa

Re: Monitoring Broker/Prod/Cons on Kubernetes

2019-09-29 Thread Otis Gospodnetić
Hi, I missed the original email, but I know Sematext can monitor your containerized/k8sized Kafka. Maybe of interest: https://sematext.com/blog/monitoring-kafka-with-sematext/ Otis On Sun, Sep 29, 2019 at 6:26 AM Pere Urbón Bayes wrote: > Hi, > personally speaking my usual monitoring of ch

Achiving at least once Delivery on top of Kafka

2019-09-29 Thread Isuru Boyagane
Hi, We are implementing a use case that needs tight at least once delivery. Even in the case of failures of nodes, no messages must be lost. We are trying to find out the least restrictive configurations that can give us at least once delivery. Following is what we found. - If we use ack

Re: Monitoring Broker/Prod/Cons on Kubernetes

2019-09-29 Thread Liam Clarke
I'm using the Prometheus jmx_exporter to export key metrics from the brokers. Takes a bit of fiddling to get the regexes right, but works well. On Fri, 27 Sep. 2019, 2:02 am M. Manna, wrote: > Hello, > > Has anyone got any experience in using monitoring tool (e.g. Prometheus, > DataDog, or custo

Re: Monitoring Broker/Prod/Cons on Kubernetes

2019-09-29 Thread M. Manna
Hey thanks for your response. I was thinking along the same lines. Cruise Control comes with good set of monitoring functions, but not ready for K8s. You would need to deploy that as a container (no official image yet). Do you know if any k8s flavour has been made available yet ? Prometheus/graf

Re: Apache Kafka training

2019-09-29 Thread Pere Urbón Bayes
Disclaimer: I work for Confluent, but have you consider the online or onsite trainings we provide? the team is really good at delivering and the content is really deep in details. -- Pere Missatge de M. Manna del dia dj., 26 de set. 2019 a les 13:54: > If you have access to Pluralsight, Apache

Re: Monitoring Broker/Prod/Cons on Kubernetes

2019-09-29 Thread Pere Urbón Bayes
Hi, personally speaking my usual monitoring of choice for k8s is Prometheus + Grafana, but end of the day most of the monitoring solutions do a good job, so I would mostly ask what do you know the most. For this environments I would be more interested on proper container management for kafka and

Re: Only use SSL to encrypt the authentication messages

2019-09-29 Thread Pere Urbón Bayes
not that I'm aware of, but my experience in the field is really true. it would be nice to write such article On Sun, 29 Sep 2019, 11:56 M. Manna wrote: > Hello, > > Just curious, is there any article about Java 11 reducing SSL perf impact > ? > > Thanks, > > On Sun, 29 Sep 2019 at 10:38, Pere Ur

Re: Only use SSL to encrypt the authentication messages

2019-09-29 Thread M. Manna
Hello, Just curious, is there any article about Java 11 reducing SSL perf impact ? Thanks, On Sun, 29 Sep 2019 at 10:38, Pere Urbón Bayes wrote: > Hi, > if you're worried about the performance impact i would suggest : > > * first benchmark the impacts, by experience i would see around 30 to

Re: Only use SSL to encrypt the authentication messages

2019-09-29 Thread Pere Urbón Bayes
Hi, if you're worried about the performance impact i would suggest : * first benchmark the impacts, by experience i would see around 30 to 40% performance degradation. * use java 11, and you will see a lot less performance impact when using ssl. -- Pere On Sun, 29 Sep 2019, 08:28 张祥 <18133622