Re: Kafka for IoT ingestion pipeline

Hi Paolo, for use cases, high-level view you can read https://www.confluent.io/blog/iot-streaming-use-cases-with-kafka-mqtt-confluent-and-waterstream/ (disclaimer, this is marketing material from a vendor, but would be a good high level view of real cases). from your other question: > I'm wonde

Re: Monitoring Kafka

Hi Iirop, question: > If not, what is the best way to know how many messages with a specific key got inside a topic? why do require this information? is this a business metric? or are you looking at it from an operational point of view? I'm not aware of any metric that will give you the per-k

Re: Kafka metrics - Requests.

Hi David, these two metrics have two different objectives. > kafka.server:type=BrokerTopicMetrics,name=TotalProduceRequestsPerSec < Produce request rate per topic. Omitting 'topic=(...)' will yield the all-topic rate. * while >kafka.network:type=RequestMetrics,name=RequestsPerSec,re

Re: Introducing JulieOps, a tool to help you build your GitOps and self-service with Apache Kafka and related

has ever use it. -- Pere Missatge de Andrew Otto del dia dl., 1 de març 2021 a les 14:28: > This looks amazing, thanks Pere! I just filed a ticket > <https://phabricator.wikimedia.org/T276088> for Wikimedia to consider > adopting JulieOps. > > > > On Sat, Feb 27, 2021

Introducing JulieOps, a tool to help you build your GitOps and self-service with Apache Kafka and related

Hi, I hope this message finds everyone healthy and good. I wanted to reach out to the community to introduce JulieOps ( https://github.com/kafka-ops/julie). This is a project I have been working in recent times to help bring automation and self-service when using Apache Kafka. >From here I woul

Re: Latest Logstash 7.8 and compatibility with latest Kafka 2.5.0

HI Sebastian, I would check for that https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html, where it says: This plugin uses Kafka Client 2.1.0. For broker compatibility, see the official Kafka compatibility reference

Re: Apache Kafka training

Disclaimer: I work for Confluent, but have you consider the online or onsite trainings we provide? the team is really good at delivering and the content is really deep in details. -- Pere Missatge de M. Manna del dia dj., 26 de set. 2019 a les 13:54: > If you have access to Pluralsight, Apache

Re: Monitoring Broker/Prod/Cons on Kubernetes

Hi, personally speaking my usual monitoring of choice for k8s is Prometheus + Grafana, but end of the day most of the monitoring solutions do a good job, so I would mostly ask what do you know the most. For this environments I would be more interested on proper container management for kafka and

Re: Only use SSL to encrypt the authentication messages

n, 29 Sep 2019 at 10:38, Pere Urbón Bayes > wrote: > >> Hi, >> if you're worried about the performance impact i would suggest : >> >> * first benchmark the impacts, by experience i would see around 30 to 40% >> performance degradation. >> >&g

Re: Only use SSL to encrypt the authentication messages

Hi, if you're worried about the performance impact i would suggest : * first benchmark the impacts, by experience i would see around 30 to 40% performance degradation. * use java 11, and you will see a lot less performance impact when using ssl. -- Pere On Sun, 29 Sep 2019, 08:28 张祥 <18133622

Re: OOM for large messages with compression?

hu, Aug 22, 2019 at 12:49 AM Pere Urbón Bayes > wrote: > > > Hi, > > i agree with Liam, the OOM look to happen during produce time. I would > > look into that. > > > > But with your message size, i would recommend investigating to implement > > ClaimChec

Re: kafka security using ssl

2.5.29.37 Criticality=false > ExtendedKeyUsages [ > serverAuth > ] > > The certificate itself has the CA in the chain. > > > > On Thu, Aug 22, 2019 at 6:51 AM Pere Urbón Bayes > wrote: > >> can you share your certificate content somehow? i should ask, is it

Re: kafka security using ssl

nd the SERVER CERT. > > Sent from my iPhone > > > On Aug 22, 2019, at 6:44 AM, Pere Urbón Bayes > wrote: > > > > you should verify a proper chain of validation. is your private ca cert > in > > your trust store? > > > >> On Thu, 22 Aug 2019, 14:4

Re: kafka security using ssl

security_ssl > > The issue I am facing is when I used my internal CA. Not sure what I am > missing when I am creating the certificate. > > Thanks. > > Sent from my iPhone > > > On Aug 21, 2019, at 10:16 PM, Pere Urbón Bayes > wrote: > > > > Hi, > &g

Re: OOM for large messages with compression?

Hi, i agree with Liam, the OOM look to happen during produce time. I would look into that. But with your message size, i would recommend investigating to implement ClaimChecks. It will be much easier and reduce the avg message size. -- Pere On Thu, 22 Aug 2019, 01:12 Liam Clarke wrote: > Hi

Re: kafka security using ssl

Hi, the error looks like a missing configuration value. A good source of examples how to set up security can be found at https://github.com/purbon/kafka-security-playbook or https://docs.confluent.io/current/kafka/authentication_ssl.html. i would verify them and see if you're using the same con

Re: Doubts in Kafka

The good question, Aruna, is why would you like to do that? -- Pere Missatge de aruna ramachandran del dia dl., 15 de jul. 2019 a les 14:17: > Is there a way to get the Consumer to only read one message at a time and > commit the offset after processing a single message. > -- Pere Urbon-Baye

Re: Kafka Topic Volume and (possibly ACL) question

Hi, as Evelin has already said, I would recommend you to think about your topic / data modeling, from your email > We have a requirement where, based on business requirementes, we need to publish data only for a specific set of clients. For example, an invoice update shouldn't go to all clients,

Re: poll() question

rying to read all records > from a compacted topic on application startup and it was incorrect. > > Due to latency, threading GC pauses etc it would return 0 when there were > still records on the topic. > > Mark > > On Mon, 4 Feb 2019, 18:02 Pere Urbón Bayes >

Re: Kafka connect FieldPartitioner with scheduled rotation

ect.runtime.WorkerTask.doRun(WorkerTask.java:146) > at > org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:190) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(F

Re: poll() question

Hi, see from the javadocs: > timeout - The time, in milliseconds, spent waiting in poll if data is not available in the buffer. If 0, returns immediately with any records that are available currently in the buffer, else returns empty. Must not be negative. for a network problem it will return a

Re: enable.auto.commit semantics

Hi, from the confluent docs, > Offset Management: The two main settings affecting offset management are whether auto-commit is enabled and the offset reset policy. First, if you set enable.auto.commit (which is the default), then the consumer will automatically commit offsets periodically at the

Re: Kafka connect FieldPartitioner with scheduled rotation

Hi Manu, can you share your s3 connector config as well the exception you are getting? with only this info, I do need more details to understand your issue. Keep in mind the option you are using "rotate.schedule.interval.ms" from the docs says: > This configuration is useful when you have to com

Re: Monitoring Mirror Maker Lag & source

Hi, as otis have already shared this is accessible through the jmx metrics in Apache Kafka. You can find a comprehensive list from https://kafka.apache.org/documentation/#monitoring at the end of the day you can pull this into any monitoring solution that can pull from jmx, for example Jolokia+E

Re: Kafka tests on a remote cluster

Hi, if I understand your question properly you are aiming to validate failure scenarios. I usually see this done for learning purposes, basically to answer how Kafka would react to such and such situation? or to validate the current setup / configuration an Organisation have. For the first one, o

A Kafka cluster health api idea, does it makes sense?

Good night, After being for a long time working with Elasticsearch one gets used to it’s great REST api where you can check all sort of things from the server. As I usually get ask during my day life helping companies with their Kafka I decided to build an small Kafka Health API POC. This is a