Re: Quotas to throttle the MBytes/sec for a kafka topic

Yes, we can throttle the clients based on byte rate threshold quotas. More details here: http://kafka.apache.org/documentation/#design_quotas http://kafka.apache.org/documentation.html#quotas On Thu, Oct 5, 2017 at 3:32 AM, Anu P wrote: > Hi All, > > I am new to using quotas for Kafka. > > Is i

Re: Kafka ACL - Access Denied while publishing the message

maybe you can list the created ACLs and cross-check the permissions. On Thu, Oct 5, 2017 at 9:51 AM, Ted Yu wrote: > From the example off: > https://cwiki.apache.org/confluence/display/KAFKA/ > Kafka+Authorization+Command+Line+Interface > > it seems following 'User:', the formation is te...@exam

Re: Kafka ACL - Access Denied while publishing the message

>From the example off: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Authorization+Command+Line+Interface it seems following 'User:', the formation is te...@example.com Can you double check ? On Wed, Oct 4, 2017 at 8:55 PM, Awadhesh Gupta wrote: > Hi, > > I am working on Kafka Author

Kafka ACL - Access Denied while publishing the message

Hi, I am working on Kafka Authorization using Kafka-ACL. I had enabled the aithorization configuring following in server.properties file authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer super.users=User:CN=pxrdc.myhost.com I executed following command to add ACL on topic test ./bin

Quotas to throttle the MBytes/sec for a kafka topic

Hi All, I am new to using quotas for Kafka. Is it possible to throttle data ingested to a particular Kafka topic based on ingested MBytes/sec? If yes, please let me know the steps. Appreciate any help. Regards Anu

Re: Correct way to increase replication factor for kafka-streams internal topics

That is hard to do... Just deleting the topic might result in data loss, if not all data was processed by the application yet (note, that repartitioning topics are also kind of a buffer between subtopologies). Just manually changing the number of partitions via kafka-topics.sh will break partitio

timeout value for consumer.poll

Hi all, In our implementation, instead of reading from kafka's automatic offsets everytime, we rely on the offsetsForTimes to get the offset for our interested timestamp and seek to it before we do our consumer.poll() For quite some time, we could comfortably use 100ms for consumer.poll timeout.

Re: Kafka stream api to consume json values using kafka connect json deserializer

Have you taken a look at streams/examples/src/main/java/org/apache/kafka/streams/examples/pageview/PageViewUntypedDemo.java ? final Deserializer jsonDeserializer = new JsonDeserializer(); final Serde jsonSerde = Serdes.serdeFrom(jsonSerializer, jsonDeserializer); final Co

Re: Kafka - leadership

Hi Tomer, 1. Yes, one broker can (will) be a leader for multiple partitions - you'll usually have a relatively small number of brokers in your cluster, but they will store data for many topics, with many partitions, so (assuming auto.leader.rebalance.enable is set to true; I think it's thedefault

Re: Kafka - leadership

One broker can be leader for many partitions. Partitions are not only used to scale the brokers but also to scale the clients. So in many cases it makes perfect sense to have many partitions running from the same broker. (Of course having only one broker in total means no HA etc., but I guess you u

Fwd: Kafka - leadership

and i have an additional question, can partion be without leader ? and if so, what does it mean ? -- Forwarded message -- From: Tomer Shavit Date: 2017-10-04 16:30 GMT+03:00 Subject: Kafka - leadership To: ‫users@kafka.apache.org‬ Hi, I have a question regarding kafka. if i hav

Kafka - leadership

Hi, I have a question regarding kafka. if i have cluster which contains only one broker, and i have topic which contains 2 partions. does it mean that the same broker will be the leader for those 2 partitions ? can broker be the leader for more than one partition ? Thanks, Tomer

Kafka stream api to consume json values using kafka connect json deserializer

Hi Everyone, I am a newbie to kafka and kafka streams api. I am trying to consume a json message using kafka connect api in kafka streams. I tried searching in google but i could not find any substantial information on how to read json message in streams api. Therefore, with the limited knowl

Configuration of Kafka Connect REST interface

Hi, I have some questions about the configuration of the Kafka Connect REST interface. 1) What are the configuration options rest.advertised.host.name and rest.advertised.port useful for? The documentation says "If this is set, this is the hostname that will be given out to other workers to conne

Correct way to increase replication factor for kafka-streams internal topics

Hi all, What is correct way to increase RF for existing internal topics that kafka-streams create (re-partitioning streams)? We are increasing RF for source topics and would like to align kafka-streams as well. App part configuration is simple, but what to do with existing internal topics? Remov

Fwd: TimeoutException: Timed out waiting for a node assignment. when AdminClient attempts to create topics

(Updated, and x-posted from dev@) Hi everyone, New user to Kafka, and am psyched to get it working in our environment! I'm on 0.11.0.0, and using spring-kafka 2.0.0.rc1. I'm struggling to get things working in our unit/integration tests. I don't want to overload everyone with the context of the

Re: out of order sequence number in exactly once streams

I suggest filing a JIRA with as much information as possible (logs, configs, code, etc.) so that we can try to reproduce it. Ismael On Wed, Oct 4, 2017 at 9:48 AM, Sameer Kumar wrote: > I am receiving this exception lot more frequently, also this occurs > generally at the start of starting the

Re: out of order sequence number in exactly once streams

I am receiving this exception lot more frequently, also this occurs generally at the start of starting the streams app. I was also running 3 seperate instances of my streams app all with different state stores on 3 different machines. I am using Kafka 11.0.1. -Sameer. On Tue, Oct 3, 2017 at 12:0