Re: creating partitions programmatically

overs. -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Sun, Feb 26, 2017 at 5:44 PM, VIVEK KUMAR MISHRA 13BIT0066 < vivekkumar.mishra2...@vit.ac.in> wrote: > My question is can we create partitions in topic using an

Re: Kafka Connect

Maybe look at this Kafka source connector for salesforce https://github.com/jcustenborder/kafka-connect-salesforce -hans Sent from my iPhone > On Feb 27, 2017, at 4:06 PM, VIVEK KUMAR MISHRA 13BIT0066 > wrote: > > Actually my data sources are salesforce and mailchimp. i have developed an > a

Re: kafka-connect-salesforce

Contact the author via github if the readme is not clear. -hans Sent from my iPhone > On Mar 1, 2017, at 7:01 AM, VIVEK KUMAR MISHRA 13BIT0066 > wrote: > > Hi All, > > I want to use kafka-connect-salesforce but i am not able to use it . > can any one provide steps to how to use it. > > Than

Re: Having 4 Node Kafka Cluster

A 4 node zookeeper ensemble will not even work. It MUST be an odd number of zookeeper nodes to start. For Kafka you can start with any number of nodes (including 4). Remember that it is the partitions that are replicated, not the entire broker so if you have a Kafka node crash, its only the par

Re: Having 4 Node Kafka Cluster

could fail. So it SHOULD be an odd number of zookeeper nodes (not MUST). -hans > On Mar 6, 2017, at 12:20 AM, Jens Rantil wrote: > > Hi Hans, > >> On Mon, Mar 6, 2017 at 12:10 AM, Hans Jespersen wrote: >> >> A 4 node zookeeper ensemble will not even work.

Re: Having 4 Node Kafka Cluster

ou think a stretch cluster will work? That seems wrong. -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Mon, Mar 6, 2017 at 5:37 AM, Le Cyberian wrote: > Hi Guys, > > Thank you very much for you reply. > > The

Re: Having 4 Node Kafka Cluster

aker / Replicator then i will not be able to use them at > the same time for writes./ producers. because the consumers / producers > will request from all of them > > BR, > > Lee > > On Mon, Mar 6, 2017 at 2:50 PM, Hans Jespersen wrote: > >> What do you mean when

Re: Performance and encryption

Its not a single message at a time that is encrypted with TLS its the entire network byte stream so a Kafka broker can’t even see the Kafka Protocol tunneled inside TLS unless it’s terminated at the broker. It is true that losing the zero copy optimization impacts performance somewhat but it

Re: Having 4 Node Kafka Cluster

;> * You can run any number of Kafka brokers on a ZK cluster. In your case >> this could be 4 Kafka brokers on 3 ZK nodes. >> * You should set topic replication to 2 (can be done at any time) and some >> other producer/broker settings to ensure your messages will not get lost

Re: Question on Metadata

You may also be interested to try out the new Confluent JMS client for Kafka. It implements the JMS 1.1. API along with all the JMS metadata fields and access methods. It does this by putting/getting the JMS metadata into the body of an underlying Kafka message which is defined with a special JM

Re: Kafka Retention Policy to Indefinite

ed in 0.10.1 -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Tue, Mar 14, 2017 at 10:09 AM, Joe San wrote: > Dear Kafka Users, > > What are the arguments against setting the retention plociy on a Kafka > topic to infi

Re: Kafka Retention Policy to Indefinite

latest value for each key forever, but removes the older messages with the same key in order to reduce the total about of messages stored. How much data do you expect to store in your largest topic over the life of the cluster? -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent

Re: Common Identity between brokers

This might be useful reading as it outlines why Cluster ID was added and lists a few ways that clusters can be identifies prior to that feature enhancement. https://cwiki.apache.org/confluence/display/KAFKA/KIP-78%3A+Cluster+Id

Re: Performance and Encryption

You are correct that a Kafka broker is not just writing to one file. Jay Kreps wrote a great blog post with lots of links to even greater detail on the topic of Kafka and disk write performance. Still a good read many years later. https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-

Re: kafka-topics[.sh]: fail to support connecting via broker / v0.10 style

. -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Fri, Mar 17, 2017 at 1:20 PM, Andrew Pennebaker < andrew.penneba...@gmail.com> wrote: > If I understand Kafka correctly, since v0.9 / v0.10, users are often > re

Re: Out of order message processing with Kafka Streams

sorry I mixed up Message A and B wrt the to question but the answer is still valid. -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Sat, Mar 18, 2017 at 11:07 AM, Hans Jespersen wrote: > The only way to make sure A is consu

Re: Out of order message processing with Kafka Streams

the added benefit of getting intermediary result at much lower latency. -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Sat, Mar 18, 2017 at 10:29 AM, Ali Akhtar wrote: > Is it possible to have Kafka Streams order messages corre

Re: Out of order message processing with Kafka Streams

. -hans > On Mar 18, 2017, at 11:36 AM, Sabarish Sasidharan > wrote: > > Hans > > What you state would work for aggregations, but not for state machines and > CEP. > > Regards > Sab > >> On 19 Mar 2017 12:01 a.m., "Hans Jespersen" wrote: >>

Re: validate identity of producer in each record

work. -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Mon, Mar 20, 2017 at 3:54 PM, Matt Magoffin wrote: > Hello, > > I am new to Kafka and am looking for a way for consumers to be able to > identify the producer of eac

Re: validate identity of producer in each record

afka itself on the receiving of records, before they are > stored/distributed? I was thinking along the lines of > > org.apache.kafka.clients.producer.ProducerInterceptor > > but on the server side? > > — m@ > >> On 21/03/2017, at 12:22 PM, Hans Jespersen wrote: &g

Re: Out of order message processing with Kafka Streams

r >> the >>>>>> proper >>>>>>>>> time/session window as if message B were consumed in the >>> timestamp >>>>>> order >>>>>>>>> before message A. >>>>>>>> >>>>>

Re: Out of order message processing with Kafka Streams

; > Are those 2 things possible w/ kafka streams? (java) > > On Tue, Mar 21, 2017 at 7:43 PM, Hans Jespersen wrote: > >> While it's not exactly the same as the window start/stop time you can >> store (in the state store) the earliest and latest timestamps of any &g

Re: kafka is not accepting number of partitions from configuration

The num.partitions parameter is a server/broker config but you are using it as a client/producer parameter so it will not work and will be ignored. http://stackoverflow.com/questions/22152269/how-to-specify-number-of-partitions-on-kafka-2-8 I assume the CLI command you are using is the administr

Re: kafka not throwing any exception not any response in call back

In your producer have you set acks to be “all” and retries to be something higher than “0”? props.put("acks", "all"); props.put("retries", 2147483647); Also if the order of the messages matter you should also set the max inflight messages to 1 so retries happen and succeed before any other mes

Re: How to assign client-id to a particular kafka producer or topic?

username comes from authenticated clients client.id can be assigned by any client (no authentication required). It’s hard to enforce a quota on a client.id when the clients can just change the code to use a difference client.id, hence the recent enhancement to add user quotas based on the authen

Re: Which is True? Kafka site vs Confluent 3.2 site upgrade doc details contradiction regarding 0.10.2 clients backward compatible to resp. 0.10.0 vs 0.10.1?

They are both true. The Apache text is talking about the compatibility of the Producer/Consumer API and the Confluent text is talking about the Streams API. -hans > On Mar 31, 2017, at 11:46 PM, Roger Vandusen > wrote: > > Read below and answer: So which is the source of truth ? > Is 0.10.2.0

Re: How to increase network throughput of Kafka cluster?

What replication factor are you using? If you have a default replication factor = 3 then a publish rate of 1.4 Gbps is actually 1.4 Gbps *3 = 4.2 Gbps of network traffic. If you are also consuming at the same time then it’s actually 4.2 Gbps + 1.4 Gbps = 5.6 Gbps. You would completely saturat

Re: How to increase network throughput of Kafka cluster?

Then you will need even more parallel producers to saturate a 10 GigE network (if you don't hit you disk I/O limit first) -hans > On Apr 1, 2017, at 3:15 PM, Archie wrote: > > My replication factor is 1. > > Thanks, > Archie > >> On Sat, Apr 1, 2017 a

Re: Kafka connector

nd N3 have different names. -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Thu, Apr 6, 2017 at 4:26 PM, Tushar Sudhakar Jee wrote: > Hello Sir/Ma'am, > I was trying to write a simple case of using kafka connector. My s

Re: Kafka producer and consumer within on sync execution

You posted the same question to Stack Overflow so I answered it there https://stackoverflow.com/questions/43302857/handling-sync-api-call-rest-spring-and-async-message-kafka-in-the-same-execu/43312070#43312070

Re: Kafka MTLS Support?

Are you asking about Multiplexed Transport Layer Security (MTLS) - https://en.wikipedia.org/wiki/Multiplexed_Transport_Layer_Security or Mutual TLS authentication (mTLS) https://en.wikipedia.org/wiki/Mutual_authentication

Re: ZK and Kafka failover testing

kafka cluster. -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Tue, Apr 18, 2017 at 4:10 PM, Shrikant Patel wrote: > Hi All, > > I am seeing strange behavior between ZK and Kafka. We have 5 node in ZK > and Kafka

Re: Re: ZK and Kafka failover testing

l.keystore.location= {appropriate value as per your case} > > ssl.keystore.password= {appropriate value as per your case} > > ssl.truststore.location= {appropriate value as per your case} > > ssl.truststore.password= {appropriate value as per your case} > > enable.auto.commit

Re: Re: Re: ZK and Kafka failover testing

VS fronting Kafka cluster} > >> > > key.serializer= {appropriate value as per your cases} > >> > > value.serializer= {appropriate value as per your case} acks= all > >> > > retries=3 > >> > > ssl.key.password= {appropriate value

Re: How does replication affect kafka quota?

Replication will not effect the users quota as it is done under a different replication quota (which you can control separately). The user should still see a 50 MBps maximum rate enforced into each broker. -hans > On Apr 23, 2017, at 11:39 PM, Archie wrote: > > So by specifying a kafka quo

Re: Does Kafka producer waits till previous batch returns responce before sending next one?

There is a parameter that controls this behavior called max.in. flight.requests.per.connection If you set max.in. flight.requests.per.connection = 1 then the producer waits until previous produce requests returns a response before sending the next one (or retrying). The retries parameter contro

Re: Does Kafka producer waits till previous batch returns responce before sending next one?

> 1, then even if the topic only have > one partition, there’s still no guarantee of the ordering? > > Thanks, > Jun > On Apr 30, 2017, at 7:57 AM, Hans Jespersen wrote: > > There is a parameter that controls this behavior called max.in. > flight.requests.per.con

Re: Why do I need to specify replication factor when creating a topic?

If you enable auto topic creation that that is exactly what will happen. There are pros and cons to creating topics with defaults values but if you fell strongly that is the way that you want Kafka to work it is entire possible to setup the system to work that way. -hans > On May 11, 2017,

Re: Kafka Read Data from All Partition Using Key or Timestamp

The timeindex was added in 0.10 so I think you need to use the new Consumer API to access this functionality. Specifically you should call offsetsForTimes() https://kafka.apache.org/0102/javadoc/org/apache/kafka/clients/consumer/Consumer.html#offsetsForTimes(java.util.Map) -hans > On May 25, 20

Re: 0.10.0.0 cluster : segments getting latest ts

together with log.roll.ms to avoid frequent log segment roll out. During the migration phase, if the first message in a segment does not have a timestamp, the log rolling will still be based on the (current time - create time of the segment)." -hans /** * Hans Jespersen, Principal Systems Engin

Re: 0.10.0.0 cluster : segments getting latest ts

en if it is expired, unless all the older segment has been expired." If none of the messages in a segment has a timestamp, last modified time will be used. -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Thu, May 25, 2017

Re: 0.10.0.0 cluster : segments getting latest ts

xforversion/12334962>>. I > even tried that. On QA hosts it retains TS for .log files across restart. > But when tried the new version on one of the prod host, same old story. > > So internal or File system ts, it should get deleted when expired. What > could be other rea

Re: Producer Async Issue

The producer is asynchronous (assuming you mean the Java Producer) https://kafka.apache.org/0102/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html -hans > On May 27, 2017, at 5:15 AM, Abhimanyu Nagrath > wrote: > > Hi, > I am using Kafka 0.10.2 single node cluster and I

Re: Producer Async Issue

hat further code is executed. What I am > looking for is that whether the broker is down or not it should not get > stuck. > > > > Regards, > Abhimanyu > >> On Sat, May 27, 2017 at 10:30 PM, Hans Jespersen wrote: >> >> The producer is asynchronou

Re: [E] Re: Kafka Configuration Question

ookeeper and kafka brokers to help in debugging further -hans -- /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ > On May 29, 2017, at 1:33 AM, Bennett, Conrad > wrote: > > Hello – anyone had a chance to take a look at this f

Re: Trouble with querying offsets when using new consumer groups API

It is definitely expected behavior that the new consumer version of kafka-consumer-groups.sh —describe only returns metadata for ‘active’ members. It will print an error message if the consumer group you provide has no active members. https://github.com/confluentinc/kafka/blob/trunk/core/src/ma

Re: Trouble with querying offsets when using new consumer groups API

I can confirm that in 0.10.2.1 I get offset information for disconnected consumers. The note in the output is a bit misleading because it also works with non-Java clients as long as they implement the new consumer. For example below is what I get when using the blizzard/node-rdkafka client which

Re: Java APIs for ZooKeeper related operations

Probably important to read and understand these enhancements coming in 0.11 https://cwiki.apache.org/confluence/display/KAFKA/KIP-117%3A+Add+a+public+AdminClient+API+for+Kafka+admin+operations -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924

Re: Java APIs for ZooKeeper related operations

Target is sometime in June. Apache Kafka releases are every 4 months so February, June, and October of each year -hans > On May 30, 2017, at 3:58 PM, Raghav wrote: > > Hans > > When will this version (0.11) be available ? > > On Tue, May 30, 2017 at 3:54 PM, Hans

Re: Data in kafka topic in Json format

Check which serializer you have configured in your producer. You are probably using an Avro serializer which will add the schema and modify the payload to avro data. You can use a String serializer or a ByteArray serializer and the data will either be Base64 encoded or not encoded at all. -han

Re: Data in kafka topic in Json format

My earlier comment still applies but in Kafka Connect the equivalent of a serializer/deserializer (serdes) is called a “converter”. Check which converter you have configured for your source connector and if it is overriding whatever the default converter is configured for the connect worker it

Re: Data in kafka topic in Json format

which has extra "\" and is not json any more? > > Best regards, > Mina > > On Fri, Jun 2, 2017 at 11:18 AM, Hans Jespersen wrote: > >> >> My earlier comment still applies but in Kafka Connect the equivalent of a >> serializer/deserializer (serdes) i

<    1   2