Re: Kafka SSL encryption plus external CA

Stephane, I believe that should work, though I haven't tried it myself. On Wed, Dec 21, 2016 at 12:11 AM, Stephane Maarek < steph...@simplemachines.com.au> wrote: > Thanks Rajini. > > I used a CNAME broker-bootstrap-A.example.com that round robins to the > actual brokers broker-1.example.com, br

spontaneous / unwanted exits from KStream apps

I upgraded my app(s) to 0.10.1.1-rc1 but I'm still seeing these error-free shutdowns. Something is taking my app down after varying lengths of time (10 minutes to several hours). Doesn't matter if Im running one or many instances. Suggestions on where to look? I've sent several debug logs.

Re: Error in Kafka Streams (that shouldn't be there)

Hi Eno, Just when I'm running. I think the root cause is losing a connection to zookeeper, I occasionally see: "Terminate ZkClient event thread." In the zookeeper logs I see something like: 2016-12-21 11:40:32,915 [myid:] - WARN [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] -

Re: spontaneous / unwanted exits from KStream apps

Found this treasure lurking in my app folder: Shows that the process was OOM-killed by the OS. Have restarted to see if it will reproduce - so far memory usage seems stable. On Wed, Dec 21, 2016 at 3:05 AM, Jon Yeargers wrote: > I upgraded my app(s) to 0.10.1.1-rc1 but I'm still seeing these

Re: Kafka SSL encryption plus external CA

run openssl on your cert and pipe to grep or awk to look for Host Alt-Names e.g. openssl x509 -in mycertfile.crt -text -noout \ -certopt no_subject,no_header,no_version,no_serial,no_signame,no_validity,no_subject,no_issuer,no_pubkey,no_sigdump,no_aux \| awk '/X509v3 Subject Alternative Name

Re: spontaneous / unwanted exits from KStream apps

Did I say memory usage was stable? Lies. After ~15min it's up to 50% - and climbing. 20 min: 63% Eventually the OS killed it. Didn't generate a log this time though. Found this snip in /var/log/messages: Dec 21 12:56:19 ip-172-16-101-108 kernel: [2901342.207241] java invoked oom-killer: gfp_mas

Unable to consume compressed topics

Hi guys, I'm trying to consume our Kafka topics using Kafkacat . We were able to successfully consume messages using the the default compression.type (producer, i.e without compression in our case). When we changed the compression type to either compression al

Re: Questions about single consumer per partition approach

Hi Alexi Typically you would use a key to guarantee that messages with the same key have a global ordering, rather than using manual assignment. Kafka will send all messages with the same key to the same partition. If you need global ordering, spanning all messages from a single producer, you can

Re: Halting because log truncation is not allowed for topic __consumer_offsets

Hi Peter, We’ve seen this happen under normal operation in our virtualized environment as well. Our network is not very stable, blips happen pretty frequently. Your explanation sounds reasonable to me, I’m very interested in your further thought on this. In our case, we’re using quorum based r

Unable to consume compressed topics

Hi guys, I'm trying to consume our Kafka topics using Kafkacat . We were able to successfully consume messages using the the default compression.type (producer, i.e without compression in our case). When we changed the compression type to either compression al

Time dependency for Kafka Sync

Hi Team, Can you please let us know what would be the accepted time lag between two kafka compoents servers. Thanks Umar Shaik This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are n

Re: Unable to consume compressed topics

Does it work if you use the console consumer from 0.10.1 and/or 0.10.0? Ismael On Wed, Dec 21, 2016 at 5:06 AM, Ofir Sharony wrote: > Hi guys, > > I'm trying to consume our Kafka topics using Kafkacat > . > We were able to successfully consume messages usin

Re: [VOTE] 0.10.1.1 RC1

Hi Guozhang, Thanks for driving this release as it includes a few important fixes. I belatedly tested Scala 2.12 binaries using the quickstart, +1 (non-binding) from me. Ismael On Thu, Dec 15, 2016 at 1:29 PM, Guozhang Wang wrote: > Hello Kafka users, developers and client-developers, > > Thi

Multi-topic consumer configuration

Hello, We are using a multi-topic approach for partitioning data. Each topic will follow a pattern naming convention. We want to design our consumer configuration so that a slow topic will not block a fast topic, as each topic will consume at a different rate. However I am seeing an issue where a

Re: [VOTE] 0.10.1.1 RC1

Thanks for everyone who voted. Having community vote of confidence in bugfix releases is hugely important, especially as we are trying to support more in-flight versions :) On Wed, Dec 21, 2016 at 8:49 AM, Ismael Juma wrote: > Hi Guozhang, > > Thanks for driving this release as it includes a few

Re: Questions about single consumer per partition approach

Thank you Ben, I appreciate your answer. I AM using the key to send messages from the selected producer to a selected partition – Q1 covered. If Q2 seems obscure I could add that I am talking about Java client library and basically asked for clarification to the API doc statement that in case of

Re: Handling Leap second delay

Our method of handling the leap second (for everything, not just Kafka) is to shut off the NTP servers that all the systems refer to before the leap second, and turn it back on afterwards. The NTP server will then update from a stratum 1 server and will eventually step the time correctly on all the

Re: Questions about single consumer per partition approach

Newbie here, Q2) Think, there is no rebalance if you go with manual partition assignment (assign(List) ), but were you looking for subscribe(List, ConsumerRebalanceListener)

How does Kafka emulate exactly once processing?

How does Kafka emulate exactly once processing currently? Does it require the producer to send at least once and consumer to de dupe? I did do my research but I feel like I am going all over the place so a simple short answer would be great! Thanks!

Re: How does Kafka emulate exactly once processing?

Exactly once Kafka Sink Connectors typically store the offset externally in the same atomic write as they store the messages. That way after a crash, they can check the external store (HSFS, JSDC, etc) retrieve the last committed offset and seek the the next message and continue processing with

Processing time series data in order

- I'm receiving a batch of messages to a Kafka topic. Each message has a timestamp, however the messages can arrive / get processed out of order. I.e event 1's timestamp could've been a few seconds before event 2, and event 2 could still get processed before event 1. - I know the number of messag

Re: How does Kafka emulate exactly once processing?

Hi Hans, Thats a great answer compared to the paragraphs I read online! I am assuming you meant HDFS? what is JSDC ? Any idea on which is more common for this kind of use case? Also can I store offsets to zookeeper using ZAB instead of using external store? I am not sure how zookeeper stores data

Re: Processing time series data in order

The batch size can be large, so in memory ordering isn't an option, unfortunately. On Thu, Dec 22, 2016 at 7:09 AM, Jesse Hodges wrote: > Depending on the expected max out of order window, why not order them in > memory? Then you don't need to reread from Cassandra, in case of a problem > you ca

Re: Processing time series data in order

Depending on the expected max out of order window, why not order them in memory? Then you don't need to reread from Cassandra, in case of a problem you can reread data from Kafka. -Jesse > On Dec 21, 2016, at 7:24 PM, Ali Akhtar wrote: > > - I'm receiving a batch of messages to a Kafka topi

Is it possible for consumers within a single consumer group to have different subscriptions?

Searched for a while and not finding a clear answer. Is it possible for consumers within a single consumer group to have different topic subscriptions? If no, if any one of the consumers calls subscribe() with new topic list, how is that subscription propagated to the other consumers in the group

Re: How does Kafka emulate exactly once processing?

Please read the following: https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdoIgetexactly-oncemessagingfromKafka? https://kafka.apache.org/0101/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html (usage Examples). It describes all the use cases. If you store offs

Re: How does Kafka emulate exactly once processing?

I meant to say HDFS and JDBC which are two of the Confluent Sink Connectors that do exactly once processing. The Elasticsearch connector also has exactly once semantics but in a very different way. The Elastic API is idempotent if you update the same record multiple times using the same docume

Re: Is it possible for consumers within a single consumer group to have different subscriptions?

It is possible for them to have different subscriptions. Consumers will only be assigned partitions from topics to which they are subscribed. So if you need to modify your app use data from an additional topic, you can safely do a rolling deploy of the updated version and during the period where th

Kafka Backup Strategy

Hi, I have Kafka running on EC2 in AWS. I would like to backup my data volumes daily in order to recover to a point in time in case of a disaster. One thing I’m worried about is that if I do an EBS snapshot while Kafka is running, it seems a Kafka that recovers on it will have to deal with corrup

Re: Multi-topic consumer configuration

Do each of your topics have only 1 partition? The default partition assignor is org.apache.kafka.clients.consumer.RangeAssignor which will give the behavior you describe if each topic only has a single partition. You probably want to switch it to RoundRobinAssignor which guarantees a more even dis

[ANNOUCE] Apache Kafka 0.10.1.1 Released

The Apache Kafka community is pleased to announce the release for Apache Kafka 0.10.1.1. This is a bug fix release that fixes 30 issues in 0.10.1.0. All of the changes in this release can be found in the release notes: *https://archive.apache.org/dist/kafka/0.10.1.1/RELEASE_NOTES.html

Re: Questions about single consumer per partition approach

Hello, I appreciate your advice. My major concern is - if consumer thread didn’t send heartbeat for time period set in *session.timeout.ms * property it should look dead for the broker. Re-balancing process is not triggered for manually assigned consumers as API doc say

NotLeaderForPartitionException while doing repartitioning

Hi, I’m doing a repartitioning from broker 4 5 6 to broker 7 8 9. I’m getting a LOT of the following errors (for all topics): [2016-12-22 04:47:21,957] ERROR [ReplicaFetcherThread-0-9], Error for partition [__consumer_offsets,29] to broker 9:org.apache.kafka.common.errors.NotLeaderForPartitionExc

Re: Kafka Backup Strategy

Hi Stephane, I say this not to be condescending in any way, but simple replication *might* cover your needs. This will cover most node failures (causing unclean shutdown) like disk or power failure. This assumes that one of the replicas of your data survives (see the configs min.insync.replicas, a

Re: Unable to consume compressed topics

Yes, The console consumer works great, on all compression types. *Ofir Sharony* BackEnd Tech Lead Mobile: +972-54-7560277 | ofir.shar...@myheritage.com | www.myheritage.com MyHeritage Ltd., 3 Ariel Sharon St., Or Yehuda 60250, Israel