offset.storage.filename configuration in kafka-connect-hdfs

Hi, I'm running kafka-connect-hdfs 3.1.1. offset.storage.filename is a required configuration for standalone connector. But when I set this parameter in worker configuration file for kafka-connect-hdfs with standlone mode: offset.storage.file.filename=/mnt/data/connect.offsets It never works. Th

Re: Unsubscribe

Unsubscribe On Wed, Mar 8, 2017 at 8:06 AM Vijay Kukkala wrote: > -- > Best Regards, > Vijay kukkala > 318-243-9725 > -- Best Regards, Vijay kukkala 318-243-9725

Re: How to set offset for a consumer in Kafka 0.10.0.X

Yeah, that gist looks like it *should* work. I haven't tested it so can't guarantee. On Tue, Mar 7, 2017 at 7:04 PM, Glen Ogilvie wrote: > Hi Jeff, > > Yes, the work I am doing is ops work. Logstash is consuming from the topic > + consumer group, and I don't want it to start at the beginning, bu

Re: How to set offset for a consumer in Kafka 0.10.0.X

The best approach would be: - Have all consumers in your group shut down - Have an offset reset tool join with the same group name as above - Offset tool subscribes to all topic-partitions, seeks to the desired offset, and commits. - Offset tool shuts down - Consumers then restart and re-join the c

Configuring an IKEv2 endpoint with PAM authentication

I'm currently stuck trying to configure strongswan to support road warriors and for workstations operated by other remote users. I believe I accomplish this with IKEv2, EAP and xauth-pam. To that end: 1. I've setup an Ubuntu 16.04 box, 2. installed strongswan, strongswan-plugin-xauth, strong

Performance and Encryption

Hi everyone, I understand one of the reasons why Kafka is performant is by using zero-copy. I often hear that when encryption is enabled, then Kafka has to copy the data in user space to decode the message, so it has a big impact on performance. If it is true, I don t get why the message has to

Settings at cluster level: retention and cleanup policy

Hi everyone, Is there any reason why retention and cleanup policy are defined at cluster level and not topic level? I can t see why it would not be possible from a technical point of view... Cheers Nico

Re: Performance and encryption

Hi everyone, I have another question. Is there any reason why retention and cleanup policy are defined at cluster level and not topic level? I can t see why it would not be possible from a technical point of view... 2017-03-06 14:38 GMT+01:00 Nicolas Motte : > Hi everyone, > > I understand one of

RE: Performance and Encryption

I think because the product batches messages which could be for different topics. -Dave -Original Message- From: Nicolas MOTTE [mailto:nicolas.mo...@amadeus.com] Sent: Wednesday, March 8, 2017 2:41 PM To: users@kafka.apache.org Subject: Performance and Encryption Hi everyone, I underst

Re: How to set offset for a consumer in Kafka 0.10.0.X

Thank you Jeff and Robert. I've had success in getting the offset position to seek to a position in the queue, when all other consumers in the group are off. Would anyone like me to tidy the tool up enough that it could be included in Kafka or the docs? Regards -- Glen Ogilvie Open Syste

Re: Settings at cluster level: retention and cleanup policy

Actually they can be at topic level configs as well. If they are not defined then the server level configs are used. From Apache Kafka doc: "The server's default configuration for this property is given under the Server Default Property heading. A given server default config value only applies to

Re: Performance and Encryption

Nicholas, this appears to be a duplicate of your question from 2 days ago. Please review that for discussion on this question. -Todd On Wed, Mar 8, 2017 at 1:08 PM, Tauzell, Dave wrote: > I think because the product batches messages which could be for different > topics. > > -Dave > > -Ori

Re: [DISCUSS] KIP-120: Cleanup Kafka Streams builder API

Hi, sorry for not replying earlier and thanks for all your feedback. After some more discussions I updated the KIP. The new proposal puts some other design considerations into account, that I want to highlight shortly. Those considerations, automatically resolve the concerns raised. First some an

Re: Performance and encryption

I believe these are defaults you can set at the broker level so that if the topic doesn’t have that setting set, it will inherit those But you can definitely override your topic configuration at the topic level On 9 March 2017 at 7:42:14 am, Nicolas Motte (lingusi...@gmail.com) wrote: Hi everyone

Re: Performance and encryption

They are defined at the broker level as a default for all topics that do not have an override for those configs. Both (and many other configs) can be overridden for individual topics using the command line tools. -Todd On Wed, Mar 8, 2017 at 12:36 PM, Nicolas Motte wrote: > Hi everyone, I have

Re: How to set offset for a consumer in Kafka 0.10.0.X

Did you have to do anything different beyond what was already in the Gist? I'd be curious to see the code as I've considered putting together a small repo of various python scripts I've found useful when working with Kafka On Wed, Mar 8, 2017 at 1:11 PM, Glen Ogilvie wrote: > Thank you Jeff and

Re: Settings at cluster level: retention and cleanup policy

Hi Nico, They can be defined at both cluster and topics levels. Refer https://kafka.apache.org/documentation/#topic-config for topic-level overrides available. Cheers! -Z > On Mar 8, 2017, at 12:41 PM, Nicolas MOTTE wrote: > > Hi everyone, > > Is there any reason why retention and cleanup

performance regression with 0.10.2.0 client talking to 0.10.0.1 servers

I noticed a significant reduction in throughput when I upgraded my app to 0.10.2.0. I am talking to 0.10.0.1 server and I wanted to take advantage of the compatibility of the newest release with old server, but I noticed that my performance went from 20K msg/sec to 2K msg/sec in one case and fro

Re: How to set offset for a consumer in Kafka 0.10.0.X

A tool for reseting consumer group offsets is proposed here: KIP-122: https://cwiki.apache.org/confluence/display/KAFKA/KIP-122%3A+Add+Reset+Consumer+Group+Offsets+tooling KIP is currently in the voting stage. On Thu, Mar 9, 2017 at 7:35 AM, Jeff Widman wrote: > Did you have to do anything diff

Re: How to set offset for a consumer in Kafka 0.10.0.X

Hi Glen, Maybe have a look at this, slightly different version. Not doing anything more than what you have seen in gist. You should pass broker url as first parameter and and a json containing desired offsets as second parameter. https://github.com/goibibo/woof/tree/master/scripts/offset_commit_

How to know which consumer member causes rebalance

Hi, We are running a group of Kafka consumers on 200 mesos instances and we are observing constant Revoking/Rejoining in our consumer logs. But it's hard to tell which consumer member initially caused this issue as every consumer needs re-join in this case. Is there a good way to find out that 'ba

Re: How to know which consumer member causes rebalance

Some more details about the consumer setup: - version 0.10.1.1 - the topic has 400 partitions - consumers are running on 200 mesos instances, and each instance is running 2 KafkaConsumer threads Yifan On Wed, Mar 8, 2017 at 11:46 PM, Yifan Ying wrote: > Hi, > > We are running a gr