Re: Kafka running on AWS - how to retain broker.id on new instance spun-up in-place of instance/broker failed

2018-11-16 Thread Kaufman Ng
> > the broker.id needs to be set. > > > > Do we have any standards/script/way defined/preferred for doing this or > > suggested by Kafka experts if we are not using EBS? > > > > Thanks and Regards, > > Srinivas > > > > On Thu, Nov 15, 2018

Re: Kafka running on AWS - how to retain broker.id on new instance spun-up in-place of instance/broker failed

2018-11-15 Thread Kaufman Ng
roker is failed, new > > > broker/instance > > > > > > > spun-up in AWS get assigned with new broker.id. The issue is, > > with > > > > > this > > > > > > > approach, we need to re-assign the topics/replications on to > the > > > new > >

Re: Kafka brokers listen on multiple ports

2018-05-19 Thread Kaufman Ng
r a transition period. > > Is it possible to use 2 different ports for the same protocol (PLAINTEXT) > in the broker configuration? Can I simply put 2 connection strings in the > *listeners* config? > > Thank you! > Dan > -- Kaufman Ng +1 646 961 8063 Solutions Architect | Confluent | www.confluent.io

Re: Removing the Kafka DEBUG logs in catalina.out using log4j.properties

2018-05-19 Thread Kaufman Ng
tch for partitions > [apptivodb8-campaign-tracker-email-0] to broker xx.xx.xx.xx:9092 (id: 2 > rack: null) > DEBUG Fetcher:180 - Sending fetch for partitions > [apptivodb5-campaign-tracker-email-1] to broker xx.xx.xx.xx:9092 (id: 2 > rack: null) > > > > On Mon, May 14, 2018 a

Re: Removing the Kafka DEBUG logs in catalina.out using log4j.properties

2018-05-13 Thread Kaufman Ng
nalyzed in log file debug logs are taking more space. So i'm > having > > disk space issue. > > > > I'm using *log4j.properties* for managing the logs, Now I want to remove > > the DEBUG logs from my logger file. > > > > Anyone, please guide me to

Re: Solution for clients with long-lived sustained SSL connections using JKS

2018-03-13 Thread Kaufman Ng
as anybody else run into this problem and found a good solution? I'm > > interested to hear any other solutions for tearing down and rebuilding > SSL > > connections on the fly. > > > > > > Thanks, > > Alex > > > > -- > Sönke Liebau > Partner > Tel. +49 179 7940878 > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany > -- Kaufman Ng +1 646 961 8063 Solutions Architect | Confluent | www.confluent.io

Re: monitor consumer offset lag script/code

2018-01-30 Thread Kaufman Ng
e're using 0.9 ( CDH ) and consumer offsets are stored within Kafka. > What > > is the preferred way to get consumer offset from code or script for > > monitoring ? Is there any sample code/ script to do so ? > > > > Thanks, > > Sunil Parmar > -- Kaufman Ng +1 646 961 8063 Solutions Architect | Confluent | www.confluent.io

Re: [ANNOUNCE] New committer: Matthias J. Sax

2018-01-14 Thread Kaufman Ng
> > beyond mailing list: he's getting the close to 1000 up votes and 100 > > > helpful flags on SO for answering almost all questions about Kafka > > Streams. > > > > > > Thank you for your contribution and welcome to Apache Kafka, Matthias! > > > > > > > > > > > > Guozhang, on behalf of the Apache Kafka PMC > > > -- Kaufman Ng +1 646 961 8063 Solutions Architect | Confluent | www.confluent.io

Re: Listeners and reference/docs

2017-11-10 Thread Kaufman Ng
5:9094,INTERNAL_SASL://10.1.1.5:9095 On Fri, Nov 10, 2017 at 7:10 PM, Thomas Stringer wrote: > Yep I'm familiar with that. Just curious where it's documented that, for > instance, the CLIENT listener is for client connections. > > On Fri, Nov 10, 2017, 12:08 PM Kaufman Ng wrot

Re: Listeners and reference/docs

2017-11-10 Thread Kaufman Ng
EXT, SASL/SSL, etc. I see the encryption part of the > documentation, but is it just inferred what these listeners apply to? > > Thank you in advance! > -- Kaufman Ng +1 646 961 8063 Solutions Architect | Confluent | www.confluent.io

Re: do i need to restart the brokers if I changed the retention time for a specific topic

2017-08-06 Thread Kaufman Ng
> time in LogManager are static: > https://github.com/apache/kafka/blob/0.10.0/core/src/ > main/scala/kafka/server/KafkaServer.scala#L597-L620 > > Kafka version: kafka_2.11-0.10.0.1 > > Thanks > -- > haitao.yao > -- Kaufman Ng +1 646 961 8063 Solutions Architect | Confluent | www.confluent.io

Re: Schema Registry on DC/OS

2017-07-25 Thread Kaufman Ng
egards. > > On Tue, Jul 25, 2017 at 9:50 AM, Kaufman Ng wrote: > >> Confluent Schema Registry is available in the DC/OS Universe, see here >> for the package definitions https://github.com >> /mesosphere/universe/tree/dcd777a7e429678fd74fc7306945cdd27b >> da3b

Re: Schema Registry on DC/OS

2017-07-24 Thread Kaufman Ng
> Debasish Ghosh > http://manning.com/ghosh2 > http://manning.com/ghosh > > Twttr: @debasishg > Blog: http://debasishg.blogspot.com > Code: http://github.com/debasishg > -- Kaufman Ng +1 646 961 8063 Solutions Architect | Confluent | www.confluent.io

Re: why more open files

2017-07-05 Thread Kaufman Ng
, > >I have created a topic with 500 partitions in 3 node > > cluster with replication factor 3. kafka version is 0.11. I executed lsof > > command and it lists more 1 lakh open files. why these many open files > and > > how to reduce it ?. > >

Re: SASL and SSL

2017-05-25 Thread Kaufman Ng
> > > > If the latter is true, then is it correct to assume that encryption will > > take place using SSL if a client authenticates using a Kerberos ticket so > > long as they have a trust store configured? > > > > Thank you. > > > > Waleed > > > -- Kaufman Ng +1 646 961 8063 Solutions Architect | Confluent | www.confluent.io

Re: Bridging activeMQ and Kafka

2017-05-25 Thread Kaufman Ng
ithub projects like kalinka ( > https://github.com/dcsolutions/kalinka) and also seen that I could do with > Apache Camel. > > I would like to ask you about some experience or advice you can provide in > this bridging between ActiveMQ and Kafka. > > Thanks in advance, > David. >

Re: Settings at cluster level: retention and cleanup policy

2017-03-08 Thread Kaufman Ng
t would not be possible from a technical point of view... > > Cheers > Nico > > -- Kaufman Ng +1 646 961 8063 Solutions Architect | Confluent | www.confluent.io

Re: Creating-kafka-connect

2017-03-04 Thread Kaufman Ng
require its own specific way of keep tracking of offsets. On Sat, Mar 4, 2017 at 1:23 AM, VIVEK KUMAR MISHRA 13BIT0066 < vivekkumar.mishra2...@vit.ac.in> wrote: > Hi All, > > I want to create my own kafka connector which will connect multiple data > source. > Could anyone please hel

Re: when will the messsages be sent to broker

2017-03-01 Thread Kaufman Ng
/KafkaProducer.html On Wed, Mar 1, 2017 at 3:14 AM, Yuanjia wrote: > Hi all, > When will the messsages be sent in kafka0.10.0?If I use KafkaProducer.send > to send one message, the messsages isn't sent immediately except invoke > flush or close. > > Thanks. > > >

Re: can i delete topic like this ?

2017-01-12 Thread Kaufman Ng
opicCommand.scala:53) > at kafka.admin.TopicCommand.main(TopicCommand.scala) > > > > > > ** > > *Regards,* > *Laxmi Narayan Patel* > *MCA NIT Durgapur (2011-2014)* > *Mob:-9741292048,8345847473* > -- Kaufman Ng +1 646 961 8063 Solutions Architect | Confluent | www.confluent.io

Re: [ANNOUNCE] New committer: Grant Henke

2017-01-11 Thread Kaufman Ng
technical judgment, high-quality work and willingness > > to contribute where needed to make Apache Kafka awesome. > > > > Thank you for your contributions, Grant :) > > > > -- > > Gwen Shapira > > Product Manager | Confluent > > 650.450.2760 | @gw

Re: question about property advertised.listeners

2016-11-30 Thread Kaufman Ng
a port forwarding from this configured > port to the actual broker port within the container so that the broker > itself can also find itself, right? > > thanks. > regards, aki > -- Kaufman Ng Solutions Architect | Confluent +1 646 961 8063 | @kaufmanng www.confluent.io

Re: Send a AVRO message from kafka producer

2016-10-31 Thread Kaufman Ng
ssages using > script "kafka-console-producer.sh"? Thanks! > > > > > > > Best Regards > > Johnny > > -- Kaufman Ng Solutions Architect | Confluent +1 646 961 8063 | @kaufmanng www.confluent.io

Re: How can I delete a topic programatically?

2016-10-14 Thread Kaufman Ng
e processing on the > >> message > >> >>> available in the source topic and i merge both topic. > >> >>> That is; > >> >>> > >> >>> builder.stream(sourceTopic).to(targetTopic) > >> >>> > >> >&g

Re: Where kafka stores consumer offsets in version 0.9.0.0?

2016-09-26 Thread Kaufman Ng
r.subscribe(Arrays.asList(topicName)); > //.. > > - > > when application run once, I can get consumer offset by > *kafka-run-class kafka.tools.ConsumerOffsetChecke,

Re: Benchmarking kafka performance

2016-09-23 Thread Kaufman Ng
t; Thanks > > > > On Mon, Sep 19, 2016 at 8:18 AM, Vadim Keylis > wrote: > > > >> Good morning. Which benchmarking tools we should use to compare > >> performance of 0.8 and 0.10 versions? Which metrics should we monitor ? > >> > >> Thanks in advance, > >> Vadim > >> > > -- Kaufman Ng Solutions Architect | Confluent +1 646 961 8063 | @kaufmanng www.confluent.io

Re: Kafka forum register

2016-07-05 Thread Kaufman Ng
m > 发送时间: 2016-06-02 09:19 > 收件人: users > 主题: Kafka forum register > Hello, > > My project is using kafka ,and I want register a user in the forum,what > can I do ? > > > > Tong SS > -- Kaufman Ng | Solutions Architect | Confluent kauf...@confluent.io | +1 646 961 8063

Re: Skipping assignment for topic * since no metadata is available

2016-06-09 Thread Kaufman Ng
> This only happens when the topic does not exist. When we restart the > failing consumer it then can connect correctly to the topic and consume it. > How can this error be prevented? > > Best regards > > Patrick -- Kaufman Ng | Solutions Architect | Confluent kauf...@confluent.io | +1 646 961 8063