Re: cleanup.policy - doesn't accept compact,delete

we need to use square brackets only for command line tool. In your code, you just need to supply "compact,delete" string to props object. On Wed, Jul 11, 2018 at 8:52 AM Jayaraman, AshokKumar (CCI-Atlanta-CON) < ashokkumar.jayara...@cox.com> wrote: > Hi Matthias, > > Kept as [compact,delete] and

Re: [VOTE] 2.0.0 RC2

+1 (non binding) rolling upgrade of tiny shared staging multitenacy (200+ consumer groups) cluster from 1.1 to 2.0.0-rc1 to 2.0.0-rc2. cluster looks healthy after upgrade. Lack of burrow lag suggests consumers are still happy, and incoming messages remains the same. Will monitor. On Wed, Jul 11,

RE: cleanup.policy - doesn't accept compact,delete

Hi Matthias, Kept as [compact,delete] and still got the same exception. Thanks & Regards, Ashok -Original Message- From: Matthias J. Sax [mailto:matth...@confluent.io] Sent: Tuesday, July 10, 2018 10:09 PM To: users@kafka.apache.org Subject: Re: cleanup.policy - doesn't accept compact,

Re: Kafka Streams Application Failing to Start Due to State Store Recovery Time Exceeding Producer Transaction Timeout

Not sure atm. Can you share the whole stacktrace? -Matthias On 7/10/18 11:18 AM, David Chu wrote: > Yes, https://issues.apache.org/jira/browse/KAFKA-6634 > seems to explain the issue > I’m seeing; however, I’m running Kafka and Kafka Streams

Re: Flink ignoring -yn

Which flink version do you use ? Garrett Barton 于2018年7月11日周三 上午1:09写道： > Hey all, > I am running flink in batch mode on yarn with independant jobs creating > their own clusters. > > I have a flow defined that scales parallelism based on input size (to keep > overall processing time somewhat co

Re: cleanup.policy - doesn't accept compact,delete

Try to remove the space after the comma. -Matthias On 7/10/18 10:43 AM, Jayaraman, AshokKumar (CCI-Atlanta-CON) wrote: > Hi, > > When we try to use the same (square brackets), the internal topics are > failing to get created. Any suggestions? > > changelogConfig.put("cleanup.policy", "[compac

Re: flatMapValues does not calculate timestamp for each record generated

By default, Kafka uses the record metadata timestamp. Thus, I assume that you use a custom timestamp extractor? If yes, what timestamp do you extract, as it seems you have multiple? To make it work, you will need to write the data back to Kafka into a new topic after the flatpMap() and read each r

Re: Kafka disk recommendation for cloud

Hi Dan, Thanks for the reply. It's not many publishers right now, but it should become many publishers. Message size is pretty small, but we use batch writing. Based on architecture, Kafka relies on seq read/write, but still SSD might be a cost effective option in the case of too many publishers?

Stream processing meetup at LinkedIn (Sunnyvale) on Thursday, July 19 at 6pm

Hi everyone, We would like to invite you to a Stream Processing Meetup at LinkedIn’s Sunnyvale campus, LinkedIn Building F (LSNF) 605 West Maude Avenuee, on Thursday, July 19 at 6pm. Please RSVP here (if you intend to attend in person): *https://www.meetup.com/Stream-Processing-Meetup-LinkedIn/ev

Re: how to use SMT to transform a filed value in kafka connect?

ReplaceField works with fields themselves, not the values. You could either write your own SMT to do what you want, or you could use KStreams or KSQL to process the data in the topic once ingested through Connect. The simplest way would be with the CASE statement

Re: Create a Topic if Not Exists

Hey Chinchu, There's a --if-not-exists option on the kafka-topics script in the confluent platform. Here's the docs from confluent-3.0.0/bin/kafka-topics --help: --if-not-exists if set when creating topics, the action will only

RE: Kafka disk recommendation for cloud

In Azure we recommend using managed disks for Kafka. HD Insight Kafka uses them. I generally see SSD for Kafka, but I guess part of that could depend on if you write larger writes from fewer publishers or small writes from many publishers. What does your workload look like? Kind Regards, -Dan

Re: Create a Topic if Not Exists

The properties Dan is referring are auto.create.topics.enable which is true by default and num.partitions which is 1 by default. They are both configuration parameters for the broker. Thanks Paolo Get Outlook for Android From: Dan Rosanova Sent: Tuesday, 10 July, 20:

RE: Create a Topic if Not Exists

Hi Chinchu, I thought the default behavior of Kafka was to create topics if you send to them and they don't exist. I could be wrong. We had to support that in one of our products, so I am pretty sure that's the default. You do need to be mindful of the default partition count though, which I thi

Re: Kafka Streams Application Failing to Start Due to State Store Recovery Time Exceeding Producer Transaction Timeout

Yes, https://issues.apache.org/jira/browse/KAFKA-6634 seems to explain the issue I’m seeing; however, I’m running Kafka and Kafka Streams on version 1.1.0 so I wonder why this issue is still occurring? -David > On Jul 10, 2018, at 9:38 AM, Mat

RE: cleanup.policy - doesn't accept compact,delete

Hi, When we try to use the same (square brackets), the internal topics are failing to get created. Any suggestions? changelogConfig.put("cleanup.policy", "[compact, delete]"); org.apache.kafka.streams.errors.StreamsException: Could not create topic stream_digital_01-hrly-changelog. a

Re: flatMapValues does not calculate timestamp for each record generated

This is a specific example: We are sending metrics to Kafka Stream with the following layout: Record Key: -- |metric name|tags| -- Record Value: ---

[VOTE] 2.0.0 RC2

Hello Kafka users, developers and client-developers, This is the third candidate for release of Apache Kafka 2.0.0. This is a major version release of Apache Kafka. It includes 40 new KIPs and several critical bug fixes. Please see the 2.0.0 release plan for more details: https://cwiki.apach

Flink ignoring -yn

Hey all, I am running flink in batch mode on yarn with independant jobs creating their own clusters. I have a flow defined that scales parallelism based on input size (to keep overall processing time somewhat constant). Right now the flow initializes with around ~22k tasks for a flow that sets p

Re: Kafka Streams Application Failing to Start Due to State Store Recovery Time Exceeding Producer Transaction Timeout

Can it be, that you hit: https://issues.apache.org/jira/browse/KAFKA-6634 -Matthias On 7/9/18 7:58 PM, David Chu wrote: > I have a Kafka Streams application which is currently failing to start due to > the following ProducerFencedException: > > "Caused by: org.apache.kafka.common.errors.Produce

Re: flatMapValues does not calculate timestamp for each record generated

Not sure what you mean by "does not reset recordContext". Note, that the "contract" for `flatMapValues` is that the output records inherit the timestamp of the input record. Not sure what behavior you expect? Maybe you can elaborate? -Matthias On 7/9/18 7:27 PM, Sicheng Liu wrote: > Hi, > > I

Create a Topic if Not Exists

Hi All, I am looking for a way to script(bash) kafka topic creation . We have around 30 to 40 topics that need to be created on different envs . The intention is to make this process automated ,so looking for something like 'create topic if not exists' or similar by looking at a return code . D

RE: Monitoring Kafka

Thank it is very useful for me, because I already have a kibana and grafana installed. I will try it. Thank again Daniel Adrien De : Daniel Aparecido Martins Rosa Envoyé : mardi 10 juillet 2018 14:47:06 À : users@kafka.apache.org Objet : Re: Monitoring Kafk

Steadily increasing open file descriptors on one node a cluster

Hi, I’m running a Kafka cluster with many topics and constant input of data. The cluster is running for over one year but now (since 2 weeks) there is one node where I see a steady increase of open file descriptors of the Kafka server process. All other nodes have a constant number of this metri

Re: Monitoring Kafka

Hi Adrien, Take a look at this post that I wrote. Maybe can guide you. Enjoy, https://medium.com/@danielmrosa/monitoring-kafka-b97d2d5a5434 2018-07-09 12:09 GMT-03:00 adrien ruffie : > Great ! Thank a lot Daniel ! > > I will try it. > > Best Regards > > Adrien >

how to use SMT to transform a filed value in kafka connect?

Hi all, I'm new to kafak. I'm using kafka to build a ETL program. Here's my scinaro, My jdbc connect table t_user have a column sex which is int in mysql, and use 0 to represent male, 1 to represent female. I want change sex value when 0 then 100, when 1 then 200. below is my connecter con

Re: Real time streaming as a microservice

Thanks Rahul. This is the outcome of [root@rhes75 ~]# iptables -t nat -L -n Chain PREROUTING (policy ACCEPT) target prot opt source destination DOCKER all -- 0.0.0.0/00.0.0.0/0ADDRTYPE match dst-type LOCAL Chain INPUT (policy ACCEPT) target prot

Re: Real time streaming as a microservice

Seems like you need to expose your port via docker run or docker-compose . https://docs.docker.com/v17.09/engine/userguide/networking/default_network/binding/ -- Rahul Singh rahul.si...@anant.us Anant Corporation On Jul 9, 2018, 2:21 PM -0500, Mich Talebzadeh , wrote: > Hi, > > I have now suc