value.subject.name.strategy for multiple event types not applied to internal topics in kafka

hello, in kafka, we have a requirement for multiple event types in the same topic (we use avro w/ schema registry). we were able to achieve this using the following configuration: [value.subject.name.strategy=io.confluent.kafka.serializers.subject.TopicRecordNameStrategy](https://www.confluent.i

Re: Repartition topics in Kafka Streams when there is only one partition

Hi Frank, It doesn't make a lot of sense to create repartition topics for single partition input topics. However the way it currently works we don't have the partition information at hand when making the decision that we need to repartition. It is purely based on the key potentially changing. The

Repartition topics in Kafka Streams when there is only one partition

Hi, I'm using quite a lot of topics, and the speed on each topic isn't extreme, so I generally use a single partition. In this situation, the high level API of Kafka Streams still makes repartition topics, but I don't see there use. Wouldn't it make sense to skip repartition topics in this case,

Re: Commits of slow moving topics in Kafka Streams time out

Ok, yes increasing the retention minutes to something silly is indeed what we intend to do, and for now, that should work fine as we don't use many short lived consumers. For the long term, the change in semantics in the link you provide looks good to us. regards, Frank On Tue, Mar 21, 2017 at 8

Re: Commits of slow moving topics in Kafka Streams time out

Hi Frank, There is a similar discussion here https://www.mail-archive.com/users@kafka.apache.org/msg25089.html , with a JIRA. Have you tried to increase the retention minutes to something large? Thanks Eno > On 21 Mar 2017, at

Commits of slow moving topics in Kafka Streams time out

Hi Kafka people, We have a Kafka Streams application that replicates a database, and transforms it to a different data model. Some tables/topics move fast, with many changes a second, some might be dormant for months. For those slow moving topics, we have some trouble with the 'offsets.retention.m

Force delete tombstone markers in the log compacted topics in kafka 0.9.0.1

Hi everybody, I'm newbie in this mail list , hi everybody! I'm trying to improve the documentation for my operator guys and it could be useful to know some method to force the deletion of tombstone markers in the log compacted topics. I only know the properties log.cleaner.delete.retention.ms

Re: How to merge two topics in kafka?

I checked my targetTopic for available messages, but it says '0' . What might cause issue here to merge two topics with custom type messages ? On 11 October 2016 at 14:44, Ratha v wrote: > Thanks, this demo works perfectly for string messages. > > I have custom messageType defined( a java pojo

Re: How to merge two topics in kafka?

Thanks, this demo works perfectly for string messages. I have custom messageType defined( a java pojo class). And i have SerDe implemented for that. Now after merging sourceTopic-->Target Topic, I could not consume the messages..Means, Consumer does not return any messages. What might be the caus

Re: How to merge two topics in kafka?

Check this example https://github.com/apache/kafka/blob/trunk/streams/examples/src/main/java/org/apache/kafka/streams/examples/pipe/PipeDemo.java On Mon, Oct 10, 2016 at 11:34 AM, Ratha v wrote: > Hi Sachin; > I went through the KStream/KTable Documentation. My scenario is very > simple..I want

Re: How to merge two topics in kafka?

Hi Sachin; I went through the KStream/KTable Documentation. My scenario is very simple..I want to merge two topics( ie: Send messages available in the topic A -->topic B , in my case i'll be having only single message in that topicA) Do I need Stateful processing (KStream)? Thanks. On 10 October

Re: How to merge two topics in kafka?

Thank you.I'll try the solution. But in the highlevel consumer API, topics will be created automatically? , We are not using zookeeper? On 10 October 2016 at 12:34, Sachin Mittal wrote: > You can use topicA.leftJoin (topicB).to (new-topic). > > You can the consume message from that new topic via

Re: How to merge two topics in kafka?

You can use topicA.leftJoin (topicB).to (new-topic). You can the consume message from that new topic via second process. Note you need to create all three topics in zookeeper first. On 10 Oct 2016 5:19 a.m., "Ratha v" wrote: Hi all; I have two topics in the broker. Without consuming from one t

How to merge two topics in kafka?

Hi all; I have two topics in the broker. Without consuming from one topic, I want to merge both topics, and will consume messages from the second topic. It is because, I have two processes, one process, pushes messages to topic A. And the second process once finished processing, it wants to merge

Re: Regex topics in kafka connect?

Barry, Actually, it's not exposed in 0.10 either. https://issues.apache.org/jira/browse/KAFKA-3073 is filed to track this. We know people will want this, it just hasn't made it in quite yet. -Ewen On Fri, Jun 10, 2016 at 4:40 AM, Barry Kaplan wrote: > The kafka connect docs say > > The Kafka C

Regex topics in kafka connect?

The kafka connect docs say The Kafka Connect framework manages any changes to the Kafka input, such as > when the set of input topics changes because of a regex subscription. But I can find no other information how to define a wildcard/regex topic subscription. I tried giving the config topic pr

Aw: Re: Re: Topics in Kafka

Uhr Von: "Manikumar Reddy" An: "users@kafka.apache.org" Betreff: Re: Re: Topics in Kafka Hi, you can have single topic with multiple partitions. It looks like you have messages with key=region and value=sensor reading, and you want to run some aggregate, windowing opera

Re: Re: Topics in Kafka

ark Streaming afterwards? > > Thank you in advance. > > Regards, > Daniela > > > > Gesendet: Mittwoch, 23. März 2016 um 09:42 Uhr > Von: "Manikumar Reddy" > An: "users@kafka.apache.org" > Betreff: Re: Topics in Kafka > Hi, > > 1. Based

Aw: Re: Topics in Kafka

    Gesendet: Mittwoch, 23. März 2016 um 09:42 Uhr Von: "Manikumar Reddy" An: "users@kafka.apache.org" Betreff: Re: Topics in Kafka Hi, 1. Based on your design, it can be one or more topics. You can design one topic per region or one topic for all region devices. 2. Yes, you n

Re: Topics in Kafka

Hi, 1. Based on your design, it can be one or more topics. You can design one topic per region or one topic for all region devices. 2. Yes, you need to listen to web socket messages and write to kafka server using kafka producer. In your use case, you can also send messages using Kafka Re

Topics in Kafka

Hi I am new and I have a question regarding Kafka. I have devices in different regions. Every device opens a websocket connection when it gets active and it sends its messages over the opened websocket connection to a server. My question is: is every region a topic or is every websocket connect

Re: Question regarding compression of topics in Kafka

, - R P From: Ben Stopford Sent: Friday, March 18, 2016 9:21 AM To: users@kafka.apache.org Subject: Re: Question regarding compression of topics in Kafka Assuming you’re using the new producer (org.apache.kafka.clients.producer) the property is called

Re: Question regarding compression of topics in Kafka

gt; _ >> From: Ben Stopford > <mailto:b...@confluent.io><mailto:b...@confluent.io> >> <mailto:b...@confluent.io>> >> Sent: Friday, March 18, 2016 7:50 AM >> Subject: Re: Question regarding compression of topics in Kafka >>

Re: Question regarding compression of topics in Kafka

Yes it will compress the data stored on the file system if you specify compression in the producer. You can check whether the data is compressed on disk by running the following command in the data directory. kafka-run-class kafka.tools.DumpLogSegments --print-data-log --files latest-log-file.l

Re: Question regarding compression of topics in Kafka

on? Thanks, R P _ From: Ben Stopford mailto:b...@confluent.io>> Sent: Friday, March 18, 2016 7:50 AM Subject: Re: Question regarding compression of topics in Kafka To: mailto:users@kafka.apache.org>> Yes it will compress the data stored on the file system if

Question regarding compression of topics in Kafka

Hello All, Does kafka support compressing storage logs stored in log dir? What does compression.type=(gzip/snappy) in server.properties do? Based on documents I am assuming that it will compress the logs on local file system. I ran a quick experiment and found that my logs stored on local disk