Re: Logs truncated hourly

logs look normal to me. It looks like you are creating a new topic for every hour? from logs: 16:00:01 : Created log for partition [mytopic.2017-07-13-16,0] (this is replica log for partition 0) 16:00:01: Truncating log mytopic.2017-07-13-16-0 to offset 0 (This should be harmless) Not sure, Why

Re: Kafka with SSL enabled - Not able to publish messages (org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms)

it looks like SSL configuration issue. Brokers are not able to authenticate with each other. Hope you followed the instructions given at http://kafka.apache.org/ documentation/#security_configbroker You can enable SSL debug logs by using JVM flag -Djavax.net.debug=all On Wed, Jul 26, 2017 at 11:3

RE: colocating zookeeper on the kafka broker nodes

Ok Robert. For the scale you mentioned, is should be fine in my opinion. Regards, Srinath On Jul 27, 2017 6:23 AM, "Robert Friberg" wrote: > > Thank you Srinath, > > I see now that my question was unclear and actually had an incomplete > sentence... > > My idea is to have 3 VM cluster and each

RE: colocating zookeeper on the kafka broker nodes

Thank you Srinath, I see now that my question was unclear and actually had an incomplete sentence... My idea is to have 3 VM cluster and each VM will run both zookeeper and kafka. -- Robert Friberg Building OrigoDB at Devrex Labs +46733839080 -Original Message- From: Srinath C [mailto

Re: colocating zookeeper on the kafka broker nodes

Hi Robert, By colocating the zookeeper servers on the same VM we could have a downtime if the VM happened to go down. Having the servers on different VMs will reduce the risk factor. For example, if the zookeeper cluster has 5 servers you can tolerate upto 2 servers going down and still have no d

colocating zookeeper on the kafka broker nodes

Hi, I'm preparing a production grade install of Kafka. The message volume will be very low, peak load in messages/second would be measured in the 100's. Message size is on average 0.3kb. The number of topics will be < 100 and the typical number of consumers per topic 2 or 3. Reading the docs a

Re: Confluent Control Center UI - not coming up

Hello all- i was able to fix this by modifying the following entry confluent.controlcenter.rest.listeners=http://0.0.0.0:9021 -> confluent.controlcenter.rest.listeners=http://:9021 On Wed, Jul 26, 2017 at 3:35 PM, karan alang wrote: > hi All, > > I've installed Confluent

Confluent Control Center UI - not coming up

hi All, I've installed Confluent Kafka 3.2.2 & am using the following link to start the services - http://docs.confluent.io/current/control-center/docs/quickstart.html I've started the services shown below, however the Contraol Center UI is not coming up. Any ideas on this ? screen -ls > There

Re: Consumer group metadata retention

The offsets.retention.minutes value (1440 = 24 hours = 1 day) is a broker level configuration, and can't be changed dynamically during runtime. You would have to modify the broker configurations, and restart the brokers. -James > On Jul 25, 2017, at 9:43 PM, Raghu Angadi wrote: > > I am writi

Kafka streams regex match

Hello, I am able to get the kstream to ktable join work. I have some use cases where the key is not always a exact match. I was wondering if there is a way to lookup keys based on regex. For example, I have these entries for a ktable: test_host1,{ "source": "test_host", "UL1": "test1_l1" } test_

Re: Logs truncated hourly

still I haven't been able to figure out what's the problem... > On 13/07/17 17:09, mosto...@gmail.com > wrote: > > Hi > > With swiss precission, our kafka test environment seems to truncate topics > at o'clock hours. > > This might be confirmed with the following trace, which states "Truncating

Re: Merging windowed KStreams with regular KStreams

I guess it depend what you want as an output... But what you suggest would work. You can also apply a .map() to the windowed stream and extract the actual record key from the window (ie, strip away the window) -Matthias On 7/26/17 6:15 PM, Sameer Kumar wrote: > I wanted to merge two KStreams on

Re: Kafka Connect distributed mode rebalance

Ewen, thanks for the reply. I will admit that in the current state my connector use is not optimal. Today it is a combination of JDBC source connectors and a custom sink connector. The original rollout was before single message transforms (SMT) weren't available (2.0.1 was our first deployment, an

Re: Without data loss migration | Kafka 10 to Kafka 11

Sure, will do that by tomorrow. -Sameer. On Wed, Jul 26, 2017 at 8:39 PM, Ismael Juma wrote: > Hi Sameer, > > Yes, the upgrade should be seamless. Can you please share the log entries > with the errors? > > Ismael > > On Wed, Jul 26, 2017 at 1:35 PM, Sameer Kumar > wrote: > > > Hi , > > > > I

Merging windowed KStreams with regular KStreams

I wanted to merge two KStreams one of them is a windowed stream and another one is of type , what is the preferred way of merging them. One way I thought was to run a map phase and create a windowed instance based on system.currentmillis. -Sameer.

Suggestion re. SessionWindows: make Merger extend Aggregator

Hi, I suggest that Merger should extend Aggregator. reason: Both classes usually do very similar things. A merger takes two sessions and combines them, an aggregator takes an existing session and aggregates new values into it. in some use cases it is actually the same thing, e.g.: -> .map() to

Upgrade by replacing brokers?

​Hi all, We’re planning a big upgrade from 0.9.0.1 to 0.11. As part of this upgrade, we’ll be replacing the all the hardware in the cluster. We are considering doing this as follows: One by one, we’d shut down an original broker machine, and then start up a broker​ on a new node with the same b

Re: Without data loss migration | Kafka 10 to Kafka 11

Hi Sameer, Yes, the upgrade should be seamless. Can you please share the log entries with the errors? Ismael On Wed, Jul 26, 2017 at 1:35 PM, Sameer Kumar wrote: > Hi , > > I wanted to understand the process for production upgrade of Kafka. As > documented in the https://kafka.apache.org/docum

Re: handling exceptions in a Kafka Streams application ..

One of the brokers died. The good thing is that it's not a production cluster, it's just a demo cluster. I have no replicas. But I can knock off the current Kafka instance and have a new one. Just for my understanding, if I don't have a replica, how should such situations be handled ? And if I hav

Re: handling exceptions in a Kafka Streams application ..

Hi, It looks to me that there is currently no leader for the partition, i.e., leader -1. Also there are no replicas? Something up with your brokers? Thanks, Damian On Wed, 26 Jul 2017 at 12:34 Debasish Ghosh wrote: > Hi Damian - > > Yes, it exists .. It's actually a change log topic correspond

Re: Merging Two KTables

Merging two tables does not make too much sense because each table might contain an entry for the same key. So it's unclear, which of both values the merged table should contain. KTable.toStream() is just a semantic change and has no runtime overhead. -Matthias On 7/26/17 1:34 PM, Sameer Kumar

Re: struggling with runtime Schema in connect

just out of curiosity, why does kafka streams not use this runtime data api defined in kafka connect? On Wed, Jul 26, 2017 at 3:10 AM, Ewen Cheslack-Postava wrote: > Stephen's explanation is great and accurate :) > > One of the design goals for Kafka Connect was to not rely on any specific > ser

Without data loss migration | Kafka 10 to Kafka 11

Hi , I wanted to understand the process for production upgrade of Kafka. As documented in the https://kafka.apache.org/documentation/#upgrade, it should be seamless. I had a 3 node cluster(single topic, partitions=60, replication factor =2 )on which i was trying the same. As suggested, I first st

Re: Key Value State Store value retention

This might help: https://kafka.apache.org/documentation/#compaction On Wed, 26 Jul 2017 at 12:37 Sameer Kumar wrote: > Damian, > > Does this mean data is retained for infinite time limited only by disk > space. > > -Sameer. > > On Wed, Jul 26, 2017 at 3:53 PM, Sameer Kumar > wrote: > > > got it

Re: Key Value State Store value retention

Damian, Does this mean data is retained for infinite time limited only by disk space. -Sameer. On Wed, Jul 26, 2017 at 3:53 PM, Sameer Kumar wrote: > got it. Thanks. > > On Wed, Jul 26, 2017 at 3:24 PM, Damian Guy wrote: > >> The changelog is one created by kafka streams, then it is a compact

Merging Two KTables

Hi, Is there a way I can merge two KTables just like I have in KStreams api. KBuilder.merge(). I understand I can use KTable.toStream(), if I choose to use it, is there any performance cost associated with this conversion or is it just a API conversion. -Sameer.

Re: handling exceptions in a Kafka Streams application ..

Hi Damian - Yes, it exists .. It's actually a change log topic corresponding to the state store log-count $ dcos confluent-kafka topic describe kstream-log-count-log-counts-changelog { "partitions": [ { "0": { "leader": -1, "controller_epoch": 3, "isr": [],

Re: Key Value State Store value retention

got it. Thanks. On Wed, Jul 26, 2017 at 3:24 PM, Damian Guy wrote: > The changelog is one created by kafka streams, then it is a compacted topic > and the retention period is irrelevant. If it is one you have created > yourself and isn't compacted, then the data will be retained in the topic > f

Re: handling exceptions in a Kafka Streams application ..

The exception indicates that streams was unable to find that topic-partition on the kafka brokers. Can you verify that it exists? Also, i'm assuming you are on 0.10.2.x? On Wed, 26 Jul 2017 at 10:54 Debasish Ghosh wrote: > Thanks Damien .. this worked. But now after the application restarts, I >

Re: handling exceptions in a Kafka Streams application ..

Thanks Damien .. this worked. But now after the application restarts, I see the following exception .. 09:41:26.516 TKD [StreamThread-1] ERROR > c.l.fdp.sample.kstream.WeblogDriver$ - Stream terminated because of > uncaught exception .. Shutting down app > org.apache.kafka.streams.errors.StreamsEx

Re: Key Value State Store value retention

The changelog is one created by kafka streams, then it is a compacted topic and the retention period is irrelevant. If it is one you have created yourself and isn't compacted, then the data will be retained in the topic for as long as the retention period. If you use a non-compacted topic and the k

Re: Key Value State Store value retention

ok. Thanks. Actually, I had this confusion. Changelog like every Kafka topic would have its retention period, lets say 2 days. and if the value on day1 for key1 = 4 and data for key1 doesnt come for next 3 days. Would it still retail the same value(key1=4) on day4. -Sameer. On Wed, Jul 26, 2017

Re: Kafka Streams 0.10.2.1 client crash - .checkpoint.tmp (No such file or directory)

Hi, Sorry, yes this is a bug to do with file locking and the clean-up thread. For now the workaround is to configure StreamsConfig.STATE_CLEANUP_DELAY_MS_CONFIG to a very large value, i.e., Long.MAX_VALUE. So it is effectively disabled. There are a couple of related JIRAs https://issues.apache.or

Re: Key Value State Store value retention

Sameer, For a KeyValue store the changelog topic is a compacted topic so there is no retention period. You will always retain the latest value for a key. Thanks, Damian On Wed, 26 Jul 2017 at 08:36 Sameer Kumar wrote: > Hi, > > Retention period for state stores are clear(default, otherwise spe

Re: handling exceptions in a Kafka Streams application ..

Hi Debasish, It might be that it is blocked in `streams.close()` You might want to to try the overload that has a long and TimeUnit as params, i.e., `streams.close(1, TimeUnit.MINUTES)` Thanks, Damian On Wed, 26 Jul 2017 at 09:11 Debasish Ghosh wrote: > Hi - > > I have a Kafka streams applicat

handling exceptions in a Kafka Streams application ..

Hi - I have a Kafka streams application deployed on a Mesos DC/OS cluster. While the application was running, Kafka suddenly reported to be unhealthy and the application got an exception .. 07:45:16.606 TKD [StreamThread-1] ERROR c.l.f.s.kstream.WeblogProcessing$ - > Stream terminated because of

kafka-consumer-groups.sh with ssl

Hi, I use kafka_2.11-0.10.0.1 with ssl. I try to see lag between consummers and producers with the kafka-consumer-groups.sh script. #bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server myserver:9092 --list Error while executing consumer group command Java heap space java.lang.OutOfMemo

Key Value State Store value retention

Hi, Retention period for state stores are clear(default, otherwise specified by TimeWindows.until). Intrigued to know the retention period for key values. The use case is something like I am reading from a windowed store, and using plain reduce() with out any time windows. Would the values be ret

Re: Kafka Connect distributed mode rebalance

Btw, if you can share, I would be curious what connectors you're using and why you need so many. I'd be interested if a modification to the connector could also simplify things for you. -Ewen On Wed, Jul 26, 2017 at 12:33 AM, Ewen Cheslack-Postava wrote: > Stephen, > > Cool, that is a *lot* of

Re: Kafka Connect distributed mode rebalance

Stephen, Cool, that is a *lot* of connectors! Regarding rebalances, the reason this happens is that Kafka Connect is trying to keep the total work of the cluster balanced across the workers. If you add/remove connectors or the # of workers change, then we need to go through another round deciding

Re: struggling with runtime Schema in connect

Stephen's explanation is great and accurate :) One of the design goals for Kafka Connect was to not rely on any specific serialization format since that is really orthogonal to getting/sending data from/to other systems. We define the generic *runtime* data API, which is what you'll find in the Ka

Re: Kafka Connect Embedded API

Thanks for the information. I also had the same feeling after spending some time with the current code base. On Wed, Jul 26, 2017 at 12:30 PM, Ewen Cheslack-Postava wrote: > The vast majority of KIP-26 has been implemented. Unfortunately, the > embedded API is still one of the gaps that has not

Re: Kafka Connect Embedded API

The vast majority of KIP-26 has been implemented. Unfortunately, the embedded API is still one of the gaps that has not yet been implemented. It likely requires some additional design work as only a prototype API was proposed in the KIP describing the framework as a whole. -Ewen On Wed, Jul 12, 2