Kafka Streams: finding a solution to a particular use case

Hi, I am a newbie to Kafka Streams and I am using it trying to solve a particular use case. Let me explain. I have two sources of data both like that: Key (string) DateTime (hourly granularity) Value I need to join the two sources by key and date (hour of day) to obtain: Key (string) DateTime

KStreams - reading from the start of a stream

Hi all, I spoke at the London Kafka meetup last night about searching streaming data (write-up here: http://www.flax.co.uk/blog/2016/04/14/apache-kafka-london-meetup-real-time-search-insights/), and as part of the preparation for the talk I tried porting some Samza code I have to the KStreams

Re: Kafka Connect - Source Connector for Apache Cassandra

And this one. We’re adding a source to the sink plus ssl and user/name password support. Kerberos support is planned. https://github.com/datamountaineer/stream-reactor On 14/04/16 01:54, "Joe Stein" wrote: >There is one being worked on here >https://github.com/tuplejump/kafka-connect-cass

Re: Help understanding a failure please.

:) Not lame. Valid question! Part of the problem is that the exception doesn't tell me where the connection refused is coming from. No IP address or hostname or application name is part of the error, so I have no idea to which system the problem is occurring! I was able to ssh to the broker se

Re: Kafka Connect - Source Connector for Apache Cassandra

Thank you all. Andrew, so at this point one can not go this way: Cassandra > Connector_A > Kafka > Connector_B > Some other storage? On Thu, Apr 14, 2016 at 6:08 AM, Andrew Stevenson < and...@datamountaineer.com> wrote: > And this one. We’re adding a source to the sink plus ssl and user/name >

users@kafka.apache.org

Hi Ismael, Thank you for providing link. I was able to resolve the issue. Regards, Bharat On Wed, Apr 13, 2016 at 1:41 AM, Ismael Juma wrote: > Hi Bharat, > > It looks authorization is not configured correctly. I suggest taking a look > at the following blog post that configures authenticatio

Reg: Kafka With Kerberos/SSL [Enhancement to add option, Need suggestions on this]

Hi All, When Kafka is running on kerberoized cluster/ SSL. Can we add an option security.protocol. So, that user can given PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL. This will be helpful during running console producer and console consumer. ./bin/kafka-console-producer.sh --broker-list --topic --

Re: Kafka Streams: finding a solution to a particular use case

Hi Guillermo, 1) Yes in your case, the streams are really a "changelog" stream, hence you should create the stream as KTable, and do KTable-KTable join. 2) Could elaborate about "achieving this"? What behavior do require in the application logic? Guozhang On Thu, Apr 14, 2016 at 1:30 AM, Guil

Re: KStreams - reading from the start of a stream

Hello Alan, Kafka Streams library rely on the underlying Kafka Consumer client for reading data, and you can pass any consumer configs into the StreamsConfig as well. For example in this example you can see that developers can set both streams configs as well as underlying client configs into the

Consumer Client - How to simulate heartbeats ?

Hi everyone, I have a use case where a message can take longer than ' heartbeat.interval.ms' to be processed by my application. As I understand the heartbeats of consumer are done while the poll method is invoked. I would like to instantiate a worker thread to process the messages but I need to w

Re: Kafka Streams: context.forward() with downstream name

Hi Josh, As we chatted offline, would you like to summarize your proposed Transform APIs in a separate JIRA so we can continue our discussions there? Guozhang On Tue, Apr 5, 2016 at 4:13 PM, Guozhang Wang wrote: > HI Josh, > > Re 1): transform is only for usage in the higher-level DSL, while i

Consumer Group not visible for Kafka 0.9 Consumer

Hi all, I am having trouble getting details on a consumer group I am using. I would appreciate the help! Here is what I'm doing. *1. Launch a Kafka 0.9.0.1 and Consumer with group.id =testGroup* *2. I start receiving messages just fine. * *3. I can see my group in the 0.9.0.1 Bro

Re: Consumer Client - How to simulate heartbeats ?

Hi Florian, It's actually OK if processing takes longer than the heartbeat interval, but it does need to finish before the session timeout expires or the consumer will be kicked out of the group (which typically is revealed by commit failures). If the problem is just that the consumer is handling

Re: Consumer Group not visible for Kafka 0.9 Consumer

Hi Alexander, The group management of the new Kafka Consumer is not using Zookeeper. Can you add new-consumer to the command line arguments of /kafka-consumer-groups.sh? Thanks, Liquan On Thu, Apr 14, 2016 at 2:01 PM, Alexander Cook wrote: > Hi all, > > I am having trouble getting details on a

please add me

Re: please add me

Automated reply: thank you for attempting to subscribe to Kafka mailing list. To finish the subscription process send email to users-subscr...@kafka.apache.org < users-subscr...@kafka.apache.org> :) Marko Bonaći Monitoring | Alerting | Anomaly Detection | Centralized Log Management Solr & Elasti

Re: please add me

Also sent to: ggol...@hortonworks.com Marko Bonaći Monitoring | Alerting | Anomaly Detection | Centralized Log Management Solr & Elasticsearch Support Sematext | Contact On Fri, Apr 15, 2016 at 1:47 AM, Guruditta Golani wrote: > >

consumer (java client 2.11 0.9.0.1) poll does not return record in between

I am using 0.9.0.1 JAVA KafkaCosnumer poll API with timeout as long as 1000ms. I am connecting to a topic with 7 million messages. The message is FIX message. When I poll - I see that in between it returns blank records. Why is that happening. Is there a possibility that poll API is returning the

how to config kafka security with plaintext && acl?

how to config kafka security with plaintext && acl? i just want to deny some ips.

RE: Kafka Connect - Source Connector for Apache Cassandra

In a few days you can! Regards Andrew From: Kaz Chehresa Sent: ‎14/‎04/‎2016 19:24 To: users@kafka.apache.org Subject: Re: Kafka Connect - Source Connector for Apache Cassandra Thank you all. Andrew, so at th

RE: Kafka Connect - Source Connector for Apache Cassandra

To clarify is few days we'll have the first cut of the cassandra source to match our sink. You can then mix and match to create your flow. We are working on hosting live demos at www.landoop.com. They have also built a Cloudera CSD to integrate Confluent into Cloudera Manager. We have a number