Re: programmatic way to check for topic existence?

You could just catch the exception but if this is per row, that is probably prohibitively expensive. Doesn't the binlog get "create table" events? Wouldn't that be a better time to create the topic? -- Andy On Mon, Oct 24, 2016 at 2:32 PM, Ben Osheroff wrote: > Hiya! > > I've been trying to me

RE: customised event time

Merci Guoahang. Bonne journée. De : Guozhang Wang Envoyé : lundi 24 octobre 2016 16:50:45 À : users@kafka.apache.org Objet : Re: customised event time Hi Hamza, You can create a windowed store in the processor API via the Stores factory class: org.apache.kafka.

Re: Consumer error : This consumer has already been closed

I can see that you have done consumer.subscribe(Arrays.asList(topic)); I don't know if this is a problem but try consumer.subscribe(Collections. singletonList(this.topic)); You can refer to official example code of Kafka at github[1]. Hope that helps. [1] https://github.com/apache/kafka/blob/trun

handleFetchRequest throw exception

Hi all, I use Kafka 0.9.0.0, and we have a cluster with 6 nodes, when I restart a broker，we find there are many logs as below， [2016-10-24 15:29:00,914] ERROR [KafkaApi-2141642] error when handling request Name: FetchRequest; Version: 1; CorrelationId: 4928; ClientId: ReplicaFetcherThre

Re: programmatic way to check for topic existence?

Have a look at the Cluster which has a "topic" method to get a set of all the topics. https://kafka.apache.org/0100/javadoc/org/apache/kafka/common/Cluster.html In version 8/9, there was also the ZKUtils, but the desire is to have clients not to interrogate ZK directly. On 10/24/16, 4:32 PM, "

KafkaProducer.java topics() call?

Hey all, Bit of curiosity that came out of the background to this thread: http://search-hadoop.com/m/Kafka/uyzND1lLYXb136UfJ1?subj=programmatic+way+to+check+for+topic+existence+ I noticed that KafkaConsumer.java has the ability to list topics ( https://github.com/apache/kafka/blob/trunk/clients/s

Re: customised event time

Thanks! :) On Tue, Oct 25, 2016 at 12:20 AM, Hamza HACHANI wrote: > Merci Guoahang. > > Bonne journée. > > > De : Guozhang Wang > Envoyé : lundi 24 octobre 2016 16:50:45 > À : users@kafka.apache.org > Objet : Re: customised event time > > Hi Hamza, > > You can c

Re: kafka streaming rocks db lock bug?

Logs would be very helpful, I can look further into it. Thanks Ara. On Mon, Oct 24, 2016 at 11:04 PM, Ara Ebrahimi wrote: > This was in 10.1.0. What happened was that a kafka broker went down and > then this happened on the kafka streaming instance which had connection to > this broker. I can s

Re: programmatic way to check for topic existence?

Jayesh, Thanks, but AFAICT that's just the metadata "shell" -- the Cluster object just holds information about a cluster, it has no functionality to actually retrieve it. I looked briefly around the source and couldn't find a (public) API that might fill out the Cluster object. KafkaConsumer has s

Re: Exception when accessing partition, offset and timestamp in processor class

-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Hi, sorry for late reply. Seems like a bug to me; within Processor#process() accessing the context should work. Can you open a JIRA for it? - -Matthias On 10/23/16 10:28 PM, saiprasad mishra wrote: > Sorry for the email again > > I was expecting

Re: programmatic way to check for topic existence?

We won't proceed in the face of a missing table, we'll just crash. But it's still a bad experience for us and the end user; we have to guess that maybe a TimeoutException means a missing topic, and we also have to wait the N seconds (default 60) for the thing to "timeout". On Tue, Oct 25, 2016 at

Re: SSL between Kafka and Spark Streaming API

Hi Everyone, I am trying to enable the SSL between my Apache Spark 1.4.1 and Kafka 0.9.0.0 and I am using spark-streaming-kafka_2.10 Jar to connect to Kafka and I am using KafkaUtils.createDirectStream method to read the data from Kafka topic. Initially, I got OOM issue and I have resolved

Re: programmatic way to check for topic existence?

Ben, You can list all the available topic information and do a simple look up from the returned list. Map> topics = consumer.listTopics(); topics.contains () - isn't enough? -- Kamal On 25 Oct 2016 22:56, "Ben Osheroff" wrote: > We won't proceed in the face of a missing table, we'll just crash

Re: Exception when accessing partition, offset and timestamp in processor class

Hi Matthias Thanks for the reply. I think I don't have permission for this. If you can grant me permission I can create one (my handle is saimishra). Or you can go ahead and create one I may need permission to create JIRA as I might report more issues after discussing with you over here. Regards

Re: programmatic way to check for topic existence?

There's a variety of ways (including consumer.listTopics(), or catching the TimeoutException) that are "kinda OK", I was hoping for something a bit cleaner. The problems with the consumer.listTopics() method: - I have to instantiate a Consumer object in an application that only ever produces ob

Re: Exception when accessing partition, offset and timestamp in processor class

-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 What is your JIRA ID? We can add you to the contributor list to give you permission. - -Matthias On 10/25/16 10:48 AM, saiprasad mishra wrote: > Hi Matthias Thanks for the reply. I think I don't have permission > for this. If you can grant me perm

Re: Exception when accessing partition, offset and timestamp in processor class

My JIRA id is saimishra Regards Sai On Tue, Oct 25, 2016 at 10:55 AM, Matthias J. Sax wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA512 > > What is your JIRA ID? We can add you to the contributor list to give > you permission. > > - -Matthias > > > On 10/25/16 10:48 AM, saiprasad mishr

Re: Exception when accessing partition, offset and timestamp in processor class

Just created the JIRA https://issues.apache.org/jira/browse/KAFKA-4344 Regards Sai On Tue, Oct 25, 2016 at 11:59 AM, saiprasad mishra < saiprasadmis...@gmail.com> wrote: > My JIRA id is saimishra > > Regards > Sai > > On Tue, Oct 25, 2016 at 10:55 AM, Matthias J. Sax > wrote: > >> -BEGIN P

[VOTE] Add REST Server to Apache Kafka

Hi All, We are proposing to have a REST Server as part of Apache Kafka to provide producer/consumer/admin APIs. We Strongly believe having REST server functionality with Apache Kafka will help a lot of users. Here is the KIP that Mani Kumar wrote https://cwiki.apache.org/confluence/disp

issue with custom processor flush to rocksdb store

Hi This is with version 10.1.0 kafka streams (server running in remote and streams app running local in my laptop). I have a kafka stream pipeline like this source topic(with 10 partitions) stream -> filter for null value ->map to make it keyed by id ->custom processor to mystore(persistent) I

Re: [VOTE] Add REST Server to Apache Kafka

+ 1 On Tue, Oct 25, 2016 at 10:16 PM, Harsha Chintalapani wrote: > Hi All, >We are proposing to have a REST Server as part of Apache Kafka > to provide producer/consumer/admin APIs. We Strongly believe having > REST server functionality with Apache Kafka will help a lot of users. >

Re: [VOTE] Add REST Server to Apache Kafka

-1 I think the REST server for Kafka that already exists is quite good and getting contributions. Moving this into the core project doesn't solve a problem that I see. -Jay On Tue, Oct 25, 2016 at 2:16 PM, Harsha Chintalapani wrote: > Hi All, >We are proposing to have a REST Server

Re: [VOTE] Add REST Server to Apache Kafka

+1. On Tue, Oct 25, 2016 at 2:58 PM, Jay Kreps wrote: > -1 > > I think the REST server for Kafka that already exists is quite good and > getting contributions. Moving this into the core project doesn't solve a > problem that I see. > > -Jay > > On Tue, Oct 25, 2016 at 2:16 PM, Harsha Chintalapan

Re: [VOTE] Add REST Server to Apache Kafka

-1 As an end-user, while I like the idea in theory, in practice I don't think it's a good idea just yet. Certainly, it'd be useful, enabling things like https://github.com/Landoop/kafka-topics-ui to work without needing anything outside of Kafka core. But there are already enough things in the e

Re: [VOTE] Add REST Server to Apache Kafka

-1 I fully agree with Jay and Jeff. On Wed, Oct 26, 2016 at 12:03 AM, Jeff Widman wrote: > -1 > > As an end-user, while I like the idea in theory, in practice I don't think > it's a good idea just yet. > > Certainly, it'd be useful, enabling things like > https://github.com/Landoop/kafka-topics

Re: [VOTE] Add REST Server to Apache Kafka

Jeff, Thanks for participating. We already have the discussion thread going on. Please add your thoughts there . I'll keep this as interest check vote thread. Thanks, Harsha On Tue, Oct 25, 2016 at 3:12 PM Stevo Slavić wrote: > -1 > > I fully agree with Jay and Jeff. > > On Wed, Oct 26, 20

Re: [VOTE] Add REST Server to Apache Kafka

+1. This is an http access to core Kafka. This is very much needed as part of Apache Kafka under ASF governance model. This would be great for the community instead of duplicated and splintered efforts that may spring up. Get Outlook for iOS

Re: [VOTE] Add REST Server to Apache Kafka

+1 On Tue, Oct 25, 2016 at 3:17 PM, Harsha Chintalapani wrote: > Jeff, > Thanks for participating. We already have the discussion thread going > on. Please add your thoughts there . I'll keep this as interest check vote > thread. > Thanks, > Harsha > > On Tue, Oct 25, 2016 at 3:12 PM Stev

Re: [VOTE] Add REST Server to Apache Kafka

-1 for all the reasons that have been described before. This does not need to be part of the core project. On Tue, Oct 25, 2016 at 3:25 PM, Suresh Srinivas wrote: > +1. > > This is an http access to core Kafka. This is very much needed as part of > Apache Kafka under ASF governance model. This

Re: Exception when accessing partition, offset and timestamp in processor class

-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Thx! :) On 10/25/16 2:03 PM, saiprasad mishra wrote: > Just created the JIRA > > https://issues.apache.org/jira/browse/KAFKA-4344 > > Regards Sai > > On Tue, Oct 25, 2016 at 11:59 AM, saiprasad mishra < > saiprasadmis...@gmail.com> wrote: > >>

Re: [VOTE] Add REST Server to Apache Kafka

-1 On Tuesday, October 25, 2016, Sriram Subramanian wrote: > -1 for all the reasons that have been described before. This does not need > to be part of the core project. > > On Tue, Oct 25, 2016 at 3:25 PM, Suresh Srinivas > > wrote: > > > +1. > > > > This is an http access to core Kafka. This

Re: [VOTE] Add REST Server to Apache Kafka

+1 On Wed, 26 Oct 2016 at 8:00 AM craig w wrote: > -1 > > On Tuesday, October 25, 2016, Sriram Subramanian wrote: > > > -1 for all the reasons that have been described before. This does not > need > > to be part of the core project. > > > > On Tue, Oct 25, 2016 at 3:25 PM, Suresh Srinivas > >

Re: [VOTE] Add REST Server to Apache Kafka

+1. I hope there will be a corresponding Java library for doing admin functionality. On Wed, Oct 26, 2016 at 4:10 AM, Jungtaek Lim wrote: > +1 > > > On Wed, 26 Oct 2016 at 8:00 AM craig w wrote: > > > -1 > > > > On Tuesday, October 25, 2016, Sriram Subramanian > wrote: > > > > > -1 for all the

Re: [VOTE] Add REST Server to Apache Kafka

And this will make adding health checks via Kubernetes easy. On Wed, Oct 26, 2016 at 4:12 AM, Ali Akhtar wrote: > +1. I hope there will be a corresponding Java library for doing admin > functionality. > > On Wed, Oct 26, 2016 at 4:10 AM, Jungtaek Lim wrote: > >> +1 >> >> >> On Wed, 26 Oct 2016

Re: [VOTE] Add REST Server to Apache Kafka

I am not able to link the whole thread on REST server discussion thread from dev list properly. But if you go here http://mail-archives.apache.org/mod_mbox/kafka-dev/201610.mbox/thread look for subject "[DISCUSS] KIP-80: Kafka REST Server" Thanks, Harsha On Tue, Oct 25, 2016 at 4:13 PM Ali Akhta

Re: [VOTE] Add REST Server to Apache Kafka

+1 Thanks, Jun On Tue, Oct 25, 2016 at 2:16 PM, Harsha Chintalapani wrote: > Hi All, >We are proposing to have a REST Server as part of Apache Kafka > to provide producer/consumer/admin APIs. We Strongly believe having > REST server functionality with Apache Kafka will help a lot

Re: [VOTE] Add REST Server to Apache Kafka

Sorry a typo. -1 instead. Thanks, Jun On Tue, Oct 25, 2016 at 4:24 PM, Jun Rao wrote: > +1 > > Thanks, > > Jun > > On Tue, Oct 25, 2016 at 2:16 PM, Harsha Chintalapani > wrote: > >> Hi All, >>We are proposing to have a REST Server as part of Apache Kafka >> to provide producer/co

{Kafka V 0.10.1}Producing message takes approximately 1 minute if any error occurs at async producer

Hi; Im using asynchronous producer to send messages to the broker. If there is no any error in my callback, i see it takes approximately 0.3 seconds to produce message. But when I get below error [1], I see it takes 60 seconds to produce message. But I do not see any message loss . All messages are