Re: Architecture recommendations for a tricky use case

> On Sep 29, 2016, at 16:39, Ali Akhtar wrote: > > Why did you choose Druid over Postgres / Cassandra / Elasticsearch? Well, to be clear, we haven’t chosen it yet — we’re evaluating it. That said, it is looking quite promising for our use case. The Druid docs say it well: > Druid is an open

Delayed Queue usecase

Hi, I have a use case where I need to process certain messages only after a certain amount time has elapsed. Does Kafka have any support for time delays? Currently I am putting messages in different queues based on when the message should get processed and at any given time the consumers only po

Re: kafka new consumer is ready for production setup?

The new consumer is absolutely ready for production use. I think this is leftover warning from older releases and should be updated to better reflect the current reality. 0.10.1 is currenlty in RC status (see https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+0.10.1) so that could be wh

Re: Kafka Streams dynamic partitioning

Thanks, I got the point. That solves my problem. On Wed, Oct 5, 2016 at 10:58 PM, Matthias J. Sax wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA512 > > Hi, > > even if you have more distinct keys than partitions (ie, different key > go to the same partition), if you do "aggregate by k

Re: Kafka consumer configuration / performance issues

Sorry to bump this up, can anyone provide some input on this ? I need to make a call soon whether kafka is a good fit to our requirement. On Tuesday, October 4, 2016 8:57 PM, Shamik Banerjee wrote: Hi, I'm a newbie trying out kafka as an alternative to AWS SQS. The motivation primarily is

Re: Kafka Streams dynamic partitioning

-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Hi, even if you have more distinct keys than partitions (ie, different key go to the same partition), if you do "aggregate by key" Streams will automatically separate the keys and compute an aggregate per key. Thus, you do not need to worry about wh

Re: Kafka Streams dynamic partitioning

Hi, @Ali IMO, Yes. That is the job of kafka server to assign kafka instances partition(s) to process. Each instance can process more than one partition but one partition cannot be processed by more than one instance. @Michael, Thanks for reply. >Rather, pick the number of partitions in a way

Re: kafka stream to new topic based on message key

Hello Gary, 1. The InternalTopicManager is only used by the Streams-instantiated PartitionAssignor to create internal topics for auto-repartitioning and changelog. 2. About "RecordCollector.Supplier": you are right, and as I wrote in the above email you have to force casting it to RecordCollecto

Re: difficulty to delete a topic because of its syntax

You *could* go in to zookeeper and nuke the topic, then delete the files on disk Slightly more risky but it should work On Wednesday, 5 October 2016, Manikumar wrote: > Kafka doesn't support white spaces in topic names. Only support '.', '_' > and '-' these are allowed. > Not sure how you got

Kafka 0.8.1.1 and JDK1.8

Team, Currently we have kafka 0.8.1.1 running with Idk 1.6. Wanted to verify the compatibility of JDK 1.8 with kafka version 0.8.1.1. is Kafka 0.8.1.1 is compatible with jdk1.8, or any other patches have to be applied before upgrading idk 1.6 to idk 1.8. NOTE: along with upgrade to jdk1.8 , we ar

Re: Kafka Streams dynamic partitioning

> It's often a good idea to over-partition your topics. For example, even if today 10 machines (and thus 10 partitions) would be sufficient, pick a higher number of partitions (say, 50) so you have some wiggle room to add more machines (11...50) later if need be. If you create e.g 30 partitions,

Re: Kafka Streams dynamic partitioning

> So, in this case I should know the max number of possible keys so that > I can create that number of partitions. Assuming I understand your original question correctly, then you would not need to do/know this. Rather, pick the number of partitions in a way that matches your needs to process the

Re: difficulty to delete a topic because of its syntax

Kafka doesn't support white spaces in topic names. Only support '.', '_' and '-' these are allowed. Not sure how you got white space in topic name. On Wed, Oct 5, 2016 at 8:19 PM, Hamza HACHANI wrote: > Well ackwardly when i list the topics i find it but when i do delete it it > says that this

Re: kafka stream to new topic based on message key

Guozhang. I was just looking at the source for this, and it looks like the RecordCollector.Supplier is part of the internal ProcessorContextImpl class. I don't think that's exposed to me, is it? If I create a processor class that extends AbstractProcess, it only has access to the ProcessorContext

RE: difficulty to delete a topic because of its syntax

Well ackwardly when i list the topics i find it but when i do delete it it says that this topic does not exist. De : Ben Davison Envoyé : mercredi 5 octobre 2016 02:37:14 À : users@kafka.apache.org Objet : Re: difficulty to delete a topic because of its syntax T

Re: difficulty to delete a topic because of its syntax

Try putting "" or '' around the string when running the command. On Wed, Oct 5, 2016 at 3:29 PM, Hamza HACHANI wrote: > It's between "the" and "metric" > > > De : Ali Akhtar > Envoyé : mercredi 5 octobre 2016 02:16:33 > À : users@kafka.apache.org > Objet : Re: d

kafka new consumer is ready for production setup?

Hi, I want to use kafka new consumer. But in the documentation of 0.10.0 version, it is mentioned that "The code is considered beta quality." I am using the latest version which is 0.10.1. I am not able to find out any documentation for this version. Can you please tell me, if the new consumer is

Support for Kafka

Dear Kafka team. I am in the Implementation stage of Kafka cluster and looking to find out does Apache Kafka supported for Ubuntu 16.04 LTS - Xenial. Would be great if you please let us know. [The Exchange Lab] Syed Hussaini Infrastructure Engineer 1 Ne

RE: difficulty to delete a topic because of its syntax

It's between "the" and "metric" De : Ali Akhtar Envoyé : mercredi 5 octobre 2016 02:16:33 À : users@kafka.apache.org Objet : Re: difficulty to delete a topic because of its syntax I don't see a space in that topic name On Wed, Oct 5, 2016 at 6:42 PM, Hamza HACHA

Re: difficulty to delete a topic because of its syntax

I don't see a space in that topic name On Wed, Oct 5, 2016 at 6:42 PM, Hamza HACHANI wrote: > Hi, > > I created a topic called device-connection-invert-key-value-the > metric-changelog. > > I insit that there is a space in it. > > > > Now that i want to delete it because my cluster can no longe

difficulty to delete a topic because of its syntax

Hi, I created a topic called device-connection-invert-key-value-the metric-changelog. I insit that there is a space in it. Now that i want to delete it because my cluster can no longer work correctly i can't do it as it only reads the first part of it : ( device-connection-invert-key-valu

Re: kafka stream to new topic based on message key

What if we were to use kafka connect instead of streams? Does it have the ability to specify partitions, rf, segment size etc? On 5 October 2016 at 09:42, Gary Ogden wrote: > Thanks Guozhang. > > So there's no way we could also use InternalTopicManager to specify the > number of partitions and R

Re: kafka stream to new topic based on message key

Thanks Guozhang. So there's no way we could also use InternalTopicManager to specify the number of partitions and RF? https://github.com/apache/kafka/blob/0.10.1/streams/src/ main/java/org/apache/kafka/streams/processor/internals/ InternalTopicManager.java On 4 October 2016 at 19:34, Guozhang Wa

Re: kafka streams with dynamic content and filtering

Sorry. I responded to the wrong message On 5 October 2016 at 09:40, Gary Ogden wrote: > Thanks Guozhang. > > So there's no way we could also use InternalTopicManager to specify the > number of partitions and RF? > > https://github.com/apache/kafka/blob/0.10.1/streams/src/ > main/java/org/apa

Re: kafka streams with dynamic content and filtering

Thanks Guozhang. So there's no way we could also use InternalTopicManager to specify the number of partitions and RF? https://github.com/apache/kafka/blob/0.10.1/streams/src/main/java/org/apache/kafka/streams/processor/internals/InternalTopicManager.java On 4 October 2016 at 19:25, Guozhang Wa

Re: Kafka Streams windowed aggregation

Hi Davood, The behaviour is indeed as you say. Recently we checked in KIP-63 in trunk (it will be part of the 0.10.1 release coming up). That should reduce the amount of downstream traffic you see (https://cwiki.apache.org/confluence/display/KAFKA/KIP-63%3A+Unify+store+and+downstream+caching+in

Re: Consumer offsets reset for _all_ topics after increasing partitions for one topic

Does anyone know about this? Altering topic partitions seems to reset consumer offsets. On Tue, Sep 27, 2016 at 1:28 PM, Juho Autio wrote: > I increased partitions for one existing topic (2->10), but was surprised > to see that it entirely reset the committed offsets of my consumer group. > > Al

Re: 0.10.1.0 RC0 release notes fix suggestion

Thanks Enrico. In general, we should avoid setting "Fix version" for duplicate issues for the reason you state. I fixed the case you reported, but I haven't checked if there are other ones. Ismael On Wed, Oct 5, 2016 at 8:35 AM, Enrico Olivelli wrote: > Hi Jason, > I see in the release notes th

Kafka Streams windowed aggregation

Hi, I want to do windowed aggregation with streams library. However, I get the output from particular operator immediately, independent of window size. This makes sense for unlimited windows or sometimes for event time windows. However, for ingestion time or processing time windows, users may want

0.10.1.0 RC0 release notes fix suggestion

Hi Jason, I see in the release notes that this issue seems to be fixed, but is marked as 'duplicate' https://issues.apache.org/jira/browse/KAFKA-156 Maybe you can consider removing this kind of un-fixed issues from the release notes. In my case I'm waiting for that fix and I 'was' very happy to se

Re: Kafka Streams dynamic partitioning

Hi Guozhang, So, in this case I should know the max number of possible keys so that I can create that number of partitions. Thanks Adrienne On Wed, Oct 5, 2016 at 1:00 AM, Guozhang Wang wrote: > By default the partitioner will use murmur hash on the key and mode on > current num.partitions to