Why are the per second metrics for messages and bytes not useful for
capacity planning? I can't think of a situation where knowing the number of
metrics would be more useful. If you really want that, you can always
extrapolate the per second number and get an approximation.
--
Sharninder
O
27;t think it makes sense to have total metrics per day for
brokers/producers. There are per second metrics such as MessagesInPerSec at
brokers that you can use to gauge throughput. If you really want total
metrics, measure at your client/producer end yourself.
--
Sharninder
d then it should be taken care of
> properly.
>
> I googled out but didn’t find any satisfactory answers.
> Please suggest.
>
> Thanks,
> NJ Dash
--
--
Sharninder
I don't have benchmarks but multiple consumer groups are possible. For Kafka
the performance should be similar or close to as having multiple consumers
using a single group.
_
From: Senthil Kumar
Sent: Tuesday, January 24, 2017 10:38 PM
Subject: Kaf
2017 at 6:26 AM, Cas Apanowicz wrote:
> Hi,
>
> I have general understanding of main Kafka functionality as a streaming
> tool.
> However, I'm trying to figure out if I can use Kafka to read Hadoop file.
> Can you please advise?
> Thanks
>
> Cas
>
>
--
--
Sharninder
com/> | *jsm...@clearsense.com
> * | Follow on Twitter
> <https://twitter.com/ClearsenseCan> | Connect on LinkedIn
> <https://www.linkedin.com/>
>
>
>
--
--
Sharninder
learning. We
> > need to backup the messages somewhere so that the data scientists can
> > query/load them.
> >
> > So we need something like a router that just opens up a new consumer
> group
> > which just keeps on storing them to S3.
> >
> > On Tue, Dec 6
Why not just have a parallel consumer read all messages from whichever topics
you're interested in and store them wherever you want to? You don't need to
"backup" Kafka messages.
_
From: Aseem Bansal
Sent: Tuesday, December 6, 2016 4:55 PM
Subject: S
t; pace. Any idea what could be the reason?
>
> Thanks
> Achintya
>
>
--
--
Sharninder
Do you have only one partition in the topic? The way Kafka works is that all
messages are first distributed into partitions in the topic and then the
consumers are distributed among them and they read them sequentially.
If you have only one partition in the topic, all your messages will be in it
processes for each will
> increase
> > the overall CPU available and therefore the throughput.
> >
> > One day I will spend time improving the CPU performance of kafka-python,
> > but probably not in the near term.
> >
> > -Dana
>
>
--
--
Sharninder
I'd second Tom here. Create a ram disk or just let Kafka write to disk. Use
compression and batch messages and the OS fscache would take care of the rest.
Kafka is pretty fast and you probably won't notice.
_
From: Tom Crayford
Sent: Friday, July 1
requires 0.10
On Thu, Jul 14, 2016 at 6:08 AM, Matt Anderson
wrote:
> Is the new Kafka Streams API compatible with Kafka 0.9.x API and Broker or
> does it require v0.10.x?
>
> Thanks,
> Matt
>
--
--
Sharninder
llion messages a second of discrete 150 byte
> messages?
>
>
>
> Kind regards,
>
>
>
> Jahn Roux
>
>
>
>
>
> ---
> This email has been checked for viruses by Avast antivirus software.
> https://www.avast.com/antivirus
>
--
--
Sharninder
itions, is
> that right?
>
> Does anyone know more about the key and your role inside kafka?
>
> []s
>
--
--
Sharninder
What kind of queue are you looking for? Kafka works as a nice FIFO queue by
default anyway.
On Mon, Mar 28, 2016 at 5:19 PM, Vinod Kakad wrote:
> Can anybody share any good example(code) for kafka as a queue
> implementation?
>
> Thanks & Regards,
> Vinod Kakad.
>
--
--
Sharninder
This topic comes up often on this list. Kafka can be used as a datastore if
that’s what your application wants with the caveat that Kafka isn’t designed to
keep data around forever. There is a default retention time after which older
data gets deleted. The high level consumer essentially reads d
t need
kafka or any other component, don't get it. All said and done, it still is
one extra component to manage.
--
Sharninder
On Mon, Feb 15, 2016 at 6:07 AM, Todd Snyder wrote:
> So long as you put some basic monitoring in place, it should run nicely
> with very little interventio
/2014/10/12/kafka-high-level-consumer-frequently-missing-pieces/
--
Sharninder
On Wed, Feb 10, 2016 at 9:11 PM, Joe San wrote:
> I'm following the ConsumerGroup example,
>
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
>
> How can I specify the b
Kafka uses and relies on OS level caching heavily. Likely you're just
measuring that. No need to worry unless you're actually seeing any GC
related problems.
--
Sharninder
On Tue, Nov 10, 2015 at 4:44 PM, Birendra Kumar Singh
wrote:
> I am looking to understand the RAM usage of
Sounds like an app design decision. What help can this list give you ?
> On 20-Oct-2015, at 8:07 PM, Kudumula, Surender
> wrote:
>
> Dear sir/madam
> I have a query. We are working on POC at the moment and we are using kafka to
> produce and consume messages. I have one component which consu
Sorry, you're correct. Looks like a silly mistake on my side. I'll check my
imports
--
Sharninder
On Mon, Sep 28, 2015 at 6:24 PM, Gwen Shapira wrote:
> KafkaProducer takes ProducerRecord as an argument to send().
>
> See the code:
>
> https://github.com/apache/kaf
kafka/clients/producer/KafkaProducer.html
Is there an equivalent Java API for 0.8.2 yet or is the older one the most
current?
--
Sharninder
On Mon, Sep 28, 2015 at 9:15 AM, Gwen Shapira wrote:
> KafkaProducer is the most current and full-featured one, and it should be
> used.
&g
As far as I know, with a consumer group implementation you cannot pin consumers
to partitions. That logic is taken care of by the high level API on its own.
> On 23-Sep-2015, at 6:38 AM, Spandan Harithas Karamchedu
> wrote:
>
> Hi,
>
> We created a topic with 3 partitions and a replication f
t; Many thanks,
> Petr
>
--
--
Sharninder
Well, you're right. But that's not what I meant. I meant using consumers
with the same consumer id. With different IDs, obviously the problem (and
solution) changes. But then I think the OP is just trying to twist his
problem to fit in the kafka way of things.
--
Sharninder
On Sat, A
You can't. Kafka is essentially a queue, so you always read messages one by
one. What you can do is disable auto offset commit, read 100 messages, process
them and then manually commit offset.
--
Sharninder
> On 04-Aug-2015, at 9:07 pm, shahab wrote:
>
> Hi,
>
> While
I don't know of any limits as such but I don't think your problem is suitable
for Kafka. The third point especially wouldn't work with Kafka. Using Kafka,
only one consumer will get a message out of the 30k
> On 03-Aug-2015, at 10:39 am, Vaibhav Kirte wrote:
>
> Hi,
>
> I need to know how m
The producer can use a key to determine which partition to put data in. For
example we use a partitioning scheme based on the header of the message so that
all messages with a specific header go to a specific partition. The consumer
can also include the same logic to process those messages.
>
You can have the same consumer id and Kafka will balance partitions across the
two instances automatically. When one of them dies the partitions are
rebalanced and assigned to the remaining alive consumers.
_
From: Panda, Samaresh
Sent: Friday, June 5, 2015 7:
nly commit
when you're sure the message has been acknowledged by the next stage.
I think the link is this ->
http://ingest.tips/2014/10/12/kafka-high-level-consumer-frequently-missing-pieces/
Regards,
Sharninder
another process will decide when to spawn new customers.
--
Sharninder
On Wed, Apr 29, 2015 at 11:58 PM, Nimi Wariboko Jr
wrote:
> Hi,
>
> I was wondering what options there are/what other people are doing for
> horizontally scaling kafka consumers? Basically if I have 100 partiti
I haven't been following closely but getting rid of zookeeper is in the
pipeline. Look up 0.9 plans. They're somewhere on the wiki.
Sent from my iPhone
> On 14-Nov-2014, at 5:18 pm, Stevo Slavić wrote:
>
> Hello Apache Kafka community,
>
> Is it already possible to configure/use a different
gt; > > > > > What is that I can do to keep both of my consumer busy?
> > > > > > >
> > > > > > > Thank you
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Regards,
> > > > > > > Sandeep Palur
> > > > > > > Data-Intensive Distributed Systems Laboratory, CS/IIT
> > > > > > > Department of Computer Science, Illinois Institute of
> Technology
> > > > (IIT)
> > > > > > > Phone : 312-647-9833
> > > > > > > Email : psand...@hawk.iit.edu
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Regards,
> > > > > Sandeep Palur
> > > > > Data-Intensive Distributed Systems Laboratory, CS/IIT
> > > > > Department of Computer Science, Illinois Institute of Technology
> > (IIT)
> > > > > Phone : 312-647-9833
> > > > > Email : psand...@hawk.iit.edu
> > > > >
> > > >
> > >
> >
>
>
>
> --
> Regards,
> Sandeep Palur
> Data-Intensive Distributed Systems Laboratory, CS/IIT
> Department of Computer Science, Illinois Institute of Technology (IIT)
> Phone : 312-647-9833
> Email : psand...@hawk.iit.edu
>
--
--
Sharninder
ra.com-1412792871089-cabd4934-0
> flume t1 2 66829740
> 8273338015903640
> flume_kafkacdh-4.ent.cloudera.com-1412793053882-9bfddff9-0
>
> Isn't Kafka the best thing ever? :)
>
> Gwen
>
> On Wed, Oct 8, 2014 at 11:23 AM, Gwen Shapira
Thanks Gwen.
When you're saying that I can add consumers to the same group, does that
also hold true if those consumers are running on different machines? Or in
different JVMs?
--
Sharninder
On Wed, Oct 8, 2014 at 11:35 PM, Gwen Shapira wrote:
> If you use the high level
On similar lines,
how do you guys handle consumer failures? Suppose one consumer process gets
an exception and crashes, is it possible for me to somehow make sure that
there is another process that is still reading the queue for me?
--
Sharninder
consumer example was that replaying needs a byte parameter, but maybe I
didn't look hard enough.
--
Sharninder
On Thu, Sep 25, 2014 at 10:15 PM, pankaj ojha
wrote:
> Thank You. I will try this out.
>
> On Thu, Sep 25, 2014 at 10:01 PM, Gwen Shapira
> wrote:
>
> > Usi
Agree. Why not write a service that reads from existing topics and writes
to new ones. Point existing producers to the new partitions wait for the
service to finish reading. Since you have only one partition right now, I'd
assume you don't have any partitioning logic per se. Think about that since
For your first question, I don't know of a way to do that.
For the second question, consumers commit the offset after every/a few
reads. When the consumer restarts, it starts from the last committed offset.
On Tue, Sep 9, 2014 at 7:23 PM, siddharth ubale
wrote:
> Hi Guys,
>
> I wanted to know
s for a certain
time, don't think of it as a data store. Kafka is a streaming system, think
of it as a fast queue that gives you the ability to move your pointer back.
--
Sharninder
On Fri, Sep 5, 2014 at 4:27 PM, Aris Alexis
wrote:
> Thanks for the reply. If I use it only for activi
itely.
So, for example all activity streams can go into kafka from where consumers
will pick up messages to parse and put them to hbase or other clients.
--
Sharninder
On Fri, Sep 5, 2014 at 12:05 AM, Aris Alexis wrote:
> Hello,
>
> I am building a big web application that I want t
java??
>
> Thanks,
> Siddharth Ubale
>
>
> On Thu, Aug 21, 2014 at 11:48 AM, Sharninder wrote:
>
> > there are kafka producers/consumers in a lot of languages. There is an
> > Ecosystem page somewhere in the wiki. Take a look at that.
> >
> > Finally, I
e the other way round.
--
Sharninder
On Thu, Aug 21, 2014 at 11:40 AM, siddharth ubale wrote:
> Hi ,
> Thanks for the quick follow up Philip.
> Also , can you let me know whether the kafka implementation can be as
> versatile as ActiveMQ with regards to connectivity?
> I m
, which ironically, is more complex than the high level consumer.
In short, if you have a usecase where you want to read from a specific
partition, you will need to implement a simple consumer.
--
Sharninder
Josh
> .
>
> On Thu, Aug 14, 2014 at 4:27 PM, Neha Narkhede
> wrote:
Implement the low level "Simple Consumer".
--
Sharninder
On Thu, Aug 14, 2014 at 2:16 PM, Josh J wrote:
> Hi,
>
> Suppose I have N partitions. I would like to have X different consumer
> threads ( X < N) read from a specified set of partitions. How can I achieve
> this?
>
> Thanks,
>
> Josh
>
like while iterator.hasNext() to get
messages from the queue. This will not return unless you explicitly call
shutdown which means you will have to write a condition which calls
shutdown based on the message contents.
--
Sharninder
On Tue, Aug 5, 2014 at 10:51 AM, anand jain wrote:
> Hi,
>
> I just
https://github.com/baniuyao/flume-ng-kafka-source
On Sun, Aug 3, 2014 at 6:15 PM, rafeeq s wrote:
> Hi,
>
> We are planning to use kafka as *flume source*. Please advice me, how to
> use kafka as source in flume.
>
> please share if there is any best example of *flume- kafka source- hdfs
> sin
You can add partitions at runtime but not delete (AFAIK, someone correct me
if I'm wrong). The reason it's usually discouraged is that when partitions
get added to a topic, the message assignment changes and if you're
application depends on some partitioning logic, that will fail.
49 matches
Mail list logo
