and the leader may not have been elected yet. You
> shouldn't see it often though.
>
> Thanks,
>
> Jun
>
> On Tue, Nov 27, 2012 at 1:11 PM, Chris Curtin >wrote:
>
> > Hi,
> >
> > I noticed several errors when writing to a topic with 5 partition
Hi,
First, thanks for 0.8.0 I'm really impressed with the redundancy
and simplification of the producer and consumer models.
I've upgraded my consumers from 0.7.2 to 0.8.0 and have some questions.
I am using the Simple Consumer since I need to support replay of messages
at request from the clien
n
>
> On Wed, Nov 28, 2012 at 5:10 AM, Chris Curtin >wrote:
>
> > Hi Jun,
> >
> > No, all 9 brokers are up and when I look at the files in
> /opt/kafka-[]-logs
> > there is data for partition 0 of that topic on 3 different brokers.
> >
> > After confir
Hi,
I was able to implement my own lookup code but have a few concerns about
this long term:
- the Broker class is marked as 'private' in the Scala code. IntelliJ gives
me an error about using it, but the runtime lets me use it and get the
host/port out.
- I have to know a lot about the structure
Hi Jun,
Couldn't make either happen again with a clean start (removed all Kafka and
Zookeeper configuration and data files).
Thanks,
Chris
On Wed, Nov 28, 2012 at 12:12 PM, Chris Curtin wrote:
> Hi Jun,
>
> Sorry, neither the missing 0 leader or all those WARN messag
,
>
> Jun
>
> On Mon, Dec 3, 2012 at 7:53 AM, Chris Curtin
> wrote:
>
> > Hi,
> >
> > I was able to implement my own lookup code but have a few concerns about
> > this long term:
> > - the Broker class is marked as 'private' in the Scala cod
know.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+papers+and+presentations
>
> -Jay
>
>
> On Wed, Jan 16, 2013 at 7:26 AM, Chris Curtin >wrote:
>
> > Hi,
> >
> > I'm going to be presenting an introduction to Kafka to the Atlanta Java
>
Zookeeper
>
> In 0.8, there is a jmx bean (ConsumerLag) in the consumer under
> kafka.server that monitors the lag of each partition in terms of messages.
> We also have a command line tool ConsumerOffsetChecker.
>
> Thanks,
>
> Jun
>
> On Mon, Feb 18, 2013 at
nd to balance the leaders again.
>
> Also, I recommend that you try the latest code in 0.8. A bunch of issues
> have been fixes since Jan. You will have to wipe out all your ZK and Kafka
> data first though.
>
> Thanks,
>
> Jun
>
> On Mon, Mar 4, 2013 at 8:32 AM, Chris C
I was able to reproduce today after upgrading to HEAD (as of 3/4/2013)
JIRA:
https://issues.apache.org/jira/browse/KAFKA-784
On Mon, Dec 3, 2012 at 11:09 AM, Chris Curtin wrote:
> Hi Jun,
>
> Couldn't make either happen again with a clean start (removed all Kafka
&
1, which means that the producer has to wait for the message to be
> received by the leader. More recently, we did the actual implementation of
> ack=0, which means the producer doesn't wait for the message to reach the
> leader and therefore it is much faster.
>
> Thanks,
>
Hi,
0.8.0 HEAD from 3/4/2013.
As I think through building a robust SimpleConsumer I ran some failure
tests today and want to make sure I understand what is going on.
FYI I know that I should be doing a metadata lookup to find the leader, but
I wanted to see what happens if things are going well
Joe Stein
> http://www.linkedin.com/in/charmalloc
> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> */
>
> On Tue, Mar 5, 2013 at 8:30 AM, Chris Curtin
> wrote:
>
> > Hi Jun,
> >
> > I wasn't explicitly setting the ack anywhere.
> >
>
What about making it explicit in the Producer Constructor? So in addition
to passing the Config object you set the ACK rule?
Someone with a working 0.7x application is going to have to make a number
of changes anyway so this shouldn't significantly impact the upgrade
process.
I know you're pushin
sed for?
Thanks,
Chris
On Mon, Mar 4, 2013 at 2:21 PM, Chris Curtin wrote:
> I was able to reproduce today after upgrading to HEAD (as of 3/4/2013)
>
> JIRA:
>
> https://issues.apache.org/jira/browse/KAFKA-784
>
>
>
> On Mon, Dec 3, 2012 at 11:09 AM, Chris Curtin wrote
I'm curious the use case, but I think you'll need to use a combination of
ZooKeeper meta data and the TopicMetadata API.
The list of known topics for the cluster is in ZooKeeper:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper
Also in ZooKeeper is the list of
Hi,
I noticed the same thing. In 0.8.0 the offset passed to the fetch is where
you want to start, not where you left off. So the last offset read from the
previous batch is truly the 'last offset' so you need to save it and ask
for it +1. Otherwise you keep asking for that last offset, which is va
Thanks Neha,
I some how missed the 'nextOffset' when converting my logic. I'm assuming
the +1 trick works by chance and I shouldn't assume the next Offset is +1?
(It is minor for me to fix, I'm just curious where +1 might not work.)
Thanks,
Chris
On Wed, Mar 13, 2013 at 3:14 PM, Neha Narkhede
Hi Jun,
I've been thinking for a while about how to contribute to the project and
thought that working on some documentation for the website might be a good
way. Do you have an outline of what you'd like the site to look like that I
(AND OTHERS hint, hint) could pick a topic, write the article and
Okay, how do we do this logistically? I've take the Producer code that I
wrote for testing purposes and wrote a description around it. How do I get
it to you guys?
Simple Consumer is going to take a little longer since my test Consumers
are non-trivial and I'll need to simplify them.
Thanks,
Chr
Or can I do it in the Wiki until you release 0.8.0 so people can comment on
them? I think I can edit the Wiki with my Apache login.
On Thu, Mar 21, 2013 at 12:17 AM, Jun Rao wrote:
> Our webpage source is at https://svn.apache.org/repos/asf/kafka/site . You
> can file a jira and attach a patch.
On Thu, Mar 21, 2013 at 1:38 PM, Neha Narkhede wrote:
> Yes, that works as well.
>
> Thanks,
> Neha
>
>
> On Thu, Mar 21, 2013 at 10:33 AM, Chris Curtin >wrote:
>
> > Or can I do it in the Wiki until you release 0.8.0 so people can comment
> on
> > t
Thanks for finding those. Looks like a copy and paste issue. I've updated
the document.
Thanks,
Chris
On Sat, Mar 23, 2013 at 11:27 AM, Jonathan Hodges wrote:
> Many thanks for contributing! The docs are very helpful. I found a couple
> small possible typos. The partitioning code example l
Hi David,
Thanks for the feedback. I've seen the example before and after in
different books/articles and it doesn't matter to me.
Anyone else want to help define a style guide or is there one I didn't see
already?
Thanks,
Chris
On Thu, Mar 21, 2013 at 7:46 PM, David Arthur wrote:
> This lo
Thanks Jun,
I've updated the example with this information.
I've also removed some of the unnecessary newlines.
Thanks,
Chris
On Mon, Mar 25, 2013 at 12:04 PM, Jun Rao wrote:
> Chris,
>
> This looks good. One thing about partitioning. Currently, if a message
> doesn't have a key, we always
Hi,
Before I submit an enhancement JIRA, is there are reason I can't use a
colon (:) or parenthesis in a client name for Simple Consumer?
I wanted to do something like 'Web:topic(partition)' so I know this is the
Web process for topic and partition.
Thanks,
Chris
ow.
https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
Thanks,
Chris
On Mon, Mar 25, 2013 at 8:02 AM, Chris Curtin wrote:
>
> Hi David,
>
> Thanks for the feedback. I've seen the example before and after in
> different books/articles and it doesn't ma
Now with Video:
http://vimeo.com/63040812
(I did notice that I misspoke about reading from replicas, sorry).
On Wed, Mar 20, 2013 at 8:11 AM, Chris Curtin wrote:
> Hi,
>
> It went really well last night. Lots of good questions. Here are the
> slides, and hopefully the video will
Hi,
Does Kafka support encrypting data at rest? During my AJUG presentation
someone asked if the files could be encrypted to address PII needs?
Thanks,
Chris
You need to reference the version of Yammer shipping with Kafka. It is
under \core\lib\metrics-*
On Thu, Apr 4, 2013 at 11:41 AM, Oleg Ruchovets wrote:
> I am executing a simple code like this:
>
> public class FirstKafkaTester {
> public Producer initProducer(){
> Properties props =
erty too?
>
> In addition to the 0.8 SimpleConsumer wiki, could you write up one for the
> 0.8 high level consumer?
>
> Thanks,
>
> Jun
>
>
>
> On Fri, Mar 29, 2013 at 8:28 AM, Chris Curtin >wrote:
>
> > Hi,
> >
> > I've added an
I following these instructions to get the first 'play' cluster going:
https://cwiki.apache.org/KAFKA/kafka-08-quick-start.html
Instead of running the 3 brokers on the same machine, I ran on on each
machine.
Note that you will need to do a little bit of zookeeper setup to get a
cluster running, I
t; > /tmp/zookeeper/myid`
> >
> >
> > Cheers,
> > Eric Sites
> >
> >
> > On 4/23/13 9:50 AM, "Jason Huang" wrote:
> >
> > >Thanks Chris and Neha.
> > >
> > >Chris - I've been through the link you mentioned be
Also keep in mind that anything done at the transport (SSL for example)
layer won't solve your 'at rest' problems.
All messages are written to disk, so unless the broker does some encryption
logic you haven't solved the data visibility issues.
I also think this should be a producer/consumer probl
Did you create the topic without a # of partitions then try to
delete/recreate it? I've had that happen to me before. Try shutting down
everything (including zookeeper) and restarting.
On Tue, Apr 23, 2013 at 9:08 PM, Jun Rao wrote:
> Does this happen on every message that you type in producer
; offset 5 and above. Otherwise, the client may get duplicates. This is
> handled in the high level consumer and we need to handle that in
> SimpleConsumer as well.
>
> Thanks,
>
> Jun
>
>
>
> On Fri, Mar 29, 2013 at 8:28 AM, Chris Curtin >wrote:
>
> >
I'm pretty sure a replication factor of 2 means leader and 2 slaves.
Shutting down one means the 2 slaves requirement isn't met.
On Wed, Apr 24, 2013 at 3:42 PM, Yu, Libo wrote:
> I figured out the scenario. I have three machines, one server on each of
> them.
> I created a topic with three par
If using the high level consumer you'll need to run two different Groups to
have both of your consumers receive every message. I'm assuming you want
two different types of business logic to process each message? If so you
need to treat them as separate groups.
On Fri, Apr 26, 2013 at 11:28 AM, Ju
ion.
> Is there any book or good documentation ?
>
> Thanks
> Oleg.
>
>
> On Fri, Apr 26, 2013 at 6:34 PM, Chris Curtin >wrote:
>
> > If using the high level consumer you'll need to run two different Groups
> to
> > have both of your consumers recei
In a nutshell: High Level uses Consumer Groups to handle the tracking of
message offset consumption. SimpleConsumer leaves it all up to you.
The 0.7.x quick start shows examples of both:
http://kafka.apache.org/quickstart.html
On Fri, Apr 26, 2013 at 12:32 PM, Oleg Ruchovets wrote:
> By the
Hi Jun,
I finished and published it this morning:
https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
One question: when documenting the ConsumerConfig parameters I couldn't
find a description for the 'auto.commit.interval.ms' setting. I found one
for 'autocommit.interval.ms
Just added the High Level Consumer example.
On Mon, Apr 29, 2013 at 1:52 AM, Jun Rao wrote:
> We have updated the 0.8 documentation in our website (
> http://kafka.apache.org/index.html). Please review the docs. We have the
> following blockers for the 0.8 beta release:
>
> additional docs:
> *
nect,
> zk.session.timeout.ms and zk.sync.time.ms are changed to
> zookeeper.connect,
> zookeeper.session.timeout.ms, and zookeeper.sync.time.ms, respectively.
>
> I will add a link to your wiki in our website.
>
> Thanks again.
>
> Jun
>
>
> On Mon, Apr 29, 2013
details are described in the "consumer rebalancing algorithm"
> section of http://kafka.apache.org/07/design.html
>
> Thanks,
>
> Jun
>
>
> On Mon, Apr 29, 2013 at 8:16 AM, Chris Curtin >wrote:
>
> > Jun, can you explain this a little better? I thought when usi
Hi,
If you get the following errors, check your server.properties file and find
all the 'zk.' and change to 'zookeeper.'
I know the change was posted yesterday, but here is the error in case
someone missed that email.
[2013-04-30 15:24:31,165] FATAL Fatal error during KafkaServerStable
startup.
Hi,
I pulled 0.8.0 head at 3 pm eastern 4/30, did the sbt update; sbt package;
sbt assembly-package-dependency and my code won't compile.
It is missing the Yammer Metrics libraries.
Last pull was about 45 days ago and they were stored in core\lib\metrics-*
Do I now need to pull them myself?
etadata.broker.list).
>
> Jun
>
>
> On Mon, Apr 29, 2013 at 11:19 AM, Chris Curtin >wrote:
>
> > Thanks, I missed that the addition of consumers can cause a re-balance.
> > Thought it was only on Leader changes.
> >
> > I've updated the wording in the
ermark(topic: String, partition: Int)
>
> Finally, it would be great if you can extend the wiki with customized
> encoder (Producer) and decoder (Consumer) at some point.
> Thanks,
>
> Jun
>
>
> On Wed, May 1, 2013 at 6:44 AM, Chris Curtin
> wrote:
>
> > I&#x
I'll try to answer some, the Kafka team will need to answer the others:
On Wed, May 8, 2013 at 12:17 PM, Yu, Libo wrote:
> Hi,
>
> I read this link
> https://cwiki.apache.org/KAFKA/consumer-group-example.html
> and have a few questions (if not too many).
>
> 1 When you say the iterator may bloc
On Thu, May 9, 2013 at 12:36 AM, Rob Withers wrote:
>
>
> > -Original Message-
> > From: Chris Curtin [mailto:curtin.ch...@gmail.com]
>
> > > 1 When you say the iterator may block, do you mean hasNext() may block?
> > >
> >
> > Yes
Hmm, that looks a lot like the lost+found directory on a Linux file system.
Is your Kafka data directory a mount point that has a lost and found
directory?
On Thu, May 9, 2013 at 12:10 PM, Neha Narkhede wrote:
> The topic name should only consist of ascii characters and '_' and '-'. I
> agree th
Yes. However be aware that starting and stopping processes will cause a
rebalance of the consumers, so your code may find itself receiving events
from a different partition suddenly (so don't assume the partition you are
reading isn't going to change!) Also as things are starting up you may
find a
Hi Tim,
On Wed, May 22, 2013 at 3:25 PM, Timothy Chen wrote:
> Hi,
>
> I'm currently trying to understand how Kafka (0.8) can scale with our usage
> pattern and how to setup the partitioning.
>
> We want to route the same messages belonging to the same id to the same
> queue, so its consumer wi
I'd look at a variation of #2. Can your messages by grouped into a 'class
(for lack of a better term)' that are consumed together? For example a
'class' of 'auditing events' or 'sensor events'. The idea would to then
have a topic for 'class'.
A couple of benefits to this:
- you can define your con
ducer is also hosting a web server, but perhaps not as dire on the
> consumer side.
>
> thanks,
> rob
>
> From: Chris Curtin [curtin.ch...@gmail.com]
> Sent: Wednesday, May 29, 2013 7:36 AM
> To: users
> Subject: Re: one consumerConn
Hi,
I'm working through a production-level High Level Consumer app and have a
couple of error/shutdown questions to understand how the offset storage is
handled.
Test case - simulate an error writing to destination application, for
example a database, offset is 'lost'
Scenario
- write 500 messag
es before the
> message is fully processed, and "auto-commit" updates the offsets in
> Zookeeper, you will drop that message.
>
> The solution to this to call commitOffsets() explicitly.
>
> Philip
>
> On Tue, Jul 9, 2013 at 11:16 AM, Chris Curtin >wrote:
>
>
ry creating a
> ConsumerConnection per partition (your clients will then need to know the
> number of partitions out there). That way commitOffsets() will actually
> only apply to that partition. Auto-commit the same way. It might give you
> the level of control you need.
>
> Philip
&
Enhancement submitted: https://issues.apache.org/jira/browse/KAFKA-966
On Tue, Jul 9, 2013 at 3:53 PM, Chris Curtin wrote:
> Thanks. I know I can write a SimpleConsumer to do this, but it feels like
> the High Level consumer is _so_ close to being robust enough tohandle
> what
Thanks Ian.
Is your consumer multi-threaded? If so can you share how you coordinated
each of the threads so you knew it was 'okay' to commit across all the
threads? I'm stuck on how to do this without really complicating the
consumer.
Thanks,
Chris
On Tue, Jul 9, 2013 at 5:51 PM, Ian Friedman
Hi Jay,
Do they record these meetups?
Thanks,
Chris
On Thu, Oct 17, 2013 at 5:03 PM, Jay Kreps wrote:
> FYI.
>
> -- Forwarded message --
> From: Jakob Homan
> Date: Thu, Oct 17, 2013 at 11:08 AM
> Subject: Special Bay Area HUG: Tajo and Samza
> To: d...@samza.incubator.apach
If you look at the example simple consumer:
https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
You'll see:
if (currentOffset < readOffset) {
System.out.println("Found an old offset: " + currentOffset + "
Expecting: " + readOffset);
continue;
}
an
62 matches
Mail list logo
