Wonder if you can listen to the zkPath for topics via a zk watch (
https://zookeeper.apache.org/doc/r3.3.3/api/org/apache/zookeeper/Watcher.html)
to let you know when the structure of the tree changes (ie, add/remove)?
The zkPath for topics is "/brokers/topics"
https://github.com/chris
kafka-0.9.0.1-candidate1/javadoc/
> >
> > * The tag to be voted upon (off the 0.9.0 branch) is the 0.9.0.1 tag
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=2c17685a45efe665bf5f24c0296cb8f9e1157e89
> >
> > * Documentation
> > http://kafka.apache.org/090/documentation.html
> >
> > Thanks,
> >
> > Jun
> >
> >
>
--
*Christian Posta*
twitter: @christianposta
http://www.christianposta.com/blog
http://fabric8.io
t;group.id", groupId)
> > props.put("auto.commit.enabled", "false")
> > // this timeout is needed so that we do not block on the stream!
> > props.put("consumer.timeout.ms", "1")
> > props.put("zookeeper.sync.time.ms", "200")
> >
>
--
*Christian Posta*
twitter: @christianposta
http://www.christianposta.com/blog
http://fabric8.io
setting in my email above is responsible for
> this auto reconnect mechanism?
>
> On Wed, Feb 17, 2016 at 8:04 PM, Christian Posta <
> christian.po...@gmail.com>
> wrote:
>
> > Yep, assuming you haven't completely partitioned that client from the
> > cluster,
0 1
>
> Can someone help me clarify or point me at a doc that explains what is
> getting counted here? You can shoot me if you like for attempting the
> hack-ish solution of re-setting the offset through the Zookeeper API, but I
> would still like to understand what, exactly, is represented by that number
> 30024.
>
> I need to hand off to IT for the Disaster Recovery portion and saying
> "trust me, it just works" isn't going to fly very far...
>
> Thanks.
>
--
*Christian Posta*
twitter: @christianposta
http://www.christianposta.com/blog
http://fabric8.io
I believe so. Happy to be corrected.
On Wed, Feb 17, 2016 at 12:31 PM, Joe San wrote:
> So if I use the High Level Consumer API, using the ConsumerConnector, I get
> this automatic zookeeper connection for free?
>
> On Wed, Feb 17, 2016 at 8:25 PM, Christian Posta <
> christi
lease even if a critical issue is reported from
> a non-committer.
>
> Thanks,
>
> Jun
>
> On Tue, Feb 16, 2016 at 11:05 PM, Christian Posta <
> christian.po...@gmail.com
> > wrote:
>
> > BTW, what's the etiquette for votes (non-binding) for this co
; a non-committer.
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, Feb 16, 2016 at 11:05 PM, Christian Posta <
> > christian.po...@gmail.com
> > > wrote:
> >
> > > BTW, what's the etiquette for votes (non-binding) for this community
Can someone add Karma to my user id for contributing to the wiki/docs?
userid is 'ceposta'
thanks!
--
*Christian Posta*
twitter: @christianposta
http://www.christianposta.com/blog
http://fabric8.io
in case you are aware of any issue that might cause it?
> I'm chasing this leak for several days, and managed to track it down to the
> code writing to Kafka, so I'm a little desperate :) any help will do.
>
> Thanks!
>
> Asaf
>
--
*Christian Posta*
twitter: @christianposta
http://www.christianposta.com/blog
http://fabric8.io
; consumer level?
>
--
*Christian Posta*
twitter: @christianposta
http://www.christianposta.com/blog
http://fabric8.io
Would love to have the docs in gitbook/markdown format so they can easily
be viewed from the source repo (or mirror, technically) on github.com. They
can also be easily converted to HTML, have a side-navigation ToC, and still
be versioned along with the src code.
Thoughts?
--
*Christian Posta
For sure! Will take a look!
On Wednesday, March 2, 2016, Gwen Shapira wrote:
> Hey!
>
> Yes! We'd love that too! Maybe you want to help us out with
> https://issues.apache.org/jira/browse/KAFKA-2967 ?
>
> Gwen
>
> On Wed, Mar 2, 2016 at 2:39 PM, Christian Posta
>
b.com/christian-posta/kafka/tree/ceposta-doco
On Thu, Mar 3, 2016 at 6:28 AM, Marcos Luis Ortiz Valmaseda <
marcosluis2...@gmail.com> wrote:
> I love that too
> +1
>
> 2016-03-02 21:15 GMT-05:00 Christian Posta :
>
> > For sure! Will take a look!
> >
> > On W
e a message and to ensure it makes it to Mongo. (Redelivery is
> ok)
>
> Thanks for any help or pointers in the right direction.
>
> Michael
>
--
*Christian Posta*
twitter: @christianposta
http://www.christianposta.com/blog
http://fabric8.io
able to demonstrate that addressing
> this case solves my stuck consumer problem.
>
> How do I submit a bug report for this issue, or does this email constitute
> a bug report?
>
> --Larkin
>
--
*Christian Posta*
twitter: @christianposta
http://www.christianposta.com/blog
http://fabric8.io
s in Mongo. That's either as a successful
> message in a collection or an unsuccessful message in an error collection.
>
> Right now I let the consumer die and don't create a new one for x minutes.
> This causes a lot of rebalancing.
>
> Michael
>
> > On 9 Mar 2016, at 21:
page
>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+papers+and+presentations
> is
> outdated..
>
> Thanks in advance..
>
--
*Christian Posta*
twitter: @christianposta
http://www.christianposta.com/blog
http://fabric8.io
.partition.fetch.bytes.
> >
> > For now I'm just going to shut the consumer down and restart after x
> > period of time.
> >
> > Thanks for your insights.
> >
> > Michael
> >
> > > On 10 Mar 2016, at 18:33, Christian Posta
> > wrote:
>
gt; -Jason
>
> On Mon, Mar 14, 2016 at 11:21 AM, Christian Posta <
> christian.po...@gmail.com
> > wrote:
>
> > Jason,
> >
> > Can you link to the proposal so I can take a look? Would the "sticky"
> > proposal prefer to keep partitions assig
to this list to learn and implement Kafka.
>
> Thanks,
> Punya
>
--
*Christian Posta*
twitter: @christianposta
http://www.christianposta.com/blog
http://fabric8.io
gt;
> bookmarks80[0, 1][0, 1]
>
> bookmarks91[1, 0][1, 0]
>
> We monitor the disk writes and I only see writes at broker 0, and broker 1
> sees none (not comparable at all). I do see comparable network traffic at
&
You need to send a mail to users-subscr...@kafka.apache.org
http://kafka.apache.org/contact.html
On Sat, Mar 19, 2016 at 4:14 AM, Andreas Thoelke
wrote:
> Hi,
>
> please add me to the Kafka list.
>
> Andreas
>
--
*Christian Posta*
twitter: @christianposta
http://www.chris
gt; end
> > > up with bugs in production.
> > >
> > > I can volunteer for the release management of the LTS release but as a
> > > community, can we follow the rigour of back-porting the bug-fixes to
> the
> > > LTS branch?
> > >
> > > --
> > > Regards
> > > Vamsi Subhash
> > >
> >
> >
> >
> > --
> > Regards
> > Vamsi Subhash
> >
>
--
*Christian Posta*
twitter: @christianposta
http://www.christianposta.com/blog
http://fabric8.io
ides.
>
> I think the overhead which happens while establishing connection from
> consumer/producer to kafka broker(s) seems a little heavy.
>
> Thanks in advance!
>
> Best regards
>
> bgkim
>
--
*Christian Posta*
twitter: @christianposta
http://www.christianposta.com/blog
http://fabric8.io
consumer. The respective
> consumers do their fetches only on their assigned partitions but the old
> consumer still commits the same last offset it had when it was holding on
> to that partition At the same time the added consumer commits the correct
> offsets.
>
> Any inputs on what co
a.tools.JmxTool --object-name
> "kafka.consumer:type=FetchRequestAndResponseMetrics,name=FetchRequestRateAndTimeMs,clientId=ReplicaFetcherThread*,brokerHost=hostname*.
> cluster.com,brokerPort=*" --jmx-url
> service:jmx:rmi:///jndi/rmi://`hostname`:/jmxrmi
>
> There may
u, Mar 31, 2016 at 4:36 PM, craig w wrote:
> >
> > > Including jolokia would be great, I've used for kafka and it worked
> well.
> > > On Mar 31, 2016 6:54 PM, "Christian Posta"
> > > wrote:
> > >
> > > > What if we added some
6, 2016 at 9:41 AM, Mudit Kumar wrote:
> How can i get the list for all the class names i can run through
> ./kafka-run-class.sh [class-name] command?
>
> Thanks,
> Mudit
--
*Christian Posta*
twitter: @christianposta
http://www.christianposta.com/blog
http://fabric8.io
1, initiating session
> (org.apache.zookeeper.ClientCnxn)
> > > > [2016-05-10 15:41:03,337] INFO Unable to read additional data from
> server sessionid 0x1549b308dd20002, likely server has closed socket,
> closing socket connection and attempting reconnect
> (org.apache.zookeeper.ClientCnxn)
> > > > [2016-05-10 15:41:05,121] INFO Opening socket connection to server
> 10.0.0.184/10.0.0.184:2181. Will not attempt to authenticate using SASL
> (unknown error) (org.apache.zookeeper.ClientCnxn)
> > > > [2016-05-10 15:41:05,121] INFO Socket connection established to
> 10.0.0.184/10.0.0.184:2181, initiating session
> (org.apache.zookeeper.ClientCnxn)
> > > > [2016-05-10 15:41:05,122] INFO Unable to read additional data from
> server sessionid 0x1549b308dd20002, likely server has closed socket,
> closing socket connection and attempting reconnect
> (org.apache.zookeeper.ClientCnxn)
> > > >
> > > > You can see when the first zookeeper dies and connection is lost ...
> and all the retries by kafka server in order to connect to the new one
> (same IP, same port).
> > > >
> > > > Why the zookeeper server closes the connection (I can see FIN ACK
> frames on Wireshark)
> > > >
> > > > Thanks,
> > > > Paolo.
> > > >
> > > > Paolo PatiernoSenior Software Engineer (IoT) @ Red Hat
> > > > Microsoft MVP on Windows Embedded & IoTMicrosoft Azure Advisor
> > > > Twitter : @ppatierno
> > > > Linkedin : paolopatierno
> > > > Blog : DevExperience
>
>
--
*Christian Posta*
twitter: @christianposta
http://www.christianposta.com/blog
http://fabric8.io
tps://twitter.com/anas24aj> [image: linkedin]
> > <http://in.linkedin.com/in/anas24aj> [image: googleplus]
> > <https://plus.google.com/u/0/+anasA24aj/>
> > +917736368236
> > anas.2...@gmail.com
> > Bangalore
> >
>
--
*Christian Posta*
twitter: @christianposta
http://www.christianposta.com/blog
http://fabric8.io
If you're using KafkaConnect, it does it for you!
basically you set the sourceRecord's "sourcePartition" and "sourceOffset"
fields (
https://github.com/christian-posta/kafka/blob/8db55618d5d5d5de97feab2bf8da4dc45387a76a/connect/api/src/main/java/org/apache/kafka/co
> > > > internal services which need to process the event to be "done"
> > > > processing before returning a response.
> > >
> > >
> > >
> > > In theory that's possible - the producer can return the offset of the
> > > message
ther projects, I know that
> without the initial pitch / discussion, it could be difficult to get such
> feature in. I can create a jira in the morning, no electricity again
> tonight :-/
>
> Get Outlook for iOS
>
>
>
>
> On Tue, May 17, 2016 at 4:53 PM -0700, &qu
s? Let's say I want to push 3 messages
> > > > atomically but the producer process crashes after sending only 2
> > > messages,
> > > > is it possible to "rollback" the first 2 messages (e.g. "all or
> > nothing"
> > > > semantics)?
> > > >
> > > > 3) Does it support request/response style semantics or can they be
> > > > simulated? My system's primary interface with the outside world is an
> > > HTTP
> > > > API so it would be nice if I could publish an event and wait for all
> > the
> > > > internal services which need to process the event to be "done"
> > > > processing before returning a response.
> > > >
> > > > PS: I'm a Node.js/Go developer so when possible please avoid Java
> > centric
> > > > terminology.
> > > >
> > > > Thanks!
> > > >
> > > > - Oli
> > > >
> > > > --
> > > > - Oli
> > > >
> > > > Olivier Lalonde
> > > > http://www.syskall.com <-- connect with me!
> > > >
> > >
> >
>
>
>
> --
> - Oli
>
> Olivier Lalonde
> http://www.syskall.com <-- connect with me!
>
--
*Christian Posta*
twitter: @christianposta
http://www.christianposta.com/blog
http://fabric8.io
a – considering that different consumers within the group
> commit to either kafka or Zk ?
>
> Regards
> Sathya,
>
--
*Christian Posta*
twitter: @christianposta
http://www.christianposta.com/blog
http://fabric8.io
72.17.0.1/172.17.0.1
> mirrormaker, local.general.example, 0, unknown, 0, unknown,
> mirrormaker-0_172.17.0.1/172.17.0.1
>
> On Wed, May 18, 2016 at 2:36 PM Christian Posta >
> wrote:
>
> > Maybe give it a try with the kafka-consumer-groups.sh tool and the
> >
apache.org/dyn/closer.cgi?path=/kafka/0.10.0.0/kafka_2.10-0.10.0.0
> > .tgz
> >
> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.0.0/kafka_2.11-0.10.0.0
> > .tgz
> >
> > A big thank you for the following people who have contributed to the
> > 0.10.0.0 rele
he multi-threaded to one-thread and subscribing multi-topics?... I'm
> just
> > wonder whether a KafkaConsumer is able to stand the bunch of data without
> > performance degradation.
> >
> > Thanks in advance!
> >
> > Best regards
> >
> > KIM
> >
>
--
*Christian Posta*
twitter: @christianposta
http://www.christianposta.com/blog
http://fabric8.io
Kafka topics for now. So, as Gwen
> > mentioned, Connect would be the way to go to bring the data to a Kafka
> > Topic first.
>
> Got it — thank you!
>
>
--
*Christian Posta*
twitter: @christianposta
http://www.christianposta.com/blog
http://fabric8.io
gt; props.put("autooffset.reset", "smallest"); //
> props.put("autocommit.enable", false); KafkaConsumer String> consumer = new KafkaConsumer<>(props);
> consumer.subscribe(Arrays.asList("say-hello-test1")); final int
&g
Might be worth describing your use case a bit to see if there's another way
to help you?
On Tue, Jun 14, 2016 at 5:29 AM, Mudit Kumar wrote:
> Hey,
>
> How can I delete particular messages from particular topic?Is that
> possible?
>
> Thanks,
> Mudit
>
>
it in some
> topic till as such time that a service can dequeue, process it and/or
> investigate it.
>
> Thanks.
>
> Best,
> Krish
>
--
*Christian Posta*
twitter: @christianposta
http://www.christianposta.com/blog
http://fabric8.io
; resources; I can also ensure that it runs on a dedicated machine.
>
> Thanks.
>
> Best,
> Krish
>
--
*Christian Posta*
twitter: @christianposta
http://www.christianposta.com/blog
http://fabric8.io
44 matches
Mail list logo
