RE: Re : A specific use case

[Hamza HACHANI]

Thanks Guozhang Wang.


Hamza


De : Guozhang Wang 
Envoyé : jeudi 4 août 2016 06:58:22
À : users@kafka.apache.org
Objet : Re: Re : A specific use case

Yeah, if you can buffer yourself in the process() function and then rely on
punctuate() for generating the outputs that would resolve your issue.

Remember that punctuate() function itself is event-time driven so if you do
not have any data coming in then it may not be triggered. Details:

https://github.com/apache/kafka/pull/1689

Guozhang

On Wed, Aug 3, 2016 at 8:53 PM, Hamza HACHANI 
wrote:

> Hi,
> Yes in fact .
> And ï found à solution.
> It was in editing the method punctuate in kafka stream processor.
>
> - Message de réponse -
> De : "Guozhang Wang" 
> Pour : "users@kafka.apache.org" 
> Objet : A specific use case
> Date : mer., août 3, 2016 23:38
>
> Hello Hamza,
>
> By saying "broker" I think you are actually referring to a Kafka Streams
> instance?
>
>
> Guozhang
>
> On Mon, Aug 1, 2016 at 1:01 AM, Hamza HACHANI 
> wrote:
>
> > Good morning,
> >
> > I'm working on a specific use case. In fact i'm receiving messages from
> an
> > operator network and trying to do statistics on their number per
> > minute,perhour,per day ...
> >
> > I would like to create a broker that receives the messages and generates
> a
> > message every minute. These producted messages are consumed by a consumer
> > from in one hand and also se,t to an other topic which receives them and
> > generates messages every minute.
> >
> > I've  been doing that for a while without a success. In fact the first
> > broker in any time it receives a messages ,it produces one and send it to
> > the other topic.
> >
> > My question is ,what i'm trying to do,Is it possible without passing by
> an
> > intermediate java processus which is out of kafka.
> >
> > If yes , How ?
> >
> > Thanks In advance.
> >
>
>
>
> --
> -- Guozhang
>



--
-- Guozhang

Kafka compaction that can aggregate/count messages by key?

[R Krishna]

Is it possible to use Kafka to track counts instead of deletion on
compaction? I know we can aggregate ourself and add it to a different topic
but that won't make sense if the time window is more than few seconds.

Say, I can then, use it to count based on a key containing minute, hour,
day.
https://cwiki.apache.org/confluence/display/KAFKA/Log+Compaction

-- 
Krishna

Re: Kafka Streams on Windows?

[Eno Thereska]

Hi Mathieu,

It is true that the DSL currently does not support configuration of the stores.

Sounds like it might be worth trying to build RocksDb and dropping into 
classpath for now.

Eno

> On 4 Aug 2016, at 17:42, Mathieu Fenniak  wrote:
> 
> Hi Eno,
> 
> Yes, I've looked at that.  RocksDB can be built and run in Windows, but,
> the JNI wrapper does not include Windows binarie (
> https://github.com/facebook/rocksdb/issues/703).  rocksdbjni-4.4.1.jar
> includes librocksdbjni-linux32.so, librocksdbjni-linux64.so, and
> librocksdbjni-osx.jnilib, so only supports Linux x86 & x64 and OS X.  It is
> probably possible for me to build it myself and drop it in my classpath,
> but, I'm looking for a lower friction approach if one exists. :-)
> 
> It looks like this was discussed recently on the Confluent Platform mailing
> list (https://groups.google.com/forum/#!topic/confluent-platform/Z1rsfSNrVJk)
> and the conclusion there was that high-level streams DSL doesn't support
> configuration of the stores.
> 
> Mathieu
> 
> 
> On Thu, Aug 4, 2016 at 10:28 AM, Eno Thereska 
> wrote:
> 
>> Hi Mathieu,
>> 
>> Have you had a chance to look at http://rocksdb.org/blog/2033/
>> rocksdb-is-now-available-in-windows-platform/? <
>> http://rocksdb.org/blog/2033/rocksdb-is-now-available-in-windows-platform/?>
>> Curious to hear your and other's comments on whether that worked.
>> 
>> It is possible to have Kafka Streams use an in-memory store (included with
>> Kafka Streams) for development purposes. In that scenario RocksDb would not
>> be needed.
>> 
>> Eno
>> 
>> 
>>> On 4 Aug 2016, at 16:14, Mathieu Fenniak 
>> wrote:
>>> 
>>> Hey all,
>>> 
>>> Is it anyone developing Kafka Streams applications on Windows?
>>> 
>>> It seems like the RocksDB Java library doesn't include a native JNI
>> library
>>> for Windows, which prevents a Kafka Streams app from running on
>> Windows.  I
>>> was just wondering if others have run into this, and if so, what approach
>>> you took to resolve it.
>>> 
>>> I'm favouring the idea of running my applications in a Vagrant VM to
>> avoid
>>> the issue.  It makes the Windows development environment a little less
>>> pleasant, but, seems plausible.
>>> 
>>> Other ideas that occurred to me:
>>> 
>>>  - RocksDB does support Windows, but, there don't seem to be any
>> binaries
>>>  available or packaged for it or the jni library.  I could probably
>> build
>>>  these myself, but it sounds a little painful.
>>>  - Not developing on Windows.  Works for me, but, won't work as well for
>>>  my colleagues.
>>>  - Is it possible to make Kafka Streams not use any local state storage
>>>  in some kind of development mode?  Not sure...
>>> 
>>> Thanks,
>>> 
>>> Mathieu
>> 
>> 

Re: Kafka Streams input

[Jeyhun Karimov]

Thanks in advance Eno.

-Jeyhun


On Thu, Aug 4, 2016 at 6:25 PM Eno Thereska  wrote:

> Hi Jeyhun,
>
> You can use Kafka Connect to bring data into Kafka from HDFS then use
> Kafka Streams as usual. I believe there is a desire to make the Connect +
> Streams more integrated so it doesn't feel like you're running two
> different things, and hopefully that integration happens soon.
>
> Thanks
> Eno
>
> > On 4 Aug 2016, at 17:11, Jeyhun Karimov  wrote:
> >
> > Hi community,
> >
> > Can Kafka Streams use input sources other than Kafka? For example can it
> > read data from HDFS? You can assume that for each Streams instance
> > initiating, I will have separate  partition (in this case file path
> > containing data) and each instance will read from one file path.
> >
> > Cheers
> > Jeyhun
> > --
> > -Cheers
> >
> > Jeyhun
>
> --
-Cheers

Jeyhun

Log compaction leaves empty files?

[Harald Kirsch]

Hi,

experimenting with log compaction, I see Kafka go through all the steps, 
in particular I see positive messages in log-cleaner.log and *.deleted 
files. Yet once the *.deleted segment files have disappeared, the 
segment and index files with size 0 are still kept.


I stopped and restarted Kafka and saw several rounds of the log cleaner 
go by, but the empty files stay:


-rw-rw-r-- 1 0 Aug  5 11:58 .index
-rw-rw-r-- 1 0 Aug  5 11:58 .log
-rw-rw-r-- 1 0 Aug  5 11:58 0051.index
-rw-rw-r-- 1 0 Aug  5 11:58 0051.log
-rw-rw-r-- 1 0 Aug  5 11:58 0096.index
-rw-rw-r-- 1 0 Aug  5 11:58 0096.log
[... snip ...]
-rw-rw-r-- 1 92041 Aug  5 11:58 0680.log
-rw-rw-r-- 1 0 Aug  5 11:58 0727.index
-rw-rw-r-- 1199822 Aug  5 11:58 0727.log
-rw-rw-r-- 1  10485760 Aug  5 11:58 0781.index
-rw-rw-r-- 1 95972 Aug  5 11:58 0781.log

Is this expected behavior or is there yet another configuration option 
that defines when these get purged?


Harald.

Re: Kafka compaction that can aggregate/count messages by key?

[Tom Crayford]

Kafka can't by itself do aggregation (nor does it really make sense for it
to). You can build such a feature on top of log compaction relatively
easily (by sending the new count as a message under an individual key), or
you can use the KTable and aggregation features of Kafka Streams.

Thanks

Tom Crayford
Heroku Kafka

On Friday, 5 August 2016, R Krishna  wrote:

> Is it possible to use Kafka to track counts instead of deletion on
> compaction? I know we can aggregate ourself and add it to a different topic
> but that won't make sense if the time window is more than few seconds.
>
> Say, I can then, use it to count based on a key containing minute, hour,
> day.
> https://cwiki.apache.org/confluence/display/KAFKA/Log+Compaction
>
> --
> Krishna
>

Re: Compress before or after serialization?

[Tom Crayford]

Kafka compresses batches in the producer before sending them to the broker.
You'll get notably better compression from this than you will from per
message compression. I'd recommend checking your producer config and maybe
looking at the log segments on the broker with DumpLogSegments.

If you have sample producer configs and code, maybe this would help in
diagnosing the issue.

Thanks

Tom Crayford
Heroku Kafka

On Friday, 5 August 2016, David Yu  wrote:

> We are using Avro as our format when writing to a Kafka topic. After we
> enabled snappy compression on the producer, we don't see a change in the
> compression ratio (still 1). I was wondering if we should compress the
> message before serialization.
>
> Thanks,
> David
>

Re: [VOTE] 0.10.0.1 RC2

[Jim Jagielski]

Looks good here: +1

> On Aug 4, 2016, at 9:54 AM, Ismael Juma  wrote:
> 
> Hello Kafka users, developers and client-developers,
> 
> This is the third candidate for the release of Apache Kafka 0.10.0.1. This
> is a bug fix release and it includes fixes and improvements from 53 JIRAs
> (including a few critical bugs). See the release notes for more details:
> 
> http://home.apache.org/~ijuma/kafka-0.10.0.1-rc2/RELEASE_NOTES.html
> 
> When compared to RC1, RC2 contains a fix for a regression where an older
> version of slf4j-log4j12 was also being included in the libs folder of the
> binary tarball (KAFKA-4008). Thanks to Manikumar Reddy for reporting the
> issue.
> 
> *** Please download, test and vote by Monday, 8 August, 8am PT ***
> 
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS
> 
> * Release artifacts to be voted upon (source and binary):
> http://home.apache.org/~ijuma/kafka-0.10.0.1-rc2/
> 
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging
> 
> * Javadoc:
> http://home.apache.org/~ijuma/kafka-0.10.0.1-rc2/javadoc/
> 
> * Tag to be voted upon (off 0.10.0 branch) is the 0.10.0.1-rc2 tag:
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=f8f56751744ba8e55f90f5c4f3aed8c3459447b2
> 
> * Documentation:
> http://kafka.apache.org/0100/documentation.html
> 
> * Protocol:
> http://kafka.apache.org/0100/protocol.html
> 
> * Successful Jenkins builds for the 0.10.0 branch:
> Unit/integration tests: *https://builds.apache.org/job/kafka-0.10.0-jdk7/182/
> *
> System tests: *https://jenkins.confluent.io/job/system-test-kafka-0.10.0/138/
> *
> 
> Thanks,
> Ismael

Leader not available when using ACLs on Kafka 0.10

[Wannes De Smet]

Hi all

We are getting 'Leader not available' exception' when using ACLs with TLS
on a three node Kafka cluster, configured as [1]. The error occurs both
when trying to produce and consume from a topic, to which the producer
principal and all hosts have been granted access for testing, using the
following:

./kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer
--authorizer-properties zookeeper.connect=localhost:2181 --add
--allow-principal User:* --producer --topic topicName

The same issue appears in another thread on this mailing list [2], though
no information is present on how to resolve this issue. We also tried using
0.10.0.1 RC2, unfortunately to no effect. When the ACLs are not active,
everything works as expected.

Another attempt to explicitly allow access to all Kafka cluster hosts with
the 'All' principal did not have any effect.

Please advise how we might debug and resolve this issue.

Thanks
Wannes

[1] listeners=PLAINTEXT://:9092,SSL://:9093 ; inter-broker communication is
using the PLAINTEXT default
[2]
http://mail-archives.apache.org/mod_mbox/kafka-users/201608.mbox/%3CCANZ-JHHmL_E5xhcEdHeW0ZYME+M8iZsaz-D59UKL8HeWh3=p...@mail.gmail.com%3E

Re: Leader not available when using ACLs on Kafka 0.10

[Tom Crayford]

Hi,

I'd recommend turning up broker logs to DEBUG and looking at the
controller's logs. The controller talks to nodes over the network and if it
can't reach them because of ACLs, then you won't get a leader.

The only other note is to check if your brokers are talking to each other
over TLS or plaintext. If they're going over plaintext you'll need to
authenticate those hosts. If they're going over TLS, you'll need to ensure
they're using the right client certs.

Thanks

Tom Crayford
Heroku Kafka

On Friday, 5 August 2016, Wannes De Smet  wrote:

> Hi all
>
> We are getting 'Leader not available' exception' when using ACLs with TLS
> on a three node Kafka cluster, configured as [1]. The error occurs both
> when trying to produce and consume from a topic, to which the producer
> principal and all hosts have been granted access for testing, using the
> following:
>
> ./kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer
> --authorizer-properties zookeeper.connect=localhost:2181 --add
> --allow-principal User:* --producer --topic topicName
>
> The same issue appears in another thread on this mailing list [2], though
> no information is present on how to resolve this issue. We also tried using
> 0.10.0.1 RC2, unfortunately to no effect. When the ACLs are not active,
> everything works as expected.
>
> Another attempt to explicitly allow access to all Kafka cluster hosts with
> the 'All' principal did not have any effect.
>
> Please advise how we might debug and resolve this issue.
>
> Thanks
> Wannes
>
> [1] listeners=PLAINTEXT://:9092,SSL://:9093 ; inter-broker communication
> is
> using the PLAINTEXT default
> [2]
> http://mail-archives.apache.org/mod_mbox/kafka-users/201608.
> mbox/%3CCANZ-JHHmL_E5xhcEdHeW0ZYME+M8iZsaz-D59UKL8HeWh3=PSw@
> mail.gmail.com%3E
>

Connections to Zookeeper

[Krzysztof Nawara]

Hello,

How many connections to Zookeeper should correctly working Kafka broker
have opened at the same time?

Krzysiek

Re: Kafka Streams on Windows?

[Mathieu Fenniak]

I took that approach.  It was painful, but, ultimately did get me a working
Windows development environment.

To any who follow in my footsteps, here is my trail:

   1. Upgrade to at least Kafka Streams 0.10.0.1 (currently only in RC).
  - This is necessary because .1 bumps the rocksdb dependency to 4.8.0,
  where the previous 4.4 dependency did not yet support loading a
Windows JNI
  library.
  2. Follow the instructions here (
   https://github.com/facebook/rocksdb/blob/v4.8/CMakeLists.txt) to build
   rocksdb.
  - That link is for the v4.8 tag; be sure to match this with the
  version of rocksdb that Kafka Streams depends upon in the
future.  0.10.0.1
  -> v4.8, but future releases of Kafka Streams will likely depend on newer
  versions.
  - Be sure to include the "-DJNI=1" compile option to build the JNI
  wrapper.
  - None of the third-party dependencies (eg. snappy, jemalloc, etc.)
  seem to be required to get something functional, but, it
probably isn't the
  most efficient build.
  - Ensure that "javac" and "javah" are on your path when running
  cmake; if there are any errors in the cmake output your JNI wrapper
  probably won't build.
  - You may or may not need to make minor patches to make the project
  build in Windows.  It appears that the Windows build is often broken; for
  example, I had to apply this patch:
  https://github.com/facebook/rocksdb/pull/1223/files
  - Phew, that should give you a build\java\Debug\rocksdbjni.dll.  So
  close to the summit... just a horrible hack left...
   3. Copy rocksdbjni.dll to librocksdbjni-win64.dll.
   4. Insert librocksdbjni-win64.dll into your rocksdbjni-4.8.0.jar.
  - Ugh, this is the horrible hack.  rocksdbjni seems to only look
  inside its own jar for its native libraries, so, this is where
it needs to
  be.
  - If you're a gradle user on Windows, you'd find this jar file
  in 
C:\Users\...your-windows-user...\.gradle\caches\modules-2\files-2.1\org.rocksdb\rocksdbjni\4.8.0\b543fc4ea5b52ad790730dee376ba0df06d9f5f7.

And there you go, almost a working Kafka Streams app in Windows.  One other
detail is that the default state storage directory doesn't seem to be
created on demand, so I had to mkdir C:\tmp\kafka-streams myself before my
app would work.

Mathieu


On Fri, Aug 5, 2016 at 3:13 AM, Eno Thereska  wrote:

> Hi Mathieu,
>
> It is true that the DSL currently does not support configuration of the
> stores.
>
> Sounds like it might be worth trying to build RocksDb and dropping into
> classpath for now.
>
> Eno
>
> > On 4 Aug 2016, at 17:42, Mathieu Fenniak 
> wrote:
> >
> > Hi Eno,
> >
> > Yes, I've looked at that.  RocksDB can be built and run in Windows, but,
> > the JNI wrapper does not include Windows binarie (
> > https://github.com/facebook/rocksdb/issues/703).  rocksdbjni-4.4.1.jar
> > includes librocksdbjni-linux32.so, librocksdbjni-linux64.so, and
> > librocksdbjni-osx.jnilib, so only supports Linux x86 & x64 and OS X.  It
> is
> > probably possible for me to build it myself and drop it in my classpath,
> > but, I'm looking for a lower friction approach if one exists. :-)
> >
> > It looks like this was discussed recently on the Confluent Platform
> mailing
> > list (https://groups.google.com/forum/#!topic/confluent-
> platform/Z1rsfSNrVJk)
> > and the conclusion there was that high-level streams DSL doesn't support
> > configuration of the stores.
> >
> > Mathieu
> >
> >
> > On Thu, Aug 4, 2016 at 10:28 AM, Eno Thereska 
> > wrote:
> >
> >> Hi Mathieu,
> >>
> >> Have you had a chance to look at http://rocksdb.org/blog/2033/
> >> rocksdb-is-now-available-in-windows-platform/? <
> >> http://rocksdb.org/blog/2033/rocksdb-is-now-available-in-
> windows-platform/?>
> >> Curious to hear your and other's comments on whether that worked.
> >>
> >> It is possible to have Kafka Streams use an in-memory store (included
> with
> >> Kafka Streams) for development purposes. In that scenario RocksDb would
> not
> >> be needed.
> >>
> >> Eno
> >>
> >>
> >>> On 4 Aug 2016, at 16:14, Mathieu Fenniak  >
> >> wrote:
> >>>
> >>> Hey all,
> >>>
> >>> Is it anyone developing Kafka Streams applications on Windows?
> >>>
> >>> It seems like the RocksDB Java library doesn't include a native JNI
> >> library
> >>> for Windows, which prevents a Kafka Streams app from running on
> >> Windows.  I
> >>> was just wondering if others have run into this, and if so, what
> approach
> >>> you took to resolve it.
> >>>
> >>> I'm favouring the idea of running my applications in a Vagrant VM to
> >> avoid
> >>> the issue.  It makes the Windows development environment a little less
> >>> pleasant, but, seems plausible.
> >>>
> >>> Other ideas that occurred to me:
> >>>
> >>>  - RocksDB does support Windows, but, there don't seem to be any
> >> binaries
> >>>  available or packaged for it or the jni library.  I could probably
> >> build
> >>>  these myself, but it sound

Re: Kafka Streams on Windows?

[Damian Guy]

Thanks for providing the instructions - appreciated.

On Fri, 5 Aug 2016 at 14:10 Mathieu Fenniak 
wrote:

> I took that approach.  It was painful, but, ultimately did get me a working
> Windows development environment.
>
> To any who follow in my footsteps, here is my trail:
>
>1. Upgrade to at least Kafka Streams 0.10.0.1 (currently only in RC).
>   - This is necessary because .1 bumps the rocksdb dependency to 4.8.0,
>   where the previous 4.4 dependency did not yet support loading a
> Windows JNI
>   library.
>   2. Follow the instructions here (
>https://github.com/facebook/rocksdb/blob/v4.8/CMakeLists.txt) to build
>rocksdb.
>   - That link is for the v4.8 tag; be sure to match this with the
>   version of rocksdb that Kafka Streams depends upon in the
> future.  0.10.0.1
>   -> v4.8, but future releases of Kafka Streams will likely depend on
> newer
>   versions.
>   - Be sure to include the "-DJNI=1" compile option to build the JNI
>   wrapper.
>   - None of the third-party dependencies (eg. snappy, jemalloc, etc.)
>   seem to be required to get something functional, but, it
> probably isn't the
>   most efficient build.
>   - Ensure that "javac" and "javah" are on your path when running
>   cmake; if there are any errors in the cmake output your JNI wrapper
>   probably won't build.
>   - You may or may not need to make minor patches to make the project
>   build in Windows.  It appears that the Windows build is often
> broken; for
>   example, I had to apply this patch:
>   https://github.com/facebook/rocksdb/pull/1223/files
>   - Phew, that should give you a build\java\Debug\rocksdbjni.dll.  So
>   close to the summit... just a horrible hack left...
>3. Copy rocksdbjni.dll to librocksdbjni-win64.dll.
>4. Insert librocksdbjni-win64.dll into your rocksdbjni-4.8.0.jar.
>   - Ugh, this is the horrible hack.  rocksdbjni seems to only look
>   inside its own jar for its native libraries, so, this is where
> it needs to
>   be.
>   - If you're a gradle user on Windows, you'd find this jar file
>   in
> C:\Users\...your-windows-user...\.gradle\caches\modules-2\files-2.1\org.rocksdb\rocksdbjni\4.8.0\b543fc4ea5b52ad790730dee376ba0df06d9f5f7.
>
> And there you go, almost a working Kafka Streams app in Windows.  One other
> detail is that the default state storage directory doesn't seem to be
> created on demand, so I had to mkdir C:\tmp\kafka-streams myself before my
> app would work.
>
> Mathieu
>
>
> On Fri, Aug 5, 2016 at 3:13 AM, Eno Thereska 
> wrote:
>
> > Hi Mathieu,
> >
> > It is true that the DSL currently does not support configuration of the
> > stores.
> >
> > Sounds like it might be worth trying to build RocksDb and dropping into
> > classpath for now.
> >
> > Eno
> >
> > > On 4 Aug 2016, at 17:42, Mathieu Fenniak  >
> > wrote:
> > >
> > > Hi Eno,
> > >
> > > Yes, I've looked at that.  RocksDB can be built and run in Windows,
> but,
> > > the JNI wrapper does not include Windows binarie (
> > > https://github.com/facebook/rocksdb/issues/703).  rocksdbjni-4.4.1.jar
> > > includes librocksdbjni-linux32.so, librocksdbjni-linux64.so, and
> > > librocksdbjni-osx.jnilib, so only supports Linux x86 & x64 and OS X.
> It
> > is
> > > probably possible for me to build it myself and drop it in my
> classpath,
> > > but, I'm looking for a lower friction approach if one exists. :-)
> > >
> > > It looks like this was discussed recently on the Confluent Platform
> > mailing
> > > list (https://groups.google.com/forum/#!topic/confluent-
> > platform/Z1rsfSNrVJk)
> > > and the conclusion there was that high-level streams DSL doesn't
> support
> > > configuration of the stores.
> > >
> > > Mathieu
> > >
> > >
> > > On Thu, Aug 4, 2016 at 10:28 AM, Eno Thereska 
> > > wrote:
> > >
> > >> Hi Mathieu,
> > >>
> > >> Have you had a chance to look at http://rocksdb.org/blog/2033/
> > >> rocksdb-is-now-available-in-windows-platform/? <
> > >> http://rocksdb.org/blog/2033/rocksdb-is-now-available-in-
> > windows-platform/?>
> > >> Curious to hear your and other's comments on whether that worked.
> > >>
> > >> It is possible to have Kafka Streams use an in-memory store (included
> > with
> > >> Kafka Streams) for development purposes. In that scenario RocksDb
> would
> > not
> > >> be needed.
> > >>
> > >> Eno
> > >>
> > >>
> > >>> On 4 Aug 2016, at 16:14, Mathieu Fenniak <
> mathieu.fenn...@replicon.com
> > >
> > >> wrote:
> > >>>
> > >>> Hey all,
> > >>>
> > >>> Is it anyone developing Kafka Streams applications on Windows?
> > >>>
> > >>> It seems like the RocksDB Java library doesn't include a native JNI
> > >> library
> > >>> for Windows, which prevents a Kafka Streams app from running on
> > >> Windows.  I
> > >>> was just wondering if others have run into this, and if so, what
> > approach
> > >>> you took to resolve it.
> > >>>
> > >>> I'm favouring the idea of running my applications i

Re: Log compaction leaves empty files?

[Dustin Cote]

Harald,

I note that your last modified times are all the same.  Are you maybe using
Java 7?  There's some details here that a JDK bug in Java 7 causes the last
modified time to get updated on broker restart:
https://issues.apache.org/jira/browse/KAFKA-3802



On Fri, Aug 5, 2016 at 6:12 AM, Harald Kirsch 
wrote:

> Hi,
>
> experimenting with log compaction, I see Kafka go through all the steps,
> in particular I see positive messages in log-cleaner.log and *.deleted
> files. Yet once the *.deleted segment files have disappeared, the segment
> and index files with size 0 are still kept.
>
> I stopped and restarted Kafka and saw several rounds of the log cleaner go
> by, but the empty files stay:
>
> -rw-rw-r-- 1 0 Aug  5 11:58 .index
> -rw-rw-r-- 1 0 Aug  5 11:58 .log
> -rw-rw-r-- 1 0 Aug  5 11:58 0051.index
> -rw-rw-r-- 1 0 Aug  5 11:58 0051.log
> -rw-rw-r-- 1 0 Aug  5 11:58 0096.index
> -rw-rw-r-- 1 0 Aug  5 11:58 0096.log
> [... snip ...]
> -rw-rw-r-- 1 92041 Aug  5 11:58 0680.log
> -rw-rw-r-- 1 0 Aug  5 11:58 0727.index
> -rw-rw-r-- 1199822 Aug  5 11:58 0727.log
> -rw-rw-r-- 1  10485760 Aug  5 11:58 0781.index
> -rw-rw-r-- 1 95972 Aug  5 11:58 0781.log
>
> Is this expected behavior or is there yet another configuration option
> that defines when these get purged?
>
> Harald.
>



-- 
Dustin Cote
confluent.io

Kafka cluster with a different version that the java API

[Sergio Gonzalez]

Hi users,

Is there some issue if I create the kafka cluster using the
kafka_2.10-0.8.2.0 version  and I have my java producers and consumers with
the 0.10.0.0 version?


org.apache.kafka
kafka-clients
0.10.0.0


   org.apache.kafka
   kafka-streams
   0.10.0.0


What are the repercutions that i could have if I use this enviroment?

Thanks,
Sergio GQ

Kafka consumer getting duplicate message

[Ghosh, Achintya (Contractor)]

Hi there,

We are using Kafka 1.0.0.M2 with Spring and we see a lot of duplicate message 
is getting received by the Listener onMessage() method .
We configured :

enable.auto.commit=false
session.timeout.ms=15000
factory.getContainerProperties().setSyncCommits(true);
factory.setConcurrency(5);

So what could be the reason to get the duplicate messages?

Thanks
Achintya

How to leverage kafka-connect-hdfs in HDP

[Banias H]

Hi,

We are using Hortonworks HDP 2.4 with Apache Kafla 0.9 and we have an
in-house solution to pull messages from Kafka to HDFS. I would like to try
using kakfa-connector-hdfs to push messages to HDFS. As far as I concern,
Apache Kafka 0.9 doesn't come with kafka-connector-hdfs. What is a solid
way to run kafka-connector-hdfs in HDP? I won't be able to install
Confluent platform there though... I would appreciate any pointers. Thanks.

-B

Re: Kafka Streams on Windows?

[Michael Noll]

Thanks a lot for investing and also for sharing back your findings, Mathieu!

-Michael


On Fri, Aug 5, 2016 at 3:10 PM, Mathieu Fenniak <
mathieu.fenn...@replicon.com> wrote:

> I took that approach.  It was painful, but, ultimately did get me a working
> Windows development environment.
>
> To any who follow in my footsteps, here is my trail:
>
>1. Upgrade to at least Kafka Streams 0.10.0.1 (currently only in RC).
>   - This is necessary because .1 bumps the rocksdb dependency to 4.8.0,
>   where the previous 4.4 dependency did not yet support loading a
> Windows JNI
>   library.
>   2. Follow the instructions here (
>https://github.com/facebook/rocksdb/blob/v4.8/CMakeLists.txt) to build
>rocksdb.
>   - That link is for the v4.8 tag; be sure to match this with the
>   version of rocksdb that Kafka Streams depends upon in the
> future.  0.10.0.1
>   -> v4.8, but future releases of Kafka Streams will likely depend on
> newer
>   versions.
>   - Be sure to include the "-DJNI=1" compile option to build the JNI
>   wrapper.
>   - None of the third-party dependencies (eg. snappy, jemalloc, etc.)
>   seem to be required to get something functional, but, it
> probably isn't the
>   most efficient build.
>   - Ensure that "javac" and "javah" are on your path when running
>   cmake; if there are any errors in the cmake output your JNI wrapper
>   probably won't build.
>   - You may or may not need to make minor patches to make the project
>   build in Windows.  It appears that the Windows build is often
> broken; for
>   example, I had to apply this patch:
>   https://github.com/facebook/rocksdb/pull/1223/files
>   - Phew, that should give you a build\java\Debug\rocksdbjni.dll.  So
>   close to the summit... just a horrible hack left...
>3. Copy rocksdbjni.dll to librocksdbjni-win64.dll.
>4. Insert librocksdbjni-win64.dll into your rocksdbjni-4.8.0.jar.
>   - Ugh, this is the horrible hack.  rocksdbjni seems to only look
>   inside its own jar for its native libraries, so, this is where
> it needs to
>   be.
>   - If you're a gradle user on Windows, you'd find this jar file
>   in C:\Users\...your-windows-user...\.gradle\caches\modules-2\
> files-2.1\org.rocksdb\rocksdbjni\4.8.0\b543fc4ea5b52ad790730dee376ba0
> df06d9f5f7.
>
> And there you go, almost a working Kafka Streams app in Windows.  One other
> detail is that the default state storage directory doesn't seem to be
> created on demand, so I had to mkdir C:\tmp\kafka-streams myself before my
> app would work.
>
> Mathieu
>
>
> On Fri, Aug 5, 2016 at 3:13 AM, Eno Thereska 
> wrote:
>
> > Hi Mathieu,
> >
> > It is true that the DSL currently does not support configuration of the
> > stores.
> >
> > Sounds like it might be worth trying to build RocksDb and dropping into
> > classpath for now.
> >
> > Eno
> >
> > > On 4 Aug 2016, at 17:42, Mathieu Fenniak  >
> > wrote:
> > >
> > > Hi Eno,
> > >
> > > Yes, I've looked at that.  RocksDB can be built and run in Windows,
> but,
> > > the JNI wrapper does not include Windows binarie (
> > > https://github.com/facebook/rocksdb/issues/703).  rocksdbjni-4.4.1.jar
> > > includes librocksdbjni-linux32.so, librocksdbjni-linux64.so, and
> > > librocksdbjni-osx.jnilib, so only supports Linux x86 & x64 and OS X.
> It
> > is
> > > probably possible for me to build it myself and drop it in my
> classpath,
> > > but, I'm looking for a lower friction approach if one exists. :-)
> > >
> > > It looks like this was discussed recently on the Confluent Platform
> > mailing
> > > list (https://groups.google.com/forum/#!topic/confluent-
> > platform/Z1rsfSNrVJk)
> > > and the conclusion there was that high-level streams DSL doesn't
> support
> > > configuration of the stores.
> > >
> > > Mathieu
> > >
> > >
> > > On Thu, Aug 4, 2016 at 10:28 AM, Eno Thereska 
> > > wrote:
> > >
> > >> Hi Mathieu,
> > >>
> > >> Have you had a chance to look at http://rocksdb.org/blog/2033/
> > >> rocksdb-is-now-available-in-windows-platform/? <
> > >> http://rocksdb.org/blog/2033/rocksdb-is-now-available-in-
> > windows-platform/?>
> > >> Curious to hear your and other's comments on whether that worked.
> > >>
> > >> It is possible to have Kafka Streams use an in-memory store (included
> > with
> > >> Kafka Streams) for development purposes. In that scenario RocksDb
> would
> > not
> > >> be needed.
> > >>
> > >> Eno
> > >>
> > >>
> > >>> On 4 Aug 2016, at 16:14, Mathieu Fenniak <
> mathieu.fenn...@replicon.com
> > >
> > >> wrote:
> > >>>
> > >>> Hey all,
> > >>>
> > >>> Is it anyone developing Kafka Streams applications on Windows?
> > >>>
> > >>> It seems like the RocksDB Java library doesn't include a native JNI
> > >> library
> > >>> for Windows, which prevents a Kafka Streams app from running on
> > >> Windows.  I
> > >>> was just wondering if others have run into this, and if so, what
> > approach
> > >>> you took to resol

Re: Unable to write, leader not available

[Bryan Baugher]

Found I still hit this issue without VPN. I had to make the cluster's user
a super user or at least give it appropriate privileges

On Thu, Aug 4, 2016 at 11:39 AM Bryan Baugher  wrote:

> Figured this out. This had to do with me being on a VPN and running
> everything locally
>
> On Thu, Aug 4, 2016 at 11:26 AM Bryan Baugher  wrote:
>
>> I managed to replicate this issue using the default provided config for
>> kafka/zookeeper in the binary artifact and using the console producer to
>> write a message
>>
>> On Thu, Aug 4, 2016 at 10:23 AM Bryan Baugher  wrote:
>>
>>> Using the --producer option in kafka-acls.sh it looks to have allowed
>>> create operations on the cluster. Turning on trace logging for
>>> authorization shows repeated mentions of my user and that its allowed to
>>> create on the cluster and describe the topic.
>>>
>>> Looks like I might not be the only one with this issue[1] so I'm
>>> wondering if its not kerberos related
>>>
>>> [1] -
>>> http://mail-archives.apache.org/mod_mbox/kafka-users/201608.mbox/%3CBLU184-W1930FDD3A39241FFDA0E6AB3040%40phx.gbl%3E
>>>
>>> On Wed, Aug 3, 2016 at 11:09 PM Manikumar Reddy <
>>> manikumar.re...@gmail.com> wrote:
>>>
 Hi,

 Can you enable Authorization debug logs and check for logs related to
 denied operations..
 we should also enable operations on Cluster resource.


 Thanks,
 Manikumar

 On Thu, Aug 4, 2016 at 1:51 AM, Bryan Baugher  wrote:

 > Hi everyone,
 >
 > I was trying out kerberos on Kafka 0.10.0.0 by creating a single node
 > cluster. I managed to get everything setup and past all the
 authentication
 > errors but whenever I try to use the console producer I get 'Error
 while
 > fetching metadata ... LEADER_NOT_AVAILABLE'. In this case I've
 created the
 > topic ahead of time (1 replica, 1 partition) and I can see that
 broker 0 is
 > in the ISR and is the leader. I have also opened an ACL to the topic
 for my
 > user to produce and was previously seeing authentication errors
 prior. I
 > don't see any errors or helpful logs on the broker side even after
 turning
 > on debug logging. Turning on debug logging on the client the only
 thing
 > that stands out is that it lists the broker as 'node -1' instead of
 0. It
 > does mention the correct hostname/port and that it was able to
 successfully
 > connect. Any ideas?
 >
 > Bryan
 >

>>>

Re: [VOTE] 0.10.0.1 RC2

[Tom Crayford]

Heroku has tested this using the same performance testing setup we used to
evaluate the impact of 0.9 -> 0.10 (see https://engineering.
heroku.com/blogs/2016-05-27-apache-kafka-010-evaluating-
performance-in-distributed-systems/).

We see no issues at all with them, so +1 (non-binding) from here.

On Fri, Aug 5, 2016 at 12:58 PM, Jim Jagielski  wrote:

> Looks good here: +1
>
> > On Aug 4, 2016, at 9:54 AM, Ismael Juma  wrote:
> >
> > Hello Kafka users, developers and client-developers,
> >
> > This is the third candidate for the release of Apache Kafka 0.10.0.1.
> This
> > is a bug fix release and it includes fixes and improvements from 53 JIRAs
> > (including a few critical bugs). See the release notes for more details:
> >
> > http://home.apache.org/~ijuma/kafka-0.10.0.1-rc2/RELEASE_NOTES.html
> >
> > When compared to RC1, RC2 contains a fix for a regression where an older
> > version of slf4j-log4j12 was also being included in the libs folder of
> the
> > binary tarball (KAFKA-4008). Thanks to Manikumar Reddy for reporting the
> > issue.
> >
> > *** Please download, test and vote by Monday, 8 August, 8am PT ***
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > http://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > http://home.apache.org/~ijuma/kafka-0.10.0.1-rc2/
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging
> >
> > * Javadoc:
> > http://home.apache.org/~ijuma/kafka-0.10.0.1-rc2/javadoc/
> >
> > * Tag to be voted upon (off 0.10.0 branch) is the 0.10.0.1-rc2 tag:
> > https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
> f8f56751744ba8e55f90f5c4f3aed8c3459447b2
> >
> > * Documentation:
> > http://kafka.apache.org/0100/documentation.html
> >
> > * Protocol:
> > http://kafka.apache.org/0100/protocol.html
> >
> > * Successful Jenkins builds for the 0.10.0 branch:
> > Unit/integration tests: *https://builds.apache.org/job
> /kafka-0.10.0-jdk7/182/
> > *
> > System tests: *https://jenkins.confluent.io/
> job/system-test-kafka-0.10.0/138/
> > *
> >
> > Thanks,
> > Ismael
>
>

Re: [VOTE] 0.10.0.1 RC2

[Dana Powers]

passed kafka-python integration tests, +1

-Dana


On Fri, Aug 5, 2016 at 9:35 AM, Tom Crayford  wrote:
> Heroku has tested this using the same performance testing setup we used to
> evaluate the impact of 0.9 -> 0.10 (see https://engineering.
> heroku.com/blogs/2016-05-27-apache-kafka-010-evaluating-
> performance-in-distributed-systems/).
>
> We see no issues at all with them, so +1 (non-binding) from here.
>
> On Fri, Aug 5, 2016 at 12:58 PM, Jim Jagielski  wrote:
>
>> Looks good here: +1
>>
>> > On Aug 4, 2016, at 9:54 AM, Ismael Juma  wrote:
>> >
>> > Hello Kafka users, developers and client-developers,
>> >
>> > This is the third candidate for the release of Apache Kafka 0.10.0.1.
>> This
>> > is a bug fix release and it includes fixes and improvements from 53 JIRAs
>> > (including a few critical bugs). See the release notes for more details:
>> >
>> > http://home.apache.org/~ijuma/kafka-0.10.0.1-rc2/RELEASE_NOTES.html
>> >
>> > When compared to RC1, RC2 contains a fix for a regression where an older
>> > version of slf4j-log4j12 was also being included in the libs folder of
>> the
>> > binary tarball (KAFKA-4008). Thanks to Manikumar Reddy for reporting the
>> > issue.
>> >
>> > *** Please download, test and vote by Monday, 8 August, 8am PT ***
>> >
>> > Kafka's KEYS file containing PGP keys we use to sign the release:
>> > http://kafka.apache.org/KEYS
>> >
>> > * Release artifacts to be voted upon (source and binary):
>> > http://home.apache.org/~ijuma/kafka-0.10.0.1-rc2/
>> >
>> > * Maven artifacts to be voted upon:
>> > https://repository.apache.org/content/groups/staging
>> >
>> > * Javadoc:
>> > http://home.apache.org/~ijuma/kafka-0.10.0.1-rc2/javadoc/
>> >
>> > * Tag to be voted upon (off 0.10.0 branch) is the 0.10.0.1-rc2 tag:
>> > https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
>> f8f56751744ba8e55f90f5c4f3aed8c3459447b2
>> >
>> > * Documentation:
>> > http://kafka.apache.org/0100/documentation.html
>> >
>> > * Protocol:
>> > http://kafka.apache.org/0100/protocol.html
>> >
>> > * Successful Jenkins builds for the 0.10.0 branch:
>> > Unit/integration tests: *https://builds.apache.org/job
>> /kafka-0.10.0-jdk7/182/
>> > *
>> > System tests: *https://jenkins.confluent.io/
>> job/system-test-kafka-0.10.0/138/
>> > *
>> >
>> > Thanks,
>> > Ismael
>>
>>

Re: [VOTE] 0.10.0.1 RC2

[Grant Henke]

+1 (non-binding)

On Fri, Aug 5, 2016 at 2:04 PM, Dana Powers  wrote:

> passed kafka-python integration tests, +1
>
> -Dana
>
>
> On Fri, Aug 5, 2016 at 9:35 AM, Tom Crayford  wrote:
> > Heroku has tested this using the same performance testing setup we used
> to
> > evaluate the impact of 0.9 -> 0.10 (see https://engineering.
> > heroku.com/blogs/2016-05-27-apache-kafka-010-evaluating-
> > performance-in-distributed-systems/).
> >
> > We see no issues at all with them, so +1 (non-binding) from here.
> >
> > On Fri, Aug 5, 2016 at 12:58 PM, Jim Jagielski  wrote:
> >
> >> Looks good here: +1
> >>
> >> > On Aug 4, 2016, at 9:54 AM, Ismael Juma  wrote:
> >> >
> >> > Hello Kafka users, developers and client-developers,
> >> >
> >> > This is the third candidate for the release of Apache Kafka 0.10.0.1.
> >> This
> >> > is a bug fix release and it includes fixes and improvements from 53
> JIRAs
> >> > (including a few critical bugs). See the release notes for more
> details:
> >> >
> >> > http://home.apache.org/~ijuma/kafka-0.10.0.1-rc2/RELEASE_NOTES.html
> >> >
> >> > When compared to RC1, RC2 contains a fix for a regression where an
> older
> >> > version of slf4j-log4j12 was also being included in the libs folder of
> >> the
> >> > binary tarball (KAFKA-4008). Thanks to Manikumar Reddy for reporting
> the
> >> > issue.
> >> >
> >> > *** Please download, test and vote by Monday, 8 August, 8am PT ***
> >> >
> >> > Kafka's KEYS file containing PGP keys we use to sign the release:
> >> > http://kafka.apache.org/KEYS
> >> >
> >> > * Release artifacts to be voted upon (source and binary):
> >> > http://home.apache.org/~ijuma/kafka-0.10.0.1-rc2/
> >> >
> >> > * Maven artifacts to be voted upon:
> >> > https://repository.apache.org/content/groups/staging
> >> >
> >> > * Javadoc:
> >> > http://home.apache.org/~ijuma/kafka-0.10.0.1-rc2/javadoc/
> >> >
> >> > * Tag to be voted upon (off 0.10.0 branch) is the 0.10.0.1-rc2 tag:
> >> > https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
> >> f8f56751744ba8e55f90f5c4f3aed8c3459447b2
> >> >
> >> > * Documentation:
> >> > http://kafka.apache.org/0100/documentation.html
> >> >
> >> > * Protocol:
> >> > http://kafka.apache.org/0100/protocol.html
> >> >
> >> > * Successful Jenkins builds for the 0.10.0 branch:
> >> > Unit/integration tests: *https://builds.apache.org/job
> >> /kafka-0.10.0-jdk7/182/
> >> > *
> >> > System tests: *https://jenkins.confluent.io/
> >> job/system-test-kafka-0.10.0/138/
> >> > *
> >> >
> >> > Thanks,
> >> > Ismael
> >>
> >>
>



-- 
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke

Re: How to leverage kafka-connect-hdfs in HDP

[Gwen Shapira]

The installation instructions from Confluent will still work for you :)

If you are using deb/rpm packages, basically add the repositories as
explained here:
http://docs.confluent.io/3.0.0/installation.html#rpm-packages-via-yum

and then:
sudo yum install confluent-kafka-connect-hdfs
or
sudo apt-get install confluent-kafka-connect-hdfs

This will put the connector config in /etc/kafka-connect-hdfs and the
connector jars in /usr/share/java/

You may need to move the jar so it is on the classpath for connect
(I'm not sure what's the default kafka classpath for HDP).

BTW. We (Confluent) are testing the HDFS connector with HDP (we
basically install Confluent platform on one machine and HDP on another
and use Connect to move data) - so this setup should work :)

Gwen


On Fri, Aug 5, 2016 at 8:47 AM, Banias H  wrote:
> Hi,
>
> We are using Hortonworks HDP 2.4 with Apache Kafla 0.9 and we have an
> in-house solution to pull messages from Kafka to HDFS. I would like to try
> using kakfa-connector-hdfs to push messages to HDFS. As far as I concern,
> Apache Kafka 0.9 doesn't come with kafka-connector-hdfs. What is a solid
> way to run kafka-connector-hdfs in HDP? I won't be able to install
> Confluent platform there though... I would appreciate any pointers. Thanks.
>
> -B



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [VOTE] 0.10.0.1 RC2

[Neha Narkhede]

+1 (binding)

On Fri, Aug 5, 2016 at 12:29 PM, Grant Henke  wrote:

> +1 (non-binding)
>
> On Fri, Aug 5, 2016 at 2:04 PM, Dana Powers  wrote:
>
> > passed kafka-python integration tests, +1
> >
> > -Dana
> >
> >
> > On Fri, Aug 5, 2016 at 9:35 AM, Tom Crayford 
> wrote:
> > > Heroku has tested this using the same performance testing setup we used
> > to
> > > evaluate the impact of 0.9 -> 0.10 (see https://engineering.
> > > heroku.com/blogs/2016-05-27-apache-kafka-010-evaluating-
> > > performance-in-distributed-systems/).
> > >
> > > We see no issues at all with them, so +1 (non-binding) from here.
> > >
> > > On Fri, Aug 5, 2016 at 12:58 PM, Jim Jagielski 
> wrote:
> > >
> > >> Looks good here: +1
> > >>
> > >> > On Aug 4, 2016, at 9:54 AM, Ismael Juma  wrote:
> > >> >
> > >> > Hello Kafka users, developers and client-developers,
> > >> >
> > >> > This is the third candidate for the release of Apache Kafka
> 0.10.0.1.
> > >> This
> > >> > is a bug fix release and it includes fixes and improvements from 53
> > JIRAs
> > >> > (including a few critical bugs). See the release notes for more
> > details:
> > >> >
> > >> > http://home.apache.org/~ijuma/kafka-0.10.0.1-rc2/RELEASE_NOTES.html
> > >> >
> > >> > When compared to RC1, RC2 contains a fix for a regression where an
> > older
> > >> > version of slf4j-log4j12 was also being included in the libs folder
> of
> > >> the
> > >> > binary tarball (KAFKA-4008). Thanks to Manikumar Reddy for reporting
> > the
> > >> > issue.
> > >> >
> > >> > *** Please download, test and vote by Monday, 8 August, 8am PT ***
> > >> >
> > >> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > >> > http://kafka.apache.org/KEYS
> > >> >
> > >> > * Release artifacts to be voted upon (source and binary):
> > >> > http://home.apache.org/~ijuma/kafka-0.10.0.1-rc2/
> > >> >
> > >> > * Maven artifacts to be voted upon:
> > >> > https://repository.apache.org/content/groups/staging
> > >> >
> > >> > * Javadoc:
> > >> > http://home.apache.org/~ijuma/kafka-0.10.0.1-rc2/javadoc/
> > >> >
> > >> > * Tag to be voted upon (off 0.10.0 branch) is the 0.10.0.1-rc2 tag:
> > >> > https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
> > >> f8f56751744ba8e55f90f5c4f3aed8c3459447b2
> > >> >
> > >> > * Documentation:
> > >> > http://kafka.apache.org/0100/documentation.html
> > >> >
> > >> > * Protocol:
> > >> > http://kafka.apache.org/0100/protocol.html
> > >> >
> > >> > * Successful Jenkins builds for the 0.10.0 branch:
> > >> > Unit/integration tests: *https://builds.apache.org/job
> > >> /kafka-0.10.0-jdk7/182/
> > >> > *
> > >> > System tests: *https://jenkins.confluent.io/
> > >> job/system-test-kafka-0.10.0/138/
> > >> > *
> > >> >
> > >> > Thanks,
> > >> > Ismael
> > >>
> > >>
> >
>
>
>
> --
> Grant Henke
> Software Engineer | Cloudera
> gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
>



-- 
Thanks,
Neha

Re: How to leverage kafka-connect-hdfs in HDP

[Banias H]

Thanks Gwen. I went with Confluent 2.0 as it has Kakfa 0.9 that matches
with that in HDP 2.4. I installed confluent-kafka-connect-hdfs and
confluent-common and softlinked a couple jar into kafka libs/.

I was able to start Kafka Connect but kafka.out was showing the following
error:

[2016-08-05 20:57:01,187] ERROR Exception emitting metrics
(org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter)
org.apache.hadoop.metrics2.sink.timeline.UnableToConnectException:
java.net.ConnectException: Connection refused
at
org.apache.hadoop.metrics2.sink.timeline.AbstractTimelineMetricsSink.emitMetrics(AbstractTimelineMetricsSink.java:87)
at
org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter.access$200(KafkaTimelineMetricsReporter.java:58)
at
org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter$TimelineScheduledReporter.report(KafkaTimelineMetricsReporter.java:253)
at
org.apache.hadoop.metrics2.sink.kafka.ScheduledReporter.report(ScheduledReporter.java:185)
at
org.apache.hadoop.metrics2.sink.kafka.ScheduledReporter$1.run(ScheduledReporter.java:137)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Is there any configuration I should also look into? I started with

./connect-standalone.sh ../config/connect-standalone.properties
/etc/kafka-connect-hdfs/quickstart-hdfs.properties

And here is my quickstart-hdfs.properties:

name=hdfs-sink
connector.class=io.confluent.connect.hdfs.HdfsSinkConnector
tasks.max=1
topics=hdfs
hdfs.url=hdfs://sandbox.hortonworks.com:8020
flush.size=3

Thanks,
-B

On Fri, Aug 5, 2016 at 3:31 PM, Gwen Shapira  wrote:

> The installation instructions from Confluent will still work for you :)
>
> If you are using deb/rpm packages, basically add the repositories as
> explained here:
> http://docs.confluent.io/3.0.0/installation.html#rpm-packages-via-yum
>
> and then:
> sudo yum install confluent-kafka-connect-hdfs
> or
> sudo apt-get install confluent-kafka-connect-hdfs
>
> This will put the connector config in /etc/kafka-connect-hdfs and the
> connector jars in /usr/share/java/
>
> You may need to move the jar so it is on the classpath for connect
> (I'm not sure what's the default kafka classpath for HDP).
>
> BTW. We (Confluent) are testing the HDFS connector with HDP (we
> basically install Confluent platform on one machine and HDP on another
> and use Connect to move data) - so this setup should work :)
>
> Gwen
>
>
> On Fri, Aug 5, 2016 at 8:47 AM, Banias H  wrote:
> > Hi,
> >
> > We are using Hortonworks HDP 2.4 with Apache Kafla 0.9 and we have an
> > in-house solution to pull messages from Kafka to HDFS. I would like to
> try
> > using kakfa-connector-hdfs to push messages to HDFS. As far as I concern,
> > Apache Kafka 0.9 doesn't come with kafka-connector-hdfs. What is a solid
> > way to run kafka-connector-hdfs in HDP? I won't be able to install
> > Confluent platform there though... I would appreciate any pointers.
> Thanks.
> >
> > -B
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>

Re: How to leverage kafka-connect-hdfs in HDP

[Gwen Shapira]

This is wierd, and looks like an Ambari class that ended up in the
classpath and that somehow we are trying to load?

Perhaps Sriharsha or one of the HDP dudes can help.

Does it happen without the Connector too? It looks like it has to do
with Kafka broker metrics in general:
https://issues.apache.org/jira/browse/AMBARI-9185

Gwen

On Fri, Aug 5, 2016 at 2:16 PM, Banias H  wrote:
> Thanks Gwen. I went with Confluent 2.0 as it has Kakfa 0.9 that matches
> with that in HDP 2.4. I installed confluent-kafka-connect-hdfs and
> confluent-common and softlinked a couple jar into kafka libs/.
>
> I was able to start Kafka Connect but kafka.out was showing the following
> error:
>
> [2016-08-05 20:57:01,187] ERROR Exception emitting metrics
> (org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter)
> org.apache.hadoop.metrics2.sink.timeline.UnableToConnectException:
> java.net.ConnectException: Connection refused
> at
> org.apache.hadoop.metrics2.sink.timeline.AbstractTimelineMetricsSink.emitMetrics(AbstractTimelineMetricsSink.java:87)
> at
> org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter.access$200(KafkaTimelineMetricsReporter.java:58)
> at
> org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter$TimelineScheduledReporter.report(KafkaTimelineMetricsReporter.java:253)
> at
> org.apache.hadoop.metrics2.sink.kafka.ScheduledReporter.report(ScheduledReporter.java:185)
> at
> org.apache.hadoop.metrics2.sink.kafka.ScheduledReporter$1.run(ScheduledReporter.java:137)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> Is there any configuration I should also look into? I started with
>
> ./connect-standalone.sh ../config/connect-standalone.properties
> /etc/kafka-connect-hdfs/quickstart-hdfs.properties
>
> And here is my quickstart-hdfs.properties:
>
> name=hdfs-sink
> connector.class=io.confluent.connect.hdfs.HdfsSinkConnector
> tasks.max=1
> topics=hdfs
> hdfs.url=hdfs://sandbox.hortonworks.com:8020
> flush.size=3
>
> Thanks,
> -B
>
> On Fri, Aug 5, 2016 at 3:31 PM, Gwen Shapira  wrote:
>
>> The installation instructions from Confluent will still work for you :)
>>
>> If you are using deb/rpm packages, basically add the repositories as
>> explained here:
>> http://docs.confluent.io/3.0.0/installation.html#rpm-packages-via-yum
>>
>> and then:
>> sudo yum install confluent-kafka-connect-hdfs
>> or
>> sudo apt-get install confluent-kafka-connect-hdfs
>>
>> This will put the connector config in /etc/kafka-connect-hdfs and the
>> connector jars in /usr/share/java/
>>
>> You may need to move the jar so it is on the classpath for connect
>> (I'm not sure what's the default kafka classpath for HDP).
>>
>> BTW. We (Confluent) are testing the HDFS connector with HDP (we
>> basically install Confluent platform on one machine and HDP on another
>> and use Connect to move data) - so this setup should work :)
>>
>> Gwen
>>
>>
>> On Fri, Aug 5, 2016 at 8:47 AM, Banias H  wrote:
>> > Hi,
>> >
>> > We are using Hortonworks HDP 2.4 with Apache Kafla 0.9 and we have an
>> > in-house solution to pull messages from Kafka to HDFS. I would like to
>> try
>> > using kakfa-connector-hdfs to push messages to HDFS. As far as I concern,
>> > Apache Kafka 0.9 doesn't come with kafka-connector-hdfs. What is a solid
>> > way to run kafka-connector-hdfs in HDP? I won't be able to install
>> > Confluent platform there though... I would appreciate any pointers.
>> Thanks.
>> >
>> > -B
>>
>>
>>
>> --
>> Gwen Shapira
>> Product Manager | Confluent
>> 650.450.2760 | @gwenshap
>> Follow us: Twitter | blog
>>



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [VOTE] 0.10.0.1 RC2

[Guozhang Wang]

Verified artifact md5 and ran quick start / unit test. +1 (binding).

On Fri, Aug 5, 2016 at 1:49 PM, Neha Narkhede  wrote:

> +1 (binding)
>
> On Fri, Aug 5, 2016 at 12:29 PM, Grant Henke  wrote:
>
> > +1 (non-binding)
> >
> > On Fri, Aug 5, 2016 at 2:04 PM, Dana Powers 
> wrote:
> >
> > > passed kafka-python integration tests, +1
> > >
> > > -Dana
> > >
> > >
> > > On Fri, Aug 5, 2016 at 9:35 AM, Tom Crayford 
> > wrote:
> > > > Heroku has tested this using the same performance testing setup we
> used
> > > to
> > > > evaluate the impact of 0.9 -> 0.10 (see https://engineering.
> > > > heroku.com/blogs/2016-05-27-apache-kafka-010-evaluating-
> > > > performance-in-distributed-systems/).
> > > >
> > > > We see no issues at all with them, so +1 (non-binding) from here.
> > > >
> > > > On Fri, Aug 5, 2016 at 12:58 PM, Jim Jagielski 
> > wrote:
> > > >
> > > >> Looks good here: +1
> > > >>
> > > >> > On Aug 4, 2016, at 9:54 AM, Ismael Juma 
> wrote:
> > > >> >
> > > >> > Hello Kafka users, developers and client-developers,
> > > >> >
> > > >> > This is the third candidate for the release of Apache Kafka
> > 0.10.0.1.
> > > >> This
> > > >> > is a bug fix release and it includes fixes and improvements from
> 53
> > > JIRAs
> > > >> > (including a few critical bugs). See the release notes for more
> > > details:
> > > >> >
> > > >> > http://home.apache.org/~ijuma/kafka-0.10.0.1-rc2/RELEASE_
> NOTES.html
> > > >> >
> > > >> > When compared to RC1, RC2 contains a fix for a regression where an
> > > older
> > > >> > version of slf4j-log4j12 was also being included in the libs
> folder
> > of
> > > >> the
> > > >> > binary tarball (KAFKA-4008). Thanks to Manikumar Reddy for
> reporting
> > > the
> > > >> > issue.
> > > >> >
> > > >> > *** Please download, test and vote by Monday, 8 August, 8am PT ***
> > > >> >
> > > >> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > >> > http://kafka.apache.org/KEYS
> > > >> >
> > > >> > * Release artifacts to be voted upon (source and binary):
> > > >> > http://home.apache.org/~ijuma/kafka-0.10.0.1-rc2/
> > > >> >
> > > >> > * Maven artifacts to be voted upon:
> > > >> > https://repository.apache.org/content/groups/staging
> > > >> >
> > > >> > * Javadoc:
> > > >> > http://home.apache.org/~ijuma/kafka-0.10.0.1-rc2/javadoc/
> > > >> >
> > > >> > * Tag to be voted upon (off 0.10.0 branch) is the 0.10.0.1-rc2
> tag:
> > > >> > https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
> > > >> f8f56751744ba8e55f90f5c4f3aed8c3459447b2
> > > >> >
> > > >> > * Documentation:
> > > >> > http://kafka.apache.org/0100/documentation.html
> > > >> >
> > > >> > * Protocol:
> > > >> > http://kafka.apache.org/0100/protocol.html
> > > >> >
> > > >> > * Successful Jenkins builds for the 0.10.0 branch:
> > > >> > Unit/integration tests: *https://builds.apache.org/job
> > > >> /kafka-0.10.0-jdk7/182/
> > > >> > *
> > > >> > System tests: *https://jenkins.confluent.io/
> > > >> job/system-test-kafka-0.10.0/138/
> > > >> > *
> > > >> >
> > > >> > Thanks,
> > > >> > Ismael
> > > >>
> > > >>
> > >
> >
> >
> >
> > --
> > Grant Henke
> > Software Engineer | Cloudera
> > gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
> >
>
>
>
> --
> Thanks,
> Neha
>



-- 
-- Guozhang

Re: [kafka-clients] [VOTE] 0.10.0.1 RC2

[Jun Rao]

Thanks for running the release. +1

Jun

On Thu, Aug 4, 2016 at 6:54 AM, Ismael Juma  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the third candidate for the release of Apache Kafka 0.10.0.1. This
> is a bug fix release and it includes fixes and improvements from 53 JIRAs
> (including a few critical bugs). See the release notes for more details:
>
> http://home.apache.org/~ijuma/kafka-0.10.0.1-rc2/RELEASE_NOTES.html
>
> When compared to RC1, RC2 contains a fix for a regression where an older
> version of slf4j-log4j12 was also being included in the libs folder of the
> binary tarball (KAFKA-4008). Thanks to Manikumar Reddy for reporting the
> issue.
>
> *** Please download, test and vote by Monday, 8 August, 8am PT ***
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> http://home.apache.org/~ijuma/kafka-0.10.0.1-rc2/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging
>
> * Javadoc:
> http://home.apache.org/~ijuma/kafka-0.10.0.1-rc2/javadoc/
>
> * Tag to be voted upon (off 0.10.0 branch) is the 0.10.0.1-rc2 tag:
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
> f8f56751744ba8e55f90f5c4f3aed8c3459447b2
>
> * Documentation:
> http://kafka.apache.org/0100/documentation.html
>
> * Protocol:
> http://kafka.apache.org/0100/protocol.html
>
> * Successful Jenkins builds for the 0.10.0 branch:
> Unit/integration tests: *https://builds.apache.org/job/kafka-0.10.0-jdk7/182/
> *
> System tests: *https://jenkins.confluent.io/job/system-test-kafka-0.10.0/138/
> *
>
> Thanks,
> Ismael
>
> --
> You received this message because you are subscribed to the Google Groups
> "kafka-clients" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kafka-clients+unsubscr...@googlegroups.com.
> To post to this group, send email to kafka-clie...@googlegroups.com.
> Visit this group at https://groups.google.com/group/kafka-clients.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/kafka-clients/CAD5tkZYMMxDEjg_2jt4x-mVZZHgJ6EC6HKSf4Hn%
> 2Bi59DbTdVoQ%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

Re: [kafka-clients] [VOTE] 0.10.0.1 RC2

[Jason Gustafson]

Verified checksums and signatures. Ran quickstart. +1 (non-binding)

On Fri, Aug 5, 2016 at 2:51 PM, Jun Rao  wrote:

> Thanks for running the release. +1
>
> Jun
>
> On Thu, Aug 4, 2016 at 6:54 AM, Ismael Juma  wrote:
>
> > Hello Kafka users, developers and client-developers,
> >
> > This is the third candidate for the release of Apache Kafka 0.10.0.1.
> This
> > is a bug fix release and it includes fixes and improvements from 53 JIRAs
> > (including a few critical bugs). See the release notes for more details:
> >
> > http://home.apache.org/~ijuma/kafka-0.10.0.1-rc2/RELEASE_NOTES.html
> >
> > When compared to RC1, RC2 contains a fix for a regression where an older
> > version of slf4j-log4j12 was also being included in the libs folder of
> the
> > binary tarball (KAFKA-4008). Thanks to Manikumar Reddy for reporting the
> > issue.
> >
> > *** Please download, test and vote by Monday, 8 August, 8am PT ***
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > http://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > http://home.apache.org/~ijuma/kafka-0.10.0.1-rc2/
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging
> >
> > * Javadoc:
> > http://home.apache.org/~ijuma/kafka-0.10.0.1-rc2/javadoc/
> >
> > * Tag to be voted upon (off 0.10.0 branch) is the 0.10.0.1-rc2 tag:
> > https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
> > f8f56751744ba8e55f90f5c4f3aed8c3459447b2
> >
> > * Documentation:
> > http://kafka.apache.org/0100/documentation.html
> >
> > * Protocol:
> > http://kafka.apache.org/0100/protocol.html
> >
> > * Successful Jenkins builds for the 0.10.0 branch:
> > Unit/integration tests: *https://builds.apache.org/
> job/kafka-0.10.0-jdk7/182/
> > *
> > System tests: *https://jenkins.confluent.io/
> job/system-test-kafka-0.10.0/138/
> > *
> >
> > Thanks,
> > Ismael
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "kafka-clients" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to kafka-clients+unsubscr...@googlegroups.com.
> > To post to this group, send email to kafka-clie...@googlegroups.com.
> > Visit this group at https://groups.google.com/group/kafka-clients.
> > To view this discussion on the web visit https://groups.google.com/d/
> > msgid/kafka-clients/CAD5tkZYMMxDEjg_2jt4x-mVZZHgJ6EC6HKSf4Hn%
> > 2Bi59DbTdVoQ%40mail.gmail.com
> >  mVZZHgJ6EC6HKSf4Hn%2Bi59DbTdVoQ%40mail.gmail.com?
> utm_medium=email&utm_source=footer>
> > .
> > For more options, visit https://groups.google.com/d/optout.
> >
>

Re: [VOTE] 0.10.0.1 RC2

[Vahid S Hashemian]

+1 (non-binding)

Tried the build and ran the quick start successfully on Ubuntu, Mac, and 
Windows.

--Vahid 




From:   Ismael Juma 
To: users@kafka.apache.org, d...@kafka.apache.org, kafka-clients 

Date:   08/04/2016 06:55 AM
Subject:[VOTE] 0.10.0.1 RC2
Sent by:isma...@gmail.com



Hello Kafka users, developers and client-developers,

This is the third candidate for the release of Apache Kafka 0.10.0.1. This
is a bug fix release and it includes fixes and improvements from 53 JIRAs
(including a few critical bugs). See the release notes for more details:

http://home.apache.org/~ijuma/kafka-0.10.0.1-rc2/RELEASE_NOTES.html

When compared to RC1, RC2 contains a fix for a regression where an older
version of slf4j-log4j12 was also being included in the libs folder of the
binary tarball (KAFKA-4008). Thanks to Manikumar Reddy for reporting the
issue.

*** Please download, test and vote by Monday, 8 August, 8am PT ***

Kafka's KEYS file containing PGP keys we use to sign the release:
http://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
http://home.apache.org/~ijuma/kafka-0.10.0.1-rc2/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging

* Javadoc:
http://home.apache.org/~ijuma/kafka-0.10.0.1-rc2/javadoc/

* Tag to be voted upon (off 0.10.0 branch) is the 0.10.0.1-rc2 tag:
https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=f8f56751744ba8e55f90f5c4f3aed8c3459447b2


* Documentation:
http://kafka.apache.org/0100/documentation.html

* Protocol:
http://kafka.apache.org/0100/protocol.html

* Successful Jenkins builds for the 0.10.0 branch:
Unit/integration tests: 
*https://builds.apache.org/job/kafka-0.10.0-jdk7/182/
*
System tests: 
*https://jenkins.confluent.io/job/system-test-kafka-0.10.0/138/
*

Thanks,
Ismael

Re: Kafka cluster with a different version that the java API

[Alex Loddengaard]

Hi Sergio, clients have to be the same version or older than the brokers. A
newer client won't work with an older broker.

Alex

On Fri, Aug 5, 2016 at 7:37 AM, Sergio Gonzalez <
sgonza...@cecropiasolutions.com> wrote:

> Hi users,
>
> Is there some issue if I create the kafka cluster using the
> kafka_2.10-0.8.2.0 version  and I have my java producers and consumers with
> the 0.10.0.0 version?
>
> 
> org.apache.kafka
> kafka-clients
> 0.10.0.0
> 
> 
>org.apache.kafka
>kafka-streams
>0.10.0.0
> 
>
> What are the repercutions that i could have if I use this enviroment?
>
> Thanks,
> Sergio GQ
>

Re: [kafka-clients] [VOTE] 0.10.0.1 RC2

[Joel Koshy]

+1 (binding)

Thanks Ismael!

On Thu, Aug 4, 2016 at 6:54 AM, Ismael Juma  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the third candidate for the release of Apache Kafka 0.10.0.1. This
> is a bug fix release and it includes fixes and improvements from 53 JIRAs
> (including a few critical bugs). See the release notes for more details:
>
> http://home.apache.org/~ijuma/kafka-0.10.0.1-rc2/RELEASE_NOTES.html
>
> When compared to RC1, RC2 contains a fix for a regression where an older
> version of slf4j-log4j12 was also being included in the libs folder of the
> binary tarball (KAFKA-4008). Thanks to Manikumar Reddy for reporting the
> issue.
>
> *** Please download, test and vote by Monday, 8 August, 8am PT ***
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> http://home.apache.org/~ijuma/kafka-0.10.0.1-rc2/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging
>
> * Javadoc:
> http://home.apache.org/~ijuma/kafka-0.10.0.1-rc2/javadoc/
>
> * Tag to be voted upon (off 0.10.0 branch) is the 0.10.0.1-rc2 tag:
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
> f8f56751744ba8e55f90f5c4f3aed8c3459447b2
>
> * Documentation:
> http://kafka.apache.org/0100/documentation.html
>
> * Protocol:
> http://kafka.apache.org/0100/protocol.html
>
> * Successful Jenkins builds for the 0.10.0 branch:
> Unit/integration tests: *https://builds.apache.org/job/kafka-0.10.0-jdk7/182/
> *
> System tests: *https://jenkins.confluent.io/job/system-test-kafka-0.10.0/138/
> *
>
> Thanks,
> Ismael
>
> --
> You received this message because you are subscribed to the Google Groups
> "kafka-clients" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kafka-clients+unsubscr...@googlegroups.com.
> To post to this group, send email to kafka-clie...@googlegroups.com.
> Visit this group at https://groups.google.com/group/kafka-clients.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/kafka-clients/CAD5tkZYMMxDEjg_2jt4x-mVZZHgJ6EC6HKSf4Hn%
> 2Bi59DbTdVoQ%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

Testing broker failover

[Alper Akture]

I'm using 0.10.0.0 and testing some failover scenarios. For dev, i have
single kafka node and a zookeeper instance. While sending events to a
topic, I shutdown the broker to see if my failover handling works. However,
I don't see any indication that the send failed, but I do see the
connection refused errors logged at debug. What is the standard way to
detect a message send failure, and handle it for offline processing later?

Here's the debug output I see:

19:20:00.906 [kafka-producer-network-thread | producer-1] DEBUG
org.apache.kafka.clients.NetworkClient - Initialize connection to node -1
for sending metadata request
19:20:00.906 [kafka-producer-network-thread | producer-1] DEBUG
org.apache.kafka.clients.NetworkClient - Initiating connection to node -1
at localhost:9092.
19:20:00.907 [kafka-producer-network-thread | producer-1] DEBUG
org.apache.kafka.common.network.Selector - Connection with localhost/
127.0.0.1 disconnected
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:1.8.0_66]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
~[?:1.8.0_66]
at
org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:51)
~[kafka-clients-0.10.0.0.jar:?]
at
org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:73)
~[kafka-clients-0.10.0.0.jar:?]
at
org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:309)
[kafka-clients-0.10.0.0.jar:?]
at org.apache.kafka.common.network.Selector.poll(Selector.java:283)
[kafka-clients-0.10.0.0.jar:?]
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:260)
[kafka-clients-0.10.0.0.jar:?]
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:229)
[kafka-clients-0.10.0.0.jar:?]
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:134)
[kafka-clients-0.10.0.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_66]
19:20:00.907 [kafka-producer-network-thread | producer-1] DEBUG
org.apache.kafka.clients.NetworkClient - Node -1 disconnected.
19:20:00.907 [kafka-producer-network-thread | producer-1] DEBUG
org.apache.kafka.clients.NetworkClient - Give up sending metadata request
since no node is available

Re: [kafka-clients] [VOTE] 0.10.0.1 RC2

[Manikumar Reddy]

+1 (non-binding).
verified quick start and artifacts.

On Sat, Aug 6, 2016 at 5:45 AM, Joel Koshy  wrote:

> +1 (binding)
>
> Thanks Ismael!
>
> On Thu, Aug 4, 2016 at 6:54 AM, Ismael Juma  wrote:
>
>> Hello Kafka users, developers and client-developers,
>>
>> This is the third candidate for the release of Apache Kafka 0.10.0.1.
>> This is a bug fix release and it includes fixes and improvements from 53
>> JIRAs (including a few critical bugs). See the release notes for more
>> details:
>>
>> http://home.apache.org/~ijuma/kafka-0.10.0.1-rc2/RELEASE_NOTES.html
>>
>> When compared to RC1, RC2 contains a fix for a regression where an older
>> version of slf4j-log4j12 was also being included in the libs folder of the
>> binary tarball (KAFKA-4008). Thanks to Manikumar Reddy for reporting the
>> issue.
>>
>> *** Please download, test and vote by Monday, 8 August, 8am PT ***
>>
>> Kafka's KEYS file containing PGP keys we use to sign the release:
>> http://kafka.apache.org/KEYS
>>
>> * Release artifacts to be voted upon (source and binary):
>> http://home.apache.org/~ijuma/kafka-0.10.0.1-rc2/
>>
>> * Maven artifacts to be voted upon:
>> https://repository.apache.org/content/groups/staging
>>
>> * Javadoc:
>> http://home.apache.org/~ijuma/kafka-0.10.0.1-rc2/javadoc/
>>
>> * Tag to be voted upon (off 0.10.0 branch) is the 0.10.0.1-rc2 tag:
>> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
>> f8f56751744ba8e55f90f5c4f3aed8c3459447b2
>>
>> * Documentation:
>> http://kafka.apache.org/0100/documentation.html
>>
>> * Protocol:
>> http://kafka.apache.org/0100/protocol.html
>>
>> * Successful Jenkins builds for the 0.10.0 branch:
>> Unit/integration tests: *https://builds.apache.org/job/kafka-0.10.0-jdk7/182/
>> *
>> System tests: *https://jenkins.confluent.io/job/system-test-kafka-0.10.0/138/
>> *
>>
>> Thanks,
>> Ismael
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "kafka-clients" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to kafka-clients+unsubscr...@googlegroups.com.
>> To post to this group, send email to kafka-clie...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/kafka-clients.
>> To view this discussion on the web visit https://groups.google.com/d/ms
>> gid/kafka-clients/CAD5tkZYMMxDEjg_2jt4x-mVZZHgJ6EC6HKSf4Hn%2
>> Bi59DbTdVoQ%40mail.gmail.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "kafka-clients" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kafka-clients+unsubscr...@googlegroups.com.
> To post to this group, send email to kafka-clie...@googlegroups.com.
> Visit this group at https://groups.google.com/group/kafka-clients.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/kafka-clients/CAAOfhrAUcmrFRH2PpsLLmv579WDOi
> oMOcpy1LBrLJfdWff5iFA%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

Re: Kafka consumer getting duplicate message

[Ewen Cheslack-Postava]

Achintya,

1.0.0.M2 is not an official release, so this version number is not
particularly meaningful to people on this list. What platform/distribution
are you using and how does this map to actual Apache Kafka releases?

In general, it is not possible for any system to guarantee exactly once
semantics because those semantics rely on the source and destination
systems coordinating -- the source provides some sort of retry semantics,
and the destination system needs to do some sort of deduplication or
similar to only "deliver" the data one time.

That said, duplicates should usually only be generated in the face of
failures. If you're seeing a lot of duplicates, that probably means
shutdown/failover is not being handled correctly. If you can provide more
info about your setup, we might be able to suggest tweaks that will avoid
these situations.

-Ewen

On Fri, Aug 5, 2016 at 8:15 AM, Ghosh, Achintya (Contractor) <
achintya_gh...@comcast.com> wrote:

> Hi there,
>
> We are using Kafka 1.0.0.M2 with Spring and we see a lot of duplicate
> message is getting received by the Listener onMessage() method .
> We configured :
>
> enable.auto.commit=false
> session.timeout.ms=15000
> factory.getContainerProperties().setSyncCommits(true);
> factory.setConcurrency(5);
>
> So what could be the reason to get the duplicate messages?
>
> Thanks
> Achintya
>



-- 
Thanks,
Ewen