Hi All, Thanks for the help I have worked out what the issue was. The Kafka producer I guess was trying to connect to the producer using the machines hostname. I was using an IP Address in the server list. So it could connect into get the meta data. As soon as it then tried to connect to the leader for that partition it was using the hostname, which wasn't resolvable form my local machine. Adding it to the hosts file allowed it work.
Out of interest whats the reason behind using the hostname rather than an IP address in the leader information? I guess it adds a layer of indirection? Thanks again for the help. Charlie M On Thu, May 28, 2015 at 9:31 PM, Joel Koshy <jjkosh...@gmail.com> wrote: > I think you can also set this dynamically via an mbean > (kafka.Log4jController) > > On Thu, May 28, 2015 at 08:26:00PM +0000, Jiangjie Qin wrote: > > Actually the name should be log4j.logger.kafka.network.RequestChannel$ > > It should be there in 0.8.2.1. Can you check it again? > > > > From: Charlie Mason <charlie....@gmail.com<mailto:charlie....@gmail.com > >> > > Reply-To: "charlie....@gmail.com<mailto:charlie....@gmail.com>" < > charlie....@gmail.com<mailto:charlie....@gmail.com>> > > Date: Thursday, May 28, 2015 at 1:09 PM > > To: Jiangjie Qin <j...@linkedin.com<mailto:j...@linkedin.com>> > > Cc: "users@kafka.apache.org<mailto:users@kafka.apache.org>" < > users@kafka.apache.org<mailto:users@kafka.apache.org>> > > Subject: Re: Kafka Not Commiting Messages > > > > Hi Jiangjie, > > > > Thanks for you message. Unfortunately there doesn't appear to have that > setting in log4j.properties in Kafka 0.8.2.1. > > > > I tried adding that property to the log file however it doesn't seem to > have any effect. > > > > Is there a different logger I need to configure for 0.8.2.1? > > > > > > Thanks, > > > > Charlie M > > > > On Thu, May 28, 2015 at 1:15 AM, Jiangjie Qin <j...@linkedin.com<mailto: > j...@linkedin.com>> wrote: > > Can you turn on TRACE level logging for kafka-request.log and see if > > broker received the producer request or not? > > You can go to KAKFA_FOLDER/config/log4j.properties and change > > log4j.logger.kafka.network.RequestChannels to TRACE. > > > > Jiangjie (Becket) Qin > > > > On 5/27/15, 12:12 PM, "Charlie Mason" <charlie....@gmail.com<mailto: > charlie....@gmail.com>> wrote: > > > > >Hi All, > > > > > >So I have done some more tests and found something I really don't > > >understand. > > > > > >I found a simple example of the Kafka Java producer so I ran that > pointing > > >at the same topic as my last test. That failed when run from my local > > >machine. I uploaded it to the VM where Kafka is installed and it worked > > >perfectly. Bare in mind this exactly the same code configured to talk to > > >exactly the same IP address. > > > > > >The thing I don't understand is I can see a connection via Wireshark to > > >Kafka when the code is running on my machine. I can even query for the > > >topic metadata successfully from my local machine. The only thing it > won't > > >do is commit a message. From what I have read the Kafka protocol is a > > >fairly straight forward single TCP connection to the server from the > > >client. > > > > > >Does anyone have any ideas what be causing this? Obviously people run in > > >production with the producer on different nodes from the brokers, so I > > >can't see whats different here. > > > > > >Thanks, > > > > > >Charlie M > > > > > > > > > > > > > > >On Tue, May 26, 2015 at 9:09 PM, Charlie Mason <charlie....@gmail.com > <mailto:charlie....@gmail.com>> > > >wrote: > > > > > >> Hi All, > > >> > > >> I have been trying to get started with Kafka. I have set up an 0.8.2 > > >> broker as per the quick start. With a single node broker I am able to > > >>run > > >> the scripts in the bin folder to successfully produce and consume > > >>messages. > > >> > > >> I then tried to write some Scala code to use the new 0.8 Producer API > to > > >> produce messages. However nothing seems to appear on the consumer. I > > >> modified the producer code to wait for the broker's metadata response. > > >>That > > >> blocks and then times out. > > >> > > >> The only difference I can see apart from using the new Producer API is > > >>all > > >> my previous tests were run inside the VM Kafka is installed on. Where > > >>as my > > >> code is running on the host machine and connecting into Kafka. I put > > >>some > > >> code into get the topics metadata to confirm connectivity to Kafka. > That > > >> prints the topic metadata correctly before hanging on the > > >>send(...).get(). > > >> I have also checked the topic is set for a replication factor of 1. I > > >>can't > > >> see anything in the Kafka logs either. All I see on the broker is a > > >>message > > >> when the Producer times out saying the client disconnected. > > >> > > >> Anyone got any ideas what might be making Kafka fail to commit the > > >> messages? > > >> > > >> I really want to start playing with Kafka however I seem to have > fallen > > >>at > > >> the first hurdle. > > >> > > >> Thanks, > > >> > > >> Charlie M > > >> > > > > > > -- > Joel >
