Re: Thread safety of Encoder implementations

Hi Liz, Do you have multiple producer clients that use the same Kryo serializer objects? Each client will only have one background thread that tries to call serialize(), and hence in that case you will have concurrent access. Guozhang On Mon, Jan 12, 2015 at 5:32 PM, Elizabeth Bennett wrote:

Thread safety of Encoder implementations

Hi Kafka Users, I have written my own implementation of the kafka Encoder class for serializing objects to Messages. It uses Kryo, which is a non-thread safe java serialization library. I'm using Kafka 0.7.2. We recently ran into an issue where we increased the number of kafka brokers for our kafk

Re: Monitor data to missing topic

I don't remember seeing a metric for that but I'd look at logs first. Not sure if anything is logged about it, but that's where I'd look for errors. Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/ On Thu, Jan

Re: Kafka Cluster Monitoring and Documentation of Internals (JMX Metrics) of Rejected Events

Hi, I think you could just email user@? There was no attached image. I think Jun committed something about this: https://issues.apache.org/jira/browse/KAFKA-1481?focusedCommentId=14272057&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14272057 Otis -- Monitoring * A

Re: New 0.8.2 client compatibility with 0.8.1.1 during failure cases

Thanks Ewan! I highly recommend this is clearly specified in the 0.8.2 producer docs. Regarding test cases, I will be pushing some I have in a KAFKA-1835. However, this doesn't enable running 0.8.2 producer integration tests against 0.8.1.1 server since the test cases are going in Kafka core inte

Re: Kafka 0.8.2 new producer blocking on metadata

Hi Steven, Speaking only for myself, I agree with you. I think these settings/tweaks are the easiest short term way to get some proper non-blocking behavior. Long term, it seems like having a secondary queue in the client to hold raw messages until meta is available and then start blocking or dro

Kafka Cluster Monitoring and Documentation of Internals (JMX Metrics) of Rejected Events

Hi Kafka Team, I am trying to find out Kafka Internal and how a message can be corrupted or lost at brokers side. I have refer to following documentations for monitoring: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Internals http://kafka.apache.org/documentation.html#monitoring I am

Re: new to kafka

Hi, Check http://search-hadoop.com/?q=benchmark&fc_project=Kafka for info about Kafka benchmarking. Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/ On Sun, Jan 11, 2015 at 8:03 PM, Alpesh Desai wrote: > I am fa

Re: Get replication and partition count of a topic

I think the closest thing to what you want is ZkUtils.getPartitionsForTopics, which returns a list of partition IDs for each topic you specify. -Ewen On Mon, Jan 12, 2015 at 12:55 AM, Manikumar Reddy wrote: > Hi, > > kafka-topics.sh script can be used to retrieve topic information. > > Ex: sh k

Re: Consumer and offset management support in 0.8.2 and 0.9

Catching up on email - yes the wire protocol change was a big mistake and +1 on this approach. I'm reviewing KAFKA-1841 right now. On Thu, Jan 08, 2015 at 10:07:44PM -0800, Dana Powers wrote: > This approach makes sense to me -- thanks, Jun. > > -Dana > > On Thu, Jan 8, 2015 at 3:57 PM, Jun Rao

Re: Using Kafka for Event Sourcing

up, as I am not sure you receive this email. Le Sun Jan 11 2015 at 5:34:17 PM, Yann Simon a écrit : > Hi, > > after having read > http://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying, > I am considering Kafka for an appli

Re: latency - how to reduce?

Thank you for the guidance, yet I want to know I am doing the test Jun had in mind On Thu, Jan 8, 2015 at 5:36 PM, Jayesh Thakrar wrote: > I do see the Windows based scripts in the tar file - but haven't them > though.You should find them under bin/windows. > Also you can always use other Window

Re: Get replication and partition count of a topic

Hi, kafka-topics.sh script can be used to retrieve topic information. Ex: sh kafka-topics.sh --zookeeper localhost:2181 --describe --topic TOPIC1 You can look into TopicCommand.scala code https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=blob;f=core/src/main/scala/kafka/admin/TopicCommand.sc

Get replication and partition count of a topic

Hi All, I want to get the replication and partition count of a topic. I tried the following piece of code: java.util.Set topics = new HashSet(); topics.add("topicName"); Set topicMetadatas = AdminUtils.fetchTopicMetadataFromZk(JavaConversions.asScalaSet(topics), zkCli