Max. storage for Kafka and impact

Hi, We are using Kafka for our messaging system and we have an estimate for 200 TB/week in the coming months. Will it impact any performance for Kafka? PS: We will be having greater than 2 lakh partitions. -- Regards Vamsi Subhash

Re: Max. storage for Kafka and impact

We definitely need a retention policy of a week. Hence. On Fri, Dec 19, 2014 at 7:40 PM, Achanta Vamsi Subhash < achanta.va...@flipkart.com> wrote: > > Hi, > > We are using Kafka for our messaging system and we have an estimate for > 200 TB/week in the coming month

Re: Max. storage for Kafka and impact

ds, > Nitin Kumar Sharma. > > > On Fri, Dec 19, 2014 at 9:12 AM, Achanta Vamsi Subhash < > achanta.va...@flipkart.com > wrote: > > > > We definitely need a retention policy of a week. Hence. > > > > On Fri, Dec 19, 2014 at 7:40 PM, Achanta Vamsi Subhash <

Re: Max. storage for Kafka and impact

nd these many topics will make that book-keeping significant. > As for storage, I don't think it should be an issue with sufficient > spindles, servers and higher than default memory configuration. > Jayesh > From: Achanta Vamsi Subhash > To: "users@kafka.apache.org&

Re: Max. storage for Kafka and impact

gt; > On Fri, Dec 19, 2014 at 12:01 PM, Joe Stein wrote: > > > > see some comments inline > > > > On Fri, Dec 19, 2014 at 11:30 AM, Achanta Vamsi Subhash < > > achanta.va...@flipkart.com> wrote: > >> > >> We require: > >> - many topi

Re: Apache Kafka 0.8.2 Consumer Example

High level consumer of 0.8.1 works fine with 0.8.2. In extra, you can change the config to use kafka for offsets storage instead of zookeeper. There are some extra config parameters added as well as explained in the wiki. http://kafka.apache.org/documentation.html#consumerconfigs For low-level con

New subscriber offset

Hi, We are using HighLevelConsumer and when a new subscription is added to the topic, the HighLevelConsumer for the same group starts from the start of the Kafka topic log. Is there anyway we could set the offset of the HighLevelConsumer to the end of the log instead? We don't want to move to Low

Re: New subscriber offset

Thanks a lot Xiao. Somehow missed reading about the config parameter. On Tue, Mar 3, 2015 at 6:51 PM, tao xiao wrote: > You can set the consumer config auto.offset.reset=largest > Ref: http://kafka.apache.org/documentation.html#consumerconfigs > > On Tue, Mar 3, 2015 at 8:30 PM, A

Producer Timeout

Hi, I am intermittently getting the following exception when producing the messages using 0.8.2 new producer: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 6 ms. Network connectivity is fine and the brokers are all u

Re: Producer Timeout

#x27;t seen any " + " partition leadership changes to proactively discover any new brokers or partitions."; Even though the *max.age.ms <http://max.age.ms>* is set to 30 minutes, it still tries to update almost every 10 seconds when it fails. Am I doing something wrong with produ

Consumer offsets in offsets topic 0.8.2

Hi, We are using 0.8.2.1 currently. - How to get the consumer offsets from the offsets topic? ​- Is there any built-in function which I could use? (like in AdminUtils.scala) - Is it ok to start a simple consumer and read the offsets from the topic? ​We used to read the offsets from zookeeper pre

Kafka Zookeeper queries

Hi, How often does Kafka query zookeeper while producing and consuming? Ex: If there is a single partition to which we produce and a HighLevel consumer running on it, how many read/write queries to zookeeper happen. Extending further, multiple topics with ~100 partitions each, how many zookeeper

Re: Kafka Zookeeper queries

Hi, Could anyone help with this? Thanks. On Sun, Apr 19, 2015 at 12:58 AM, Achanta Vamsi Subhash < achanta.va...@flipkart.com> wrote: > Hi, > > How often does Kafka query zookeeper while producing and consuming? > > Ex: > If there is a single partition to which we

Re: Kafka Zookeeper queries

ar frequency. > > The limitation of Zookeeper usage for Kafka I am aware of is probably the > size of each zNode. As long as you don¹t have so many partitions that > zNode cannot handle, it should be fine. > > Thanks. > > Jiangjie (Becket) Qin > > On 4/20/15, 5:58 AM, &quo

Re: Kafka Zookeeper queries

e you can > sync up your consumer offsets in kafka itself instead of Zk which further > brings down write load on ZKs. > > Regards, > Pushkar > > On Tue, Apr 21, 2015 at 1:13 PM, Jiangjie Qin > wrote: > > > 2 partitions should be OK. > > > > On 4/2

Simple Consumer Exceptions

Hi, We are getting the below exceptions while trying to get the last offset of a topic. The code being used is taken from here: https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example We are finding the lead broker and getting the offsets and noticing the failures due to th

Log end offset

Hi, What is the best way for finding out the log end offset for a topic? Currently I am using the SimpleConsumer getLastOffset logic mentioned in: https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example But we are running into ClosedChannelException for some of the topics.

Re: Log end offset

e end of a partition (consumed > all messages) when using the high-level consumer?" > http://search-hadoop.com/m/uyzND1Eb3e42NMCWl > > -James > > On May 10, 2015, at 11:48 PM, Achanta Vamsi Subhash < > achanta.va...@flipkart.com> wrote: > > > Hi, >

Re: Log end offset

issue request. I have also increased the socket-timeout-ms to a very high number in Simple consumer and the issue still persists. Did anyone face this issue before? Any help in this would be great. Thanks. On Tue, May 12, 2015 at 11:47 AM, Achanta Vamsi Subhash < achanta.va...@flipkart.com>

Kafka broker - Ip-address instead of host naem

Hi, Currently Kakfa brokers register the hostname in zookeeper. [zk: localhost:2181(CONNECTED) 5] get /varadhi/kafka/brokers/ids/0 {"jmx_port":,"timestamp":"1427704934158","host":"currHostName","version":1,"port":9092} ​Is there any config to make it use ip-address instead so that we don't m

Re: How to verify /update offsets in 0.8.2.1 ?

Should you be doing: get /consumers/elastic_search_group/offsets/my_log_topic/0 On Sat, May 23, 2015 at 1:42 AM, Marina wrote: > Hi, > I would like to inspect current offsets for my topic/partitions from a > command line, and update them when needed. > > I can use the kafka.tools.ConsumerOffset

Re: Kafka broker - Ip-address instead of host naem

https://issues.apache.org/jira/browse/KAFKA-1229 On Sun, May 24, 2015 at 1:14 PM, Gwen Shapira wrote: > If you set advertised.hostname in server.properties to the ip address, the > IP will be registered in ZooKeeper. > > > On Fri, May 22, 2015 at 2:20 PM, Achanta Vamsi Subhash < >

Re: Kafka broker - Ip-address instead of host naem

: > You can't dynamically re-load server properties. > > However, the norm in zookeeper is to configure the connection string with > all the nodes in the zk cluster, so there will be no need to modify > properties when you replace zk nodes. > > On Sun, May 24, 2015 at 4:13

Re: Kafka broker - Ip-address instead of host naem

> > wrote: > > > > > You can't dynamically re-load server properties. > > > > > > However, the norm in zookeeper is to configure the connection string > with > > > all the nodes in the zk cluster, so there will be no need to modify > >

Batch producer latencies and flush()

Hi, We are using the batch producer of 0.8.2.1 and we are getting very bad latencies for the topics. We have ~40K partitions now in a 20-node cluster. - We have many topics and each with messages published to them varying. Ex: some topics take 10k/sec and other 2000/minute. - We are seeing latenc

Is trunk safe for production?

I am planning to use for the producer part. How stable is trunk generally? -- Regards Vamsi Subhash -- -- This email and any files transmitted with it are c

Re: Is trunk safe for production?

t of curiosity, why do you want to run trunk? > > General fondness for cutting edge stuff? Or are there specific > > features in trunk that you need? > > > > Gwen > > > > On Tue, Jun 23, 2015 at 2:59 AM, Achanta Vamsi Subhash > > wrote: > > > I am

Re: Batch producer latencies and flush()

*bump* On Tue, Jun 23, 2015 at 1:03 PM, Achanta Vamsi Subhash < achanta.va...@flipkart.com> wrote: > Hi, > > We are using the batch producer of 0.8.2.1 and we are getting very bad > latencies for the topics. We have ~40K partitions now in a 20-node cluster. > > - We h

Questions reg. Kafka apis and proxy

Hi, We are considering hosting Kafka as a service in our company (with multiple clusters and a common interface) . We have the following questions with regards to hosting Kafka as a service: - The recommended Kafka clients and their api have changed since the last 2 releases - producer in 0.8.2 a

Consumer Offsets Topic cleanup.policy

Hi all, We have a __consumer_offsets topic has cleanup.policy=compact and log.cleaner.enable=false. What would happen if we change the cleanup.policy to delete? Will that treat the offsets topic as same as any other topic? We currently have a setup without log.cleaner.enable=false and we have off

Re: Consumer Offsets Topic cleanup.policy

Hi, We tested this on our stage environment and works fine if we change the policy to delete from compact. Will there be any side effects if we change it to delete for the __consumer_offsets topic? On Wed, Mar 2, 2016 at 4:43 PM, Achanta Vamsi Subhash < achanta.va...@flipkart.com> wrote:

Re: Consumer Offsets Topic cleanup.policy

the end of the log. There may be other consequences as > well that I haven't thought of... > > Can you describe in a little more detail the problem that you found > enabling the cleaner? > > -Jason > > On Sun, Mar 6, 2016 at 3:09 AM, Achanta Vamsi Subhash < > achanta

Re: Consumer Offsets Topic cleanup.policy

the log. There may be other consequences as >> well that I haven't thought of... >> >> Can you describe in a little more detail the problem that you found >> enabling the cleaner? >> >> -Jason >> >> On Sun, Mar 6, 2016 at 3:09 AM, Achanta V

Kafka LTS release

Hi all, We find that there are many releases of Kafka and not all the bugs are back ported to the older releases. Can we have a LTS (Long Term Support) release which can be supported for 2 years with all the bugs back-ported? This will be very helpful as during the last 2-3 releases, we often hav

Re: Need a help in understanding __consumer_offsets topic creation in Kafka Cluster

We changed the policy to "delete" dynamically for the __consumer_offsets topic and it was a better option than doing a cluster restart after enabling log compaction. Also, we found problems when you are replicating to a log compacted topic from a non-compacted topic (which is leader). On Mon, Mar

Re: Kafka LTS release

*bump* Any opinions on this? On Mon, Mar 14, 2016 at 4:37 PM, Achanta Vamsi Subhash < achanta.va...@flipkart.com> wrote: > Hi all, > > We find that there are many releases of Kafka and not all the bugs are > back ported to the older releases. Can we have a LTS (Long Term

Re: Kafka LTS release

.0, you need > to make sure all the 0.9.0.1 clients still work with it, and you don't > introduce new bugs by the partial merge. > I do think once there will be a 1.0.0.0 release it would be great to have a > lts release. > > On Mon, Mar 21, 2016 at 11:54 AM Achanta Vamsi

Re: Kafka LTS release

. > If you > are not the intended recipient, please notify the sender immediately, and > do not > disclose the contents to another person, use it for any purpose, or store, > or copy > the information in any medium. Please also destroy and delete the message > from > your compute

Re: Supervisord for Kafka 0.8.1

We use daemontools and this is our run file: #!/bin/bash PAC=kafka-0.8.2.x APP_HOME=/usr/share/$PAC # app options APP_CONFIG_HOME=${APP_HOME}/config APP_OPTS="${APP_CONFIG_HOME}/server.properties" JVM_OPTS="" # jvm user options if [ "abc$KAFKA_HEAP_OPTS" == "abc" ]; then export KAFKA_HEAP_OP

Re: Queue implementation

If you want 1:1, start only one subscriber for a topic from the app. Then it will act as a queue. On Mon, Mar 28, 2016 at 6:48 PM, Sharninder wrote: > What kind of queue are you looking for? Kafka works as a nice FIFO queue by > default anyway. > > > > On Mon, Mar 28, 2016 at 5:19 PM, Vinod Kaka

Re: dumping JMX data

Why not use tools like jmxtrans and send your metrics to Graphite/OpenTsdb.etc? Why do serialization/de-serialization twice? On Fri, Apr 1, 2016 at 9:51 AM, Gerard Klijs wrote: > Don't know if adding it to Kafka is a good thing. I assume you need some

Re: [ANNOUNCE] New committer: Ismael Juma

Congrats Ismael On Tue, Apr 26, 2016 at 12:58 PM, Ismael Juma wrote: > Thank you Neha and the PMC for the opportunity. I am both grateful and > excited. :) > > And thanks to everyone else in the Kafka community too. > > Ismael > > On Mon, Apr 25, 2016 at 10:52 PM, Neha Narkhede wrote: > > > The

Re: [DISCUSS] Java 8 as a minimum requirement

+1 to Java8 - we have already moved to Java8 last year. On Fri, Jun 17, 2016 at 12:48 PM, Ismael Juma wrote: > Hi Harsha, > > Are you saying that you are aware of many Kafka users still using Java 7 > who would be ready to upgrade to the next Kafka feature release (whatever > that version number

Re: handle the data loss in page cache?

By default the flush to disk in Kafka is turned off. The reason for this is to rely on the replicas instead for the data. But the OS still flushes the data to disk periodically when its buffer is full. The problem with this is that - if you force kill your Kafka process or the VM restarts - Kafka w

Message details

Hi, We are experimenting Kafka for a MQ use-case. We found it very useful but couldn't find the following info from the documentation: I have a consumer logic which can say that a message consumption failed. Is there any way I can remove the message from that partition and put it in other topic?

Sync Producer

Hi, How to use a sync producer with a KeyedMessage. The example in the documentation points to the Async Producer. What exceptions will be thrown if the producer.send() fails? Could any one point to an example of sync producer? -- Regards Vamsi Subhash

Re: Sync Producer

- Is setting type in config of the producer to sync the way? - Is the exception thrown a Runtime Exception? My IDE doesn't show that an exception is being thrown? On Sun, Jun 8, 2014 at 12:24 PM, Achanta Vamsi Subhash < achanta.va...@flipkart.com> wrote: > Hi, > > How to

Offset of last un-consumed message

Hi, I have a consumer group with multiple threads (high-level consumers) which read from a topic. I am also using a SimpleConsumer to read messages given a start offset. I am getting the offset as the last produced message using the below code. How to get the last un-consumed message? public

Re: Offset of last un-consumed message

Sorry. I want the first un-consumed message offset. On Tue, Jun 17, 2014 at 2:53 PM, Achanta Vamsi Subhash < achanta.va...@flipkart.com> wrote: > Hi, > > I have a consumer group with multiple threads (high-level consumers) which > read from a topic. > > I am also using

Re: Reliable Message Commits

What about ths pattern: message = iter.peek(); //gets the message process(message); iter.next(); //consumes the message now consumer.commit(); On Fri, Jun 20, 2014 at 11:53 AM, Kyle Banker wrote: > Thanks for the advice, Guozhang. > > Jagbir: I'll report back on my progress. I intend to have