Re: [VOTE] 3.0.0 RC1

Hi Konstantine, Some findings from running 3.0.0-RC1 with the librdkafka test suite: * Compaction seems to take slightly longer to kick in when segment sizes exceed their threshold. (Used to take less than 20 seconds, now takes 20..30 seconds.) * CreateTopic seems to take slightly longer to pr

Re: group id issue on kafka using C library

You need to explicitly set the "group.id" configuration property (with rd_kafka_conf_set()) when using the high-level consumer. Den sön 10 mars 2019 kl 18:57 skrev K R, Rakesh : > Then any idea on why I am seeing this error > RD_KAFKA_RESP_ERR__UNKNOWN_GROUP without group id? > > -Original Me

Re: [VOTE] 2.1.1 RC2

+1 Passes librdkafka test suite. Den fre 8 feb. 2019 kl 21:02 skrev Colin McCabe : > Hi all, > > This is the third candidate for release of Apache Kafka 2.1.1. This > release includes many bug fixes for Apache Kafka 2.1. > > Compared to rc1, this release includes the following changes: > * MINO

Re: Python Kafka client that has support for SASL_SSL with SCRAM-SHA-256/512

Hi Alexei, I'm a bit late to the party, but: confluent-kafka-python is a Python client based on librdkafka, it supports all SASL mechanisms, including SCRAM. https://github.com/confluentinc/confluent-kafka-python Regards, Magnus 2017-08-23 10:09 GMT+02:00 Alexei Levashov : > Hello, > > could

Re: [VOTE] 0.11.0.1 RC0

+1 (non-binding) Verified with librdkafka regression test suite 2017-09-06 11:52 GMT+02:00 Damian Guy : > Resending as i wasn't part of the kafka-clients mailing list > > On Tue, 5 Sep 2017 at 21:34 Damian Guy wrote: > > > Hello Kafka users, developers and client-developers, > > > > This is the

Re: [VOTE] 0.11.0.0 RC1

+1 (non-binding) Passes librdkafka integration tests (v0.9.5 and master) 2017-06-19 0:32 GMT+02:00 Ismael Juma : > Hello Kafka users, developers and client-developers, > > This is the second candidate for release of Apache Kafka 0.11.0.0. > > This is a major version release of Apache Kafka. It

Re: [VOTE] 0.10.2.0 RC2

Verified with librdkafka v0.9.4-RC1. 2017-02-15 9:18 GMT-08:00 Tom Crayford : > Heroku tested this with our usual round of performance benchmarks, and > there seem to be no notable regressions in this RC that we can see (for a > sample on earlier regressions we found using these benchmarks during

Re: Kafka Producer Protocol

Hi Steve, this is probably what you are looking for: http://kafka.apache.org/protocol.html /Magnus 2017-02-07 15:57 GMT+01:00 Hopson, Stephen : > Can anyone point me to documentation for the protocol exchanges between a > producer client and Kafka? I do not mean the message formats, I have thos

Re: [VOTE] 0.10.1.0 RC3

+1 (non-binding) passes librdkafka test suites 2016-10-19 15:55 GMT+02:00 Ismael Juma : > +1 (non-binding). > > Verified source and Scala 2.11 binary artifacts, ran ./gradlew test with > JDK 7u80, quick start on source artifact and Scala 2.11 binary artifacts. > > Thanks for managing the release!

Re: librdkafka

Hey Dave, yes that's the general plan. Regards, Magnus 2016-09-29 19:33 GMT+02:00 Tauzell, Dave : > Does anybody know if the librdkafka releases are kept in step with kafka > releases? > > -Dave > This e-mail and any files transmitted with it are confidential, may > contain sensitive informatio

Re: Kafka C/C++ client with consumer group support?

Hi Vince, librdkafka (master branch) supports >=0.9 balanced consumers groups: https://github.com/edenhill/librdkafka As long as you are using different consumer groups each group will receive the same set of messages. Example C++ consumer: https://github.com/edenhill/librdkafka/blob/master/exa

Re: Trying to understand the format of the LogSegment file.

Hi, messages are stored on disk in the Kafka (network) protocol format, so if you have a look at the protocol guide you'll see the pieces start coming together: https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-Messagesets Regards, Magnus

Re: Queue Full

That's librdkafka's producer queue that has run full because: a) client is not connected to broker b) no leader broker for relevant partitions c) producing messages at a higher rate than the network or broker can handle /Magnus 2015-10-26 16:49 GMT+01:00 Eduardo Costa Alfaia : > Hi Guys,

Re: Any tool to easily fetch a single message or a few messages starting from a given offset then exit after fetching said count of messages?

Hi David, kafkacat does this with ease: kafkacat -b mybroker -t mytopic -p mypartition -o myoffset -c msgcount If you are to pass the output to another program/script you may want to look in to the -f formatting option or -J for JSON. See here: https://github.com/edenhill/kafkacat Regards, M

Re: Anouncing PyKafka!

That's exciting news, looks very promising! I've added PyKafka to the Clients wiki page. Good work! 2015-06-08 21:24 GMT+02:00 Keith Bourgoin : > Hi Kafka folks, > > I'm happy to announce the 1.0 release of PyKafka > , a new Python driver for Kafka! > > This

Re: Producer waiting ~15 mins before disconnecting.

Hi Mayank, The client should expose a configuration property to enable TCP keepalives (SO_KEEPALIVE) on its broker sockets, SO_KEEPALIVE provides speedier detection of connection loss on idle connections. (as a positive side effect it also helps keeping connections alive through NAT/firewalls/LBs)

Re: kafkacat

Hi Clay, not really sure what you mean by socket, but if you want something listening on a network port and forwards/produces all data to Kafka then you might want to look at n2kafka: https://github.com/redBorder/n2kafka Another alternative would be to use inetd, socat, or similar to pipe a netwo

Re: New Producer Async - Metadata Fetch Timeout

I completely agree with Mohit, an application should not have to know or care about producer implementation internals. Given a message and its delivery constraints (produce retry count and timeout) the producer should hide any temporal failures until the message is succesfully delivered, a permanen

Re: Could this be happening?

Hi Scott, what producer client are you using? Reordering is possible in async producers in the case of temporary broker failures and the combination of request.required.acks != 0 and retries > 0. Consider the case where a producer has 20 messages in-flight to the broker, out of those messages #

Re: New Producer API - batched sync mode support

Hi Gwen, can you clarify: by batch do you mean the protocol MessageSet, or some java client internal construct? If the former I was under the impression that a produced MessageSet either succeeds delivery or errors in its entirety on the broker. Thanks, Magnus 2015-04-27 23:05 GMT+02:00 Gwen Sh

Re: Problems decoding fetch responses, getting messages that are bigger than their message sets

Hi Theo, seems like you found the answer yourself, the server may return partial messages. While parsing a MessageSet you simply ignore any message whose length > remainingBufferLength. Regards, Magnus 2015-03-01 8:55 GMT+01:00 Theo Hultberg : > That the message set size was always the same wh

Re: generate specific throughput load

Hi, the rdkafka_performance tool from librdkafka's examples [1] lets you do this with something like: rdkafka_performance -P -b -t [-p ] -r -s Thats the producer side, if you want performance measurements on the consumer side as well you do: rdkafka_performance -C -b -t -p -o There is an

Re: ping kafka server

Relying on just the TCP connection getting established seems a bit poor, the easiest non-intrusive approach is probably to query the broker for metadata, e.g.: kafkacat -b mybroker -L 2015-02-10 1:47 GMT+01:00 Koert Kuipers : > a simple nagios check_tcp works fine. as gwen indicated kafka closes

Re: [VOTE] 0.8.2.0 Candidate 3

+1 on librdkafka interop Minor nitpick: KAFKA-1781 (state required gradle version in README) is included in the Release notes but is not actually fixed 2015-01-29 6:22 GMT+01:00 Jun Rao : > This is the third candidate for release of Apache Kafka 0.8.2.0. > > Release Notes for the 0.8.2.0 rele

Re: Resilient Producer

The big syslog daemons support Kafka since a while back. rsyslog: http://www.rsyslog.com/doc/master/configuration/modules/omkafka.html syslog-ng: https://czanik.blogs.balabit.com/2015/01/syslog-ng-kafka-destination-support/#more-1013 And Bruce might be of interest aswell: https://github.com/tagg

Re: Can I run Kafka Producer in BG and then run a program that outputs to the console?

s, that is > stdout, can the producer use stdout or only stdin? > > Thanks, > > Su > > On Tue, Jan 20, 2015 at 1:10 PM, Magnus Edenhill > wrote: > > > Hi Su, > > > > It is as simple as piping to it: > > some_program_outputting_the_stuff | kafkacat

Re: Can I run Kafka Producer in BG and then run a program that outputs to the console?

Hi Su, It is as simple as piping to it: some_program_outputting_the_stuff | kafkacat -b mybroker -t mytopic That will produce one kafka message to topic 'mytopic' for each line of input. See kafkacat -h for the full range of options (e.g., delimiter, specific partition, etc). Hope that helps, M

Re: Issue size message

(duplicating the github answer for reference) Hi Eduardo, the default maximum fetch size is 1 Meg which means your 2 Meg messages will not fit the fetch request. Try increasing it by appending -X fetch.message.max.bytes=400 to your command line. Regards, Magnus 2015-01-19 17:52 GMT+01:00 E

Re: Treating a specific topic as in-memory only

Hi, you could configure the topic to reside on a tmp-fs mount (i.e., in memory filesystem with no disk backing). /Magnus 2014-12-23 15:38 GMT+01:00 Olof Johansson : > Hello, > > I have a specific use-case where I am not allowed to persist a certain kind > of events. The events can be kept in m

Re: Programmatic Kafka version detection/extraction?

onse has its own version ID embedded in the request > itself. > > Broker version, broker ID and (as Joel suggested) git hash are all > reasonable. And +1 for adding this to API to support non-JVM clients. > > Gwen > > On Fri, Nov 14, 2014 at 8:46 AM, Magnus Edenhill > wrote

Re: Programmatic Kafka version detection/extraction?

Please let's not add dependencies on more third party protocols/software, the move away from Zookeeper dependence on the clients is very much welcomed and it'd be a pity to see a new foreign dependency added for such a trivial thing as propagating the version of a broker to its client. My suggesti

Re: Kafka to transport binary files

Hi Rohit, librdkafka will produce messages with zero copy (unless compression is enabled). And kafkacat will do this if you provide it one or more files as arguments in producer mode: kafkacat -b -t -P (each file is sent as one message) Just make sure to keep message.max.bytes synchronize

Re: spikes in producer requests/sec

Hi Wes, are you monitoring librdkafka statistics as well? If so, are there any correlating spikes in the per-broker and per-partition statistics? Such as: - brokers..rtt.avg <--- broker round-trip-time (latency) - brokers..waitresp_cnt <-- requests in flight - topics..partitions..msgq_cnt <

Re: Can both uncompressed data and compressed data reside in the same topic (partition)

Hi Zuoning, since compression is a per-message(set) attribute a topic can have both compressed and uncompressed messages, as Guozhang says, and yes, this is supported by both the broker and client (librdkafka in this case). Regards, Magnus 2014-10-31 17:14 GMT+01:00 Zuoning Yin : > Hi Guozhang,

Re: C/C++ kafka client API's

Hi Svante, yes, I will keep librdkafka in pace with the official Kafka functionality/features/protocol changes as they are released upstream. Next in the pipeline is broker based consumer groups to provide a proper high levelish C/C++ consumer (and remedy the need for ZooKeeper). Regards, Magnus

Re: C/C++ kafka client API's

Hi, I might be a bit biased, but librdkafka is an established C/C++ client library. https://github.com/edenhill/librdkafka Regards, Magnus 2014-10-14 11:52 GMT+02:00 Virendra Pratap Singh < vpsi...@yahoo-inc.com.invalid>: > Is there a supported C/C++ kafka client API? > > Regards, > Virendra

Re: how is the partitioner working when compression is enabled?

Hi Lex, messages are batched (and compressed) after partitioning. 2014-08-19 15:46 GMT+02:00 Lian, Li : > I am using Kafka 0.8 and message compression can only be enabled on > producer side. > > However one batch of messages (decided by the producer configuration > batch.num.messages) is compr

Re: Performance/Stress tools

Hi Dayo, the rdkafka_performance tool from librdkafka is useful for performance measurements / stress testing. https://github.com/edenhill/librdkafka It resides in the examples/ directory. https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#test-details Regards, Magnus 2014-07

Re: Kafka latency measures

Hi, do these spikes happen to correlate with log.flush.interval.messages or log.flush.interval.ms? If so it's the file system sync blockage you are seeing. /Magnus 2014-06-18 16:31 GMT+02:00 Supun Kamburugamuva : > Hi, > > We are trying to evaluate Kafka for a real time application. We are sen

Re: console producer question

Hey Ian, this is where a tool like kafkacat comes in handy, it will use a random partitioner by default (without the need for defining a key): tail -f /my/log | kafkacat -b mybroker -t mytopic See https://github.com/edenhill/kafkacat 2014-04-10 6:13 GMT+07:00 Ian Friedman : > Hey guys. We

Re: Kafka snappy compression

Some notes on this: - remember that Messages are wrapped in a compressed MessageSet that is wrapped in an outer Message which is wrapped in an outer MessageSet (the mind boggles). - the Kafka snappy compression has a non-standard framing - snappy-java (prefix "\x82SNAPPY\x0") - this framing is sp

Re: Java vs. C++ in Client API

Hi, most third party clients don't feature the zookeeper and consumer group logic of the official scala/java clients due to the complexity of it. There are plans to move much of the consumer group logic from the client side to the broker to alleviate this, making it easier for third party clients

Re: Java vs. C++ in Client API

Hi Churu, there are at least three C++ client implementations: * for 0.7 :the older kafka-cpp for apache kafka v0.7 * for 0.8: libkafka https://github.com/adobe-research/libkafka * for 0.8: librdkafka: https://github.com/edenhill/librdkafka Which one are you referring to? Regards, Magnus 20

Re: Kafka C++ Windows Client for 0.8 Releases

gt; interesting if we could generate the librdkafka libs for the client for the > Windows environment. I am wondering if you have checked building the > package for windows - e.g. with Pthreads-win32? > > KR > > Tianning > > ---- >

Re: Kafka C++ Windows Client for 0.8 Releases

If you are okay to use Cygwin then librdkafka might be of interest: https://github.com/edenhill/librdkafka/tree/c++_configure 2014-03-19 19:10 GMT+07:00 Tianning Zhang : > Dear all, > > I am writing to this list to check if there is any new progress concerning > 0.8 Windows C++ clients and if t

Re: node.js native bindings

Producer side compression is available in librdkafka 0.8.0. The consumer side compression is available in librdkafka 0.8.2 and later. All librdkafka 0.8 releases are API and ABI safe and backwards compatible, so plugging in a new librdkafka should be fairly straight forward. 2014-02-25 4:32 GMT+

Re: How to make an async producer with c++ libkafka?

librdkafka provides an async interface: https://github.com/edenhill/librdkafka/blob/master/rdkafka.h#L709 It is written in C but is used by C++ applications, a proper C++ interface will be available shortly. Regards, Magnus 2014/1/26 Zicfy Hsu > It does not, please see Connection.cc. > > > On

Re: Kafka server occure java.nio.BufferUnderflowException

Hey Jerry, this issue was fixed on dec 14 in commit 16915a2. If the problem persists in the latest version of librdkafka (0.8.2), please file a github issue here: https://github.com/edenhill/librdkafka/issues/ Thanks, Magnus 2014/1/22 陈小军 > I use c library (librdkafka) > > Best Regards > J

Re: Velocity on local machine

Klaus Schaefers > Hi, > > I have close to 2k messages per second. My machine is just a (BG 4-core i5 > but I would expect more messages. I ran Kafka in the default settings. > > > On Fri, Jan 10, 2014 at 12:31 PM, Magnus Edenhill >wrote: > > > What performance

Re: Velocity on local machine

What performance numbers did you see? For reference you can check the following tests that were also run on the same machine as the broker: https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#performance-numbers Do they correspond to your numbers? Consumer thruput is not included

Re: About speed contorl in Kafka

Sounds to me your consumer performance is the problem, not the producer. So either make your consumers faster or make them consume but drop messages to keep up with the producer speed. This also gives you some means of keeping track of how many messages are lost, why, and when. And just for refere

Re: Anyone working on a Kafka book?

There is this one: http://www.packtpub.com/develop-custom-message-producers-and-consumers-using-apache-kafka/book 2013/12/10 S Ahmed > Is there a book or this was just an idea? > > > On Mon, Mar 25, 2013 at 12:42 PM, Chris Curtin >wrote: > > > Thanks Jun, > > > > I've updated the example wi

Re: Updated kafka client (producer and consumer)

Produce requests in 0.7 are fire-and-forget since there is no acknowledgements being returned by the broker. 0.8 fixes this by the optional use of required.acks. I dont know if a non-JVM client is relevant for your needs, but librdkafka implements all of your listed requirements for 0.8: 1. Messa

Re: are topics and partitions dynamic?

See the configuration reference here: https://kafka.apache.org/documentation.html#configuration 2013/12/4 Arjun > hi, > > I was just looking at kafka 0.8, I could not find any option in > config/server.properties with key "auto.create.topics.enable" or > "default.replication.factor". > > Can

Re: Anyone running kafka with a single broker in production? what about only 8GB ram?

Make sure the fetch batch size and the local consumer queue sizes are large enough, setting them too low will limit your throughput to the broker<->client latency. This would be controlled using the following properties: - fetch.message.max.bytes - queued.max.message.chunks On the producer side y

Re: Sending huge binary files via Kafka?

It would be possible to modify an existing client implementation to use sendfile(2) to pass message contents from/to the filesystem rather than (pre-)allocating send&receive buffers, Thus providing zero-copy file transfer over Kafka. I believe this is how the broker is implemented. Regards, Magnus

Re: C/C++ Client

Hi Matt, regarding librdkafka: - the AF_NETLINK stuff has now been removed, a helper function had support for it but it had no relevance to Kafka operation. - TLS: it would be possible to remove the need for TLS in rdkafka, but it does feel like a leap back in time; even NetBSD has support for T