Kafka consumer to unzip stream of .gz files and read

HI Team, Greeting! I am facing one issue on kafka consumer using python hope you guys help us to resolve this issue Kafka consumer to unzip stream of .gz files and read Kafka producer is sending

Unclear client-to-broker communication

Hi everybody, the communication between the client and the broker is unclear to me. The documentation states: > The client initiates a socket connection and then writes a sequence of > request messages and reads back the corresponding response message. No > handshake is required on connection or

Re: streams windowing question

Hi Peter, It depends on how you specify the JoinWindow. But using `JoinWindows.of(10 secocds)` would mean that a record will join with any other record with the matching key that arrived between 10 seconds before it arrived and 10 seconds after it arrived. So your example is correct. You would ne

Re: Best practices

If a process failed to write a message into one broker what is the best practice to write to same topic on different broker? Is there one? I should be able to get a list of brokers programmatically from zk path /brokers/ids ? On Sun, May 20, 2018, 3:21 PM Matthias J. Sax wrote: > You can registe

Tools to view logs and/or clean up problems?

I operate in a work environment where all the servers are rebooted once a week whether they need it or not. An internal subsystem which gives me the opportunity to run a command on shutdown, so I wrote a simple shell script which runs kafka-server-stop.sh, waits a bit, then stops zookeeper. Unfort

Re: Removing the Kafka DEBUG logs in catalina.out using log4j.properties

Okay Kaufman, will check on that On Sun, May 20, 2018 at 12:08 AM, Kaufman Ng wrote: > Karthick, > > I am suspecting that it could be the log4j.properties in your webapp is not > picked up by Tomcat, or somehow Tomcat is overriding your webapp's log4j > setup. > > On Tue, May 15, 2018 at 2:19 AM

Re: Best practices

If you specify some bootstrap.servers, after connecting the producer will learn about all available brokers automatically, by fetching cluster metadata from the first broker it connects. Thus, in practice it is usually sufficient to specify 3 to 5 brokers (in case one is down, the producer can "boo

Re: Tools to view logs and/or clean up problems?

> Unfortunately, Kafka's careful approach to shutdown means my script is > probably not given enough time for a proper shutdown, and upon reboot, I > often find my brokers won't accept connections. Never mind. I believe I shot myself in the foot elsewhere. Not directly a Kafka problem. Skip

RE: Kafka consumer to unzip stream of .gz files and read

You should read the message value as byte array rather than string . Other Approach is , while producing you can use the kafka compression = GZIP to have similar results. -Original Message- From: mayur shah Sent: Monday, May 21, 2018 1:50 AM To: users@kafka.apache.org; d...@kafka.apach

Kafka behind NAT

We have kafka behind NAT with *only one broker*. Let say we have internal (A) and external (B) network. When we try to reach the broker from external network (we use bootstrap.servers parameter set to B address) then what is obvious the broker responds with internal network's address (A) which is

Re: Kafka consumer to unzip stream of .gz files and read

Also, if I recall correctly - the console producer uses a BufferedReader to read from the console and assumes that a newline terminates a message, so any byte of value 0A in your gzipped file will send a message. I suggest using a Python producer to send your gzipped file. Regards, Liam Clarke