Re: Shutting down a Streams job

teful- > set/#ordered-pod-termination> > . > > On Wed, Feb 8, 2017 at 2:15 PM, Elias Levy > wrote: > > > What are folks doing to cleanly shutdown a Streams job comprised of > > multiple workers? > > > > Right now I am doing sys.addShutdownHook(streams.close())

Shutting down a Streams job

What are folks doing to cleanly shutdown a Streams job comprised of multiple workers? Right now I am doing sys.addShutdownHook(streams.close()) but that is not working well to shutdown a fleet of workers. When I signal the fleet to shutdown by sending them all a SIGTERM, some of them will shutdow

Re: Fetch offset out of range errors while testing Streams application

Guozhang, Thanks for the reply. I figured it out after a while. Indeed, the global default time based retention was tripping me. I was using older data for testing and publishing messages with explicit timestamps. It took me a while to figure out what was happening because kafka-topics.sh does

Re: Fetch offset out of range errors while testing Streams application

Suggestions? On Thu, Jan 19, 2017 at 6:23 PM, Elias Levy wrote: > In the process of testing a Kafka Streams application I've come across a > few issues that are baffling me. > > For testing I am executing a job on 20 nodes with four cores per node, > each instance configu

Fetch offset out of range errors while testing Streams application

In the process of testing a Kafka Streams application I've come across a few issues that are baffling me. For testing I am executing a job on 20 nodes with four cores per node, each instance configured to use 4 threads, against a 5 node broker cluster running 0.10.1.1. Before execution kafka-stre

Convert a KStream to KTable

I am correct in assuming there is no way to convert a KStream into a KTable, similar to KTable.toStream() but in the reverse direction, other than using KSteam.reduceByKey and a Reducer or looping back through Kafka and using KStreamBuilder.table?

Re: Time of derived records in Kafka Streams

On Sat, Sep 10, 2016 at 9:17 AM, Eno Thereska wrote: > > For aggregations, the timestamp will be that of the latest record being > aggregated. > How does that account for out of order records? What about kstream-kstream joins? The output from the join could be triggered by a record received fr

Re: Unexpected KStream-KStream join behavior with asymmetric time window

https://issues.apache.org/jira/browse/KAFKA-4153 https://github.com/apache/kafka/pull/1846 On Mon, Sep 12, 2016 at 7:00 AM, Elias Levy wrote: > Any ideas? > > > On Sunday, September 11, 2016, Elias Levy > wrote: > >> Using Kafka 0.10.0.1, I am joining records in

Re: Unexpected KStream-KStream join behavior with asymmetric time window

Any ideas? On Sunday, September 11, 2016, Elias Levy wrote: > Using Kafka 0.10.0.1, I am joining records in two streams separated by > some time, but only when records from one stream are newer than records > from the other. > > I.e. I am doing: > > stream1.join

Unexpected KStream-KStream join behavior with asymmetric time window

Using Kafka 0.10.0.1, I am joining records in two streams separated by some time, but only when records from one stream are newer than records from the other. I.e. I am doing: stream1.join(stream2, valueJoiner, JoinWindows.of("X").after(1)) I would expect that the following would be equiva

Time of derived records in Kafka Streams

The Kafka Streams documentation discussed how to assign timestamps to records received from source topic via TimestampExtractor. But neither the Kafka nor the Confluent documentation on Kafka Streams explain what timestamp is associated with a record that has been transformed. What timestamp is a

Re: Heartbeating during long processing times

har > > On Thu, Jun 30, 2016 at 6:02 PM Elias Levy > wrote: > > > What is the officially recommended method to heartbeat using the new Java > > consumer during long message processing times? > > > > I thought I could accomplish this by setting max.poll.records

Heartbeating during long processing times

What is the officially recommended method to heartbeat using the new Java consumer during long message processing times? I thought I could accomplish this by setting max.poll.records to 1 in the client, calling consumer.pause(consumer.assignment()) when starting to process a record, calling consum

Consumer pause/resume & partition assignment race condition

While performing some prototyping on 0.10.0.0 using the new client API I noticed that some some clients fail to drain their topic partitions. The Kafka cluster is comprised of 3 nodes. The topic in question has been preloaded with messages. The topic has 50 partitions. The messages were loaded

End to end compression?

I have across this blog post by Neha Narkhede from a three years ago discussing changes made in Kafka 0.8 as a result of the introduction of logical message offsets where the broker has to decompress and recompress compressed

Re: KAFKA-1499 compression.type

Anyone? On Thu, Jan 14, 2016 at 8:42 PM, Elias Levy wrote: > The description of the compression.type config property in the > documentation is somewhat confusing. It begins with "Specify the final > compression type for a given topic.", yet it is defined as a broker >

KAFKA-1499 compression.type

The description of the compression.type config property in the documentation is somewhat confusing. It begins with "Specify the final compression type for a given topic.", yet it is defined as a broker configuration property and it is not listed under topic-level configuration properties. Reading

Kafka and Btrfs

Anyone using Kafka with Brtfs successfully? Any recommendations against taking that path? Elias

Kafka on EC2 with ephemeral storage mirrored to EBS

I am curious if anyone has attempted to run Kafka on EC2 using ephemeral storage for the logs (I am looking to use the I2 or D2 instance types), but actively copying the logs into an EBS volume to aid in bringing a dead broker back to life faster, so that it doesn't have to replicate all messages f

Re: Kafka availability guarantee

On Sun, Oct 11, 2015 at 2:34 PM, Todd Palino wrote: > To answer the question, yes, it is incorrect. There are a few things you > can do to minimize problems. One is to disable unclean leader election, use > acks=-1 on the producers, have an RF of 3 or greater, and set the min ISR > to 2. This mea

Kafka availability guarantee

Reading through the Kafka documentation for statements regarding Kafka's availability guarantees one comes across this statement: *With this ISR model and f+1 replicas, a Kafka topic can tolerate f failures without losing committed messages.* In my opinion, this appears incorrect or at best misle