Re: Kafka Streams: how can I get the name of the Processor when calling `KStream.process`

No problem with that. It's perfectly explained in https://github.com/confluentinc/examples/blob/3.1.x/kafka-streams/src/test/java/io/confluent/examples/streams/StateStoresInTheDSLIntegrationTest.java . 2017-01-18 19:41 GMT+01:00 Michael Noll : > Nicolas, > > here's some information I shared on St

Re: FW: Need Help on issue with Kafka

Hi John, My name is Justin and I work for Confluent, confluent.io. We work with various teams within Cisco to support your organization on Kafka. At this time, I would recommend you contact me directly, and remove the users@kafka.apache.org from all further communication, as this is a public, wo

Streams: Global state & topic multiplication questions

Greetings Everyone, I'm just getting into the kafka world with a sample project, and I've got two conceptional issues, you might have a trivial answer already at hand to. Scenario: multiuser painting webapp, with N user working on M images simultaneously. The "brush" events go to one single k

Re: Streams: Global state & topic multiplication questions

Hi Peter, About Q1: The DSL has the "branch" API, where one stream is branched to several streams, based on a predicate. I think that could help. About Q2: I'm not entirely sure I understand the problem space. What is the definition of a "full image"? Thanks Eno > On 19 Jan 2017, at 12:07, Pet

Re: Streams: Global state & topic multiplication questions

Q1: Thank you, the branch() is what I'm looking for, I just missed it somehow. Q2: I receive something like "imageid,x,y" as key, and a color as value. I aggregate this to something like average color for example. So technically I do not have images, I have colored pixels with 3 dimensions on

General question about kafka to use in a correct way

I read about kafka, and still i am thinking about some scenarios, what is the possibilities to use kafka in a cloud POS application? is there any sense to do a live dashboard? Thanks Amtest

Re: Streams: Global state & topic multiplication questions

For Q2: one way to export the state on demand would be to use the Interactive Queries API (https://www.confluent.io/blog/unifying-stream-processing-and-interactive-queries-in-apache-kafka/ ). Tha

Re: Errors producing / consuming with kafka 0.10.0.0

I was able to make this error disappear by upgrading my client library from 0.10.0.0 to 0.10.0.1 On Wed, Jan 18, 2017 at 10:40 AM, Ryan Thompson wrote: > Hello, > > I'm attempting to upgrade an application from 0.8 to 0.10 broker / client > libs, and integrate kafka streams. I am currently usin

Re: Kafka Protocol : about "Offset" in the MessageSet

It's either meaningless and ignored (for uncompressed messages) or a relative offset for the messages in the set (for compressed messages). Field Description Offset This is the offset used in kafka as the log sequence number. When the producer is sending non compressed messages, it ca

Re: Kafka Streams: got bit by WindowedSerializer (only window.start is serialized)

Regarding KAFKA-4468, as discussed on the JIRA we intentionally did not write the end-timestamp to RocksDB for storage optimization, i.e. we will still write the combo of window-start-time and key, that is because for TimeWindow the window length is fixed and accessible in the Windows object, so we

Re: Does offsetsForTimes use createtime of logsegment file?

Another question, with getOffsetsBefore, we used to be able to get offsets for time in older versions. .10 doesn't have an equivalent method. Is there any other way to achieve the same functionality as getOffsetsBefore in .10 ? Does a .10 server respond to ListOffsetRequestV0 request? On Fri, Ja

Kafka SASL_PLAINTEXT and authentication/authorization backend failure

I have successfully gotten SASL_PLAINTEXT configured on Kafka cluster. We implemented our own LoginModule and Server with the following caveat that I am guessing I am doing something wrong. The LoginModule's login method acquires a session id from an internal security system and populates the subj

Re: Kafka SASL_PLAINTEXT and authentication/authorization backend failure

Hi, I've added kerberos support for https://github.com/gerritjvv/kafka-fast and have seen that the kafka brokers do not send any response if the SASL authentication is not correct or accepted, thus causing the client to hang while waiting for a response from kafka. Some things that might help to

Re: Kafka SASL_PLAINTEXT and authentication/authorization backend failure

Thanks for the response Gerrit! It seems like authorization has the same behavior. Have you experienced that as well? On Thu, Jan 19, 2017 at 11:48 AM, Gerrit Jansen van Vuuren < gerrit...@gmail.com> wrote: > Hi, > > I've added kerberos support for https://github.com/gerritjvv/kafka-fast > and >

Re: Kafka SASL_PLAINTEXT and authentication/authorization backend failure

If you mean authorization with kafka (not with kerberos) then yes, seems that kafka stops responding when it doesn't get exactly what it expects :/ no errors just timeouts. On Thu, Jan 19, 2017 at 8:59 PM, Christian wrote: > Thanks for the response Gerrit! It seems like authorization has the sa

Re: Kafka SASL_PLAINTEXT and authentication/authorization backend failure

Hi Gerrit, I think it's important to distinguish broker and client behaviour. The clients can hang because they keep retrying when they get certain errors. When it comes to the broker, it should give you errors as a general rule. If you are aware of certain scenarios where it should give an error

Reassigning partitions to a non-existent broker

We recently tried to rebalance partitions for a topic (via kafka.admin.ReassignPartitionsCommand). In the .json file with the desired end-state, an id for a non-existent broker was entered for a partition. Upon --execute, all partitions were moved without issue, but the partition with the bogus br

Kerberos/SASL Enabled Kafka - broker fails NoAuth due ACL

Hi I am planning to setup a Kerberos/SASL enabled kafka cluster with three brokers. Since “zookeeper.set.acl=true” is set, when running the first broker, it creates the znodes required. It also sets the ACL of nodes which locks down the nodes to the first broker. Here is the output of the ACL on

Re: Kafka SASL_PLAINTEXT and authentication/authorization backend failure

Hi, I refer to the broker behaviour, for most part without SASL the brokers do respond but as soon as you put SASL into the mix it does hang whenever there is something that goes wrong, i.e the broker doesn't provide a response on the socket and the client waits reading a response that it never ge

Fetch offset out of range errors while testing Streams application

In the process of testing a Kafka Streams application I've come across a few issues that are baffling me. For testing I am executing a job on 20 nodes with four cores per node, each instance configured to use 4 threads, against a 5 node broker cluster running 0.10.1.1. Before execution kafka-stre

Automatic Offset Committing

Hi, As per Kafka documentation, *Automatic Offset Committing,* "Setting enable.auto.commit means that offsets are committed automatically." Can you help me understand where the offsets are stored in case of automatic offset committing.. I believe it used to be on Zookeeper but not anymore as per

Kafka 0.10 quickstart: consumer fails when “primary” broker is brought down

I’ve posted this on Stackoverflow as well: http://stackoverflow.com/questions/41756435/kafka-0-10-quickstart-consumer-fails-when-primary-broker-is-brought-down So I'm trying the kafka quickstart as per the main documentation. Got the multi-cluster example a

Re: Kafka 0.10 quickstart: consumer fails when “primary” broker is brought down

The topic created in the quickstart only has a replication factor of 1 and a single partition. bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test So the one topic partition is on broker 1. When broker 1 is up, you can consume the data but

Re: General question about kafka to use in a correct way

Hi Vishnu, Assuming that the POS application will generate POS events, we could accumulate these events in Kafka and use it as a data feed for live dashboards. We could do some common events stream processing within Kafka itself using the streams API. Alternatively the POS events stream could