Re: [DISCUSS] Streams DSL/StateStore Refactoring

2017-07-24 Thread Jan Filipiak
Hi Damian, thanks for taking the time. I think you read my points individually but you seem to not understand the bigger picture I am trying to paint. From the three problems I mentioned - and that you agreed on to be problems - you are only trying to address the first. What I am trying to

how about kafka test

2017-07-24 Thread fuyou
kafka is a distributed, partitioned, replicated commit log service message queue, i am interesting how kafka test. transaction message ,exactly once message ,replication ,any document about kafka test? how to ensure kafka work as except? -- = fu

Re: Logs truncated at o'clock

2017-07-24 Thread mosto...@gmail.com
anyone? On 17/07/17 15:24, mosto...@gmail.com wrote: ping? On 13/07/17 17:09, mosto...@gmail.com wrote: Hi With swiss precission, our kafka test environment seems to truncate topics at o'clock hours. This might be confirmed with the following trace, which states "Truncating log ... to

Re: decreasing loglevel for basic creation-roll-delete events

2017-07-24 Thread mosto...@gmail.com
anyone? On 17/07/17 15:21, mosto...@gmail.com wrote: ping? On 13/07/17 16:53, mosto...@gmail.com wrote: Hi While testing Kafka in our environment, we have noticed it creates A LOT of "debug" logs (not to be confused with topic logs!). eg: Jul 13 16:44:26 host kafka[28511]: [2017-07-1

How to tune Kafka High Level Consumer for Production

2017-07-24 Thread Rachana Srivastava
I want to know how to tune/setup high level Kafka Client to a Kafka server in EC2 I set zookeeper.session.timeout.ms=5. I found that after some time I got following error in the logs. I want to know how to tune Kafka parameters to run the consumer for ever. I checked and found ZK is running

Re: Event sourcing with Kafka and Kafka Streams. How to deal with atomicity

2017-07-24 Thread Ben Stopford
No worries Jose ;-) So there are a few ways you could do this, but I think it’s important that you manage a single “stock level” state store, backed by a changelog. Use this for validation, and keep it up to date at the same time. You should also ensure the input topic(s) are partitioned by produc

Re: [DISCUSS] KIP-163: Lower the Minimum Required ACL Permission of OffsetFetch

2017-07-24 Thread Ewen Cheslack-Postava
Vahid, Thanks for the KIP. I think we're mostly in violent agreement that the lack of any Write permissions on consumer groups is confusing. Unfortunately it's a pretty annoying issue to fix since it would require an increase in permissions. More generally, I think it's unfortunate because by sque

Schema Registry on DC/OS

2017-07-24 Thread Debasish Ghosh
Hi - Is it possible to run schema registry service on DC/OS ? I checked the Confluent Kafka package on Mesosphere Universe. It doesn't have any support for running Schema Registry. Also this thread https://stackoverflow.com/questions/37322078/cant-start-confluent-2-0-apache-kafka-schema-registry-i

Kafka error after SSL enabled - Bootstrap broker-name :6667 disconnected (org.apache.kafka.clients.NetworkClient)

2017-07-24 Thread karan alang
Hello - i've enabled SSL for Kafka, and Kafka is starting up fine with SSL enable. However, when i run the Kafka console producer, it is give me error as shown below -> 1. Command : 2. 3. /usr/hdp/2.5.3.0-37/kafka/bin/kafka-console-producer.sh --broker-list nwk2-bdp-kafka-05.gdcs-qa

kafka-consumer-groups tool with SASL_PLAINTEXT

2017-07-24 Thread Meghana Narasimhan
Hi, What is the correct way to use the kafka-consumer-groups tool with SASL_PLAINTEXT security enabled ? The tool seems to work fine with PLAINTEXT port but not with SASL_PLAINTEXT. Can it be configured to work with SASL_PLAINTEXT ? If so what permissions have to enabled for it ? Thanks, Meghana

Re: Kafka error after SSL enabled - Bootstrap broker-name :6667 disconnected (org.apache.kafka.clients.NetworkClient)

2017-07-24 Thread karan alang
Here is what i see the logs .. So, it seems the Kafka Broker is starting up with SSL, however - when the Controller is not able to connect to the Broker server.log > > [2017-07-24 20:57:19,461] INFO [ThrottledRequestReaper-Produce], Starting > (kafka.server.ClientQuotaManager$ThrottledRequestReap

Re: kafka-consumer-groups tool with SASL_PLAINTEXT

2017-07-24 Thread Vahid S Hashemian
Hi Meghana, I did some experiments with SASL_PLAINTEXT and documented the results here: https://developer.ibm.com/opentech/2017/05/31/kafka-acls-in-practice/ I think it covers what you'd like to achieve. If not, please advise. Thanks. --Vahid From: Meghana Narasimhan To: users@kafka.a

Re: Tuning up mirror maker for high thruput

2017-07-24 Thread James Cheng
Todd, I have a question about the OS/broker tuning that you are talking about on the source cluster. Aside from mirrormaker (which you say should be running in the remote destination datacenter), presumably there will be other consumers in the source datacenter as well. How does the OS/broker t

Re: Tuning up mirror maker for high thruput

2017-07-24 Thread Todd Palino
We haven’t had any problem after tuning the default send/receive buffers in the OS up to 10MB. Linux uses a sliding window, so if you have short latencies, you won’t use as much of the buffer and you should see very little, if any, impact. -Todd On Mon, Jul 24, 2017 at 2:20 PM, James Cheng wrot

Re: Kafka error after SSL enabled - Bootstrap broker-name :6667 disconnected (org.apache.kafka.clients.NetworkClient)

2017-07-24 Thread karan alang
further update -> i recreated the certificates & here is the result of the verification (i read in one post that the CN should match the FQDN, else it gives the error, any ideas on how to debug this ? openssl s_client -debug -connect nwk2-bdp-kafka-04.gdcs-qa.apple.com:6667 -tls1 CONNECTED(0

Re: [DISCUSS] KIP-163: Lower the Minimum Required ACL Permission of OffsetFetch

2017-07-24 Thread Vahid S Hashemian
Hi Ewen, Thanks for reviewing the KIP. Your comment about the "food for thought" section makes sense. It seems like a bug to me, not sure how you and others feel about it. I'll remove it for now, and open a separate JIRA for it, so we have a record of it. The read vs. write discussion and fixin

Re: Consumer blocked when auth failed.

2017-07-24 Thread 陈江枫
Hi, I intentionally use the wrong credential, get the following log: *java.net.ConnectException: Connection refused* * at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)* * at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)* * at org.apache.kafka.common.network.Pla

Re: Schema Registry on DC/OS

2017-07-24 Thread Kaufman Ng
Confluent Schema Registry is available in the DC/OS Universe, see here for the package definitions https://github.com/mesosphere/universe/tree/dcd777a7e429678fd74fc7306945cdd27bda3b94/repo/packages/C/confluent-schema-registry/5 The stackoverflow thread is quite out of date, as it mentions Confluen

Kafka Streams internal repartitioning topics, retention time, and reprocessing old data

2017-07-24 Thread Gerd Behrmann
Hi, While adding a new Streams based micro service to an existing Kafka infrastructure, I have run into some issues processing older data in existing topics. I am uncertain of the exact cause of the problems, but am looking for advice to clarify how things are supposed to work to eliminate poss

Re: Schema Registry on DC/OS

2017-07-24 Thread Debasish Ghosh
Thanks a lot .. I found it from the community supported packages on the DC/OS UI. Installed it and it runs ok. One question - is there any CLI for confluent-schema-registry ? dcos package install confluent-schema-registry --cli does not give anything .. regards. On Tue, Jul 25, 2017 at 9:50 AM, K