Re: Producer connect timeouts

Yes, this is something that we could consider fixing in Kafka itself. Pretty much all timeouts can be customized if the defaults for the OS/network are larger than make sense for the system. And given the large default values for some of these timeouts, we probably don't want to rely on the default

Re: TLS

Ruben, There are step-by-step instructions explained here: http://docs.confluent. io/3.1.1/kafka/security.html For the purposes of configuring Kafka, the JAAS details basically boil down to a security configuration in a security configuration file. -Ewen On Mon, Dec 19, 2016 at 8:40 AM, Ruben Po

Re: [VOTE] 0.10.1.1 RC1

Hi Guozhang, I also verified the quickstart on Ubuntu and Mac. +1 on those. On Windows OS there are a couple of issues for which the following PRs exist: - https://github.com/apache/kafka/pull/2146 (already merged to trunk) - https://github.com/apache/kafka/pull/2238 (open) These issues are not

Re: WARN [ReplicaFetcherThread-1-1011], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@5dad549c

After a further debugging, I found the following error on broken node (1011). the error : [2016-12-19 21:07:38,081] ERROR Error while accepting connection (kafka.network.Acceptor) java.io.IOException: Too many open files at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.Se

Zookeeper configuration with ACL using SSL

Hi All, I trying to enable ACL with SSL protocol and It is giving me below exceptions(LEADER_NOT_AVAILABLE) *[2016-12-19 16:16:47,078] WARN Error while fetching metadata with correlation id 16 : {my-ssl-topic4=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)* *[2016-12-19 16:16:47

WARN [ReplicaFetcherThread-1-1011], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@5dad549c

​Hi Experts, is there anyone run into this connection error ? [2016-12-17 20:13:32,728] WARN [ReplicaFetcherThread-1-1011], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@5dad549c (kafka.server.ReplicaFetcherThread)java.io.IOException: Connection to 10.137.126.113:9092 (id: 1011 r

Re: Kafka SSL encryption plus external CA

Thanks Rajini! Also, I currently have each broker advertising as broker1.mydomain.com, broker2.mydomain.com broker6.mydomain.com etc… I have setup CNAME with round robin fashion to group brokers by availability zone i.e. broker-a.mydomain.com broker-b.mydomain.com broker-c.mydomain.com. I use them

Re: Website Update, Part 2

Hi Gwen, I am about to have a separate side bar for the subsections, but a site-level search bar is also a good idea. Just filed a JIRA to keep track of this. Guozhang On Tue, Dec 13, 2016 at 6:16 PM, Gwen Shapira wrote: > Hi, > > Since we are breaking down the docs, we can no longer use ct

Re: [VOTE] 0.10.1.1 RC1

Hi, Guozhang, Thanks for preparing the release. Verified quickstart. +1 Jun On Thu, Dec 15, 2016 at 1:29 PM, Guozhang Wang wrote: > Hello Kafka users, developers and client-developers, > > This is the second, and hopefully the last candidate for the release of > Apache Kafka 0.10.1.1 before th

Does kafka support reloading TLS certificates without downtime?

I want to use Vault[1] to manage my TLS certificates. The certificates would be issued by Vault and have a short lifetime of around 72 hours. There would be a co-process to request a new certificate before expiry. Does kafka provide any method to reload the TLS certificate without any down tim

Re: KafkaStreams StateStore as EventStore (Event Sourcing)

Hello Anatoly, Jay's understanding about the time-space tradeoff in state store HA is correct. As for your second question on "best practices": Streams high-level DSL has a notion of KTable in addition to its first-class KStream objects, which can be viewed as a materialized view of the stream, wh

Re: [VOTE] 0.10.1.1 RC1

+1 (binding) Validated signatures Ran tests Built from source distro Tested binaries using the quickstart guide Gwen On Thu, Dec 15, 2016 at 1:29 PM, Guozhang Wang wrote: > Hello Kafka users, developers and client-developers, > > This is the second, and hopefully the last candidate for the rele

Re: "support" topics for KStreams

For changelog topics, their retention policy is based on the log compaction by default, and users can customize it if necessary when creating the state store (this is in trunk, not in released versions yet). For repartition topics, their retention policy is based on whatever the broker's default r

Re: Consumers in the same group but consuming different topics?

Hello Avi, Similar questions have been discussed previously: https://issues.apache.org/jira/browse/KAFKA-3775 One concern of doing this, though is that if you want to do the change on-the-fly by rolling bounce (i.e. combining step 5 6 and 7) the instances to switch back the assignor class, then

Re: RocksDB - no locks available exception

Jon, Re: "RocksDBException: IO error ... No locks available". We have nailed a couple of rebalance issues that could cause the rocksDB lock file to be released not in time recently in trunk, and by checking from your logs I suspect you were hitting one of these issues. Could you try out the curre

Re: Another odd error

Hi Jon, When you "copied a new build up to one machine" did you mean that you swipe in the new jar, and then bounce the instance? Kafka Streams should naturally support online upgrading by simply rolling bounce your instances, so I would not expect the scenarios you described to happen. However,

Re: KStreams app - frequent broker dis/re connects

Jon, Could you share your observe debug logs here? BTW note that if the application's topology is complex or if you start multiple instance / or threads at the same time, then it may take multiple iterations of rebalances in the worst case to eventually get the assignment among all the threads.

Re: Kafka ACL's with SSL Protocol is not working

Thanks Rajani for the above Info but I want to restrict a user from performing all the operations (I think that defines ACL), I just want User_1 to produce messages and User_2 to consume messages. How can we achieve that. Thanks in advance On Mon, Dec 19, 2016 at 3:13 AM, Rajini Sivaram wrote:

Re: Kafka 0.9-Java : consumer skipping offsets during application restart

I'm very interested in the answer to this question as well - if the offsets are not preserved over the app re-start, we coudl be looking into a sizable data loss thanks!Marina From: Dhyan Muralidharan To: users@kafka.apache.org Sent: Friday, December 16, 2016 8:29 AM Subject: Ka

Re: The connection between kafka and zookeeper is often closed byzookeeper, lead to NotLeaderForPartitionException: This server is not theleader for that topic-partition.

Xiaoyuan, I am not an expert in ZK so here is what I can tell: 1. "NotLeaderForPartitionException" is not usually thrown when ZK connection timed out, it is thrown when the produce requests has arrived the broker but the brokers think themselves as not the leader (any more) for the requested part

Kafka controlled shutdown hangs when there are large number of topics in the cluster

Hi We have 9 broker instances in a kafka cluster spread across 3 linux machines. The 1st machine has 4 broker instances. 2nd machine has 4 broker instances and 3rd one has 1 broker instance. There are around 101 topics created in the cluster We start the broker as follows All 4 broke

Re: Kafka ACL's with SSL Protocol is not working

Raghu, It could be because the principal used for inter broker communication doesn't have all the necessary permissions. If you are using PLAINTEXT for inter-broker, the principal is ANONYMOUS, if using SSL, it would be similar to the one you are setting for client. You can configure broker princi

Re: Kafka SSL encryption plus external CA

Stephane, If you are using a trusted CA like Verisign, clients don't need to specify a truststore. The host names specified in advertised.listeners in the broker must match the wildcard DNS names in the certificates if clients configure ssl.endpoint.identification.algorithm=https. If ssl.endpoint.

Subscribe.

Subscribe.

TLS

Hello, I'm trying set SSL/TLS between Kafka-Zookeeper-Kafka but in the documentation you explain that the security must set as JAAS, but what is JAAS and how configure it? Un saludo, Rubén Poveda Teba Security Infrastructures rpov...@sia.es

Re: Producer connect timeouts

Makes sense, thanks Ewen. Is this something we could consider fixing in Kafka itself? I don't think the producer is necessarily doing anything wrong, but the end result is certainly very surprising behavior. It would also be nice not to have to coordinate request timeouts, retries, and the max blo

Re: How does 'TimeWindows.of().until()' work?

Yes that is one of the methods. It will be available on the 0.10.2 release which is due at the beginning of February. On Mon, 19 Dec 2016 at 12:17 Sachin Mittal wrote: > I believe you are talking about this method. > public KTable, T> aggregate(final > Initializer initializer, >

Re: getting intermittent TimeoutException at producer side in streams application

Hi, You could use this: https://github.com/apache/kafka/blob/trunk/streams/src/test/resources/log4j.properties as a starting point. You will probably want to change the level from WARN to INFO. Thanks On Mon, 19 Dec 2016 at 12:49 Sachin Mittal wrote: > Hi, > In order to enable streams logging

Re: getting intermittent TimeoutException at producer side in streams application

Hi, In order to enable streams logging we would need to add log4j.properties as part of our classpath right? Is there any sample log properties we can use. I think this issue mostly happens when we run streams app in a different DC than the brokers. However would like to see in logs if there is

Re: How does 'TimeWindows.of().until()' work?

I believe you are talking about this method. public KTable, T> aggregate(final Initializer initializer, final Aggregator aggregator, final Windows windows,

Re: getting intermittent TimeoutException at producer side in streams application

Hi Sachin, This would usually indicate that may indicate that there is a connectivity issue with the brokers. You would need to correlate the logs etc on the brokers with the streams logs to try and understand what is happening. Thanks, Damian On Sun, 18 Dec 2016 at 07:26 Sachin Mittal wrote:

Re: How does 'TimeWindows.of().until()' work?

Hi Sachin, I think we have a way of doing what you want already. If you create a custom state store you can call the enableLogging method and pass in any configuration parameters you want: For example: final StateStoreSupplier supplier = Stores.create("store") .withKeys(Serdes.String())

Re: Halting because log truncation is not allowed for topic __consumer_offsets

Hi Jun This should only be possible in situations where there is a crash or something happens to the underlying disks (assuming clean leader election). I've not come across others. The assumption, as I understand it, is that the underlying issue stems from KAFKA-1211

Re: How does 'TimeWindows.of().until()' work?

Hi, I am working towards adding topic configs as part of streams config. However I have run into an issue: Code flow is like this KStreamBuilder builder = new KStreamBuilder(); builder.stream(...) ... KafkaStreams streams = new KafkaStreams(builder, streamsProps); streams.start(); So we can see w