Re: [DISCUSS] Client Side Auto Topic Creation

Jun, thanks. I will check again. Regards Bosco On 7/17/16, 9:36 PM, "Jun Rao" wrote: Bosco, Currently, if acl is enabled, auto topic creation will succeed if the client has the CREATE permission. Thanks, Jun On Mon, Jul 11, 2016 at 6:26 PM, Don Bosc

Re: [DISCUSS] Client Side Auto Topic Creation

Bosco, Currently, if acl is enabled, auto topic creation will succeed if the client has the CREATE permission. Thanks, Jun On Mon, Jul 11, 2016 at 6:26 PM, Don Bosco Durai wrote: > Jun, my understanding is, currently if ACLs are enabled, then auto topic > creation is disabled. Is this going t

Re: [DISCUSS] Client Side Auto Topic Creation

Jun, my understanding is, currently if ACLs are enabled, then auto topic creation is disabled. Is this going to change with this requirement? Thanks Bosco On 7/11/16, 1:14 PM, "Jun Rao" wrote: Ismael, We could change the existing behavior if it's bad for most of the users. In the case of a

Re: [DISCUSS] Client Side Auto Topic Creation

Thanks for all the feedback guys. We could change the existing behavior if it's bad for most of the users. In > the case of auto topic creation in the producer, it seems that it's at > least convenient in a testing environment. So, I am not sure if that > behavior is universally bad. I agree tha

Re: [DISCUSS] Client Side Auto Topic Creation

I'd prefer to have the auto-creation on producer to false by default. Also, I would suggest not have a broker-side default configs for created topics, but rather on the admin-client side, this way users may be more aware of the default config values. Guozhang On Fri, Jul 8, 2016 at 6:06 AM, Is

Re: [DISCUSS] Client Side Auto Topic Creation

Ismael, We could change the existing behavior if it's bad for most of the users. In the case of auto topic creation in the producer, it seems that it's at least convenient in a testing environment. So, I am not sure if that behavior is universally bad. Also, I am not sure if we can rely on the cl

Re: [DISCUSS] Client Side Auto Topic Creation

Hi Jun, I agree that it's closer to the existing behaviour, which some people may be used to by now. However, I am not sure that it won't surprise people. As Grant said, auto-topic creation is a common source of confusion and it interacts badly with topic deletion. If we need to provide auto-topi

Re: [DISCUSS] Client Side Auto Topic Creation

I think the use case for not blowing up the consumer is simply to not create an implicit ordering in which your services have to come up. On 07/07/2016 04:47 PM, Jun Rao wrote: It seems that it makes sense for the writer to trigger auto topic creation, but not the reader. So, my preference is J

Re: [DISCUSS] Client Side Auto Topic Creation

It seems that it makes sense for the writer to trigger auto topic creation, but not the reader. So, my preference is Jay's option #1: add a new configuration to enable topic creation on the producer side and defaults to true. If the topic doesn't exist, the producer will send a createTopicRequest a

Re: [DISCUSS] Client Side Auto Topic Creation

I agree with Grant and Ismael regarding 1 / 2 / 3, and for 2) I would prefer having the default configs on the client side (i.e. like what we did in the kafka-topics script today) than on the cluster side, where the AdminClient will auto-set the configs if they are not specified by the user trigger

Re: [DISCUSS] Client Side Auto Topic Creation

Hi all, I think there are a few things being discussed and it would be good to make that explicit: 1. If and how we expose auto-topic creation in the client (under the assumption that the server auto-topic creation will be deprecated and eventually removed) 2. The ability to create topics with th

Re: [DISCUSS] Client Side Auto Topic Creation

Hi Roger and Constantine, Thanks for the feedback. I agree that configuration to maintain guarantees is commonly spread across enterprise teams, making it difficult to get right. That said its also hard to solve for every company structure too. I think there is room for an open discussion about w

Re: [DISCUSS] Client Side Auto Topic Creation

We currently run with auto topic creation enabled, largely to ensure that our topics all get created with the cluster defaults. My understanding is that this is the only to ensure this, since the defaults are not accessible to clients. We run a cluster per deployment, with the defaults are set

Re: [DISCUSS] Client Side Auto Topic Creation

Roger, I concur with everything you said. Couple more use cases to prove the point: 1. Some topics should always have 1 and only one partition. 2. CDC application based on Kafka Connect. Those type of application absolutely must know how to create properly configured topics: compacted,

Re: [DISCUSS] Client Side Auto Topic Creation

My comments go a bit beyond just topic creation but I'd like to see Kafka make it easier for application developers to specify their requirements declaratively in a single place. Today, for example, if your application requires strong guarantees against data loss, you must set a mix of topic-level

Re: [DISCUSS] Client Side Auto Topic Creation

Thanks for the discussion, below are some thoughts and responses. One of the problems that we currently have with > the clients is that we retry silently on unknown topics under the > expectation that they will eventually be created (automatically or not). > This makes it difficult to detect misco

Re: [DISCUSS] Client Side Auto Topic Creation

For what it's worth the use case for auto-creation isn't using a dynamic set of topics, but rather letting apps flow through different dev/staging/prod/integration_testing/unit_testing environments without having the app configure appropriate replication/partitioning stuff in each environment and h

Re: [DISCUSS] Client Side Auto Topic Creation

rsley Park, SO21 2JN IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU From: Jason Gustafson To: dev@kafka.apache.org Date: 28/06/2016 21:53 Subject: Re: [DISCUSS] Client Side

Re: [DISCUSS] Client Side Auto Topic Creation

Hey Grant, I'm also inclined to the latter two options you've suggested (and probably more to the one which uses AdminClient since it encourages separation of privileges). Raising an exception gives the user the most control over how to handle unknown topics. One of the problems that we currently

[DISCUSS] Client Side Auto Topic Creation

With the KIP-4 create topic schema voted and passed and a PR available upstream. I wanted to discuss moving the auto topic creation from the broker side to the client side (KAFKA-2410 ). This change has many benefits - Remove the need for faile