Hi Boyang,
"One thing I'm not fully convinced of is why we need to deprecate the
server side auto topic creation logic,"
The point is to eventually remove auto topic creation logic from
MetadataRequest. It makes no sense for MetadataRequest to cause topics to
be created. It's an unintuitive hack.
d client agree on turning on the switch, will a topic get
created?"
It was the same as what I thought.
-邮件原件-----
发件人: Boyang Chen
发送时间: 2020年6月24日 13:12
收件人: dev
主题: Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer
Hey Justin and Jiamei,
I read the KIP and skimmed over the
Hey Justin and Jiamei,
I read the KIP and skimmed over the discussion. One thing I'm not fully
convinced of is why we need to deprecate the server side auto topic
creation logic, which seems orthogonal towards whether a client wants to
create the topic or not. Won't it be more natural to assume th
Hi all
For
https://cwiki.apache.org/confluence/display/KAFKA/KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer
, It has not been updated for a long time. And I made some update, which has
been pushed to https://github.com/apache/kafka/pull/8831
MetadataRequest has method Builder(List
Hi folks
For
https://cwiki.apache.org/confluence/display/KAFKA/KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer
, It has not been updated for a long time. And I made some update, which has
been pushed to https://github.com/apache/kafka/pull/8831
MetadataRequest has method Builder(Li
On Fri, Aug 09, 2019 at 11:12 AM, Ismael Juma wrote:
> Hi all,
>
> A few points:
>
> 1. I think the way backwards compatibility is being used here is not
> correct. Any functionality that is only enabled if set via a config is
> backwards compatible. People may disagree with the functionality or
Hi all,
A few points:
1. I think the way backwards compatibility is being used here is not
correct. Any functionality that is only enabled if set via a config is
backwards compatible. People may disagree with the functionality or the
config, but it's not a backwards compatibility issue.
2. It's
On Thu, Aug 8, 2019, at 11:51, Jun Rao wrote:
> Hi, Justine,
>
> Thanks for the KIP. Overall, it seems to be a good improvement.
>
> However, I think Harsha's point seems reasonable. We had
> auto.create.topics.enable config on the broker to allow admins to disable
> topic creation from the produ
Hi, Justine,
Thanks for the KIP. Overall, it seems to be a good improvement.
However, I think Harsha's point seems reasonable. We had
auto.create.topics.enable config on the broker to allow admins to disable
topic creation from the producer/consumer clients before we had the
security feature. The
Hi,
If I may, perhaps you could simplify everything by using only
'auto.create.topics.enable' as a value along with true. In other words, the
public interfaces section should only have [true,auto.create.topics.enable,
false].
The reason for this is that auto.create.topics.enable is already known
Hi Harsha,
I think my message may have gotten lost in all the others.
Two of the goals of this KIP are to 1) allow auto-creation on specific
clients when the broker default is false and 2) eventually replace the
broker config.
In order to accomplish these two goals, we need the producer to be ab
On Wed, Aug 07, 2019 at 9:50 AM, Colin McCabe wrote:
> On Wed, Aug 7, 2019, at 09:24, Harsha Ch wrote:
>
> On Tue, Aug 06, 2019 at 11:46 PM, Colin McCabe < cmcc...@apache.org >
> wrote:
>
> On Tue, Aug 6, 2019, at 21:38, Harsha Ch wrote:
>
> Hi Colin,
> "Hmm... I'm not sure I follow. Users don't
On Wed, Aug 7, 2019, at 09:24, Harsha Ch wrote:
> On Tue, Aug 06, 2019 at 11:46 PM, Colin McCabe < cmcc...@apache.org > wrote:
>
> > On Tue, Aug 6, 2019, at 21:38, Harsha Ch wrote:
> >>
> >> Hi Colin,
> >> "Hmm... I'm not sure I follow. Users don't have to build their own
> >> tooling, right? The
On Tue, Aug 06, 2019 at 11:46 PM, Colin McCabe < cmcc...@apache.org > wrote:
>
>
>
> On Tue, Aug 6, 2019, at 21:38, Harsha Ch wrote:
>
>
>
>>
>>
>> Hi Colin,
>> "Hmm... I'm not sure I follow. Users don't have to build their own
>> tooling, right? They can use any of the shell scripts that
Hello all,
Thank you for all the feedback!
It seems that one of the main issues is how the client-side auto-creation
can act on its own and does not simply block or allow auto-creation as
configured by the broker.
I think I was a bit unclear about this, but the idea of this KIP is to
eventually
On Tue, Aug 6, 2019, at 21:38, Harsha Ch wrote:
> Hi Colin,
> "Hmm... I'm not sure I follow. Users don't have to build their own
> tooling, right? They can use any of the shell scripts that we've shipped
> in the last few releases. For example, if any of your users run it, this
> shell script wi
Hi Colin,
"Hmm... I'm not sure I follow. Users don't have to build their own
tooling, right? They can use any of the shell scripts that we've shipped
in the last few releases. For example, if any of your users run it, this
shell script will delete all of the topics from your non-security-enabled
On Tue, Aug 6, 2019, at 18:06, Harsha Ch wrote:
> Not sure how the AdminClient applies here, It is an external API and
> is not part of KafkaProducer so any user who updates to latest version of
> Kafka with this feature get to create the topics.
> They have to build a tooling around AdminClient al
Hi Justine,
Thanks for the clarifications.
I understand that auto-creation of topics will happen through
CreateTopic request instead of metadata request. What I meant in
earlier mail is producer client should not override broker config
about auto-creation of topics. I agree with Harsha on other ma
Hi Justin,
Thanks for making changes. I still have concern that we are
prioritizing producer config over server side which is breaking the
backward compatibility of broker's auto.topic.create.enable as far as
producer is concerned.
Also
https://cwiki.apache.org/confluence/d
Hi Colin,
There is no behavior in Kafka producer that allowed users to
delete the topics or delete the records. So
citing them as an example doesn't makes sense in this context. But there is
a functionality which allowed creation of topics if they don't exist in the
cluster and this behavi
Hi Harsha,
Thanks for taking a look.
I'm not sure I follow the discussion about AdminClient. KafkaAdminClient has
been around for about 2 years at this point as a public class. There are many
programs that use it to automatically create topics. Kafka Streams does this,
for example. If any
Hi Satish,
Thanks for looking at the KIP.
Yes, the producer will wait for the topic to be created before it can send
any messages to it.
I would like to clarify "overriding" broker behavior. If the client enables
client-side autocreation, the only difference will be that the topic
auto-creation
Hi Justine,
Thanks for the KIP. This is a nice addition to the producer client
without running admin-client’s create topic APIs. Does producer wait
for the topic to be created successfully before it tries to publish
messages to that topic? I assume that this will not throw an error
that the topic d
Hi Harsha,
After taking this all into consideration, I've updated the KIP to no longer
allow client-side configuration of replication factor and partitions.
Instead, the broker defaults will be used as long as the broker supports
KIP 464.
If the broker does not support this KIP, then the client ca
Hi,
Even with policies one needs to implement that, so for every user who
doesn't want a producer to create topics or have limits around partitions &
replication factor they have to implement a policy.
The KIP is changing the behavior , it might not be introducing the
new functionality bu
Hi Harsha,
I mentioned policies and the authorizer. For example, with
CreateTopicPolicy, you can implement the limits you describe. If you have
ideas of how that should be improved, please submit a KIP. My point is that
this KIP is not introducing any new functionality with regards to what
rogue c
Ismael,
Sure AdminClient can do that and we should've shipped a config or
use the existing one to block that. Not all users are yet to upgrade to
AdminClient and start using that to cause issues yet.
In shared environment we should allow server to set sane
defaults and not a
Harsha,
Rogue clients can use the admin client to create topics and partitions.
ACLs and policies can help in that case as well as this one.
Ismael
On Mon, Aug 5, 2019, 7:48 PM Harsha Chintalapani wrote:
> Hi Justine,
> Thanks for the KIP.
> "When server-side auto-creation is disa
Hi Justine,
Thanks for the KIP.
"When server-side auto-creation is disabled, client-side auto-creation will
try to use client-side configurations"
If I understand correctly, this KIP is removing any server-side blocking
client auto creation of topic?
if so this will present potential i
Hi all,
I made some changes to the KIP. Hopefully this configuration change will
make things a little clearer.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer
Please let me know if you have any feedback or questions!
Thank you,
Justin
Hi Mickael,
I think you bring up a good point. It would be better if we didn't ever have
to set up client-side configuration for this feature, and KIP-464 would let us
skip this entirely.
On Wed, Jul 31, 2019, at 09:19, Justine Olshan wrote:
> Hi Mickael,
> I agree that KIP-464 works on newer
Hi Mickael,
I agree that KIP-464 works on newer brokers, but I was a bit worried how
things would play out on older brokers that* do not *have KIP 464 included.
Is it enough to throw an error in this case when producer configs are not
specified?
Thanks,
Justine
On Wed, Jul 31, 2019 at 9:10 AM Mic
Hi Justine,
We can rely on KIP-464 which allows to omit the partition count or
replication factor when creating a topic. In that case, the broker
defaults are used.
On Wed, Jul 31, 2019 at 4:55 PM Justine Olshan wrote:
>
> Michael,
> That makes sense to me!
> To clarify, in the current state of
Michael,
That makes sense to me!
To clarify, in the current state of the KIP, the producer does not rely on
the broker to autocreate--if the broker's config is disabled, then the
producer can autocreate on its own with a create topic request (the same
type of request the admin client uses).
However
Hi Justine,
Thanks for the response!
In my opinion, it would be better if the producer did not rely at all
on the broker auto create feature as this is what we're aiming to
deprecate. When requesting metadata we can set the
"allow_auto_topic_creation" field to false to avoid the broker auto
creati
Currently the way it is implemented, the broker auto-creation configuration
takes precedence. The producer will not use the CreateTopics request.
(Technically it can--but the topic will already be created through the
broker, so it will never try to create the topic.)
It is possible to change this h
Hi Justine,
Thanks for the KIP!
In case auto creation is enabled on both the client and server, will
the producer still use the AdminClient (CreateTopics request) to
create topics? and not rely on the broker auto create.
I'm guessing the answer is yes but can you make it explicit.
Thank you
On
Hi,
Just a friendly reminder to take a look at this KIP if you have the time.
I was thinking about broker vs. client default precedence, and I think it
makes sense to keep the broker as the default used when both client-side
and broker-side defaults are configured. The idea is that there would be
Hello all,
I was looking at this KIP again, and there is a decision I made that I
think is worth discussing.
In the case where both the broker and producer's
'auto.create.topics.enable' are set to true, we have to choose either the
broker configs or the producer configs for the replication
factor
Just a quick update--
It seems that enabling both the broker and producer configs works fine,
except that the broker configurations for partitions, replication factor
take precedence.
I don't know if that is something we would want to change, but I'll be
updating the KIP for now to reflect this. P
Hi Colin,
Thanks for looking at the KIP. I can definitely add to the title to make it
more clear.
It makes sense that both configurations could be turned on since there are
many cases where the user can not control the server-side configurations. I
was a little concerned about how both interactin
Hi Justine,
Thanks for the KIP. This seems like a good step towards removing server-side
topic auto-creation.
We should add included "client-side" to the title of the KIP somewhere, to make
it clear that we're talking about client-side auto creation.
The KIP says:
> In order to automatically
Hi Dhruvil,
Thanks for reading the KIP!
That was the general idea for deprecation. We would log a warning when the
config is enabled on the broker.
I also believe that there would be a change to documentation.
If there is anything else that should be done, please let me know!
Justine
On Thu, Jul
Hi Justine,
Thanks for the KIP, this is great!
Could you add some more information about what deprecating the broker
configuration means? Would we log a warning in the logs when auto topic
creation is enabled on the broker, for example?
Thanks,
Dhruvil
On Thu, Jul 11, 2019 at 10:28 AM Justine O
Hello all,
I'd like to start a discussion thread for KIP-487.
This KIP plans to deprecate the current system of auto-creating topics
through requests to the metadata and give the producer the ability to
automatically create topics instead.
More information can be found here:
https://cwiki.apache.
46 matches
Mail list logo
