Hi folks,
I have a use-case and a non-trivial implementation with Apache Atlas for
this KIP and since this kip seems to be dormant for a while now, I'd take
it over and drive it to completion if you don't mind.
The current state of the PoC can be found on my fork at
https://github.com/viktorsomogyi/kafka/tree/atlas-audit-impl.
Viktor
On Wed, Jan 29, 2020 at 9:32 AM Игорь Мартемьянов <ledost...@gmail.com>
wrote:
> Hello, Nikolai.
>
>
>
> > Can you, please, make it more specific?
>
> > Why does a business want to have this information?
>
> It is very demanded for security department to know who/when/where create
> or edit ACL settings. The same situation about topics.
>
>
>
> > What are the use-cases for it?
>
> This KIP are able to help catching some unauthorized changes of cluster
> resources or save history these changes.
>
>
>
> > Who will be analyzing these events and how?
>
> Events could be analyzed by some security department or developers who
> builds applications that can change cluster resources.
>
>
>
> > Why it’s not convenient to implement it with third-party tools?
>
> It is imposible to my mind getting detailed information about these events
> through third-party tools, because we don`t have the ability to catch these
> events outside Kafka core.
>
>
>
> > 1. `audit` name sounds too general for me. How about `onEvent`?
>
> Thanks, good point. I`ll change the method name.
>
>
>
> > 2. Should we introduce a special marker interface for audit events?
>
> > `AuditableRequest`, for example?
>
> No, we shouldn`t. We are going to create these events if we process special
> request in KafkaApis class (ApiKeys.CREATE_TOPICS, ApiKeys.DELETE_TOPICS,
> ApiKeys.CREATE_ACLS, ApiKeys.DELETE_ACLS, ApiKeys.DESCRIBE_ACLS,
> ApiKeys.ALTER_CONFIGS)
>
>
>
> >> public interface AuditEvent<T extends AbstractRequest> {
>
> >> String guid();
>
>
>
> > Where this `guid` comes from?
>
> Guid generated automatically when the audit event is created.
>
>
>
> > Will it be the same on each node that receives an auditable event?
>
> Event is creating while request that changes cluster resource is processed
> and will destroyed after the prosessing will finished on each node.
>
>
>
> > Do we have `guid` for any extensions of `AbstractRequest`?
>
> No, we don`t.
>
>
>
> > If this field is `guid` why do we format this as a String on the API
> level?
>
> Thanks for the point.
>
> I have changed event interface recently.
>
>
>
> > Can you, please, add a full list of proposed new configuration properties
> and
>
> > examples for each to clarify your intentions?
>
> Yes, I have added some examples of new configuration properties.
>
> ср, 29 янв. 2020 г., 10:23 Владимир Беруненко <vvberune...@gmail.com>:
>
> > Hi Nikolai!
> >
> > >Can you, please, make it more specific?
> > Why does a business want to have this information?
> > >What are the use-cases for it?
> > >Who will be analyzing these events and how?
> > >Why it’s not convenient to implement it with third-party tools?
> >
> > This is required by the guys from information security to detect
> potential
> > threats of violation the rules for providing access to the data layer.
> > Analysis in the audit system is usually based on identifying
> > uncharacteristic integrations. The key feature of the audit, is that
> > cluster administrators do not have access to modification of audit
> events.
> > Third-party tools are great for data analysis, but its not good idea to
> use
> > it for audit events collection.
> >
> > >It’s not clear for me where and when AuditEvents will be sent?
> > Who will be the receiver of events?
> >
> > In my opinion, sending an audit event should be initiated when the broker
> > receives a request that matches the audit parameters. Each organization
> has
> > its own receiver system, so a common interface is required that the
> > organization’s development team can implement to integrate with their
> audit
> > system.
> >
> > Best wishes, Vladimir
> >
> > Hello, Igor.
> > >
> > > Thanks for the KIP.
> > > I have a couple of comments for it:
> > >
> > > > Motivation
> > > > It is highly demanded in most businesses to have the ability of
> > > obtaining audit information in case someone changes cluster
> configuration
> > > (like creation/deletion/modify/description of any topic or ACLs).
> > >
> > > Can you, please, make it more specific?
> > > Why does a business want to have this information?
> > > What are the use-cases for it?
> > > Who will be analyzing these events and how?
> > > Why it’s not convenient to implement it with third-party tools?
> > >
> > > It’s not clear for me where and when AuditEvents will be sent?
> > > Who will be the receiver of events?
> > >
> > > > void audit<T extends AbstractRequest>(AuditEvent<T> event);
> > >
> > > 1. `audit` name sounds too general for me. How about `onEvent`?
> > > 2. Should we introduce a special marker interface for audit events?
> > > `AuditableRequest`, for example?
> > >
> > > > public interface AuditEvent<T extends AbstractRequest> {
> > > > String guid();
> > >
> > > Where this `guid` comes from?
> > > Will it be the same on each node that receives an auditable event?
> > > Do we have `guid` for any extensions of `AbstractRequest`?
> > > If this field is `guid` why do we format this as a String on the API
> > level?
> > >
> > > > One or more of AuditExtension's implementations can be configured via
> > > the configuration audit.extension.classes
> > >
> > > Can you, please, add a full list of proposed new configuration
> properties
> > > and examples for each to clarify your intentions?
> > >
> > >
> > > > 24 янв. 2020 г., в 18:12, Alexander Dunayevsky <
> > a.a.dunayev...@gmail.com>
> > > написал(а):
> > > >
> > > > Hello Igor,
> > > >
> > > > Thanks for your KIP 🙌🏽
> > > > It would be great to adopt this functionality and getting the best of
> > > > tracking cluster activity.
> > > >
> > > > +1 vote from me
> > > >
> > > > Cheers,
> > > > Alex Dunayevsky
> > > >
> > > >
> > > > On Fri, 24 Jan 2020, 15:35 Игорь Мартемьянов, <ledost...@gmail.com>
> > > wrote:
> > > >
> > > >> Motivation:
> > > >>
> > > >>
> > > >> *It is highly demanded in most businesses to have the ability of
> > > obtaining
> > > >> audit information in case someone changes cluster configuration
> (like
> > > >> creation/deletion/modify/description of any topic or ACLs).We may
> add
> > > this
> > > >> ability. Since audit requirements are so broad, it's impractical to
> > > support
> > > >> all of them.Hence we have to provide ability for users to plug
> > resources
> > > >> helping to achieve required capabilities.*
> > > >>
> > > >>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-567%3A+Kafka+Cluster+Audit
> > > >>
> > > >>
> > > >> пт, 24 янв. 2020 г., 17:29 Игорь Мартемьянов <ledost...@gmail.com>:
> > > >>
> > > >>> Hello there.
> > > >>> Please review this KIP.
> > > >>> Thanks.
> > > >>>
> > > >>
> > >
> > >
> >
>
