Hi folks, It's been a few days since I last pinged and nobody replied so I assume that this KIP is abandoned and I can take this over (but please let me know if it's not). I will keep the current version of the KIP and move it to a sub-page if it's ever needed.
Thanks, Viktor On Fri, Aug 28, 2020 at 11:35 AM Viktor Somogyi-Vass < viktorsomo...@gmail.com> wrote: > Hi folks, > > I have a use-case and a non-trivial implementation with Apache Atlas for > this KIP and since this kip seems to be dormant for a while now, I'd take > it over and drive it to completion if you don't mind. > The current state of the PoC can be found on my fork at > https://github.com/viktorsomogyi/kafka/tree/atlas-audit-impl. > > Viktor > > On Wed, Jan 29, 2020 at 9:32 AM Игорь Мартемьянов <ledost...@gmail.com> > wrote: > >> Hello, Nikolai. >> >> >> >> > Can you, please, make it more specific? >> >> > Why does a business want to have this information? >> >> It is very demanded for security department to know who/when/where create >> or edit ACL settings. The same situation about topics. >> >> >> >> > What are the use-cases for it? >> >> This KIP are able to help catching some unauthorized changes of cluster >> resources or save history these changes. >> >> >> >> > Who will be analyzing these events and how? >> >> Events could be analyzed by some security department or developers who >> builds applications that can change cluster resources. >> >> >> >> > Why it’s not convenient to implement it with third-party tools? >> >> It is imposible to my mind getting detailed information about these events >> through third-party tools, because we don`t have the ability to catch >> these >> events outside Kafka core. >> >> >> >> > 1. `audit` name sounds too general for me. How about `onEvent`? >> >> Thanks, good point. I`ll change the method name. >> >> >> >> > 2. Should we introduce a special marker interface for audit events? >> >> > `AuditableRequest`, for example? >> >> No, we shouldn`t. We are going to create these events if we process >> special >> request in KafkaApis class (ApiKeys.CREATE_TOPICS, ApiKeys.DELETE_TOPICS, >> ApiKeys.CREATE_ACLS, ApiKeys.DELETE_ACLS, ApiKeys.DESCRIBE_ACLS, >> ApiKeys.ALTER_CONFIGS) >> >> >> >> >> public interface AuditEvent<T extends AbstractRequest> { >> >> >> String guid(); >> >> >> >> > Where this `guid` comes from? >> >> Guid generated automatically when the audit event is created. >> >> >> >> > Will it be the same on each node that receives an auditable event? >> >> Event is creating while request that changes cluster resource is processed >> and will destroyed after the prosessing will finished on each node. >> >> >> >> > Do we have `guid` for any extensions of `AbstractRequest`? >> >> No, we don`t. >> >> >> >> > If this field is `guid` why do we format this as a String on the API >> level? >> >> Thanks for the point. >> >> I have changed event interface recently. >> >> >> >> > Can you, please, add a full list of proposed new configuration >> properties >> and >> >> > examples for each to clarify your intentions? >> >> Yes, I have added some examples of new configuration properties. >> >> ср, 29 янв. 2020 г., 10:23 Владимир Беруненко <vvberune...@gmail.com>: >> >> > Hi Nikolai! >> > >> > >Can you, please, make it more specific? >> > Why does a business want to have this information? >> > >What are the use-cases for it? >> > >Who will be analyzing these events and how? >> > >Why it’s not convenient to implement it with third-party tools? >> > >> > This is required by the guys from information security to detect >> potential >> > threats of violation the rules for providing access to the data layer. >> > Analysis in the audit system is usually based on identifying >> > uncharacteristic integrations. The key feature of the audit, is that >> > cluster administrators do not have access to modification of audit >> events. >> > Third-party tools are great for data analysis, but its not good idea to >> use >> > it for audit events collection. >> > >> > >It’s not clear for me where and when AuditEvents will be sent? >> > Who will be the receiver of events? >> > >> > In my opinion, sending an audit event should be initiated when the >> broker >> > receives a request that matches the audit parameters. Each organization >> has >> > its own receiver system, so a common interface is required that the >> > organization’s development team can implement to integrate with their >> audit >> > system. >> > >> > Best wishes, Vladimir >> > >> > Hello, Igor. >> > > >> > > Thanks for the KIP. >> > > I have a couple of comments for it: >> > > >> > > > Motivation >> > > > It is highly demanded in most businesses to have the ability of >> > > obtaining audit information in case someone changes cluster >> configuration >> > > (like creation/deletion/modify/description of any topic or ACLs). >> > > >> > > Can you, please, make it more specific? >> > > Why does a business want to have this information? >> > > What are the use-cases for it? >> > > Who will be analyzing these events and how? >> > > Why it’s not convenient to implement it with third-party tools? >> > > >> > > It’s not clear for me where and when AuditEvents will be sent? >> > > Who will be the receiver of events? >> > > >> > > > void audit<T extends AbstractRequest>(AuditEvent<T> event); >> > > >> > > 1. `audit` name sounds too general for me. How about `onEvent`? >> > > 2. Should we introduce a special marker interface for audit events? >> > > `AuditableRequest`, for example? >> > > >> > > > public interface AuditEvent<T extends AbstractRequest> { >> > > > String guid(); >> > > >> > > Where this `guid` comes from? >> > > Will it be the same on each node that receives an auditable event? >> > > Do we have `guid` for any extensions of `AbstractRequest`? >> > > If this field is `guid` why do we format this as a String on the API >> > level? >> > > >> > > > One or more of AuditExtension's implementations can be configured >> via >> > > the configuration audit.extension.classes >> > > >> > > Can you, please, add a full list of proposed new configuration >> properties >> > > and examples for each to clarify your intentions? >> > > >> > > >> > > > 24 янв. 2020 г., в 18:12, Alexander Dunayevsky < >> > a.a.dunayev...@gmail.com> >> > > написал(а): >> > > > >> > > > Hello Igor, >> > > > >> > > > Thanks for your KIP 🙌🏽 >> > > > It would be great to adopt this functionality and getting the best >> of >> > > > tracking cluster activity. >> > > > >> > > > +1 vote from me >> > > > >> > > > Cheers, >> > > > Alex Dunayevsky >> > > > >> > > > >> > > > On Fri, 24 Jan 2020, 15:35 Игорь Мартемьянов, <ledost...@gmail.com> >> > > wrote: >> > > > >> > > >> Motivation: >> > > >> >> > > >> >> > > >> *It is highly demanded in most businesses to have the ability of >> > > obtaining >> > > >> audit information in case someone changes cluster configuration >> (like >> > > >> creation/deletion/modify/description of any topic or ACLs).We may >> add >> > > this >> > > >> ability. Since audit requirements are so broad, it's impractical to >> > > support >> > > >> all of them.Hence we have to provide ability for users to plug >> > resources >> > > >> helping to achieve required capabilities.* >> > > >> >> > > >> >> > > >> >> > > >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-567%3A+Kafka+Cluster+Audit >> > > >> >> > > >> >> > > >> пт, 24 янв. 2020 г., 17:29 Игорь Мартемьянов <ledost...@gmail.com >> >: >> > > >> >> > > >>> Hello there. >> > > >>> Please review this KIP. >> > > >>> Thanks. >> > > >>> >> > > >> >> > > >> > > >> > >> >
