Re: [DISCUSS] Eagle incubator proposal
Eagle in realtime evaluates security policies against event stream in a fully distributed way, so low latency and event partition are the two important factors for identifying malicious access instantly. So onboarding data through Falcon should consider these. Thanks Edward Zhang On 10/19/15, 22:46, "Jean-Baptiste Onofré" wrote: >It makes sense. I will try to contribute on this ;) > >Regards >JB > >On 10/19/2015 09:46 PM, Zhang, Edward (GDI Hadoop) wrote: >> Hi JB, >> >> That is a good Point. Good to know that Falcon feeds HDFS/Hive/HBase >>data >> changes, so this feature would complement Eagle which today mainly >>focuses >> on HDFS/Hive/HBase data access including view, change, delete etc. Eagle >> would benefit if Eagle can instantly capture data change from Falcon. >> >> Thanks >> Edward Zhang >> >> >> >> On 10/19/15, 8:40, "Jean-Baptiste Onofré" wrote: >> >>> Hi Arun, >>> >>> very interesting proposal. I may see some possible interaction with >>> Falcon. In Falcon, we have HDFS files (and Hive/HBase) monitoring (with >>> a kind of Change Data Capture), etc. >>> >>> So, I see a different perspective in Eagle, but Eagle could also >>> leverage Falcon somehow. >>> >>> Regards >>> JB >>> >>> On 10/19/2015 05:33 PM, Manoharan, Arun wrote: Hello Everyone, My name is Arun Manoharan. Currently a product manager in the Analytics platform team at eBay Inc. I would like to start a discussion on Eagle and its joining the ASF as an incubation project. Eagle is a Monitoring solution for Hadoop to instantly identify access to sensitive data, recognize attacks, malicious activities and take actions in real time. Eagle supports a wide variety of policies on HDFS data and Hive. Eagle also provides machine learning models for detecting anomalous user behavior in Hadoop. The proposal is available on the wiki here: https://wiki.apache.org/incubator/EagleProposal The text of the proposal is also available at the end of this email. Thanks for your time and help. Thanks, Arun Eagle Abstract Eagle is an Open Source Monitoring solution for Hadoop to instantly identify access to sensitive data, recognize attacks, malicious activities in hadoop and take actions. Proposal Eagle audits access to HDFS files, Hive and HBase tables in real time, enforces policies defined on sensitive data access and alerts or blocks user¹s access to that sensitive data in real time. Eagle also creates user profiles based on the typical access behaviour for HDFS and Hive and sends alerts when anomalous behaviour is detected. Eagle can also import sensitive data information classified by external classification engines to help define its policies. Overview of Eagle Eagle has 3 main parts. 1.Data collection and storage - Eagle collects data from various hadoop logs in real time using Kafka/Yarn API and uses HDFS and HBase for storage. 2.Data processing and policy engine - Eagle allows users to create policies based on various metadata properties on HDFS, Hive and HBase data. 3.Eagle services - Eagle services include policy manager, query service and the visualization component. Eagle provides intuitive user interface to administer Eagle and an alert dashboard to respond to real time alerts. Data Collection and Storage: Eagle provides programming API for extending Eagle to integrate any data source into Eagle policy evaluation framework. For example, Eagle hdfs audit monitoring collects data from Kafka which is populated from namenode log4j appender or from logstash agent. Eagle hive monitoring collects hive query logs from running job through YARN API, which is designed to be scalable and fault-tolerant. Eagle uses HBase as storage for storing metadata and metrics data, and also supports relational database through configuration change. Data Processing and Policy Engine: Processing Engine: Eagle provides stream processing API which is an abstraction of Apache Storm. It can also be extended to other streaming engines. This abstraction allows developers to assemble data transformation, filtering, external data join etc. without physically bound to a specific streaming platform. Eagle streaming API allows developers to easily integrate business logic with Eagle policy engine and internally Eagle framework compiles business logic execution DAG into program primitives of underlying stream infrastructure e.g. Apache Storm. For example, Eagle HDFS monitoring transforms audit log from Namenode to object and joins sensitivity metadata, security zone metadata which are generated from external programs or configured by user. Eagle
Re: Draft Report October 2015 - please review
Hi, On Mon, Oct 12, 2015 at 9:11 PM, Marvin Humphrey wrote: > Incubator PMC report for October 2015.. I just noticed that this report hasn't been posted to the board's agenda - the board meeting is tomorrow. -Bertrand - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [DISCUSS] Eagle incubator proposal
I would like to volunteer as mentor and help the project, if you are looking for more mentors. Thanks Amareshwari On Mon, Oct 19, 2015 at 9:03 PM, Manoharan, Arun wrote: > Hello Everyone, > > My name is Arun Manoharan. Currently a product manager in the Analytics > platform team at eBay Inc. > > I would like to start a discussion on Eagle and its joining the ASF as an > incubation project. > > Eagle is a Monitoring solution for Hadoop to instantly identify access to > sensitive data, recognize attacks, malicious activities and take actions in > real time. Eagle supports a wide variety of policies on HDFS data and Hive. > Eagle also provides machine learning models for detecting anomalous user > behavior in Hadoop. > > The proposal is available on the wiki here: > https://wiki.apache.org/incubator/EagleProposal > > The text of the proposal is also available at the end of this email. > > Thanks for your time and help. > > Thanks, > Arun > > > > Eagle > > Abstract > Eagle is an Open Source Monitoring solution for Hadoop to instantly > identify access to sensitive data, recognize attacks, malicious activities > in hadoop and take actions. > > Proposal > Eagle audits access to HDFS files, Hive and HBase tables in real time, > enforces policies defined on sensitive data access and alerts or blocks > user’s access to that sensitive data in real time. Eagle also creates user > profiles based on the typical access behaviour for HDFS and Hive and sends > alerts when anomalous behaviour is detected. Eagle can also import > sensitive data information classified by external classification engines to > help define its policies. > > Overview of Eagle > Eagle has 3 main parts. > 1.Data collection and storage - Eagle collects data from various hadoop > logs in real time using Kafka/Yarn API and uses HDFS and HBase for storage. > 2.Data processing and policy engine - Eagle allows users to create > policies based on various metadata properties on HDFS, Hive and HBase data. > 3.Eagle services - Eagle services include policy manager, query service > and the visualization component. Eagle provides intuitive user interface to > administer Eagle and an alert dashboard to respond to real time alerts. > > Data Collection and Storage: > Eagle provides programming API for extending Eagle to integrate any data > source into Eagle policy evaluation framework. For example, Eagle hdfs > audit monitoring collects data from Kafka which is populated from namenode > log4j appender or from logstash agent. Eagle hive monitoring collects hive > query logs from running job through YARN API, which is designed to be > scalable and fault-tolerant. Eagle uses HBase as storage for storing > metadata and metrics data, and also supports relational database through > configuration change. > > Data Processing and Policy Engine: > Processing Engine: Eagle provides stream processing API which is an > abstraction of Apache Storm. It can also be extended to other streaming > engines. This abstraction allows developers to assemble data > transformation, filtering, external data join etc. without physically bound > to a specific streaming platform. Eagle streaming API allows developers to > easily integrate business logic with Eagle policy engine and internally > Eagle framework compiles business logic execution DAG into program > primitives of underlying stream infrastructure e.g. Apache Storm. For > example, Eagle HDFS monitoring transforms audit log from Namenode to object > and joins sensitivity metadata, security zone metadata which are generated > from external programs or configured by user. Eagle hive monitoring filters > running jobs to get hive query string and parses query string into object > and then joins sensitivity metadata. > Alerting Framework: Eagle Alert Framework includes stream metadata API, > scalable policy engine framework, extensible policy engine framework. > Stream metadata API allows developers to declare event schema including > what attributes constitute an event, what is the type for each attribute, > and how to dynamically resolve attribute value in runtime when user > configures policy. Scalable policy engine framework allows policies to be > executed on different physical nodes in parallel. It is also used to define > your own policy partitioner class. Policy engine framework together with > streaming partitioning capability provided by all streaming platforms will > make sure policies and events can be evaluated in a fully distributed way. > Extensible policy engine framework allows developer to plugin a new policy > engine with a few lines of codes. WSO2 Siddhi CEP engine is the policy > engine which Eagle supports as first-class citizen. > Machine Learning module: Eagle provides capabilities to define user > activity patterns or user profiles for Hadoop users based on the user > behaviour in the platform. These user profiles are modeled using Machine > Learning algorithms and used for detection of anomalous users acti
Re: [DISCUSS] Eagle incubator proposal
Hey there, Arun! ... I have no commentary on the proposal itself, as it looks like a great proposal. I would suggest being a bit wary of the name, as "Eagle" is a *very* popular PCB design program. On Mon, Oct 19, 2015 at 10:33 AM, Manoharan, Arun wrote: > Hello Everyone, > > My name is Arun Manoharan. Currently a product manager in the Analytics > platform team at eBay Inc. > > I would like to start a discussion on Eagle and its joining the ASF as an > incubation project. > > Eagle is a Monitoring solution for Hadoop to instantly identify access to > sensitive data, recognize attacks, malicious activities and take actions in > real time. Eagle supports a wide variety of policies on HDFS data and Hive. > Eagle also provides machine learning models for detecting anomalous user > behavior in Hadoop. > > The proposal is available on the wiki here: > https://wiki.apache.org/incubator/EagleProposal > > The text of the proposal is also available at the end of this email. > > Thanks for your time and help. > > Thanks, > Arun > > > > Eagle > > Abstract > Eagle is an Open Source Monitoring solution for Hadoop to instantly > identify access to sensitive data, recognize attacks, malicious activities > in hadoop and take actions. > > Proposal > Eagle audits access to HDFS files, Hive and HBase tables in real time, > enforces policies defined on sensitive data access and alerts or blocks > user’s access to that sensitive data in real time. Eagle also creates user > profiles based on the typical access behaviour for HDFS and Hive and sends > alerts when anomalous behaviour is detected. Eagle can also import > sensitive data information classified by external classification engines to > help define its policies. > > Overview of Eagle > Eagle has 3 main parts. > 1.Data collection and storage - Eagle collects data from various hadoop > logs in real time using Kafka/Yarn API and uses HDFS and HBase for storage. > 2.Data processing and policy engine - Eagle allows users to create > policies based on various metadata properties on HDFS, Hive and HBase data. > 3.Eagle services - Eagle services include policy manager, query service > and the visualization component. Eagle provides intuitive user interface to > administer Eagle and an alert dashboard to respond to real time alerts. > > Data Collection and Storage: > Eagle provides programming API for extending Eagle to integrate any data > source into Eagle policy evaluation framework. For example, Eagle hdfs > audit monitoring collects data from Kafka which is populated from namenode > log4j appender or from logstash agent. Eagle hive monitoring collects hive > query logs from running job through YARN API, which is designed to be > scalable and fault-tolerant. Eagle uses HBase as storage for storing > metadata and metrics data, and also supports relational database through > configuration change. > > Data Processing and Policy Engine: > Processing Engine: Eagle provides stream processing API which is an > abstraction of Apache Storm. It can also be extended to other streaming > engines. This abstraction allows developers to assemble data > transformation, filtering, external data join etc. without physically bound > to a specific streaming platform. Eagle streaming API allows developers to > easily integrate business logic with Eagle policy engine and internally > Eagle framework compiles business logic execution DAG into program > primitives of underlying stream infrastructure e.g. Apache Storm. For > example, Eagle HDFS monitoring transforms audit log from Namenode to object > and joins sensitivity metadata, security zone metadata which are generated > from external programs or configured by user. Eagle hive monitoring filters > running jobs to get hive query string and parses query string into object > and then joins sensitivity metadata. > Alerting Framework: Eagle Alert Framework includes stream metadata API, > scalable policy engine framework, extensible policy engine framework. > Stream metadata API allows developers to declare event schema including > what attributes constitute an event, what is the type for each attribute, > and how to dynamically resolve attribute value in runtime when user > configures policy. Scalable policy engine framework allows policies to be > executed on different physical nodes in parallel. It is also used to define > your own policy partitioner class. Policy engine framework together with > streaming partitioning capability provided by all streaming platforms will > make sure policies and events can be evaluated in a fully distributed way. > Extensible policy engine framework allows developer to plugin a new policy > engine with a few lines of codes. WSO2 Siddhi CEP engine is the policy > engine which Eagle supports as first-class citizen. > Machine Learning module: Eagle provides capabilities to define user > activity patterns or user profiles for Hadoop users based on the user > behaviour in the platform. These user profiles are modele
Re: 回复： [VOTE] Graduate Apache Kylin from the Apache Incubator
On Tue, Oct 20, 2015 at 3:08 AM, John D. Ament wrote: > It looks weird to me that the date for the SGA is ambiguous. No one has a > more exact date than 2015? I had a look under private/documents/grants/ and I don't see any grant with "ebay" or "kylin" in its filename - it might have been filed under a different name of course. http://wiki.apache.org/incubator/KylinProposal says "Kylin has been under development since 2013 by a team of engineers at eBay Inc. It is currently hosted on Github.com under an Apache license at https://github.com/KylinOLAP/Kylin";. So it looks like a software grant should have been filed - can someone clarify the filing date and name of that document so that we can double check? -Bertrand - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [DISCUSS] Eagle incubator proposal
So glad to see one more project coming from eBay:-) Best Regards! - Luke Han On Tue, Oct 20, 2015 at 4:52 PM, Greg Stein wrote: > Hey there, Arun! ... I have no commentary on the proposal itself, as it > looks like a great proposal. I would suggest being a bit wary of the name, > as "Eagle" is a *very* popular PCB design program. > > On Mon, Oct 19, 2015 at 10:33 AM, Manoharan, Arun > wrote: > > > Hello Everyone, > > > > My name is Arun Manoharan. Currently a product manager in the Analytics > > platform team at eBay Inc. > > > > I would like to start a discussion on Eagle and its joining the ASF as an > > incubation project. > > > > Eagle is a Monitoring solution for Hadoop to instantly identify access to > > sensitive data, recognize attacks, malicious activities and take actions > in > > real time. Eagle supports a wide variety of policies on HDFS data and > Hive. > > Eagle also provides machine learning models for detecting anomalous user > > behavior in Hadoop. > > > > The proposal is available on the wiki here: > > https://wiki.apache.org/incubator/EagleProposal > > > > The text of the proposal is also available at the end of this email. > > > > Thanks for your time and help. > > > > Thanks, > > Arun > > > > > > > > Eagle > > > > Abstract > > Eagle is an Open Source Monitoring solution for Hadoop to instantly > > identify access to sensitive data, recognize attacks, malicious > activities > > in hadoop and take actions. > > > > Proposal > > Eagle audits access to HDFS files, Hive and HBase tables in real time, > > enforces policies defined on sensitive data access and alerts or blocks > > user’s access to that sensitive data in real time. Eagle also creates > user > > profiles based on the typical access behaviour for HDFS and Hive and > sends > > alerts when anomalous behaviour is detected. Eagle can also import > > sensitive data information classified by external classification engines > to > > help define its policies. > > > > Overview of Eagle > > Eagle has 3 main parts. > > 1.Data collection and storage - Eagle collects data from various hadoop > > logs in real time using Kafka/Yarn API and uses HDFS and HBase for > storage. > > 2.Data processing and policy engine - Eagle allows users to create > > policies based on various metadata properties on HDFS, Hive and HBase > data. > > 3.Eagle services - Eagle services include policy manager, query service > > and the visualization component. Eagle provides intuitive user interface > to > > administer Eagle and an alert dashboard to respond to real time alerts. > > > > Data Collection and Storage: > > Eagle provides programming API for extending Eagle to integrate any data > > source into Eagle policy evaluation framework. For example, Eagle hdfs > > audit monitoring collects data from Kafka which is populated from > namenode > > log4j appender or from logstash agent. Eagle hive monitoring collects > hive > > query logs from running job through YARN API, which is designed to be > > scalable and fault-tolerant. Eagle uses HBase as storage for storing > > metadata and metrics data, and also supports relational database through > > configuration change. > > > > Data Processing and Policy Engine: > > Processing Engine: Eagle provides stream processing API which is an > > abstraction of Apache Storm. It can also be extended to other streaming > > engines. This abstraction allows developers to assemble data > > transformation, filtering, external data join etc. without physically > bound > > to a specific streaming platform. Eagle streaming API allows developers > to > > easily integrate business logic with Eagle policy engine and internally > > Eagle framework compiles business logic execution DAG into program > > primitives of underlying stream infrastructure e.g. Apache Storm. For > > example, Eagle HDFS monitoring transforms audit log from Namenode to > object > > and joins sensitivity metadata, security zone metadata which are > generated > > from external programs or configured by user. Eagle hive monitoring > filters > > running jobs to get hive query string and parses query string into object > > and then joins sensitivity metadata. > > Alerting Framework: Eagle Alert Framework includes stream metadata API, > > scalable policy engine framework, extensible policy engine framework. > > Stream metadata API allows developers to declare event schema including > > what attributes constitute an event, what is the type for each attribute, > > and how to dynamically resolve attribute value in runtime when user > > configures policy. Scalable policy engine framework allows policies to be > > executed on different physical nodes in parallel. It is also used to > define > > your own policy partitioner class. Policy engine framework together with > > streaming partitioning capability provided by all streaming platforms > will > > make sure policies and events can be evaluated in a fully distributed > way. > > Extensible policy engin
Re: Draft Report October 2015 - please review
On Tue, Oct 20, 2015 at 12:53 AM, Bertrand Delacretaz wrote: > I just noticed that this report hasn't been posted to the board's > agenda - the board meeting is tomorrow. If Ted doesn't get to it, I plan to post it to the Board's agenda just before the 24 hour deadline. This month has been extremely busy for the IPMC Chair. During my term as Chair, the most time consuming, draining part of the job was dealing with the Board. Ted has certainly earned his money there this month. He has shown exactly the qualities that earned my support for him as Chair in the first place, and I'm delighted with the result. Nevertheless, in the future, I think we should be filing earlier, which means that we should negotiate a smaller window for whoever occupies the Chair to exercise their editorial prerogative. It's not cool to make a habit of dropping late reports on the Board just before the meeting, and our report is particularly large. The report has been in a coherent form since Monday of last week, and as of last Wednesday had been reviewed by the Incubator community for 2 days. There will be months where the Report Manager does not manage to hold tightly to the schedule, but in general, I think our expectation should be: 1. The report should ordinarily be filed on time, on the Wednesday one week before the Board meeting. 2. If the Chair has not filed the report by Saturday, someone else should file it. Marvin Humphrey - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: Draft Report October 2015 - please review
On Tue, Oct 20, 2015 at 1:26 PM, Marvin Humphrey wrote: > ...Ted has certainly earned his money there this month Definitely - there was nothing hidden behind my comment, it's just that it's good to make sure the report is filed in time for the board meeting (and I'm your shepherd this month, for the board report). -Bertrand - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [VOTE] Graduate Apache Kylin from the Apache Incubator
But Kylin come as whole as original code dump as part of going to incubator. Does eBay still need to file software grant for it? On Tuesday, October 20, 2015, Bertrand Delacretaz wrote: > On Tue, Oct 20, 2015 at 3:08 AM, John D. Ament > wrote: > > It looks weird to me that the date for the SGA is ambiguous. No one has > a > > more exact date than 2015? > > I had a look under private/documents/grants/ and I don't see any grant > with "ebay" or "kylin" in its filename - it might have been filed > under a different name of course. > > http://wiki.apache.org/incubator/KylinProposal says "Kylin has been > under development since 2013 by a team of engineers at eBay Inc. It is > currently hosted on Github.com under an Apache license at > https://github.com/KylinOLAP/Kylin";. > > So it looks like a software grant should have been filed - can someone > clarify the filing date and name of that document so that we can > double check? > > -Bertrand > > - > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > > For additional commands, e-mail: general-h...@incubator.apache.org > > >
Re: [DISCUSS] Eagle incubator proposal
Hi Greg, Thank you for reviewing the proposal. Originally we thought Eagle might be trademarked by someone already but I went thru eBay legal team to get the clearance for the name to be used. We will look into it again to see if there will be potential problems. Thanks, Arun On 10/20/15, 1:52 AM, "Greg Stein" wrote: >Hey there, Arun! ... I have no commentary on the proposal itself, as it >looks like a great proposal. I would suggest being a bit wary of the name, >as "Eagle" is a *very* popular PCB design program. > >On Mon, Oct 19, 2015 at 10:33 AM, Manoharan, Arun >wrote: > >> Hello Everyone, >> >> My name is Arun Manoharan. Currently a product manager in the Analytics >> platform team at eBay Inc. >> >> I would like to start a discussion on Eagle and its joining the ASF as >>an >> incubation project. >> >> Eagle is a Monitoring solution for Hadoop to instantly identify access >>to >> sensitive data, recognize attacks, malicious activities and take >>actions in >> real time. Eagle supports a wide variety of policies on HDFS data and >>Hive. >> Eagle also provides machine learning models for detecting anomalous user >> behavior in Hadoop. >> >> The proposal is available on the wiki here: >> https://wiki.apache.org/incubator/EagleProposal >> >> The text of the proposal is also available at the end of this email. >> >> Thanks for your time and help. >> >> Thanks, >> Arun >> >> >> >> Eagle >> >> Abstract >> Eagle is an Open Source Monitoring solution for Hadoop to instantly >> identify access to sensitive data, recognize attacks, malicious >>activities >> in hadoop and take actions. >> >> Proposal >> Eagle audits access to HDFS files, Hive and HBase tables in real time, >> enforces policies defined on sensitive data access and alerts or blocks >> user¹s access to that sensitive data in real time. Eagle also creates >>user >> profiles based on the typical access behaviour for HDFS and Hive and >>sends >> alerts when anomalous behaviour is detected. Eagle can also import >> sensitive data information classified by external classification >>engines to >> help define its policies. >> >> Overview of Eagle >> Eagle has 3 main parts. >> 1.Data collection and storage - Eagle collects data from various hadoop >> logs in real time using Kafka/Yarn API and uses HDFS and HBase for >>storage. >> 2.Data processing and policy engine - Eagle allows users to create >> policies based on various metadata properties on HDFS, Hive and HBase >>data. >> 3.Eagle services - Eagle services include policy manager, query service >> and the visualization component. Eagle provides intuitive user >>interface to >> administer Eagle and an alert dashboard to respond to real time alerts. >> >> Data Collection and Storage: >> Eagle provides programming API for extending Eagle to integrate any data >> source into Eagle policy evaluation framework. For example, Eagle hdfs >> audit monitoring collects data from Kafka which is populated from >>namenode >> log4j appender or from logstash agent. Eagle hive monitoring collects >>hive >> query logs from running job through YARN API, which is designed to be >> scalable and fault-tolerant. Eagle uses HBase as storage for storing >> metadata and metrics data, and also supports relational database through >> configuration change. >> >> Data Processing and Policy Engine: >> Processing Engine: Eagle provides stream processing API which is an >> abstraction of Apache Storm. It can also be extended to other streaming >> engines. This abstraction allows developers to assemble data >> transformation, filtering, external data join etc. without physically >>bound >> to a specific streaming platform. Eagle streaming API allows developers >>to >> easily integrate business logic with Eagle policy engine and internally >> Eagle framework compiles business logic execution DAG into program >> primitives of underlying stream infrastructure e.g. Apache Storm. For >> example, Eagle HDFS monitoring transforms audit log from Namenode to >>object >> and joins sensitivity metadata, security zone metadata which are >>generated >> from external programs or configured by user. Eagle hive monitoring >>filters >> running jobs to get hive query string and parses query string into >>object >> and then joins sensitivity metadata. >> Alerting Framework: Eagle Alert Framework includes stream metadata API, >> scalable policy engine framework, extensible policy engine framework. >> Stream metadata API allows developers to declare event schema including >> what attributes constitute an event, what is the type for each >>attribute, >> and how to dynamically resolve attribute value in runtime when user >> configures policy. Scalable policy engine framework allows policies to >>be >> executed on different physical nodes in parallel. It is also used to >>define >> your own policy partitioner class. Policy engine framework together with >> streaming partitioning capability provided by all streaming platforms >>will >> make sure polici
Re: [DISCUSS] Eagle incubator proposal
Hi Arun, Eagle sounds very promising. I just had a discussion with someone about this exact need. I do however agree with Greg on the name. As far as I can see, besides the name, your weakest point is the all eBay employed team. It's not a blocker and can be fixed during incubation. Good luck to you. Alex On Tue, Oct 20, 2015 at 5:51 PM, Manoharan, Arun wrote: > Hi Greg, > > Thank you for reviewing the proposal. > > Originally we thought Eagle might be trademarked by someone already but I > went thru eBay legal team to get the clearance for the name to be used. We > will look into it again to see if there will be potential problems. > > Thanks, > Arun > > On 10/20/15, 1:52 AM, "Greg Stein" wrote: > > >Hey there, Arun! ... I have no commentary on the proposal itself, as it > >looks like a great proposal. I would suggest being a bit wary of the name, > >as "Eagle" is a *very* popular PCB design program. > > > >On Mon, Oct 19, 2015 at 10:33 AM, Manoharan, Arun > >wrote: > > > >> Hello Everyone, > >> > >> My name is Arun Manoharan. Currently a product manager in the Analytics > >> platform team at eBay Inc. > >> > >> I would like to start a discussion on Eagle and its joining the ASF as > >>an > >> incubation project. > >> > >> Eagle is a Monitoring solution for Hadoop to instantly identify access > >>to > >> sensitive data, recognize attacks, malicious activities and take > >>actions in > >> real time. Eagle supports a wide variety of policies on HDFS data and > >>Hive. > >> Eagle also provides machine learning models for detecting anomalous user > >> behavior in Hadoop. > >> > >> The proposal is available on the wiki here: > >> https://wiki.apache.org/incubator/EagleProposal > >> > >> The text of the proposal is also available at the end of this email. > >> > >> Thanks for your time and help. > >> > >> Thanks, > >> Arun > >> > >> > >> > >> Eagle > >> > >> Abstract > >> Eagle is an Open Source Monitoring solution for Hadoop to instantly > >> identify access to sensitive data, recognize attacks, malicious > >>activities > >> in hadoop and take actions. > >> > >> Proposal > >> Eagle audits access to HDFS files, Hive and HBase tables in real time, > >> enforces policies defined on sensitive data access and alerts or blocks > >> user¹s access to that sensitive data in real time. Eagle also creates > >>user > >> profiles based on the typical access behaviour for HDFS and Hive and > >>sends > >> alerts when anomalous behaviour is detected. Eagle can also import > >> sensitive data information classified by external classification > >>engines to > >> help define its policies. > >> > >> Overview of Eagle > >> Eagle has 3 main parts. > >> 1.Data collection and storage - Eagle collects data from various hadoop > >> logs in real time using Kafka/Yarn API and uses HDFS and HBase for > >>storage. > >> 2.Data processing and policy engine - Eagle allows users to create > >> policies based on various metadata properties on HDFS, Hive and HBase > >>data. > >> 3.Eagle services - Eagle services include policy manager, query service > >> and the visualization component. Eagle provides intuitive user > >>interface to > >> administer Eagle and an alert dashboard to respond to real time alerts. > >> > >> Data Collection and Storage: > >> Eagle provides programming API for extending Eagle to integrate any data > >> source into Eagle policy evaluation framework. For example, Eagle hdfs > >> audit monitoring collects data from Kafka which is populated from > >>namenode > >> log4j appender or from logstash agent. Eagle hive monitoring collects > >>hive > >> query logs from running job through YARN API, which is designed to be > >> scalable and fault-tolerant. Eagle uses HBase as storage for storing > >> metadata and metrics data, and also supports relational database through > >> configuration change. > >> > >> Data Processing and Policy Engine: > >> Processing Engine: Eagle provides stream processing API which is an > >> abstraction of Apache Storm. It can also be extended to other streaming > >> engines. This abstraction allows developers to assemble data > >> transformation, filtering, external data join etc. without physically > >>bound > >> to a specific streaming platform. Eagle streaming API allows developers > >>to > >> easily integrate business logic with Eagle policy engine and internally > >> Eagle framework compiles business logic execution DAG into program > >> primitives of underlying stream infrastructure e.g. Apache Storm. For > >> example, Eagle HDFS monitoring transforms audit log from Namenode to > >>object > >> and joins sensitivity metadata, security zone metadata which are > >>generated > >> from external programs or configured by user. Eagle hive monitoring > >>filters > >> running jobs to get hive query string and parses query string into > >>object > >> and then joins sensitivity metadata. > >> Alerting Framework: Eagle Alert Framework includes stream metadata API, > >> scalable policy engine framework
Re: Draft Report October 2015 - please review
I will file shortly. On Tue, Oct 20, 2015 at 5:15 AM, Bertrand Delacretaz wrote: > On Tue, Oct 20, 2015 at 1:26 PM, Marvin Humphrey > wrote: > > ...Ted has certainly earned his money there this month > > Definitely - there was nothing hidden behind my comment, it's just > that it's good to make sure the report is filed in time for the board > meeting (and I'm your shepherd this month, for the board report). > > -Bertrand > > - > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org > >
Re: [RESULT] [VOTE] Accept Mynewt into the Apache Incubator
Sterling, As far as I can tell, both Bertrand Délacretaz and and Jim Jagielski are IPMC Members and as such their votes should be counted as binding. Best regards, Pierre Smits *OFBiz Extensions Marketplace* http://oem.ofbizci.net/oci-2/ On Mon, Oct 19, 2015 at 7:10 PM, Sterling Hughes wrote: > On Mon, Oct 12, 2015 at 9:04 AM, Sterling Hughes > wrote: > > Hi All, > > > > As mentioned in the DISCUSS thread, all feedback has been positive on > > the Mynewt proposal, so I'd like to call a VOTE to accept Mynewt as a > > new ASF incubator project. > > > > The full text of the proposal is available on the incubator wiki at > > the following URL: > > > > https://wiki.apache.org/incubator/MynewtProposal?action=recall&rev=20 > > > > I have also included the full text below. > > > > Vote is open until Thurs, 16th October 2015, 23:59:00 PST. > > > > [ ] +1 to accept Mynewt into the Apache Incubator > > [ ] +0 > > [ ] -1 because... > > > > > This vote is now closed and passes with 4 binding +1 votes, > 3 non-binding +1 votes and no 0 or -1 votes. > > Thanks to all who helped with the proposal and cast the vote! > > Here's a vote tally: > > Non-binding +1s: > Jim Jagielski > Marvin Humphrey > Bertrand Delacretaz > > Binding +1s: > P. Taylor Goetz > Justin Mclean > Greg Stein > Jean Baptiste Onofré > > No 0 or -1 votes. > > Thanks, > Sterling > > PS: I didn't realize that I could also vote on the proposal until too > late, but for the record, I'm also a +1 :-) > > - > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org > >
Re: [DISCUSS] Eagle incubator proposal
Hi Amareshwari, Thank you very much. We definitely like to have you as a mentor. I will edit the proposal and set up sometime with you. Thanks, Arun On 10/20/15, 1:36 AM, "Amareshwari Sriramdasu" wrote: >I would like to volunteer as mentor and help the project, if you are >looking for more mentors. > >Thanks >Amareshwari > >On Mon, Oct 19, 2015 at 9:03 PM, Manoharan, Arun >wrote: > >> Hello Everyone, >> >> My name is Arun Manoharan. Currently a product manager in the Analytics >> platform team at eBay Inc. >> >> I would like to start a discussion on Eagle and its joining the ASF as >>an >> incubation project. >> >> Eagle is a Monitoring solution for Hadoop to instantly identify access >>to >> sensitive data, recognize attacks, malicious activities and take >>actions in >> real time. Eagle supports a wide variety of policies on HDFS data and >>Hive. >> Eagle also provides machine learning models for detecting anomalous user >> behavior in Hadoop. >> >> The proposal is available on the wiki here: >> https://wiki.apache.org/incubator/EagleProposal >> >> The text of the proposal is also available at the end of this email. >> >> Thanks for your time and help. >> >> Thanks, >> Arun >> >> >> >> Eagle >> >> Abstract >> Eagle is an Open Source Monitoring solution for Hadoop to instantly >> identify access to sensitive data, recognize attacks, malicious >>activities >> in hadoop and take actions. >> >> Proposal >> Eagle audits access to HDFS files, Hive and HBase tables in real time, >> enforces policies defined on sensitive data access and alerts or blocks >> user¹s access to that sensitive data in real time. Eagle also creates >>user >> profiles based on the typical access behaviour for HDFS and Hive and >>sends >> alerts when anomalous behaviour is detected. Eagle can also import >> sensitive data information classified by external classification >>engines to >> help define its policies. >> >> Overview of Eagle >> Eagle has 3 main parts. >> 1.Data collection and storage - Eagle collects data from various hadoop >> logs in real time using Kafka/Yarn API and uses HDFS and HBase for >>storage. >> 2.Data processing and policy engine - Eagle allows users to create >> policies based on various metadata properties on HDFS, Hive and HBase >>data. >> 3.Eagle services - Eagle services include policy manager, query service >> and the visualization component. Eagle provides intuitive user >>interface to >> administer Eagle and an alert dashboard to respond to real time alerts. >> >> Data Collection and Storage: >> Eagle provides programming API for extending Eagle to integrate any data >> source into Eagle policy evaluation framework. For example, Eagle hdfs >> audit monitoring collects data from Kafka which is populated from >>namenode >> log4j appender or from logstash agent. Eagle hive monitoring collects >>hive >> query logs from running job through YARN API, which is designed to be >> scalable and fault-tolerant. Eagle uses HBase as storage for storing >> metadata and metrics data, and also supports relational database through >> configuration change. >> >> Data Processing and Policy Engine: >> Processing Engine: Eagle provides stream processing API which is an >> abstraction of Apache Storm. It can also be extended to other streaming >> engines. This abstraction allows developers to assemble data >> transformation, filtering, external data join etc. without physically >>bound >> to a specific streaming platform. Eagle streaming API allows developers >>to >> easily integrate business logic with Eagle policy engine and internally >> Eagle framework compiles business logic execution DAG into program >> primitives of underlying stream infrastructure e.g. Apache Storm. For >> example, Eagle HDFS monitoring transforms audit log from Namenode to >>object >> and joins sensitivity metadata, security zone metadata which are >>generated >> from external programs or configured by user. Eagle hive monitoring >>filters >> running jobs to get hive query string and parses query string into >>object >> and then joins sensitivity metadata. >> Alerting Framework: Eagle Alert Framework includes stream metadata API, >> scalable policy engine framework, extensible policy engine framework. >> Stream metadata API allows developers to declare event schema including >> what attributes constitute an event, what is the type for each >>attribute, >> and how to dynamically resolve attribute value in runtime when user >> configures policy. Scalable policy engine framework allows policies to >>be >> executed on different physical nodes in parallel. It is also used to >>define >> your own policy partitioner class. Policy engine framework together with >> streaming partitioning capability provided by all streaming platforms >>will >> make sure policies and events can be evaluated in a fully distributed >>way. >> Extensible policy engine framework allows developer to plugin a new >>policy >> engine with a few lines of codes. WSO2 Siddhi CEP engi
Re: [DISCUSS] Eagle incubator proposal
On Tue, Oct 20, 2015 at 10:51 AM, Manoharan, Arun wrote: > Hi Greg, > > Thank you for reviewing the proposal. > > Originally we thought Eagle might be trademarked by someone already but I > went thru eBay legal team to get the clearance for the name to be used. We > will look into it again to see if there will be potential problems. Ultimately it will be the ASF that determines the appropriateness of the name for a podling. A few pointers: http://incubator.apache.org/guides/names.html https://issues.apache.org/jira/browse/PODLINGNAMESEARCH/ > Thanks, > Arun - Sam Ruby > On 10/20/15, 1:52 AM, "Greg Stein" wrote: > >>Hey there, Arun! ... I have no commentary on the proposal itself, as it >>looks like a great proposal. I would suggest being a bit wary of the name, >>as "Eagle" is a *very* popular PCB design program. >> >>On Mon, Oct 19, 2015 at 10:33 AM, Manoharan, Arun >>wrote: >> >>> Hello Everyone, >>> >>> My name is Arun Manoharan. Currently a product manager in the Analytics >>> platform team at eBay Inc. >>> >>> I would like to start a discussion on Eagle and its joining the ASF as >>>an >>> incubation project. >>> >>> Eagle is a Monitoring solution for Hadoop to instantly identify access >>>to >>> sensitive data, recognize attacks, malicious activities and take >>>actions in >>> real time. Eagle supports a wide variety of policies on HDFS data and >>>Hive. >>> Eagle also provides machine learning models for detecting anomalous user >>> behavior in Hadoop. >>> >>> The proposal is available on the wiki here: >>> https://wiki.apache.org/incubator/EagleProposal >>> >>> The text of the proposal is also available at the end of this email. >>> >>> Thanks for your time and help. >>> >>> Thanks, >>> Arun >>> >>> >>> >>> Eagle >>> >>> Abstract >>> Eagle is an Open Source Monitoring solution for Hadoop to instantly >>> identify access to sensitive data, recognize attacks, malicious >>>activities >>> in hadoop and take actions. >>> >>> Proposal >>> Eagle audits access to HDFS files, Hive and HBase tables in real time, >>> enforces policies defined on sensitive data access and alerts or blocks >>> user¹s access to that sensitive data in real time. Eagle also creates >>>user >>> profiles based on the typical access behaviour for HDFS and Hive and >>>sends >>> alerts when anomalous behaviour is detected. Eagle can also import >>> sensitive data information classified by external classification >>>engines to >>> help define its policies. >>> >>> Overview of Eagle >>> Eagle has 3 main parts. >>> 1.Data collection and storage - Eagle collects data from various hadoop >>> logs in real time using Kafka/Yarn API and uses HDFS and HBase for >>>storage. >>> 2.Data processing and policy engine - Eagle allows users to create >>> policies based on various metadata properties on HDFS, Hive and HBase >>>data. >>> 3.Eagle services - Eagle services include policy manager, query service >>> and the visualization component. Eagle provides intuitive user >>>interface to >>> administer Eagle and an alert dashboard to respond to real time alerts. >>> >>> Data Collection and Storage: >>> Eagle provides programming API for extending Eagle to integrate any data >>> source into Eagle policy evaluation framework. For example, Eagle hdfs >>> audit monitoring collects data from Kafka which is populated from >>>namenode >>> log4j appender or from logstash agent. Eagle hive monitoring collects >>>hive >>> query logs from running job through YARN API, which is designed to be >>> scalable and fault-tolerant. Eagle uses HBase as storage for storing >>> metadata and metrics data, and also supports relational database through >>> configuration change. >>> >>> Data Processing and Policy Engine: >>> Processing Engine: Eagle provides stream processing API which is an >>> abstraction of Apache Storm. It can also be extended to other streaming >>> engines. This abstraction allows developers to assemble data >>> transformation, filtering, external data join etc. without physically >>>bound >>> to a specific streaming platform. Eagle streaming API allows developers >>>to >>> easily integrate business logic with Eagle policy engine and internally >>> Eagle framework compiles business logic execution DAG into program >>> primitives of underlying stream infrastructure e.g. Apache Storm. For >>> example, Eagle HDFS monitoring transforms audit log from Namenode to >>>object >>> and joins sensitivity metadata, security zone metadata which are >>>generated >>> from external programs or configured by user. Eagle hive monitoring >>>filters >>> running jobs to get hive query string and parses query string into >>>object >>> and then joins sensitivity metadata. >>> Alerting Framework: Eagle Alert Framework includes stream metadata API, >>> scalable policy engine framework, extensible policy engine framework. >>> Stream metadata API allows developers to declare event schema including >>> what attributes constitute an event, what is the type for each >>>attrib
Re: [VOTE] Graduate Apache Kylin from the Apache Incubator
Henry, I think that there was a garble in your message. Did you mean to point out that the original code import was code that was ASL licensed by eBay? And then if that is right, does that mean your question was "Is an SGA necessary to import ASL code?" On Tue, Oct 20, 2015 at 7:27 AM, Henry Saputra wrote: > But Kylin come as whole as original code dump as part of going to > incubator. > Does eBay still need to file software grant for it? > > On Tuesday, October 20, 2015, Bertrand Delacretaz > wrote: > > > On Tue, Oct 20, 2015 at 3:08 AM, John D. Ament > > wrote: > > > It looks weird to me that the date for the SGA is ambiguous. No one > has > > a > > > more exact date than 2015? > > > > I had a look under private/documents/grants/ and I don't see any grant > > with "ebay" or "kylin" in its filename - it might have been filed > > under a different name of course. > > > > http://wiki.apache.org/incubator/KylinProposal says "Kylin has been > > under development since 2013 by a team of engineers at eBay Inc. It is > > currently hosted on Github.com under an Apache license at > > https://github.com/KylinOLAP/Kylin";. > > > > So it looks like a software grant should have been filed - can someone > > clarify the filing date and name of that document so that we can > > double check? > > > > -Bertrand > > > > - > > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > > > > For additional commands, e-mail: general-h...@incubator.apache.org > > > > > > >
Re: [VOTE] Graduate Apache Kylin from the Apache Incubator
HI Ted, I apologize for the confusing message, I was typing it with mobile device. I was trying to figure out whether an SGA is needed when the initial code coming as part of incubator code dump with ASF license. I.E: when a project entering incubator and the original code is donated from eBay into ASF git which already come with ASF license. As I remember SGA is needed not for new projects but for new contributions from organizations after initial code submission was made. I just want to make sure my understanding is correct. - Henry On Tue, Oct 20, 2015 at 11:15 AM, Ted Dunning wrote: > Henry, > > I think that there was a garble in your message. > > Did you mean to point out that the original code import was code that was > ASL licensed by eBay? > > And then if that is right, does that mean your question was "Is an SGA > necessary to import ASL code?" > > > > On Tue, Oct 20, 2015 at 7:27 AM, Henry Saputra > wrote: > >> But Kylin come as whole as original code dump as part of going to >> incubator. >> Does eBay still need to file software grant for it? >> >> On Tuesday, October 20, 2015, Bertrand Delacretaz >> wrote: >> >> > On Tue, Oct 20, 2015 at 3:08 AM, John D. Ament > > > wrote: >> > > It looks weird to me that the date for the SGA is ambiguous. No one >> has >> > a >> > > more exact date than 2015? >> > >> > I had a look under private/documents/grants/ and I don't see any grant >> > with "ebay" or "kylin" in its filename - it might have been filed >> > under a different name of course. >> > >> > http://wiki.apache.org/incubator/KylinProposal says "Kylin has been >> > under development since 2013 by a team of engineers at eBay Inc. It is >> > currently hosted on Github.com under an Apache license at >> > https://github.com/KylinOLAP/Kylin";. >> > >> > So it looks like a software grant should have been filed - can someone >> > clarify the filing date and name of that document so that we can >> > double check? >> > >> > -Bertrand >> > >> > - >> > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org >> > >> > For additional commands, e-mail: general-h...@incubator.apache.org >> > >> > >> > >> - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [DISCUSS] Eagle incubator proposal
Hi Ted, Since Kylin almost ready to graduate, I have more bandwidth to help with Eagle. But, you are right that current proposed mentors for Eagle seemed to be very busy with other podlings, so 1 or 2 additional mentors would be great. The good news is that the team consist some people from Kylin, for example Luke, which done great job helping Kylin to understand working with Apache way. So we have some help from initial committers who have done the rodeo before. - Henry On Mon, Oct 19, 2015 at 9:00 AM, Ted Dunning wrote: > I would suggest that Owen O'Malley has not had enough time to be a viable > mentor recently and should not be on the list of mentors. > > Henry and Julian are good if their schedules permit. Henry, I know has > been mentoring a number of projects lately. > > > > On Mon, Oct 19, 2015 at 8:40 AM, Jean-Baptiste Onofré > wrote: > >> Hi Arun, >> >> very interesting proposal. I may see some possible interaction with >> Falcon. In Falcon, we have HDFS files (and Hive/HBase) monitoring (with a >> kind of Change Data Capture), etc. >> >> So, I see a different perspective in Eagle, but Eagle could also leverage >> Falcon somehow. >> >> Regards >> JB >> >> >> On 10/19/2015 05:33 PM, Manoharan, Arun wrote: >> >>> Hello Everyone, >>> >>> My name is Arun Manoharan. Currently a product manager in the Analytics >>> platform team at eBay Inc. >>> >>> I would like to start a discussion on Eagle and its joining the ASF as an >>> incubation project. >>> >>> Eagle is a Monitoring solution for Hadoop to instantly identify access to >>> sensitive data, recognize attacks, malicious activities and take actions in >>> real time. Eagle supports a wide variety of policies on HDFS data and Hive. >>> Eagle also provides machine learning models for detecting anomalous user >>> behavior in Hadoop. >>> >>> The proposal is available on the wiki here: >>> https://wiki.apache.org/incubator/EagleProposal >>> >>> The text of the proposal is also available at the end of this email. >>> >>> Thanks for your time and help. >>> >>> Thanks, >>> Arun >>> >>> >>> >>> Eagle >>> >>> Abstract >>> Eagle is an Open Source Monitoring solution for Hadoop to instantly >>> identify access to sensitive data, recognize attacks, malicious activities >>> in hadoop and take actions. >>> >>> Proposal >>> Eagle audits access to HDFS files, Hive and HBase tables in real time, >>> enforces policies defined on sensitive data access and alerts or blocks >>> user’s access to that sensitive data in real time. Eagle also creates user >>> profiles based on the typical access behaviour for HDFS and Hive and sends >>> alerts when anomalous behaviour is detected. Eagle can also import >>> sensitive data information classified by external classification engines to >>> help define its policies. >>> >>> Overview of Eagle >>> Eagle has 3 main parts. >>> 1.Data collection and storage - Eagle collects data from various hadoop >>> logs in real time using Kafka/Yarn API and uses HDFS and HBase for storage. >>> 2.Data processing and policy engine - Eagle allows users to create >>> policies based on various metadata properties on HDFS, Hive and HBase data. >>> 3.Eagle services - Eagle services include policy manager, query service >>> and the visualization component. Eagle provides intuitive user interface to >>> administer Eagle and an alert dashboard to respond to real time alerts. >>> >>> Data Collection and Storage: >>> Eagle provides programming API for extending Eagle to integrate any data >>> source into Eagle policy evaluation framework. For example, Eagle hdfs >>> audit monitoring collects data from Kafka which is populated from namenode >>> log4j appender or from logstash agent. Eagle hive monitoring collects hive >>> query logs from running job through YARN API, which is designed to be >>> scalable and fault-tolerant. Eagle uses HBase as storage for storing >>> metadata and metrics data, and also supports relational database through >>> configuration change. >>> >>> Data Processing and Policy Engine: >>> Processing Engine: Eagle provides stream processing API which is an >>> abstraction of Apache Storm. It can also be extended to other streaming >>> engines. This abstraction allows developers to assemble data >>> transformation, filtering, external data join etc. without physically bound >>> to a specific streaming platform. Eagle streaming API allows developers to >>> easily integrate business logic with Eagle policy engine and internally >>> Eagle framework compiles business logic execution DAG into program >>> primitives of underlying stream infrastructure e.g. Apache Storm. For >>> example, Eagle HDFS monitoring transforms audit log from Namenode to object >>> and joins sensitivity metadata, security zone metadata which are generated >>> from external programs or configured by user. Eagle hive monitoring filters >>> running jobs to get hive query string and parses query string into object >>> and then joins sensitivity metadata. >>> Alerting Fr
Re: [DISCUSS] Eagle incubator proposal
I should also have some improved bandwidth both now that Kylin is nearing graduation and for other reasons. I’ve been bogged down recently, but that’s starting to change. If more mentors are desired, I’d be willing to help in that respect. -Taylor > On Oct 20, 2015, at 11:49 AM, Henry Saputra wrote: > > Hi Ted, > > Since Kylin almost ready to graduate, I have more bandwidth to help with > Eagle. > > But, you are right that current proposed mentors for Eagle seemed to > be very busy with other podlings, so 1 or 2 additional mentors would > be great. > > The good news is that the team consist some people from Kylin, for > example Luke, which done great job helping Kylin to understand working > with Apache way. > So we have some help from initial committers who have done the rodeo before. > > - Henry > > On Mon, Oct 19, 2015 at 9:00 AM, Ted Dunning wrote: >> I would suggest that Owen O'Malley has not had enough time to be a viable >> mentor recently and should not be on the list of mentors. >> >> Henry and Julian are good if their schedules permit. Henry, I know has >> been mentoring a number of projects lately. >> >> >> >> On Mon, Oct 19, 2015 at 8:40 AM, Jean-Baptiste Onofré >> wrote: >> >>> Hi Arun, >>> >>> very interesting proposal. I may see some possible interaction with >>> Falcon. In Falcon, we have HDFS files (and Hive/HBase) monitoring (with a >>> kind of Change Data Capture), etc. >>> >>> So, I see a different perspective in Eagle, but Eagle could also leverage >>> Falcon somehow. >>> >>> Regards >>> JB >>> >>> >>> On 10/19/2015 05:33 PM, Manoharan, Arun wrote: >>> Hello Everyone, My name is Arun Manoharan. Currently a product manager in the Analytics platform team at eBay Inc. I would like to start a discussion on Eagle and its joining the ASF as an incubation project. Eagle is a Monitoring solution for Hadoop to instantly identify access to sensitive data, recognize attacks, malicious activities and take actions in real time. Eagle supports a wide variety of policies on HDFS data and Hive. Eagle also provides machine learning models for detecting anomalous user behavior in Hadoop. The proposal is available on the wiki here: https://wiki.apache.org/incubator/EagleProposal The text of the proposal is also available at the end of this email. Thanks for your time and help. Thanks, Arun Eagle Abstract Eagle is an Open Source Monitoring solution for Hadoop to instantly identify access to sensitive data, recognize attacks, malicious activities in hadoop and take actions. Proposal Eagle audits access to HDFS files, Hive and HBase tables in real time, enforces policies defined on sensitive data access and alerts or blocks user’s access to that sensitive data in real time. Eagle also creates user profiles based on the typical access behaviour for HDFS and Hive and sends alerts when anomalous behaviour is detected. Eagle can also import sensitive data information classified by external classification engines to help define its policies. Overview of Eagle Eagle has 3 main parts. 1.Data collection and storage - Eagle collects data from various hadoop logs in real time using Kafka/Yarn API and uses HDFS and HBase for storage. 2.Data processing and policy engine - Eagle allows users to create policies based on various metadata properties on HDFS, Hive and HBase data. 3.Eagle services - Eagle services include policy manager, query service and the visualization component. Eagle provides intuitive user interface to administer Eagle and an alert dashboard to respond to real time alerts. Data Collection and Storage: Eagle provides programming API for extending Eagle to integrate any data source into Eagle policy evaluation framework. For example, Eagle hdfs audit monitoring collects data from Kafka which is populated from namenode log4j appender or from logstash agent. Eagle hive monitoring collects hive query logs from running job through YARN API, which is designed to be scalable and fault-tolerant. Eagle uses HBase as storage for storing metadata and metrics data, and also supports relational database through configuration change. Data Processing and Policy Engine: Processing Engine: Eagle provides stream processing API which is an abstraction of Apache Storm. It can also be extended to other streaming engines. This abstraction allows developers to assemble data transformation, filtering, external data join etc. without physically bound to a specific streaming platform. Eagle streaming API allows developers to easily integrate business logic with Eagle policy engine and internally Eagle framework compiles business logic executio
Re: [RESULT] [VOTE] Accept Mynewt into the Apache Incubator
Yeah, they just didn't note that in their votes, so Sterling didn't count them that way. *shrug* ... so it passes yet again :-) On Tue, Oct 20, 2015 at 11:55 AM, Pierre Smits wrote: > Sterling, > > As far as I can tell, both Bertrand Délacretaz and and Jim Jagielski are > IPMC Members and as such their votes should be counted as binding. > > Best regards, > > Pierre Smits > > *OFBiz Extensions Marketplace* > http://oem.ofbizci.net/oci-2/ > > On Mon, Oct 19, 2015 at 7:10 PM, Sterling Hughes > wrote: > > > On Mon, Oct 12, 2015 at 9:04 AM, Sterling Hughes > > wrote: > > > Hi All, > > > > > > As mentioned in the DISCUSS thread, all feedback has been positive on > > > the Mynewt proposal, so I'd like to call a VOTE to accept Mynewt as a > > > new ASF incubator project. > > > > > > The full text of the proposal is available on the incubator wiki at > > > the following URL: > > > > > > https://wiki.apache.org/incubator/MynewtProposal?action=recall&rev=20 > > > > > > I have also included the full text below. > > > > > > Vote is open until Thurs, 16th October 2015, 23:59:00 PST. > > > > > > [ ] +1 to accept Mynewt into the Apache Incubator > > > [ ] +0 > > > [ ] -1 because... > > > > > > > > > This vote is now closed and passes with 4 binding +1 votes, > > 3 non-binding +1 votes and no 0 or -1 votes. > > > > Thanks to all who helped with the proposal and cast the vote! > > > > Here's a vote tally: > > > > Non-binding +1s: > > Jim Jagielski > > Marvin Humphrey > > Bertrand Delacretaz > > > > Binding +1s: > > P. Taylor Goetz > > Justin Mclean > > Greg Stein > > Jean Baptiste Onofré > > > > No 0 or -1 votes. > > > > Thanks, > > Sterling > > > > PS: I didn't realize that I could also vote on the proposal until too > > late, but for the record, I'm also a +1 :-) > > > > - > > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > > For additional commands, e-mail: general-h...@incubator.apache.org > > > > >
Re: [VOTE] Graduate Apache Kylin from the Apache Incubator
Yes, SGA is required. John On Tue, Oct 20, 2015 at 2:41 PM Henry Saputra wrote: > HI Ted, > > I apologize for the confusing message, I was typing it with mobile device. > > I was trying to figure out whether an SGA is needed when the initial > code coming as part of incubator code dump with ASF license. > I.E: when a project entering incubator and the original code is > donated from eBay into ASF git which already come with ASF license. > > As I remember SGA is needed not for new projects but for new > contributions from organizations after initial code submission was > made. > I just want to make sure my understanding is correct. > > - Henry > > On Tue, Oct 20, 2015 at 11:15 AM, Ted Dunning > wrote: > > Henry, > > > > I think that there was a garble in your message. > > > > Did you mean to point out that the original code import was code that was > > ASL licensed by eBay? > > > > And then if that is right, does that mean your question was "Is an SGA > > necessary to import ASL code?" > > > > > > > > On Tue, Oct 20, 2015 at 7:27 AM, Henry Saputra > > wrote: > > > >> But Kylin come as whole as original code dump as part of going to > >> incubator. > >> Does eBay still need to file software grant for it? > >> > >> On Tuesday, October 20, 2015, Bertrand Delacretaz < > bdelacre...@apache.org> > >> wrote: > >> > >> > On Tue, Oct 20, 2015 at 3:08 AM, John D. Ament >> > > wrote: > >> > > It looks weird to me that the date for the SGA is ambiguous. No one > >> has > >> > a > >> > > more exact date than 2015? > >> > > >> > I had a look under private/documents/grants/ and I don't see any grant > >> > with "ebay" or "kylin" in its filename - it might have been filed > >> > under a different name of course. > >> > > >> > http://wiki.apache.org/incubator/KylinProposal says "Kylin has been > >> > under development since 2013 by a team of engineers at eBay Inc. It is > >> > currently hosted on Github.com under an Apache license at > >> > https://github.com/KylinOLAP/Kylin";. > >> > > >> > So it looks like a software grant should have been filed - can someone > >> > clarify the filing date and name of that document so that we can > >> > double check? > >> > > >> > -Bertrand > >> > > >> > - > >> > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > >> > > >> > For additional commands, e-mail: general-h...@incubator.apache.org > >> > > >> > > >> > > >> > > - > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org > >
Re: [VOTE] Graduate Apache Kylin from the Apache Incubator
Thanks John, I will work with the Kylin and eBay team to get it filed. - Henry On Tue, Oct 20, 2015 at 2:52 PM, John D. Ament wrote: > Yes, SGA is required. > > John > > On Tue, Oct 20, 2015 at 2:41 PM Henry Saputra > wrote: > >> HI Ted, >> >> I apologize for the confusing message, I was typing it with mobile device. >> >> I was trying to figure out whether an SGA is needed when the initial >> code coming as part of incubator code dump with ASF license. >> I.E: when a project entering incubator and the original code is >> donated from eBay into ASF git which already come with ASF license. >> >> As I remember SGA is needed not for new projects but for new >> contributions from organizations after initial code submission was >> made. >> I just want to make sure my understanding is correct. >> >> - Henry >> >> On Tue, Oct 20, 2015 at 11:15 AM, Ted Dunning >> wrote: >> > Henry, >> > >> > I think that there was a garble in your message. >> > >> > Did you mean to point out that the original code import was code that was >> > ASL licensed by eBay? >> > >> > And then if that is right, does that mean your question was "Is an SGA >> > necessary to import ASL code?" >> > >> > >> > >> > On Tue, Oct 20, 2015 at 7:27 AM, Henry Saputra >> > wrote: >> > >> >> But Kylin come as whole as original code dump as part of going to >> >> incubator. >> >> Does eBay still need to file software grant for it? >> >> >> >> On Tuesday, October 20, 2015, Bertrand Delacretaz < >> bdelacre...@apache.org> >> >> wrote: >> >> >> >> > On Tue, Oct 20, 2015 at 3:08 AM, John D. Ament > >> > > wrote: >> >> > > It looks weird to me that the date for the SGA is ambiguous. No one >> >> has >> >> > a >> >> > > more exact date than 2015? >> >> > >> >> > I had a look under private/documents/grants/ and I don't see any grant >> >> > with "ebay" or "kylin" in its filename - it might have been filed >> >> > under a different name of course. >> >> > >> >> > http://wiki.apache.org/incubator/KylinProposal says "Kylin has been >> >> > under development since 2013 by a team of engineers at eBay Inc. It is >> >> > currently hosted on Github.com under an Apache license at >> >> > https://github.com/KylinOLAP/Kylin";. >> >> > >> >> > So it looks like a software grant should have been filed - can someone >> >> > clarify the filing date and name of that document so that we can >> >> > double check? >> >> > >> >> > -Bertrand >> >> > >> >> > - >> >> > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org >> >> > >> >> > For additional commands, e-mail: general-h...@incubator.apache.org >> >> > >> >> > >> >> > >> >> >> >> - >> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org >> For additional commands, e-mail: general-h...@incubator.apache.org >> >> - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [DISCUSS] Eagle incubator proposal
Thanks Taylor. I will add you to the mentor list. On 10/20/15, 11:58 AM, "P. Taylor Goetz" wrote: >I should also have some improved bandwidth both now that Kylin is nearing >graduation and for other reasons. I¹ve been bogged down recently, but >that¹s starting to change. > >If more mentors are desired, I¹d be willing to help in that respect. > >-Taylor > >> On Oct 20, 2015, at 11:49 AM, Henry Saputra >>wrote: >> >> Hi Ted, >> >> Since Kylin almost ready to graduate, I have more bandwidth to help >>with Eagle. >> >> But, you are right that current proposed mentors for Eagle seemed to >> be very busy with other podlings, so 1 or 2 additional mentors would >> be great. >> >> The good news is that the team consist some people from Kylin, for >> example Luke, which done great job helping Kylin to understand working >> with Apache way. >> So we have some help from initial committers who have done the rodeo >>before. >> >> - Henry >> >> On Mon, Oct 19, 2015 at 9:00 AM, Ted Dunning >>wrote: >>> I would suggest that Owen O'Malley has not had enough time to be a >>>viable >>> mentor recently and should not be on the list of mentors. >>> >>> Henry and Julian are good if their schedules permit. Henry, I know has >>> been mentoring a number of projects lately. >>> >>> >>> >>> On Mon, Oct 19, 2015 at 8:40 AM, Jean-Baptiste Onofré >>> wrote: >>> Hi Arun, very interesting proposal. I may see some possible interaction with Falcon. In Falcon, we have HDFS files (and Hive/HBase) monitoring (with a kind of Change Data Capture), etc. So, I see a different perspective in Eagle, but Eagle could also leverage Falcon somehow. Regards JB On 10/19/2015 05:33 PM, Manoharan, Arun wrote: > Hello Everyone, > > My name is Arun Manoharan. Currently a product manager in the >Analytics > platform team at eBay Inc. > > I would like to start a discussion on Eagle and its joining the ASF >as an > incubation project. > > Eagle is a Monitoring solution for Hadoop to instantly identify >access to > sensitive data, recognize attacks, malicious activities and take >actions in > real time. Eagle supports a wide variety of policies on HDFS data >and Hive. > Eagle also provides machine learning models for detecting anomalous >user > behavior in Hadoop. > > The proposal is available on the wiki here: > https://wiki.apache.org/incubator/EagleProposal > > The text of the proposal is also available at the end of this email. > > Thanks for your time and help. > > Thanks, > Arun > > > > Eagle > > Abstract > Eagle is an Open Source Monitoring solution for Hadoop to instantly > identify access to sensitive data, recognize attacks, malicious >activities > in hadoop and take actions. > > Proposal > Eagle audits access to HDFS files, Hive and HBase tables in real >time, > enforces policies defined on sensitive data access and alerts or >blocks > user¹s access to that sensitive data in real time. Eagle also >creates user > profiles based on the typical access behaviour for HDFS and Hive and >sends > alerts when anomalous behaviour is detected. Eagle can also import > sensitive data information classified by external classification >engines to > help define its policies. > > Overview of Eagle > Eagle has 3 main parts. > 1.Data collection and storage - Eagle collects data from various >hadoop > logs in real time using Kafka/Yarn API and uses HDFS and HBase for >storage. > 2.Data processing and policy engine - Eagle allows users to create > policies based on various metadata properties on HDFS, Hive and >HBase data. > 3.Eagle services - Eagle services include policy manager, query >service > and the visualization component. Eagle provides intuitive user >interface to > administer Eagle and an alert dashboard to respond to real time >alerts. > > Data Collection and Storage: > Eagle provides programming API for extending Eagle to integrate any >data > source into Eagle policy evaluation framework. For example, Eagle >hdfs > audit monitoring collects data from Kafka which is populated from >namenode > log4j appender or from logstash agent. Eagle hive monitoring >collects hive > query logs from running job through YARN API, which is designed to be > scalable and fault-tolerant. Eagle uses HBase as storage for storing > metadata and metrics data, and also supports relational database >through > configuration change. > > Data Processing and Policy Engine: > Processing Engine: Eagle provides stream processing API which is an > abstraction of Apache Storm. It can also be extended to other >streaming >
Re: [VOTE] Graduate Apache Kylin from the Apache Incubator
There's already one signed this week(oct 17) and I'm working on the final confirmation. Thanks. Regards! Luke Han _ From: Henry Saputra Sent: 星期三, 十月 21, 2015 06:50 Subject: Re: [VOTE] Graduate Apache Kylin from the Apache Incubator To: Thanks John, I will work with the Kylin and eBay team to get it filed. - Henry On Tue, Oct 20, 2015 at 2:52 PM, John D. Ament wrote: > Yes, SGA is required. > > John > > On Tue, Oct 20, 2015 at 2:41 PM Henry Saputra > wrote: > >> HI Ted, >> >> I apologize for the confusing message, I was typing it with mobile device. >> >> I was trying to figure out whether an SGA is needed when the initial >> code coming as part of incubator code dump with ASF license. >> I.E: when a project entering incubator and the original code is >> donated from eBay into ASF git which already come with ASF license. >> >> As I remember SGA is needed not for new projects but for new >> contributions from organizations after initial code submission was >> made. >> I just want to make sure my understanding is correct. >> >> - Henry >> >> On Tue, Oct 20, 2015 at 11:15 AM, Ted Dunning >> wrote: >> > Henry, >> > >> > I think that there was a garble in your message. >> > >> > Did you mean to point out that the original code import was code that was >> > ASL licensed by eBay? >> > >> > And then if that is right, does that mean your question was "Is an SGA >> > necessary to import ASL code?" >> > >> > >> > >> > On Tue, Oct 20, 2015 at 7:27 AM, Henry Saputra >> > wrote: >> > >> >> But Kylin come as whole as original code dump as part of going to >> >> incubator. >> >> Does eBay still need to file software grant for it? >> >> >> >> On Tuesday, October 20, 2015, Bertrand Delacretaz < >> bdelacre...@apache.org> >> >> wrote: >> >> >> >> > On Tue, Oct 20, 2015 at 3:08 AM, John D. Ament > >> > > wrote: >> >> > > It looks weird to me that the date for the SGA is ambiguous. No one >> >> has >> >> > a >> >> > > more exact date than 2015? >> >> > >> >> > I had a look under private/documents/grants/ and I don't see any grant >> >> > with "ebay" or "kylin" in its filename - it might have been filed >> >> > under a different name of course. >> >> > >> >> > http://wiki.apache.org/incubator/KylinProposal says "Kylin has been >> >> > under development since 2013 by a team of engineers at eBay Inc. It is >> >> > currently hosted on Github.com under an Apache license at >> >> > https://github.com/KylinOLAP/Kylin";. >> >> > >> >> > So it looks like a software grant should have been filed - can someone >> >> > clarify the filing date and name of that document so that we can >> >> > double check? >> >> > >> >> > -Bertrand >> >> > >> >> > - >> >> > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org >> >> > >> >> > For additional commands, e-mail: general-h...@incubator.apache.org >> >> > >> >> > >> >> > >> >> >> >> - >> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org >> For additional commands, e-mail: general-h...@incubator.apache.org >> >> - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [VOTE] Graduate Apache Kylin from the Apache Incubator
Thanks Luke. If you can get a reference out to people and have the kylin.xml updated to reflect the date, I'd be happy to add a +1 but otherwise its -1 for me for now. On Tue, Oct 20, 2015 at 8:30 PM Luke Han wrote: > There's already one signed this week(oct 17) and I'm working on the final > confirmation. > Thanks. > > Regards! > Luke Han > > _ > From: Henry Saputra > Sent: 星期三, 十月 21, 2015 06:50 > Subject: Re: [VOTE] Graduate Apache Kylin from the Apache Incubator > To: > > > Thanks John, I will work with the Kylin and eBay team to get it filed. > > - Henry > > On Tue, Oct 20, 2015 at 2:52 PM, John D. Ament > wrote: > > Yes, SGA is required. > > > > John > > > > On Tue, Oct 20, 2015 at 2:41 PM Henry Saputra > > wrote: > > > >> HI Ted, > >> > >> I apologize for the confusing message, I was typing it with mobile > device. > >> > >> I was trying to figure out whether an SGA is needed when the initial > >> code coming as part of incubator code dump with ASF license. > >> I.E: when a project entering incubator and the original code is > >> donated from eBay into ASF git which already come with ASF license. > >> > >> As I remember SGA is needed not for new projects but for new > >> contributions from organizations after initial code submission was > >> made. > >> I just want to make sure my understanding is correct. > >> > >> - Henry > >> > >> On Tue, Oct 20, 2015 at 11:15 AM, Ted Dunning > >> wrote: > >> > Henry, > >> > > >> > I think that there was a garble in your message. > >> > > >> > Did you mean to point out that the original code import was code that > was > >> > ASL licensed by eBay? > >> > > >> > And then if that is right, does that mean your question was "Is an SGA > >> > necessary to import ASL code?" > >> > > >> > > >> > > >> > On Tue, Oct 20, 2015 at 7:27 AM, Henry Saputra < > henry.sapu...@gmail.com> > >> > wrote: > >> > > >> >> But Kylin come as whole as original code dump as part of going to > >> >> incubator. > >> >> Does eBay still need to file software grant for it? > >> >> > >> >> On Tuesday, October 20, 2015, Bertrand Delacretaz < > >> bdelacre...@apache.org> > >> >> wrote: > >> >> > >> >> > On Tue, Oct 20, 2015 at 3:08 AM, John D. Ament < > johndam...@apache.org > >> >> > > wrote: > >> >> > > It looks weird to me that the date for the SGA is ambiguous. No > one > >> >> has > >> >> > a > >> >> > > more exact date than 2015? > >> >> > > >> >> > I had a look under private/documents/grants/ and I don't see any > grant > >> >> > with "ebay" or "kylin" in its filename - it might have been filed > >> >> > under a different name of course. > >> >> > > >> >> > http://wiki.apache.org/incubator/KylinProposal says "Kylin has > been > >> >> > under development since 2013 by a team of engineers at eBay Inc. > It is > >> >> > currently hosted on Github.com under an Apache license at > >> >> > https://github.com/KylinOLAP/Kylin";. > >> >> > > >> >> > So it looks like a software grant should have been filed - can > someone > >> >> > clarify the filing date and name of that document so that we can > >> >> > double check? > >> >> > > >> >> > -Bertrand > >> >> > > >> >> > > - > >> >> > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > >> >> > > >> >> > For additional commands, e-mail: general-h...@incubator.apache.org > >> >> > > >> >> > > >> >> > > >> >> > >> > >> - > >> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > >> For additional commands, e-mail: general-h...@incubator.apache.org > >> > >> > > - > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org
Re: [DISCUSS] Eagle incubator proposal
On Tue, Oct 20, 2015 at 4:14 PM, Manoharan, Arun wrote: > Thanks Taylor. I will add you to the mentor list. > Arun, Can you also do a scrub of the mentor list by asking each of the mentors whether they have been able to support other groups that they are mentoring. If they don't answer, or if they can't say that they have been supportive (at least to the extent of signing off project reports), then please remove them from your list.
Re: [DISCUSS] Eagle incubator proposal
Hi Ted, Thanks for your concern, but we have had discussions with all proposed mentors before to ask for their availability and willingness to actively mentor this project. I think we are good with existing proposed mentors. - Henry On Tue, Oct 20, 2015 at 9:10 PM, Ted Dunning wrote: > On Tue, Oct 20, 2015 at 4:14 PM, Manoharan, Arun > wrote: > >> Thanks Taylor. I will add you to the mentor list. >> > > > Arun, > > Can you also do a scrub of the mentor list by asking each of the mentors > whether they have been able to support other groups that they are > mentoring. If they don't answer, or if they can't say that they have been > supportive (at least to the extent of signing off project reports), then > please remove them from your list. - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [DISCUSS] Eagle incubator proposal
Hi Arun This looks really good and fills some obvious gaps in the security landscape. Happy to contribute anyway you want. All the best!!! Bosco On 10/20/15, 8:02 AM, "Alex Karasulu" wrote: >Hi Arun, > >Eagle sounds very promising. I just had a discussion with someone about >this exact need. I do however agree with Greg on the name. As far as I can >see, besides the name, your weakest point is the all eBay employed team. >It's not a blocker and can be fixed during incubation. Good luck to you. > >Alex > > >On Tue, Oct 20, 2015 at 5:51 PM, Manoharan, Arun >wrote: > >> Hi Greg, >> >> Thank you for reviewing the proposal. >> >> Originally we thought Eagle might be trademarked by someone already but I >> went thru eBay legal team to get the clearance for the name to be used. We >> will look into it again to see if there will be potential problems. >> >> Thanks, >> Arun >> >> On 10/20/15, 1:52 AM, "Greg Stein" wrote: >> >> >Hey there, Arun! ... I have no commentary on the proposal itself, as it >> >looks like a great proposal. I would suggest being a bit wary of the name, >> >as "Eagle" is a *very* popular PCB design program. >> > >> >On Mon, Oct 19, 2015 at 10:33 AM, Manoharan, Arun >> >wrote: >> > >> >> Hello Everyone, >> >> >> >> My name is Arun Manoharan. Currently a product manager in the Analytics >> >> platform team at eBay Inc. >> >> >> >> I would like to start a discussion on Eagle and its joining the ASF as >> >>an >> >> incubation project. >> >> >> >> Eagle is a Monitoring solution for Hadoop to instantly identify access >> >>to >> >> sensitive data, recognize attacks, malicious activities and take >> >>actions in >> >> real time. Eagle supports a wide variety of policies on HDFS data and >> >>Hive. >> >> Eagle also provides machine learning models for detecting anomalous user >> >> behavior in Hadoop. >> >> >> >> The proposal is available on the wiki here: >> >> https://wiki.apache.org/incubator/EagleProposal >> >> >> >> The text of the proposal is also available at the end of this email. >> >> >> >> Thanks for your time and help. >> >> >> >> Thanks, >> >> Arun >> >> >> >> >> >> >> >> Eagle >> >> >> >> Abstract >> >> Eagle is an Open Source Monitoring solution for Hadoop to instantly >> >> identify access to sensitive data, recognize attacks, malicious >> >>activities >> >> in hadoop and take actions. >> >> >> >> Proposal >> >> Eagle audits access to HDFS files, Hive and HBase tables in real time, >> >> enforces policies defined on sensitive data access and alerts or blocks >> >> user¹s access to that sensitive data in real time. Eagle also creates >> >>user >> >> profiles based on the typical access behaviour for HDFS and Hive and >> >>sends >> >> alerts when anomalous behaviour is detected. Eagle can also import >> >> sensitive data information classified by external classification >> >>engines to >> >> help define its policies. >> >> >> >> Overview of Eagle >> >> Eagle has 3 main parts. >> >> 1.Data collection and storage - Eagle collects data from various hadoop >> >> logs in real time using Kafka/Yarn API and uses HDFS and HBase for >> >>storage. >> >> 2.Data processing and policy engine - Eagle allows users to create >> >> policies based on various metadata properties on HDFS, Hive and HBase >> >>data. >> >> 3.Eagle services - Eagle services include policy manager, query service >> >> and the visualization component. Eagle provides intuitive user >> >>interface to >> >> administer Eagle and an alert dashboard to respond to real time alerts. >> >> >> >> Data Collection and Storage: >> >> Eagle provides programming API for extending Eagle to integrate any data >> >> source into Eagle policy evaluation framework. For example, Eagle hdfs >> >> audit monitoring collects data from Kafka which is populated from >> >>namenode >> >> log4j appender or from logstash agent. Eagle hive monitoring collects >> >>hive >> >> query logs from running job through YARN API, which is designed to be >> >> scalable and fault-tolerant. Eagle uses HBase as storage for storing >> >> metadata and metrics data, and also supports relational database through >> >> configuration change. >> >> >> >> Data Processing and Policy Engine: >> >> Processing Engine: Eagle provides stream processing API which is an >> >> abstraction of Apache Storm. It can also be extended to other streaming >> >> engines. This abstraction allows developers to assemble data >> >> transformation, filtering, external data join etc. without physically >> >>bound >> >> to a specific streaming platform. Eagle streaming API allows developers >> >>to >> >> easily integrate business logic with Eagle policy engine and internally >> >> Eagle framework compiles business logic execution DAG into program >> >> primitives of underlying stream infrastructure e.g. Apache Storm. For >> >> example, Eagle HDFS monitoring transforms audit log from Namenode to >> >>object >> >> and joins sensitivity metadata, security zone metadata which are >> >>generated
Re: Mentor disengagement - a suggestion
On Mon, Oct 19, 2015 at 6:35 PM, Roman Shaposhnik wrote: > > As a means of refocusing the mentors' efforts, and keeping them engaged, > I'd > > like to encourage each mentor (or group of mentors) to consider writing a > > running report (ie, evolving, updated every quarter) based on > > > https://community.apache.org/apache-way/apache-project-maturity-model.html > > where they evaluate each point on the maturity model, as a path towards > > graduation. This gives a concrete target, and a lens through which to > view > > the podling's progress towards that target. > > +1 this will definitely help with both responsibilities. > > But, I've got to wonder: as long as it is just a suggestion what's to > compel > a mentor to actually spend quite a bit of time on doing that? > Nothing. Is compulsion the only logic here?
Re: Draft Report October 2015 - please review
filed. Will sned to mailing list now. On Tue, Oct 20, 2015 at 8:56 AM, Ted Dunning wrote: > > I will file shortly. > > On Tue, Oct 20, 2015 at 5:15 AM, Bertrand Delacretaz < > bdelacre...@apache.org> wrote: > >> On Tue, Oct 20, 2015 at 1:26 PM, Marvin Humphrey >> wrote: >> > ...Ted has certainly earned his money there this month >> >> Definitely - there was nothing hidden behind my comment, it's just >> that it's good to make sure the report is filed in time for the board >> meeting (and I'm your shepherd this month, for the board report). >> >> -Bertrand >> >> - >> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org >> For additional commands, e-mail: general-h...@incubator.apache.org >> >> >
Re: [VOTE] Graduate Apache Kylin from the Apache Incubator
On Wed, Oct 21, 2015 at 2:30 AM, Luke Han wrote: > ...There's already one signed this week(oct 17) and I'm working on the final > confirmation... ok so let's wait until this is filed with the ASF to proceed with graduation. -Bertrand - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
