+1 (binding) On Wed, May 25, 2016 at 4:04 PM, John D. Ament <johndam...@apache.org> wrote:
> +1 > > On Wed, May 25, 2016 at 4:41 PM Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: > > > Hi all, > > > > following the discussion thread, I'm now calling a vote to accept > > CarbonData into the Incubator. > > > > [ ] +1 Accept CarbonData into the Apache Incubator > > [ ] +0 Abstain > > [ ] -1 Do not accept CarbonData into the Apache Incubator, because ... > > > > This vote is open for 72 hours. > > > > The proposal follows, you can also access the wiki page: > > https://wiki.apache.org/incubator/CarbonDataProposal > > > > Thanks ! > > Regards > > JB > > > > = Apache CarbonData = > > > > == Abstract == > > > > Apache CarbonData is a new Apache Hadoop native file format for faster > > interactive > > query using advanced columnar storage, index, compression and encoding > > techniques > > to improve computing efficiency, in turn it will help speedup queries an > > order of > > magnitude faster over PetaBytes of data. > > > > CarbonData github address: https://github.com/HuaweiBigData/carbondata > > > > == Background == > > > > Huawei is an ICT solution provider, we are committed to enhancing > > customer experiences for telecom carriers, enterprises, and consumers on > > big data, In order to satisfy the following customer requirements, we > > created a new Hadoop native file format: > > > > * Support interactive OLAP-style query over big data in seconds. > > * Support fast query on individual record which require touching all > > fields. > > * Fast data loading speed and support incremental load in period of > > minutes. > > * Support HDFS so that customer can leverage existing Hadoop cluster. > > * Support time based data retention. > > > > Based on these requirements, we investigated existing file formats in > > the Hadoop eco-system, but we could not find a suitable solution that > > satisfying requirements all at the same time, so we start designing > > CarbonData. > > > > == Rationale == > > > > CarbonData contains multiple modules, which are classified into two > > categories: > > > > 1. CarbonData File Format: which contains core implementation for file > > format such as columnar,index,dictionary,encoding+compression,API for > > reading/writing etc. > > 2. CarbonData integration with big data processing framework such as > > Apache Spark, Apache Hive etc. Apache Beam is also planned to abstract > > the execution runtime. > > > > === CarbonData File Format === > > > > CarbonData file format is a columnar store in HDFS, it has many features > > that a modern columnar format has, such as splittable, compression > > schema ,complex data type etc. And CarbonData has following unique > > features: > > > > ==== Indexing ==== > > > > In order to support fast interactive query, CarbonData leverage indexing > > technology to reduce I/O scans. CarbonData files stores data along with > > index, the index is not stored separately but the CarbonData file itself > > contains the index. In current implementation, CarbonData supports 3 > > types of indexing: > > > > 1. Multi-dimensional Key (B+ Tree index) > > The Data block are written in sequence to the disk and within each > > data blocks each column block is written in sequence. Finally, the > > metadata block for the file is written with information about byte > > positions of each block in the file, Min-Max statistics index and the > > start and end MDK of each data block. Since, the entire data in the file > > is in sorted order, the start and end MDK of each data block can be used > > to construct a B+Tree and the file can be logically represented as a > > B+Tree with the data blocks as leaf nodes (on disk) and the remaining > > non-leaf nodes in memory. > > 2. Inverted index > > Inverted index is widely used in search engine. By using this index, > > it helps processing/query engine to do filtering inside one HDFS block. > > Furthermore, query acceleration for count distinct like operation is > > made possible when combining bitmap and inverted index in query time. > > 3. MinMax index > > For all columns, minmax index is created so that processing/query > > engine can skip scan that is not required. > > > > ==== Global Dictionary ==== > > > > Besides I/O reduction, CarbonData accelerates computation by using > > global dictionary, which enables processing/query engines to perform all > > processing on encoded data without having to convert the data (Late > > Materialization). We have observed dramatic performance improvement for > > OLAP analytic scenario where table contains many columns in string data > > type. The data is converted back to the user readable form just before > > processing/query engine returning results to user. > > > > ==== Column Group ==== > > > > Sometimes users want to perform processing/query on multi-columns in one > > table, for example, performing scan for individual record in > > troubleshooting scenario. In this case, row format is more efficient > > than columnar format since all columns will be touched by the workload. > > To accelerate this, CarbonData supports storing a group of column in row > > format, so data in column group is stored together and enable fast > > retrieval. > > > > ==== Optimized for multiple use cases ==== > > > > CarbonData indices and dictionary is highly configurable. To make > > storage optimized for different use cases, user can configure what to > > index, so user can decide and tune the format before loading data into > > CarbonData. > > > > For example > > > > || Use Case || Supporting Features || > > || Interactive OLAP query || Columnar format, Multi-dimensional Key (B+ > > Tree index), Minmax index, Inverted index || > > || High throughput scan || Global dictionary, Minmax index || > > || Low latency point query || Multi-dimensional Key (B+ Tree index), > > Partitioning || > > || Individual record query || Column group, Global dictionary || > > > > === BigData Processing Framework Integration === > > > > * CarbonData provides InputFormat/OutputFormat interfaces for > > Reading/Writing data from the CarbonData files and at the same time > > provides abstract API for processing data stored as Carbondata format > > with data processing framework. > > * CarbonData provides deep integration with Apache Spark including > > predicate push down, column pruning, aggregation push down etc. So users > > can use Spark SQL to connect and query from CarbonData. > > * CarbonData can integrate with various big data Query/Processing > > framework on Hadoop eco-system such as Apache Spark,Apache Hive etc. > > > > Example: > > > > > https://github.com/HuaweiBigData/carbondata/blob/master/examples/src/main/scala/org/carbondata/examples/CarbonExample.scala > > > > == Initial Goals == > > > > Our initial goals are to bring CarbonData into the ASF, transition > > internal engineering processes into the open, and foster a collaborative > > development model according to the "Apache Way". > > > > == Current Status == > > > > CarbonData is production ready and already provide a large set of > features. > > The current license is already Apache 2.0. > > > > == Meritocracy == > > > > We intend to radically expand the initial developer and user community > > by running the project in accordance with the "Apache Way". Users and > > new contributors will be treated with respect and welcomed. By > > participating in the community and providing quality patches/support > > that move the project forward, they will earn merit. They also will be > > encouraged to provide non-code contributions (documentation, events, > > community management, etc.) and will gain merit for doing so. Those with > > a proven support and quality track record will be encouraged to become > > committers. > > > > == Community == > > > > If CarbonData is accepted for incubation, the primary initial goal is to > > build a large community. We really trust that CarbonData will become a > > key project for big data column-like platforms, and so, we bet on a > > large community of users and developers. > > > > == Known Risks == > > > > Development has been sponsored mostly by a one company.For the project > > to fully transition to the Apache Way governance model, development must > > shift towards the meritocracy-centric model of growing a community of > > contributors balanced with the needs for extreme stability and core > > implementation coherency. > > > > == Orphaned products == > > > > Huawei is fully committed CarbonData. Moreover, Huawei has a vested > > interest in making CarbonData succeed by driving its close integration > > with sister ASF projects. We expect this to further reduces the risk of > > orphaning the product. > > > > == Inexperience with Open Source == > > > > Huawei has been developing and using open source software since a long > > time. Additionally, several ASF veterans agreed to mentor the project > > and are listed in this proposal. The project will rely on their guidance > > and collective wisdom to quickly transition the entire team of initial > > committers towards practicing the Apache Way. > > > > == Reliance on Salaried Developers == > > > > Most of the contributors are paid to work in big data space. While they > > might wander from their current employers, they are unlikely to venture > > far from their core expertises and thus will continue to be engaged with > > the project regardless of their current employers. > > > > == An Excessive Fascination with the Apache Brand == > > > > While we intend to leverage the Apache ‘branding’ when talking to other > > projects as testament of our project’s ‘neutrality’, we have no plans > > for making use of Apache brand in press releases nor posting billboards > > advertising acceptance of CarbonData into Apache Incubator. > > > > == Initial Source == > > > > https://github.com/HuaweiBigData/carbondata.git > > > > == External Dependencies == > > > > All external dependencies are licensed under an Apache 2.0 license or > > Apache-compatible license. As we grow the Carbondata community we will > > configure our build process to require and validate all contributions > > and dependencies are licensed under the Apache 2.0 license or are under > > an Apache-compatible license. > > > > * Apache Spark > > * Apache Hadoop > > * Apache Maven > > * Apache Commons > > * Apache Log4j > > * Apache Thrift > > * Apache Zookeeper > > * Scala > > * Snappy > > * Kettle (Pentaho) > > * Eigenbase > > * Fastutil > > * GSON > > * Jmockit > > * Junit > > > > == Required Resources == > > > > === Mailing lists === > > > > * priv...@carbondata.incubator.apache.org (moderated subscriptions) > > * comm...@carbondata.incubator.apache.org > > * d...@carbondata.incubator.apache.org > > * iss...@carbondata.incubator.apache.org > > > > === Git Repository === > > > > * https://git-wip-us.apache.org/repos/asf/incubator-carbondata.git > > > > === Issue Tracking === > > > > * JIRA Project CarbonData (CarbonData) > > > > === Initial Committers === > > > > * Liang Chenliang > > * Jean-Baptiste Onofré > > * Henry Saputra > > * Uma Maheswara Rao G > > * Jenny MA > > * Jacky Likun > > * Vimal Das Kammath > > * Jarray Qiuheng > > > > === Affiliations === > > > > * Huawei: Liang Chenliang > > * Talend: Jean-Baptiste Onofré > > * Ebay: Henry Saputra > > * Intel: Uma Maheswara Rao G > > > > === Sponsors === > > > > === Champion === > > > > * Jean-Baptiste Onofré - Apache Member > > > > === Mentors === > > > > * Henry Saputra (eBay) > > * Jean-Baptiste Onofré (Talend) > > * Uma Maheswara Rao G (Intel) > > > > === Sponsoring Entity === > > > > The Apache Incubator > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > > For additional commands, e-mail: general-h...@incubator.apache.org > > > > >
