+1 (binding) On Feb 5, 2013, at 8:18 AM, Mattmann, Chris A (388J) wrote:
> Hi Folks, > > OK, now that discussion has settled down, I'd like to call a VOTE for > acceptance of Apache Open Climate Workbench into the Incubator. > I'll leave the VOTE open the rest of the week and close it out next > Monday, February 11th early am PT. > > [ ] +1 Accept Apache Open Climate Workbench into the Incubator > [ ] +0 Don't care. > [ ] -1 Don't accept Apache Open Climate Workbench into the Incubator > because... > > Full proposal is pasted at the bottom of this email. Only VOTEs from > Incubator PMC members are binding, but all are welcome to express their > thoughts. > > Thank you! > > Cheers, > Chris > > P.S. Here's my +1 (binding) > > ------------- > = Apache Open Climate Workbench, tool for scalable comparison of remote > sensing observations to climate model outputs, regionally and globally. = > === Abstract === > The Apache Open Climate Workbench proposal desires to contribute an > existing community of software related to the analysis and evaluation of > climate models, and related to the use of remote sensing data in that > process. > > Specifically, we will bring a fundamental software toolkit for analysis > and evaluation of climate model output against remote sensing data. The > toolkit is called the [[http://rcmes.jpl.nasa.gov|Regional Climate Model > Evaluation System (RCMES)]]. RCMES provides two fundamental components for > the easy, intuitive comparison of climate model output against remote > sensing data. The first component called RCMED (for "Regional Climate > Model Evaluation Database") is a scalable cloud database that decimates > remote sensing data and renalysis data related to climate using Apache > OODT extractors, Apache Tika, etc. These transformations make > traditionally heterogeneous upstream remote sensing data and climate model > output homogeneous and unify them into a data point model of the form > (lat, lng, time, value, height) on a per parameter basis. Latitude (lat) > and Longitude (lng) are in WGS84 format, but can be reformatted on the > fly. time is in ISO 8601 format, a string sortable format independent of > underlying store. value carries with it units, related to interpretation > and height allows for different values for different atmospheric vertical > levels. All of RCMES is built on Apache OODT, Apache Sqoop/Apache Hadoop > and Apache Hive, along with hooks to PostGIS and MySQL (traditional > relational databases). The second component of the system, RCMET (for > "Regional Climate Model Evaluation Toolkit") provides facilities for > connecting to RCMED, dynamically obtaining remote sensing data for a > space/time region of interest, grabbing associated model output (that the > user brings, or from the Earth System Grid Federation) of the same form, > and then regridding the remote sensing data to be on the model output > grid, or the model output to be on the remote sensing data grid. The > regridded data spatially is then temporally regridded using techniques > including seasonal cycle compositing (e.g., all summer months, all > Januaries, etc.), or by daily, monthly, etc. The uniform model output and > remote sensing data are then analyzed using pluggable metrics, e.g., > Probability Distribution Functions (PDFs), Root Mean Squared Error (RMSE), > Bias, and other (possibly user-defined) techniques, computing an analyzed > comparison or evaluation. This evaluation is then visualized by plugging > in to the NCAR NCL library for producing static plots (histograms, time > series, etc.) > > We also have performed a great deal of work in packaging RCMES to make the > system easy to deploy. We have working Virtual Machines (VMWare VMX and > Virtual Box OVA compatible formats) and we also have an installer built on > Python Buildout (http://buildout.org/) called "Easy RCMET" for dynamically > constructing the RCMET toolkit. > > RCMES is currently supporting a number of recognized climate projects of > (inter-)national significance. In particular, RCMES is supporting the > [[http://www.globalchange.gov/what-we-do/assessment|U.S. National Climate > Assessment (NCA) activities]] on behalf of NASA's contribution to the NCA; > is working with the [[http://www.narccap.ucar.edu/|North American Regional > Climate Change Assessment Program (NARCCAP)]]; and is also working with > the International [[http://wcrp-cordex.ipsl.jussieu.fr/|Coordinated > Regional Downscaling Experiment (CORDEX)]]. > > === Proposal === > We propose to transition the RCMES software community, which includes > developers of the RCMET and RCMED software, along with users of RCMES in > the CORDEX project across a variety of academic institutions, scientists > helping to improve the RCMES metrics, and visualizations, and regridding > algorithms, packagers making RCMES easier to install, and scientists > helping to lead some of these international projects that are already > using RCMES. > > We have been working on the RCMES project since 2009 funded initially by > the American Recovery and Reinvestment Act (ARRA) project out at NASA, and > then branching out into other sources of support and sustainability (NASA; > NSF, etc. -- see the > [[http://rcmes.jpl.nasa.gov/about/overview|acknowledgements]] section on > the RCMES website for a full list of supporting U.S. and international > partners). > > With the existing RCMES community at Apache, we will also work to > encourage other climate software projects e.g., Open Climate GIS, elements > of the Earth System Grid Federation, other NASA climate projects funded > under the Computational Modeling, Algorithms and Cyberinfrastructure > (CMAC) to contribute to the Open Climate Workbench here at Apache. > > RCMED is a Big Data project that combines several underlying Apache > software -- OODT, Tika, Hadoop, HIVE, and Sqoop -- and other related data > management software. Its primary language is Java; RCMET, on the other > hand, is a Python API, associated set of classes (framework), set of > Python Bottle Web services, and a PHP "Wizard"-based User Interface that > leverages Apache OODT Balance. > > === Background === > Bringing RCMES to Apache was the brain-child of Chris Mattmann, based on > his solid experience with Apache OODT and bringing it to the ASF. Chris > worked for a year to get the support of the JPL community including > approvals from the Software Release authority at JPL to release the > software. > > The initial code drop will include the RCMES SVN repository from JPL > including prior revisions. We anticipate also including a smaller package, > CDX, which contains some useful facilities for regridding, and command > line tools for manipulating large datasets, and working with OPeNDAP, etc. > > After the code drop, we will work with our developers, users, documentors, > and other members of the team to teach those unfamiliar with the Apache > way how it works around here at Apache. 30% of the community from RCMES > includes those intimately familiar with Apache including 6 ASF members -- > the other 70% include a range of scientific code developers, climate > scientists that use RCMES, program officers that will help make > documentation and slides for the code, and advocate for it in the > community. Their experience with Apache ranges from using various ASF > products, to contributing patches to them, to not using any ASF software > at all. > > With this diversity, we anticipate that while everything may not just work > turnkey out of the box, this represents a unique opportunity to > demonstrate Apache to the international community and to show the benefits > of its community and social models. That said, we also have a lot of ASF > experience to make sure everyone learns the Apache way. > > === Rationale === > We are bringing RCMES to Apache for a few reasons. First, we feel that it > will immediately enable our collaborators across a number of institutions > both nationally and internationally have the opportunity to work on a > common software base, and to improve it with contributions from their own > sites. Currently these are difficult to negotiate now because of varied > legal and contribution frameworks -- Apache allows us to simplify this to > a unified one. Second, using the ASF's world-wide mirroring system, we > will be able to deliver climate software broadly to the community as we > release it, rather than sneaker netting the software around or > establishing our own point release infrastructure. > > Bringing this project to Apache also immediately thrusts the ASF into the > thriving ecosystem of the Coordinated Regional Downscaling Experiment > (CORDEX), the US National Cimate Assessment, the North American Regional > Climate Change Assessment Program (the US contribution to CORDEX) and into > relevance for upcoming Intergovernmental Panel on Climate Change (IPCC) > assessment activities at a number of different institutions. We also seek > to help lead and encourage du jour standard development rather than top > down level dictating of standards for climate software and the ASF will > provide us a means for that. > > === Initial Goals === > The initial goals of the proposed project are: > > * Stand up a sustaining Apache-based community around the JPL RCMES > codebase. > * Active relationships and possible cooperation with related projects and > communities, including end user and scientific communities, CORDEX, > NARCCAP, US NCA, IPCC, ESG, etc. > * Active relationships and possible cooperation with existing Apache > communities, e.g., OODT, Hadoop/HIVE, Sqoop, Tika, SIS, etc. > * Initial Apache release. > * Leverage Apache Open Climate Workbench in climate activities at NASA, > in the international community as mentioned above, and beyond. > * Vetting all software licenses and making sure IP is clear (software > grant from JPL forthcoming). > > == Current Status == > === Meritocracy === > 30% of the proposed initial committers are familiar with the meritocracy > principles of Apache. As stated above this includes 6 ASF members. Of the > mentorship list, we have included Chris Douglas, a PMC member from Hadoop > and ASF member to help guide the community. Chris M. and Chris D. have > guided a number of projects through the Incubator over the years. The > other mentor includes Paul Ramirez, who has experience with the Incubator > -- he was a mentor for Apache Any23, and also was one of the PPMC members > and eventual mentor for Apache SIS. The 70% of proposed initial committers > that aren't as familiar with Apache have a broad range of experience in > other open source projects, and have a deep respect and affinity for the > foundation and the work that gets done here. The more experience ASF > mentors and project members will help to guide them. > === Community === > There is an existing, established community of developers and users of > this projet. This includes established communities including the > Coordinated Regional Downscaling Experiment, the U.S. National Climate > Assessment (NCA), the North American Regional Climate Change Assessment > Program (NARCCAP), and more. The Coordinated Regional Climate Downscaling > Experiment (CORDEX, http://wcrp.ipsl.jussieu.fr/cordex/about.html) is a > world wide effort of coordination of regional climate downscaling (RCD) > experiments driven by the World Climate Research Program (WRCP, > http://www.wcrp-climate.org/index.shtml). Recently, a large number of RCD > projects have been carried out on a large parts of the world. To maximize > the benefits of these research activities the WCRP designed a framework > (Giorgi, WMO-Bulletin, 2009) focused on "quality-control [of] data sets of > RCD-based information for the recent historical past and 21st century > projections, covering the majority of populated land regions on the > globe". CORDEX defined different control domains (up to 10, > http://cordex.dmi.dk/joomla/) for almost all the populated regions of the > world in a way to standardize the experiments and make them comparable. A > key region focused on Africa was also designated as the top priority by > WRCP. CORDEX also provides a a series of conventions and list of variables > that have to be followed by any project that wants to contribute to the > experiment. Each CORDEX region has a coordinator and regional and > international periodic meetings are scheduled in a way to ensure the > global well being. NARCCAP is the U.S. contribution to CORDEX. From the > [[http://www.globalchange.gov/what-we-do/assessment/nca-overview|US > National Climate Assessment]] site, work is "being conducted under the > auspices of the Global Change Research Act of 1990. The GCRA requires a > report to the President and the Congress every four years that integrates, > evaluates, and interprets the findings of the U.S. Global Change Research > Program (USGCRP); analyzes the effects of global change on the natural > environment, agriculture, energy production and use, land and water > resources, transportation, human health and welfare, human social systems, > and biological diversity; and analyzes current trends in global change, > both human-induced and natural, and projects major trends for the > subsequent 25 to 100 years." > > Apache Open Climate Workbench will support all of these communities above, > with an eye towards being a general purpose climate evaluation toolkit for > model output and remote sensing data. > > === Core Developers === > The initial set of developers comes from various NASA centers (JPL, and > Goddard Space Flight Center), NASA HQ, various Universities participating > in CORDEX (Cape Town, University of New South Wales), the Indian Institute > of Tropical Meteorology, the Free Univ. Berlin), the University of > California Los Angeles, and Howard University. As mentioned previously > several of our developers are Apache veterans and understand how it works > around here and for those that don't, they will have great mentorship. > > === Alignment === > Our proposed effort aligns with the U.S. National Climate Assessment, the > CORDEX effort, other efforts, including the Earth System Grid Federation, > other climate software including the Open Climate GIS toolkit, other > science portals for climate including the Climate Information Portal (CIP) > at the University of Cape Town, and other related projects. > > There are also a number of related Apache projects and dependencies, that > will be mentioned in the Relationships with Other Apache products section. > > == Known Risks == > === Orphaned products === > Our project has a history of funding support from JPL, NASA (Applications > program/ARRA, NCA, AIST), NSF (ExArch project), international investment > from collaborators, and from other funding sources. The funding sources > are all target future deliverables and activities, so there is little > chance this software and community will be orphaned. > > === Inexperience with Open Source === > All the initial developers have worked on open source before -- 30% of the > proposed initial community are experience with the ASF, and are PMC > members and committers on ASF project including 6 ASF members. Our mentors > are all ASF members, and we welcome any interest from additional Apache > mentors in the effort. Those 70% of our project that aren't Apache > committers, PMC members, or members will benefit from the leadership of > the other 30% of the project. > > === Homogenous Developers === > The initial developers come from a variety of backgrounds and with a > variety of needs for the proposed framework. Everyone is used to > communicating on mailing lists as the project spans timezones, > international institutions and centers of excellence for climate science. > > === Reliance on Salaried Developers === > All of the proposed initial developers are paid to work on this or related > projects, but the proposed project is not the primary task for anyone. > > === Relationships with Other Apache Products === > As mentioned above, RCMES and the Apache Open Climate Workbench already > depend on Apache OODT for facade interfaces to underlying data warehouses > for storing remote sensing data; and for metadata extraction and > transformation. The software also uses Apache Tika for this (through a > transitive dependency from OODT). In addition, we have hooks to Apache > Hadoop/HIVE, as well as dependencies on Apache Sqoop for dumping out > remote sensing data from MySQL and into HIVE. > > === A Excessive Fascination with the Apache Brand === > All of us are familiar with Apache and have a respect for its brand and > community. Chris Mattmann is a big proponent of Apache's sustainability > factor -- and it's ability to grow software communities, in an > institution, or funding source neutral manner. All of the community have > an extreme respect for Apache, including those in our communities who > aren't necessarily trained computer scientists, but are Scientists (big > "S", e.g., land, physical, Earth/Climate scientists). > > == Documentation == > The initial RCMES code base will come from the internal JPL Subversion > repository. The [[Regional Climate Model Evaluation System (RCMES) > project|http://rcmes.jpl.nasa.gov]] at [[JPL|http://www.jpl.nasa.gov/]] > has documentation on the existing software, including links to funding > support, communities, and other projects. We will continue to maintain > that site at JPL, which is part of the reason for rebranding the project > here at Apache with a new name to not interfere with the existing RCMES > one that has a following. In addition, we hope to evolve RCMES@JPL to have > increasing levels of dependency on Apache Open Climate Workbench, so that > we can incrementally transition with little impact to existing customers. > > In addition, JPL's [[http://cdx.jpl.nasa.gov|Climate Data eXchange (CDX)]] > website also has documentation on the existing software. > > == Initial Source == > The project will start with seed code donated by NASA JPL via Mattmann and > the rest of the initial committers, which consists of the Regional Climate > Model Evaluation System (RCMES) toolkit, and the Climate Data eXchange > (CDX) software. This will include the core Python API for RCMET, the RCMED > OODT catalog project (which stores remote sensing data to MySQL/PostGIS, > and HIVE), and the RCMED extractors for various climate formats. The > source will also include Easy-RCMET, the Python Buildout for RCMET. In > addition, we will bring along the CDX toolkit, which includes a CDX client > package that performs subsetting, access, regridding of climate data; and > also includes a Python Buildout installer of its own called Uber CDX. > > == Source and Intellectual Property Submission Plan == > All seed code and other contributions will be handled through the normal > Apache contribution process. Mattmann has been authorized by NASA JPL to > lead the contribution of RCMES and CDX into the Incubator via his existing > Apache CLA, and a Software Grant to be provided. > > We will also contact other related efforts for possible cooperation and > contributions. > > == External Dependencies == > Our project depends on a number of external libraries with various > licensing conditions. An initial list of such dependencies is shown below. > ||<tableclass="bodyTable"rowclass="b">'''Library''' ||'''License''' || > ||<rowclass="b">[[http://www.ncl.ucar.edu/|NCAR NCL]]||MIT compat|| > ||<rowclass="a">[[http://www.pyngl.ucar.edu/Nio.shtml|PyNIO]]||MIT compat|| > ||<rowclass="b">[[http://www.pyngl.ucar.edu/|PyNGL]]||MIT compat|| > ||<rowclass="a">[[http://matplotlib.sourceforge.net/|Matplotlib]]||Modified > PSF license|| > ||<rowclass="b">[[http://scipy.org/|Scipy]]||MIT compat|| > ||<rowclass="a">[[http://numpy.scipy.org/|NumPy]]||MIT compat|| > ||<rowclass="b">[[http://www.hdfgroup.org/HDF5/|HDF5]]||BSD|| > ||<rowclass="a">[[http://www.unidata.ucar.edu/software/netcdf/|NetCDF]]||MI > T|| > > > == Cryptography == > The project itself will not use cryptography, but it is possible that some > of the external software libraries will include cryptographic code to > handle features present in various science data formats. If we need to > provide an export control statement regarding cryptographic code per > Apache policy, we will follow a similar approach by Mattmann in > [[http://lucene.apache.org/nutch/|Apache Nutch]] and by Jukka Zitting lead > this effort in Apache Tika. Mattmann is familiar with this process. > > == Required Resources == > Mailing lists > > * d...@climate.incubator.apache.org > * comm...@climate.incubator.apache.org > * priv...@climate.incubator.apache.org > > Subversion Directory > > * https://svn.apache.org/repos/asf/incubator/climate > > Issue Tracking > > * JIRA CLIMATE (CLIMATE) > > Other Resources > > * CLIMATE Wiki http://cwiki.apache.org/CLIMATE > * Review Board instance - CLIMATE > * Jenkins instance - CLIMATE > > == Initial Committers == > ||'''Name''' ||'''Email''' ||'''Affiliation''' ||'''CLA''' || > ||Chris A. Mattmann ||mattmann at apache dot org > ||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] ||yes || > ||Cameron E. Goodale ||goodale at apache dot org > ||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] ||yes || > ||Paul Ramirez ||pramirez at apache dog org > ||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] ||yes || > ||Andrew F. Hart ||ahart at apache dot org > ||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] ||yes || > ||Jinwon Kim||jkim at atmos dot ucla dot edu > ||[[http://jifresse.ucla.edu|UCLA Joint Institute for Regional Earth > System Science and Engineering]] || no|| > ||Duane Waliser||duane dot waliser at jpl dot nasa dot gov > ||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || no || > ||Huikyo Lee||Huikyo dot Lee at jpl dot nasa dot > gov||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || no || > ||Paul Loikith|| Paul dot C dot Loikith at jpl dot nasa dot gov > ||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || no || > ||Daniel J. Crichton||crichton at apache dot > org||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || yes || > ||Kim Whitehall||Kim dot D dot Whitehall at jpl dot nasa dot gov > ||[[http://www.physics1.howard.edu/~pmisra/HUPAS/HUPAS/HUPAS%20Jenkins.html > |Howard University]] || no || > ||Paul Zimdars||pzimdars at apache dot > org||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || yes || > ||Chris Jack||cjack at csag dot uct dot ac dot > za||[[http://www.csag.uct.ac.za/|University of Cape Town]] || no || > ||Bruce Hewitson||hewitson at csag dot uct dot ac dot > za||[[http://www.csag.uct.ac.za/|University of Cape Town]] || no || > ||Lluis Fita Borrell||l dot fitaborrell at unsw dot edu dot > au||[[http://unsw.edu.au/|University of New South Wales]] || yes || > ||Jason Evans||jason dot evans at unsw dot edu dot > au||[[http://unsw.edu.au/|University of New South Wales]] || no || > ||Estani Gonzalez||estanislao dot gonzalez at met dot fu-berlin dot de > ||[[http://www.geo.fu-berlin.de/met/|Free University Berlin]] || yes || > ||Luca Cinquini||luca dot cinquini at jpl dot nasa dot gov > ||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || yes || > ||J. Sanjay||sanjay at tropmet dot res dot in || > [[http://tropmet.res.in/|Indian Institute of Tropical Meteorology]] || yes > || > ||M. V. S. Rama Rao||ramarao at tropmet dot res dot in > ||[[http://tropmet.res.in/|Indian Institute of Tropical Meteorology]] || > yes || > ||Tsengdar Lee||tsengdar dot j dot lee at nasa dot gov || > [[http://hq.nassa.gov/|NASA HQ]] || no || > ||Laura Carriere||laura dot carriere at nasa dot gov > ||[[http://www.gsfc.nasa.gov/|NASA Goddard Space Flight Center]] || no || > ||Denis Nadeau|| denis dot nadeau at nasa dot > gov||[[http://www.gsfc.nasa.gov/|NASA Goddard Space Flight Center]] || no > || > ||Michael Joyce|| Michael dot J dot Joyce at jpl dot nasa dot > gov||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] ||yes|| > ||Shakeh Khudikyan||Shakeh dot E dot Khudikyan at jpl dot nasa dot > gov||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] ||yes|| > ||Maziyar Boustani||Maziyar dot Boustani at jpl dot nasa dot > gov||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] ||no|| > ||Suresh Marru||smarru at apache dot org||[[http://pti.iu.edu/|Indiana > University]] ||yes|| > > > == Sponsors == > Champion > > * Chris Mattmann (mattmann at apache dot org) > > Nominated Mentors > > * Chris A. Mattmann (mattmann at apache dot org) > * Chris Douglas (cdouglas at apache dot org) > * Paul Ramirez (pramirez at apache dot org) > > Sponsoring Entity > > * Apache Incubator > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
