Hi Folks,
OK, now that discussion has settled down, I'd like to call a VOTE for
acceptance of Apache Open Climate Workbench into the Incubator.
I'll leave the VOTE open the rest of the week and close it out next
Monday, February 11th early am PT.
[ ] +1 Accept Apache Open Climate Workbench into the Incubator
[ ] +0 Don't care.
[ ] -1 Don't accept Apache Open Climate Workbench into the Incubator
because...
Full proposal is pasted at the bottom of this email. Only VOTEs from
Incubator PMC members are binding, but all are welcome to express their
thoughts.
Thank you!
Cheers,
Chris
P.S. Here's my +1 (binding)
-------------
= Apache Open Climate Workbench, tool for scalable comparison of remote
sensing observations to climate model outputs, regionally and globally. =
=== Abstract ===
The Apache Open Climate Workbench proposal desires to contribute an
existing community of software related to the analysis and evaluation of
climate models, and related to the use of remote sensing data in that
process.
Specifically, we will bring a fundamental software toolkit for analysis
and evaluation of climate model output against remote sensing data. The
toolkit is called the [[http://rcmes.jpl.nasa.gov|Regional Climate Model
Evaluation System (RCMES)]]. RCMES provides two fundamental components for
the easy, intuitive comparison of climate model output against remote
sensing data. The first component called RCMED (for "Regional Climate
Model Evaluation Database") is a scalable cloud database that decimates
remote sensing data and renalysis data related to climate using Apache
OODT extractors, Apache Tika, etc. These transformations make
traditionally heterogeneous upstream remote sensing data and climate model
output homogeneous and unify them into a data point model of the form
(lat, lng, time, value, height) on a per parameter basis. Latitude (lat)
and Longitude (lng) are in WGS84 format, but can be reformatted on the
fly. time is in ISO 8601 format, a string sortable format independent of
underlying store. value carries with it units, related to interpretation
and height allows for different values for different atmospheric vertical
levels. All of RCMES is built on Apache OODT, Apache Sqoop/Apache Hadoop
and Apache Hive, along with hooks to PostGIS and MySQL (traditional
relational databases). The second component of the system, RCMET (for
"Regional Climate Model Evaluation Toolkit") provides facilities for
connecting to RCMED, dynamically obtaining remote sensing data for a
space/time region of interest, grabbing associated model output (that the
user brings, or from the Earth System Grid Federation) of the same form,
and then regridding the remote sensing data to be on the model output
grid, or the model output to be on the remote sensing data grid. The
regridded data spatially is then temporally regridded using techniques
including seasonal cycle compositing (e.g., all summer months, all
Januaries, etc.), or by daily, monthly, etc. The uniform model output and
remote sensing data are then analyzed using pluggable metrics, e.g.,
Probability Distribution Functions (PDFs), Root Mean Squared Error (RMSE),
Bias, and other (possibly user-defined) techniques, computing an analyzed
comparison or evaluation. This evaluation is then visualized by plugging
in to the NCAR NCL library for producing static plots (histograms, time
series, etc.)
We also have performed a great deal of work in packaging RCMES to make the
system easy to deploy. We have working Virtual Machines (VMWare VMX and
Virtual Box OVA compatible formats) and we also have an installer built on
Python Buildout (http://buildout.org/) called "Easy RCMET" for dynamically
constructing the RCMET toolkit.
RCMES is currently supporting a number of recognized climate projects of
(inter-)national significance. In particular, RCMES is supporting the
[[http://www.globalchange.gov/what-we-do/assessment|U.S. National Climate
Assessment (NCA) activities]] on behalf of NASA's contribution to the NCA;
is working with the [[http://www.narccap.ucar.edu/|North American Regional
Climate Change Assessment Program (NARCCAP)]]; and is also working with
the International [[http://wcrp-cordex.ipsl.jussieu.fr/|Coordinated
Regional Downscaling Experiment (CORDEX)]].
=== Proposal ===
We propose to transition the RCMES software community, which includes
developers of the RCMET and RCMED software, along with users of RCMES in
the CORDEX project across a variety of academic institutions, scientists
helping to improve the RCMES metrics, and visualizations, and regridding
algorithms, packagers making RCMES easier to install, and scientists
helping to lead some of these international projects that are already
using RCMES.
We have been working on the RCMES project since 2009 funded initially by
the American Recovery and Reinvestment Act (ARRA) project out at NASA, and
then branching out into other sources of support and sustainability (NASA;
NSF, etc. -- see the
[[http://rcmes.jpl.nasa.gov/about/overview|acknowledgements]] section on
the RCMES website for a full list of supporting U.S. and international
partners).
With the existing RCMES community at Apache, we will also work to
encourage other climate software projects e.g., Open Climate GIS, elements
of the Earth System Grid Federation, other NASA climate projects funded
under the Computational Modeling, Algorithms and Cyberinfrastructure
(CMAC) to contribute to the Open Climate Workbench here at Apache.
RCMED is a Big Data project that combines several underlying Apache
software -- OODT, Tika, Hadoop, HIVE, and Sqoop -- and other related data
management software. Its primary language is Java; RCMET, on the other
hand, is a Python API, associated set of classes (framework), set of
Python Bottle Web services, and a PHP "Wizard"-based User Interface that
leverages Apache OODT Balance.
=== Background ===
Bringing RCMES to Apache was the brain-child of Chris Mattmann, based on
his solid experience with Apache OODT and bringing it to the ASF. Chris
worked for a year to get the support of the JPL community including
approvals from the Software Release authority at JPL to release the
software.
The initial code drop will include the RCMES SVN repository from JPL
including prior revisions. We anticipate also including a smaller package,
CDX, which contains some useful facilities for regridding, and command
line tools for manipulating large datasets, and working with OPeNDAP, etc.
After the code drop, we will work with our developers, users, documentors,
and other members of the team to teach those unfamiliar with the Apache
way how it works around here at Apache. 30% of the community from RCMES
includes those intimately familiar with Apache including 6 ASF members --
the other 70% include a range of scientific code developers, climate
scientists that use RCMES, program officers that will help make
documentation and slides for the code, and advocate for it in the
community. Their experience with Apache ranges from using various ASF
products, to contributing patches to them, to not using any ASF software
at all.
With this diversity, we anticipate that while everything may not just work
turnkey out of the box, this represents a unique opportunity to
demonstrate Apache to the international community and to show the benefits
of its community and social models. That said, we also have a lot of ASF
experience to make sure everyone learns the Apache way.
=== Rationale ===
We are bringing RCMES to Apache for a few reasons. First, we feel that it
will immediately enable our collaborators across a number of institutions
both nationally and internationally have the opportunity to work on a
common software base, and to improve it with contributions from their own
sites. Currently these are difficult to negotiate now because of varied
legal and contribution frameworks -- Apache allows us to simplify this to
a unified one. Second, using the ASF's world-wide mirroring system, we
will be able to deliver climate software broadly to the community as we
release it, rather than sneaker netting the software around or
establishing our own point release infrastructure.
Bringing this project to Apache also immediately thrusts the ASF into the
thriving ecosystem of the Coordinated Regional Downscaling Experiment
(CORDEX), the US National Cimate Assessment, the North American Regional
Climate Change Assessment Program (the US contribution to CORDEX) and into
relevance for upcoming Intergovernmental Panel on Climate Change (IPCC)
assessment activities at a number of different institutions. We also seek
to help lead and encourage du jour standard development rather than top
down level dictating of standards for climate software and the ASF will
provide us a means for that.
=== Initial Goals ===
The initial goals of the proposed project are:
* Stand up a sustaining Apache-based community around the JPL RCMES
codebase.
* Active relationships and possible cooperation with related projects and
communities, including end user and scientific communities, CORDEX,
NARCCAP, US NCA, IPCC, ESG, etc.
* Active relationships and possible cooperation with existing Apache
communities, e.g., OODT, Hadoop/HIVE, Sqoop, Tika, SIS, etc.
* Initial Apache release.
* Leverage Apache Open Climate Workbench in climate activities at NASA,
in the international community as mentioned above, and beyond.
* Vetting all software licenses and making sure IP is clear (software
grant from JPL forthcoming).
== Current Status ==
=== Meritocracy ===
30% of the proposed initial committers are familiar with the meritocracy
principles of Apache. As stated above this includes 6 ASF members. Of the
mentorship list, we have included Chris Douglas, a PMC member from Hadoop
and ASF member to help guide the community. Chris M. and Chris D. have
guided a number of projects through the Incubator over the years. The
other mentor includes Paul Ramirez, who has experience with the Incubator
-- he was a mentor for Apache Any23, and also was one of the PPMC members
and eventual mentor for Apache SIS. The 70% of proposed initial committers
that aren't as familiar with Apache have a broad range of experience in
other open source projects, and have a deep respect and affinity for the
foundation and the work that gets done here. The more experience ASF
mentors and project members will help to guide them.
=== Community ===
There is an existing, established community of developers and users of
this projet. This includes established communities including the
Coordinated Regional Downscaling Experiment, the U.S. National Climate
Assessment (NCA), the North American Regional Climate Change Assessment
Program (NARCCAP), and more. The Coordinated Regional Climate Downscaling
Experiment (CORDEX, http://wcrp.ipsl.jussieu.fr/cordex/about.html) is a
world wide effort of coordination of regional climate downscaling (RCD)
experiments driven by the World Climate Research Program (WRCP,
http://www.wcrp-climate.org/index.shtml). Recently, a large number of RCD
projects have been carried out on a large parts of the world. To maximize
the benefits of these research activities the WCRP designed a framework
(Giorgi, WMO-Bulletin, 2009) focused on "quality-control [of] data sets of
RCD-based information for the recent historical past and 21st century
projections, covering the majority of populated land regions on the
globe". CORDEX defined different control domains (up to 10,
http://cordex.dmi.dk/joomla/) for almost all the populated regions of the
world in a way to standardize the experiments and make them comparable. A
key region focused on Africa was also designated as the top priority by
WRCP. CORDEX also provides a a series of conventions and list of variables
that have to be followed by any project that wants to contribute to the
experiment. Each CORDEX region has a coordinator and regional and
international periodic meetings are scheduled in a way to ensure the
global well being. NARCCAP is the U.S. contribution to CORDEX. From the
[[http://www.globalchange.gov/what-we-do/assessment/nca-overview|US
National Climate Assessment]] site, work is "being conducted under the
auspices of the Global Change Research Act of 1990. The GCRA requires a
report to the President and the Congress every four years that integrates,
evaluates, and interprets the findings of the U.S. Global Change Research
Program (USGCRP); analyzes the effects of global change on the natural
environment, agriculture, energy production and use, land and water
resources, transportation, human health and welfare, human social systems,
and biological diversity; and analyzes current trends in global change,
both human-induced and natural, and projects major trends for the
subsequent 25 to 100 years."
Apache Open Climate Workbench will support all of these communities above,
with an eye towards being a general purpose climate evaluation toolkit for
model output and remote sensing data.
=== Core Developers ===
The initial set of developers comes from various NASA centers (JPL, and
Goddard Space Flight Center), NASA HQ, various Universities participating
in CORDEX (Cape Town, University of New South Wales), the Indian Institute
of Tropical Meteorology, the Free Univ. Berlin), the University of
California Los Angeles, and Howard University. As mentioned previously
several of our developers are Apache veterans and understand how it works
around here and for those that don't, they will have great mentorship.
=== Alignment ===
Our proposed effort aligns with the U.S. National Climate Assessment, the
CORDEX effort, other efforts, including the Earth System Grid Federation,
other climate software including the Open Climate GIS toolkit, other
science portals for climate including the Climate Information Portal (CIP)
at the University of Cape Town, and other related projects.
There are also a number of related Apache projects and dependencies, that
will be mentioned in the Relationships with Other Apache products section.
== Known Risks ==
=== Orphaned products ===
Our project has a history of funding support from JPL, NASA (Applications
program/ARRA, NCA, AIST), NSF (ExArch project), international investment
from collaborators, and from other funding sources. The funding sources
are all target future deliverables and activities, so there is little
chance this software and community will be orphaned.
=== Inexperience with Open Source ===
All the initial developers have worked on open source before -- 30% of the
proposed initial community are experience with the ASF, and are PMC
members and committers on ASF project including 6 ASF members. Our mentors
are all ASF members, and we welcome any interest from additional Apache
mentors in the effort. Those 70% of our project that aren't Apache
committers, PMC members, or members will benefit from the leadership of
the other 30% of the project.
=== Homogenous Developers ===
The initial developers come from a variety of backgrounds and with a
variety of needs for the proposed framework. Everyone is used to
communicating on mailing lists as the project spans timezones,
international institutions and centers of excellence for climate science.
=== Reliance on Salaried Developers ===
All of the proposed initial developers are paid to work on this or related
projects, but the proposed project is not the primary task for anyone.
=== Relationships with Other Apache Products ===
As mentioned above, RCMES and the Apache Open Climate Workbench already
depend on Apache OODT for facade interfaces to underlying data warehouses
for storing remote sensing data; and for metadata extraction and
transformation. The software also uses Apache Tika for this (through a
transitive dependency from OODT). In addition, we have hooks to Apache
Hadoop/HIVE, as well as dependencies on Apache Sqoop for dumping out
remote sensing data from MySQL and into HIVE.
=== A Excessive Fascination with the Apache Brand ===
All of us are familiar with Apache and have a respect for its brand and
community. Chris Mattmann is a big proponent of Apache's sustainability
factor -- and it's ability to grow software communities, in an
institution, or funding source neutral manner. All of the community have
an extreme respect for Apache, including those in our communities who
aren't necessarily trained computer scientists, but are Scientists (big
"S", e.g., land, physical, Earth/Climate scientists).
== Documentation ==
The initial RCMES code base will come from the internal JPL Subversion
repository. The [[Regional Climate Model Evaluation System (RCMES)
project|http://rcmes.jpl.nasa.gov]] at [[JPL|http://www.jpl.nasa.gov/]]
has documentation on the existing software, including links to funding
support, communities, and other projects. We will continue to maintain
that site at JPL, which is part of the reason for rebranding the project
here at Apache with a new name to not interfere with the existing RCMES
one that has a following. In addition, we hope to evolve RCMES@JPL to have
increasing levels of dependency on Apache Open Climate Workbench, so that
we can incrementally transition with little impact to existing customers.
In addition, JPL's [[http://cdx.jpl.nasa.gov|Climate Data eXchange (CDX)]]
website also has documentation on the existing software.
== Initial Source ==
The project will start with seed code donated by NASA JPL via Mattmann and
the rest of the initial committers, which consists of the Regional Climate
Model Evaluation System (RCMES) toolkit, and the Climate Data eXchange
(CDX) software. This will include the core Python API for RCMET, the RCMED
OODT catalog project (which stores remote sensing data to MySQL/PostGIS,
and HIVE), and the RCMED extractors for various climate formats. The
source will also include Easy-RCMET, the Python Buildout for RCMET. In
addition, we will bring along the CDX toolkit, which includes a CDX client
package that performs subsetting, access, regridding of climate data; and
also includes a Python Buildout installer of its own called Uber CDX.
== Source and Intellectual Property Submission Plan ==
All seed code and other contributions will be handled through the normal
Apache contribution process. Mattmann has been authorized by NASA JPL to
lead the contribution of RCMES and CDX into the Incubator via his existing
Apache CLA, and a Software Grant to be provided.
We will also contact other related efforts for possible cooperation and
contributions.
== External Dependencies ==
Our project depends on a number of external libraries with various
licensing conditions. An initial list of such dependencies is shown below.
||<tableclass="bodyTable"rowclass="b">'''Library''' ||'''License''' ||
||<rowclass="b">[[http://www.ncl.ucar.edu/|NCAR NCL]]||MIT compat||
||<rowclass="a">[[http://www.pyngl.ucar.edu/Nio.shtml|PyNIO]]||MIT compat||
||<rowclass="b">[[http://www.pyngl.ucar.edu/|PyNGL]]||MIT compat||
||<rowclass="a">[[http://matplotlib.sourceforge.net/|Matplotlib]]||Modified
PSF license||
||<rowclass="b">[[http://scipy.org/|Scipy]]||MIT compat||
||<rowclass="a">[[http://numpy.scipy.org/|NumPy]]||MIT compat||
||<rowclass="b">[[http://www.hdfgroup.org/HDF5/|HDF5]]||BSD||
||<rowclass="a">[[http://www.unidata.ucar.edu/software/netcdf/|NetCDF]]||MI
T||
== Cryptography ==
The project itself will not use cryptography, but it is possible that some
of the external software libraries will include cryptographic code to
handle features present in various science data formats. If we need to
provide an export control statement regarding cryptographic code per
Apache policy, we will follow a similar approach by Mattmann in
[[http://lucene.apache.org/nutch/|Apache Nutch]] and by Jukka Zitting lead
this effort in Apache Tika. Mattmann is familiar with this process.
== Required Resources ==
Mailing lists
* d...@climate.incubator.apache.org
* comm...@climate.incubator.apache.org
* priv...@climate.incubator.apache.org
Subversion Directory
* https://svn.apache.org/repos/asf/incubator/climate
Issue Tracking
* JIRA CLIMATE (CLIMATE)
Other Resources
* CLIMATE Wiki http://cwiki.apache.org/CLIMATE
* Review Board instance - CLIMATE
* Jenkins instance - CLIMATE
== Initial Committers ==
||'''Name''' ||'''Email''' ||'''Affiliation''' ||'''CLA''' ||
||Chris A. Mattmann ||mattmann at apache dot org
||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] ||yes ||
||Cameron E. Goodale ||goodale at apache dot org
||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] ||yes ||
||Paul Ramirez ||pramirez at apache dog org
||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] ||yes ||
||Andrew F. Hart ||ahart at apache dot org
||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] ||yes ||
||Jinwon Kim||jkim at atmos dot ucla dot edu
||[[http://jifresse.ucla.edu|UCLA Joint Institute for Regional Earth
System Science and Engineering]] || no||
||Duane Waliser||duane dot waliser at jpl dot nasa dot gov
||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || no ||
||Huikyo Lee||Huikyo dot Lee at jpl dot nasa dot
gov||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || no ||
||Paul Loikith|| Paul dot C dot Loikith at jpl dot nasa dot gov
||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || no ||
||Daniel J. Crichton||crichton at apache dot
org||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || yes ||
||Kim Whitehall||Kim dot D dot Whitehall at jpl dot nasa dot gov
||[[http://www.physics1.howard.edu/~pmisra/HUPAS/HUPAS/HUPAS%20Jenkins.html
|Howard University]] || no ||
||Paul Zimdars||pzimdars at apache dot
org||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || yes ||
||Chris Jack||cjack at csag dot uct dot ac dot
za||[[http://www.csag.uct.ac.za/|University of Cape Town]] || no ||
||Bruce Hewitson||hewitson at csag dot uct dot ac dot
za||[[http://www.csag.uct.ac.za/|University of Cape Town]] || no ||
||Lluis Fita Borrell||l dot fitaborrell at unsw dot edu dot
au||[[http://unsw.edu.au/|University of New South Wales]] || yes ||
||Jason Evans||jason dot evans at unsw dot edu dot
au||[[http://unsw.edu.au/|University of New South Wales]] || no ||
||Estani Gonzalez||estanislao dot gonzalez at met dot fu-berlin dot de
||[[http://www.geo.fu-berlin.de/met/|Free University Berlin]] || yes ||
||Luca Cinquini||luca dot cinquini at jpl dot nasa dot gov
||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || yes ||
||J. Sanjay||sanjay at tropmet dot res dot in ||
[[http://tropmet.res.in/|Indian Institute of Tropical Meteorology]] || yes
||
||M. V. S. Rama Rao||ramarao at tropmet dot res dot in
||[[http://tropmet.res.in/|Indian Institute of Tropical Meteorology]] ||
yes ||
||Tsengdar Lee||tsengdar dot j dot lee at nasa dot gov ||
[[http://hq.nassa.gov/|NASA HQ]] || no ||
||Laura Carriere||laura dot carriere at nasa dot gov
||[[http://www.gsfc.nasa.gov/|NASA Goddard Space Flight Center]] || no ||
||Denis Nadeau|| denis dot nadeau at nasa dot
gov||[[http://www.gsfc.nasa.gov/|NASA Goddard Space Flight Center]] || no
||
||Michael Joyce|| Michael dot J dot Joyce at jpl dot nasa dot
gov||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] ||yes||
||Shakeh Khudikyan||Shakeh dot E dot Khudikyan at jpl dot nasa dot
gov||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] ||yes||
||Maziyar Boustani||Maziyar dot Boustani at jpl dot nasa dot
gov||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] ||no||
||Suresh Marru||smarru at apache dot org||[[http://pti.iu.edu/|Indiana
University]] ||yes||
== Sponsors ==
Champion
* Chris Mattmann (mattmann at apache dot org)
Nominated Mentors
* Chris A. Mattmann (mattmann at apache dot org)
* Chris Douglas (cdouglas at apache dot org)
* Paul Ramirez (pramirez at apache dot org)
Sponsoring Entity
* Apache Incubator
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org
