I would like to propose MetaObjects as an Apache Incubator project. The text of the proposal is included below.
--carl = Abstract MetaObjects is a data model and metadata description framework to drive Model-Driven Development (MDD) across multiple environments and language stacks providing capabilities such as generated documentation, code generation, model-driven behaviors, and model-driven interfaces. == Proposal In general, Model-Driven Development describes a process which uses an abstract definition of a “model” of “something” to generate concrete artifacts related to that “something” which are consumable by end-users (client code, developers, etc). Typical use-cases provide for automatic generation of related artifacts/capabilities such as: * Data * Documentation * QA/Verification * Interface definitions * Code implementations * DTO/DAO/CRUD operations Common examples of model-driven development include: * Spring Annotations * .NET Data Annotations * Ruby on Rails * Swagger/OpenAPI * Faker.js * Protobuf MetaObjects aims to provide a generalized schema for describing data and related artifacts, along with generative templates to produce language- or environment-specific artifacts based upon the "canonical" data description. MetaObjects hoists the definition of object schemas up one level, allowing us to treat these schemas as "data" and derive further business value from the metadata in the model. The model metadata (attributes) allow for driving behaviors around the models. These behaviors can be manifested through conversion to annotations in the target language via code generation. In addition, systems can implement behavioral code by using the model metadata attributes directly to modify default behaviors. == Background Having a shared understanding of the core data models and the concepts they describe is critical to delivering value across a variety of user-facing systems. MetaObjects was born to support faster development following the 80/20 rule of most data models having 80% of the same behaviors that could be solved using MDD, while still supporting 20% with custom-written code. Over the past 15 years, the primary use cases were building web applications and APIs where the persistence, business layer, and user interface were all built using MDD techniques. As MetaObjects usage grew, its use-cases evolved to meet the needs of the users. At Liquent, it was extended to support "overlays", allowing different deployments of the same codebase to have their behaviors and interfaces modified by only modifying the MetaObjects data model and metadata (attributes) to customize for difference clients, without the need for heavy custom code modifications. At Cengage, MetaObjects evolved to support interoperable representations of the same data within both XML and JSON data formats. Through these experiences, the project adopted the perspective that XML and JSON are not themselves data models. Instead, they are just representations (serializations) of a higher-level "object". In a very real sense, MetaObjects provides metadata which describes the high-level "object" in ways that are maximally useful to a wide range of consumers. Along the way, it was recognized that more and more functionality could be efficiently generated from a core "meta-model" of the objects. Artifacts and capabilities such as DAO/DTO code, model-driven UI, validations, database schemas, API models, etc were added to the project. It was also recognized that there is a strong tendency to "solve the problem in front of you" and thereby suffer from a focus that is too narrow. As the MetaObjects project matured, architecture and capabilities evolved to solve for this myopia. However, it was also recognized that an organization such as Apache would attract a wide variety of users and use-cases and serve as the catalyst to realize the full potential of MetaObjects. == Rationale There is a strong need for common models which exist independent of the technology implementing the model. Identifying opportunities for generalization and commoditization of these models has been key to the success of many projects. The success of projects like Swagger/OpenAPI, Spring, Protobuf clearly indicates the perceived value of MDD. They also clearly point to the criticality of ensuring that these capabilities are Open Standards. MetaObjects embraces these Open Standards by providing capability and interop across as many technology standards as possible. MetaObjects also acknowledges the potential to suffer from an inverse relationship -- as more technologies are integrated, the "lowest common denominator" shrinks. As part of its design, MetaObjects has facilities for incorporating specialized behaviors on top of core functionality. This enables MetaObjects to remain generally useful across technologies while remaining maximally useful for specific technologies. For example, when implementing within a micro-services environment, the contracts defined and provided by these services are the critical points of integration between service providers and consumers. API endpoints and API payloads (objects) define the coupling between components and services within a system. In an Enterprise environment, we suffer from typical pain points including: * Accurate implementation of Core data structures across consumers and service integrations * Maintenance, change management, API versioning, API Payload updates * Backward (and forward) compatibility within and among service implementations Independent maintenance (across teams) of these common data structures and the code for manipulating these structures can be a significant part of the cost of implementation. Similar to the benefits derived from the Micro-Service Chassis pattern, having a common definition of these structures provides opportunities for automation and elimination of redundant effort. The ability to code-generate artifacts across the breadth of technology existing within an Enterprise organization can potentially save costs and increase quality/velocity. Quality, velocity, efficiency and adoption are goals every organization seeks to achieve. We believe that MetaObjects can make significant contributions to enable an organization to realize all of these goals. == Initial Goals Initial goals for the project is to complete the documentation and add use-case examples for common modeling problems. In addition, prioritization of the code generation artifacts across different consumers and application environments is key to delivering value and driving adoption within a diverse Enterprise environment. Both of these goals require input from the community in order to effectively prioritize the Roadmap. == Current Status The current codebase grew out of development on large-scale gaming and content-delivery systems, initially developed by Doug Mealing. The codebase has incorporated use-cases from a variety of industries and applications, and has found success modeling content and core data structures at the Enterprise level. As technologies improve, they have been adopted as artifact targets for the overlay and code-generation capabilities. For example, XSD Schema can be generated from the model, and the JSON Schema output has been updated to include v0.7 standards. Related to the JSON Schema, Faker overlays have been developed, allowing annotations to be added to the model enabling integration with json-schema-faker (https://github.com/json-schema-faker/json-schema-faker) for model-driven data generation. Validation metadata has been incorporated into the model, which is used to drive behaviors and generate annotated interfaces across languages and systems within the enterprise. View metadata supporting interfaces to render the data for view and edit exist, which drive an older Struts/JSP UI implementation. There is an effort underway (within Cengage) developing a similar model-driven UI implemented in Apache Wicket. Java POJOs are generated, and typical annotations (e.g. @NotNull) are supported as appropriate, depending upon metadata in the object model definition. ORM annotations on the POJOs allow usage with JPA-based persistence layers. Objects based on the Immutables library (https://immutables.github.io/) can also be generated from the same model. Support for polymorphic serialization to/from JSON is implemented natively within the MetaObjects library, as well as via Gson TypeAdapters based upon the Immutables library. Perl artifacts are currently generated, based on `moose` conventions. In the near future, targets like C#/.NET, Google Protobuf, Python are under consideration depending upon need and bandwidth. Partial (or even full) Swagger/OpenAPI generation is currently being investigated. In terms of the development community we would like to build around the MetaObjects project, we are closely aligned with Apache principles. === Meritocracy We believe that collaboration is a necessary prerequisite to great outcomes. We believe that healthy debate is an aspect of collaboration necessary for high-quality decisions. We believe that a shared view of all perspectives generally leads to alignment behind these decisions. We believe that a good idea is independent of the source of the idea. We strive to find as many good ideas as possible in part by promoting and ensuring an inclusive development community. We believe that people who are invested in the project and supported by their peers produce great outcomes for the community. === Community The MetaObjects project has grown beyond Doug Mealing’s initial development. It has been used and extended by many developers in different organizations, including: .Community Usage [%header, cols="25,10,10,55"] |=== | Company | Year | IP Release | Description | Draagon Software | 2002 | Yes | Doug Mealing's company, built a credit card gateway licensed for use by third party financial processing systems. | Liquent | 2004 | Yes | Web-based pharmaceutical regulatory product (Insight Manager) sold to Pharma companies. Each client company customized the solution with their own overlays and distributed their own versions on CDs. | Draagon Software | 2005 | Yes | Doug Mealing's company, built an online poker software system used in an online-gaming startup. Gaming company went out of business, IP transfer to Doug Mealing executed in 2010. | Sendside Networks http://sendside.com | 2006 | Yes | Licensed to a startup building a secure communications solution on the web targeted at the banking industry. Now owned by www.titleio.com. | Draagon Software | 2014 | Yes | MetaObjects software was updated to the new version and open-sourced on GitHub (https://github.com/Draagon/draagon-metaobjects) | Cengage Learning http://cengage.com/ | 2014 | No | Open-source version brought in and modified/enhanced to meet requirements for online learning applications. |=== Note, MetaObjects has been open-sourced in 2014 and is available at https://github.com/Draagon/draagon-metaobjects. One of the reasons we are pursuing Apache Incubator is in part to find and build a larger community around MetaObjects. We have benefited as developers and organizations from the capabilities that are enabled by MetaObjects, and believe that the larger community would share in those benefits. === Core Developers The core developers are a collection of individuals, both Contractors and Employees who are using and improving the MetaObjects tools. There are at least three Apache Members who work either directly on the codebase, or are building additional capabilities on top of the MetaObjects library. Other Apache Members have contributed to the project in the past in various forms. These Apache Members are committers on open-source projects such as Camel, ActiveMQ, ServiceMix, Karaf and others. === Alignment Apache is known not just for the products that are used to deliver business value, but also the tools that enable developer efficiency. We believe that MetaObjects is one such tool which also has significant alignment with other Apache projects, such as: * Apache Wicket * Apache Maven * Apache Daffodil === Known Risks Some may perceive an overlap in functionality between MetaObjects and other projects, including Swagger/OpenAPI, DFDL (Daffodil), UML, etc. As these other projects progress, there may be a natural evolution that erodes the value of MetaObjects. Each of the projects mentioned above have a slightly differing focus, and have a strong incentive to "solve the problem in front of you", which increases variation and divergence between projects. As the MetaObjects project matured, architecture and capabilities evolved to solve for this variability. Concepts such as "inheritance", "extensions" and "overlays" provides flexibility at the "edge" for supporting different target artifacts while maintaining a consistent fundamental core. These concepts mitigate the risk of misalignment as target artifacts evolve. Regardless of "competitors" in this space, our focus is on making the right decisions for the project, based on the domain and the value that we can provide to our end users. Some may perceive a conflict of interest, or potentially too much involvement from one organization which becomes overly influential on the project direction. In reality, we perceive enormous value in the ideas and contributions that a wider community can provide. === Orphaned Products As mentioned above, multiple organizations have committed to MetaObjects and have built significant functionality within these organizations that depends upon MetaObjects. Consequently, we believe there is little risk to the project in terms of stagnation or orphanage. === Inexperience with Open Source Many of the committers have experience working on significant open source projects, many within the Apache umbrella. In addition, within the current organization, committers within the MetaObjects project have been champions and sponsors of a "Corporate Source" project structure strongly based upon the Apache policies and philosophies. === Length of Incubation We have no illusions about how long the incubation period will be. We believe that the time spent in Incubation is an investment in the long-term health of the project, and is in large part dependent upon the community that is built around the project. Therefore, expectations about length of time in Incubation are premature without community input to the process. === Homogeneous Developers As mentioned above, the committers to the MetaObjects project largely come from a collection of organizations that have incorporated MetaObjects into their systems. In addition, we are fortunate to have additional contributions from current Apache Members who happen to be working within these organizations. === Reliance on Salaried Developers Currently most of the contributions to the codebase are from salaried employees. === Relationships with Other Apache Products One of the interesting aspects of this project is the potential for integration with other open source projects, as well as open standards. We see potential integrations with the following: * Swagger/OpenAPI * Hecate * Spring Data * .NET * Perl * Python * Google Protobuf * JSON Schema * Faker.js * ReactJsonSchema * AngularJsonSchema * Apache Wicket * RDBMS * Apache DFDL (Daffodil) Depending on how projects progress in the future, we see possible overlaps in functionality with: * Swagger/OpenAPI * Apache DFDL === An Excessive Fascination with the Apache Brand The ASF has a strong brand, and that brand is in itself attractive because of the attention it brings to their projects. We believe that the MetaObjects project can benefit from that attention, and that will help us achieve our long-term goals to bring more efficiency and quality to the development process through automation and MDD practices. It is not surprising that we are interested in joining the ASF in order to increase our contacts and visibility in the open source world. Furthermore, we have been enthusiastic users of Apache for a long time, and have worked side-by-side with other Apache Members. This personal connection strengthens our resolve to be sensitive to the ways in which we impact the Apache brand and will work with the Incubator PMC and the PRC to ensure the brand policies are respected. == Documentation == Initial Source The MetaObjects project has been under development by Doug Mealing et al. at various organizations for over 15 years. == Source and Intellectual Property Submission Plan The licensing of the MetaObjects project is already under an open-source umbrella. == External Dependencies Core external dependencies include: * Apache Maven * Apache Commons * W3C * JSON Schema Additional external dependencies may exist based on plugins or extensions. == Cryptography There is no cryptography involved in the core codebase. == Required Resources === Mailing Lists * priv...@metaobjects.incubator.apache.org (w/ moderated subscriptions) * d...@metaobjects.incubator.apache.org * us...@metaobjects.incubator.apache.org * comm...@metaobjects.incubator.apache.org === Subversion Directory https://svn.apache.org/repos/asf/incubator/meta-objects This should be a read-only replica of the Git repository (if it's even necessary). === Git Repositories https://git-wip-us.apache.org/repos/asf/meta-objects.git === Issue Tracking JIRA meta-objects === Other Resources None == Initial Committers * Doug Mealing (doug at dougmealing dot com) * Alejandro Mora (alejandro dot mora dot castro at gmail dot com) * James Carman (jcarman at apache dot org) * Johan Edstrom (joed at apache dot org) * Jamie Goodyear (jgoodyear at apache dot org) * Heath Kesler (hkesler at apache dot org) * Carl Nygard (cjnygard at gmail dot com) == Sponsors * Champion: Johan Edstrom - Apache, Savoirtech * Nominated Mentors: ** James Carman - Apache, Cengage ** Heath Kesler - Apache, Savoirtech ** Jamie Goodyear - Apache, Savoirtech * Sponsoring Entity: Incubator --------------------------------------------------------------------- To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
