Semantic versioning is a great tool, and we should use it as far as it
goes, but not push it.

I suggest that the Arrow specification should have a paragraph that
states the level of maturity of each part of the API; and each
implementation should have a paragraph that states which parts of the
spec are implemented, and to what quality. A lot can be accomplished
in one paragraph in terms of setting people's expectations.

And since you mentioned the open-closed principle earlier, the
robustness principle [1] should apply: be liberal in what you accept,
conservative in what you do. An arrow library should (ideally) not
fall over if it encounters a data structure that was experimental in a
previous version and has recently been removed.

Julian

[1] https://en.wikipedia.org/wiki/Robustness_principle


On Wed, Jul 26, 2017 at 12:30 PM, Wes McKinney <wesmck...@gmail.com> wrote:
> The combinatorics of code-level API stability are worrisome (with
> already 5 different language APIs in the project) while the maturity
> and development pace of different implementations may remain variable
> for some time.
>
> There are two possible things we can communicate with some form of
> major version number:
>
> * The Arrow specification (independent to implementation) is complete,
> with more than one reference implementations proving to have
> implemented it
>
> * The code is complete and stable
>
> The latter seems undesirable, at least on a 6 month horizon. I don't
> think it should keep us from making a public statement that we've
> hardened the Arrow format itself. Perhaps we need two kinds of major
> versions for the project.
>
> The worry I have is that strict semantic versioning might prove
> onerous to immature implementations. As a concrete example, suppose
> that someone starts a Go implementation shortly after we've made a 1.0
> release with integration tests for all the well-specified Arrow types.
> After a couple of months, the Go developers need to make some breaking
> API changes. Does that mean we need to bump the whole project to 2.x?
> As more languages come into the fold, this could happen more and more
> often. How would people interpret a fast escalating major version
> number?
>
> I am curious how Avro or Thrift have addressed this issue.
>
> - Wes
>
> On Wed, Jul 26, 2017 at 3:13 PM, Julian Hyde <jh...@apache.org> wrote:
>> I agree with all that. But semantic versioning only pertains to public APIs. 
>> So, for it to work, you need to declare what are your public APIs. If you 
>> don’t, people will make assumptions about what are your public APIs, and 
>> they may get it wrong.
>>
>> The ability to add experimental APIs (not subject to semantic versioning 
>> until they are officially declared public) will help the project evolve and 
>> stay relevant.
>>
>> Julian
>>
>>
>>> On Jul 26, 2017, at 12:02 PM, Wes McKinney <wesmck...@gmail.com> wrote:
>>>
>>> I see the semantic versioning like this:
>>>
>>> Major version: Format and Metadata stability
>>> Minor version: API stability within fix versions
>>> Fix version: Bug fixes
>>>
>>> So an API might be deprecated from 1.0.0 to 1.1.0, but we could not
>>> make a breaking change to the memory format without increasing the
>>> major version. We also have the added protection of a version enum in
>>> the metadata
>>>
>>> https://github.com/apache/arrow/blob/master/format/Schema.fbs#L22
>>>
>>> On Wed, Jul 26, 2017 at 2:56 PM, Wes McKinney <wesmck...@gmail.com> wrote:
>>>> Given the nature of the Arrow project, where any number of different
>>>> implementations will be in flux at any given time, claiming any sort
>>>> of API stability at the code level across the whole project seems
>>>> impossible any time soon.
>>>>
>>>> The important commitment of a 1.0 release is that the metadata and
>>>> memory format is not changing (without a change in the major version
>>>> number, i.e. Arrow 1.x.y to 2.x.y); so Arrow's "API" in a sense is the
>>>> memory format and serialized metadata representation. That is, the
>>>> files in
>>>>
>>>> https://github.com/apache/arrow/tree/master/format
>>>>
>>>> Having this kind of stability is really important so that if any
>>>> systems know how to parse or emit Arrow 1.x data, but aren't
>>>> necessarily using the libraries provided by the project, they can have
>>>> some assurance that we aren't going to break the Flatbuffers or the
>>>> arrangement of bytes in a record batch on the wire. If that makes
>>>> sense.
>>>>
>>>> - Wes
>>>>
>>>> On Wed, Jul 26, 2017 at 2:35 PM, Julian Hyde <jh...@apache.org> wrote:
>>>>> 1.0 is a Big Deal because, under semantic versioning, there is a 
>>>>> commitment to not change public APIs. If it weren’t for that, 1.0 would 
>>>>> have vague marketing connotations of robustness, adoption etc. but 
>>>>> otherwise be no different from another release.
>>>>>
>>>>> So, if API and data format lifecycle and compatibility is the goal here, 
>>>>> would it be useful to introduce explicit flags on API maturity? Call out 
>>>>> which APIs are public, and therefore bound by the semantic versioning 
>>>>> contract. This will also give Arrow some room to add experimental 
>>>>> features after 1.0, and avoid calcification.
>>>>>
>>>>> Julian
>>>>>
>>>>>
>>>>>
>>>>>> On Jul 26, 2017, at 7:40 AM, Wes McKinney <wesmck...@gmail.com> wrote:
>>>>>>
>>>>>> I created https://issues.apache.org/jira/browse/ARROW-1277 about
>>>>>> integration testing remaining data types. We are so close to having
>>>>>> everything tested and stable, we should push to complete these as soon
>>>>>> as possible (save for Map, which has only just been added to the
>>>>>> metadata)
>>>>>>
>>>>>> On Mon, Jul 24, 2017 at 5:35 PM, Wes McKinney <wesmck...@gmail.com> 
>>>>>> wrote:
>>>>>>> I agree those things would be nice to have. Hardening the memory
>>>>>>> format details probably would not take longer than a month or so if we
>>>>>>> were to focus in on it.
>>>>>>>
>>>>>>> Formalizing REST / RPC or IPC seems like it will be more work, or will
>>>>>>> require a design period and then initial implementation. I think
>>>>>>> having the streaming format implementations is a good start, but the
>>>>>>> streams are a bit monothic -- e.g. in REST you might want to request
>>>>>>> metadata only, or only record batches given a known schema. We should
>>>>>>> create a proposal document (Google docs?) for the community to comment
>>>>>>> where we can iterate on requirements
>>>>>>>
>>>>>>> Separately, I'm interested in embedding Arrow streams in other
>>>>>>> transport layers, like GRPC. The recent refactoring in C++ to make the
>>>>>>> streams less monolithic was intended to help with that.
>>>>>>>
>>>>>>> - Wes
>>>>>>>
>>>>>>> On Mon, Jul 24, 2017 at 4:01 PM, Jacques Nadeau <jacq...@apache.org> 
>>>>>>> wrote:
>>>>>>>> Top things on my list:
>>>>>>>>
>>>>>>>> - Formalize Arrow RPC and/or REST
>>>>>>>> - Some reference transformation algorithms
>>>>>>>> - Prototype IPC
>>>>>>>>
>>>>>>>> On Mon, Jul 24, 2017 at 9:47 AM, Wes McKinney <wesmck...@gmail.com> 
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> hi folks,
>>>>>>>>>
>>>>>>>>> In recent discussions, since the Arrow memory format and metadata has
>>>>>>>>> become reasonably stabilized, and we're more likely to add new data
>>>>>>>>> types than change existing ones, we may consider making a 1.0.0 to
>>>>>>>>> declare to the rest of the open source world that "Arrow is open for
>>>>>>>>> business" and can be relied upon in production applications (which
>>>>>>>>> some reasonable tolerance for library API changes from major release
>>>>>>>>> to major release). I hope we can all agree that forward and backward
>>>>>>>>> compatibility in the zero-copy wire format and metadata is the most
>>>>>>>>> essential thing.
>>>>>>>>>
>>>>>>>>> To that end, I'd like to collect ideas for what needs to be
>>>>>>>>> accomplished in the project before we'd be comfortable making a 1.0.0
>>>>>>>>> release. I think it would be a good show of project stability /
>>>>>>>>> production-readiness to do this (with the caveat the APIs will
>>>>>>>>> continue to evolve).
>>>>>>>>>
>>>>>>>>> The main things on my end are hardening the memory format and
>>>>>>>>> integration tests for the remaining data types:
>>>>>>>>>
>>>>>>>>> - Decimals
>>>>>>>>>   - Lingering issues with 128-bit decimals
>>>>>>>>>   - Need integration tests
>>>>>>>>> - Fixed size list
>>>>>>>>>   - Java has implemented, but not C++. Need integration tests
>>>>>>>>> - Union
>>>>>>>>>   - Two kinds of unions, Java only implements one. Need integration 
>>>>>>>>> tests
>>>>>>>>>
>>>>>>>>> On these, Decimals have the most work since the memory format needs to
>>>>>>>>> be specified. On Unions, we may decide to not implement the dense
>>>>>>>>> variant and focus on integration testing the sparse variant. I don't
>>>>>>>>> think this is going to be too much work, but it needs to get sorted
>>>>>>>>> out so we don't have incomplete or under-tested parts of the
>>>>>>>>> specification.
>>>>>>>>>
>>>>>>>>> There's some other things being discussed, like a Map logical type,
>>>>>>>>> but that (at least as currently proposed) won't require any disruptive
>>>>>>>>> modifications to the metadata.
>>>>>>>>>
>>>>>>>>> As far as the metadata and memory format, we would use the Open/Closed
>>>>>>>>> principle to guide our efforts
>>>>>>>>> (https://en.wikipedia.org/wiki/Open/closed_principle). For example, it
>>>>>>>>> would be possible to add compression or encoding at the field level
>>>>>>>>> without disrupting earlier versions of the software that lack these
>>>>>>>>> features.
>>>>>>>>>
>>>>>>>>> In the event that we do need to change the metadata or memory format
>>>>>>>>> in the future (which would probably be an extreme circumstance), we
>>>>>>>>> have the option of increasing the MetadataVersion which is one of the
>>>>>>>>> first tags accompanying Arrow messages
>>>>>>>>> (https://github.com/apache/arrow/blob/master/format/Schema.fbs#L22).
>>>>>>>>> So if you encounter a message that you do not support, you can raise
>>>>>>>>> an appropriate exception.
>>>>>>>>>
>>>>>>>>> There are some other things that would be nice to prototype or
>>>>>>>>> specify, like a REST protocol for exposing Arrow datasets in a
>>>>>>>>> client-server model (sending Arrow record batches via REST HTTP
>>>>>>>>> calls).
>>>>>>>>>
>>>>>>>>> Anything else that would need to go to move to a 1.x mainline for
>>>>>>>>> development? One idea would be if we need to make any breaking changes
>>>>>>>>> that we would leap from 1.x to 2.0.0 and throw the 1.x branches into
>>>>>>>>> maintenance mode.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> Wes
>>>>>>>>>
>>>>>
>>

Reply via email to