Hi Wes, Thanks for your response. In regards to the protocol negotiation your description of feature reporting (snipped below) is along the lines of what I was thinking. It might not be necessary for 1.0.0, but at some point might become useful.
> Note that we don't really have a mechanism for clients > and servers to report to each other what features they support, so > this could help with that when for applications where it might matter. Thanks, Micah On Mon, Jul 1, 2019 at 12:54 PM Wes McKinney <wesmck...@gmail.com> wrote: > hi Micah, > > Sorry for the delay in feedback. I looked at the document and it seems > like a reasonable perspective about forward- and > backward-compatibility. > > It seems like the main thing you are proposing is to apply Semantic > Versioning to Format and Library versions separately. That's an > interesting idea, my thought had been to have a version number that is > FORMAT_VERSION.LIBRARY_VERSION.PATCH_VERSION. But your proposal is > more flexible in some ways, so let me clarify for others reading > > In what you are proposing, the next release would be: > > Format version: 1.0.0 > Library version: 1.0.0 > > Suppose that 20 major versions down the road we stand at > > Format version: 1.5.0 > Library version: 20.0.0 > > The minor version of the Format would indicate that there are > additions, like new elements in the Type union, but otherwise backward > and forward compatible. So the Minor version means "new things, but > old clients will not be disrupted if those new things are not used". > We've already been doing this since the V4 Format iteration but we > have not had a way to signal that there may be new features. As a > corollary to this, I wonder if we should create a dual version in the > metadata > > PROTOCOL VERSION: (what is currently MetadataVersion, V2) > FEATURE VERSION: not tracked at all > > So Minor version bumps in the format would trigger a bump in the > FeatureVersion. Note that we don't really have a mechanism for clients > and servers to report to each other what features they support, so > this could help with that when for applications where it might matter. > > Should backward/forward compatibility be disrupted in the future, then > a change to the major version would be required. So in year 2025, say, > we might decide that we want to do: > > Format version: 2.0.0 > Library version: 21.0.0 > > The Format version would live in the project's Documentation, so the > Apache releases are only the library version. > > Regarding your open questions: > > 1. Should we clean up "warts" on the specification, like redundant > information > > I don't think it's necessary. So if Metadata V5 is Format Version > 1.0.0 (currently we are V4, but we're discussing some possible > non-forward compatible changes...) I think that's OK. None of these > things are "hurting" anything > > 2. Do we need additional mechanisms for marking some features as > experimental? > > Not sure, but I think this can be mostly addressed through > documentation. Flight will still be experimental in 1.0.0, for > example. > > 3. Do we need protocol negotiation mechanisms in Flight > > Could you explain what you mean? Are you thinking if there is some > major revamp of the protocol and you need to switch between a "V1 > Flight Protocol" and a "V2 Flight Protocol"? > > - Wes > > On Thu, Jun 13, 2019 at 2:17 AM Micah Kornfield <emkornfi...@gmail.com> > wrote: > > > > Hi Everyone, > > I think there might be some ideas that we still need to reach consensus > on > > for how the format and libraries evolve in a post-1.0.0 release world. > > Specifically, I think we need to agree on definitions for > > backwards/forwards compatibility and its implications for versioning the > > format. > > > > To this end I put some thoughts down in a Google Doc [1] for the purposes > > of discussion. Comments welcome. I will start threads for any comments > in > > the document that seem to warrant further discussion, and once we reach > > consensus I can create a patch to document what we decide on as part of > the > > specification. > > > > Thanks, > > Micah > > > > [1] > > > https://docs.google.com/document/d/1uBitWu57rDu85tNHn0NwstAbrlYqor9dPFg_7QaE-nc/edit# >
