Hey Julian,
> The rule of thumb I’ve learned integrating Calcite into Hive is that only
a branch should depend on snapshot versions of another component
This is very practical. I hadn't considered the Calcite dependency issue
when I pushed for merging into master rather than a branch.
Based on this, it seems like the best thing to do is:
1) Work off of Calcite SNAPSHOT Maven publication.
2) Create a samza-sql branch.
3) Migrate all existing samza-sql commits from master into the samza-sql
branch.
Sorry for the churn on this all. I hadn't considered the binary dependency
issue. Is everyone OK with this?
Cheers,
Chris
On Wed, Feb 11, 2015 at 10:51 AM, Julian Hyde <jul...@hydromatic.net> wrote:
> This seems more like a branch than a classifier. Calcite is developing in
> a branch, and would produce snapshots from that branch. The rule of thumb
> I’ve learned integrating Calcite into Hive is that only a branch should
> depend on snapshot versions of another component. (Hive broke this rule and
> got into a sticky situation where their trunk depended on a Calcite
> snapshot and they couldn’t release because Calcite was stuck in pre-release
> vote limbo.)
>
> I’m going to try to produce maven artifacts with {groupId:
> “org.apache.calcite”, version: "1.1.0-stream-incubating-SNAPSHOT”,
> artifactId: “calcite-core”, “calcite-avatica”, etc.} and push them to
> Apache nexus. I do recommend that you use it on a branch.
>
> However, I don’t think there would be a problem getting the functionality
> into calcite’s master branch in time for Calcite’s next release, in about 4
> weeks. (We commit to releasing approximately monthly.) The functionality
> and APIs would be marked “experimental, subject to change” but at least you
> would be able to merge your work to master at that point. Doubtless you
> would need new features & bug fixes in Calcite after that, and so work that
> depended on that would have to stay in a branch until Calcite was released.
>
> Julian
>
> On Feb 11, 2015, at 10:34 AM, Felix GV <fville...@linkedin.com.INVALID>
> wrote:
>
> > I guess one concern with Calcite SNAPSHOT releases is that they imply
> they'll be included in the next release, which may not be the case if
> they're still considered very experimental.
> >
> > Alternatively, perhaps Julian can create a new version name with a
> classifier for streaming.
> >
> >
> http://maven.apache.org/plugins/maven-deploy-plugin/examples/deploying-with-classifiers.html
> >
> > Then he could release SNAPSHOT artifacts for both the main/master code
> base as well as the streaming branch, no matter how experimental it is...
> >
> > --
> >
> > Felix GV
> > Data Infrastructure Engineer
> > Distributed Data Systems
> > LinkedIn
> >
> > f...@linkedin.com
> > linkedin.com/in/felixgv
> >
> > ________________________________________
> > From: Chris Riccomini [criccom...@apache.org]
> > Sent: Wednesday, February 11, 2015 10:23 AM
> > To: Chris Riccomini
> > Cc: dev@samza.apache.org
> > Subject: Re: [DISCUSS] SQL workflow
> >
> > Hey Julian,
> >
> > It looks like Calcite is at least setup to publish to Maven:
> >
> > https://repository.apache.org/#nexus-search;quick~calcite
> >
> > Julian, can you publish SNAPSHOT releases with your streaming changes to
> it?
> >
> > Thanks!
> > Chris
> >
> > On Wed, Feb 11, 2015 at 10:08 AM, Chris Riccomini <criccom...@apache.org
> >
> > wrote:
> >
> >> Hey Milinda,
> >>
> >> I've just committed SAMZA-484, so you should be able to re-base and get
> >> all the latest code.
> >>
> >>> But we need to add Calcite as a dependency and Calcite streaming
> >> support is only in Julian's branch
> >>
> >> We'll have to have Julian publish a release to Apache. This can be a
> >> SNAPSHOT to Apache's SNAPSHOT repo, or a real Calcite release. We can't
> >> check in binaries to the repo (licensing, release complexity, and just
> >> generally a bad idea). We'll have to coordinate with him.
> >>
> >> Cheers,
> >> Chris
> >>
> >> On Wed, Feb 11, 2015 at 9:43 AM, Milinda Pathirage <
> mpath...@umail.iu.edu>
> >> wrote:
> >>
> >>> No, SAMZA-483 doesn't depend on SAMZA-484. But we need to add Calcite
> as a
> >>> dependency and Calcite streaming support is only in Julian's branch (
> >>> https://github.com/julianhyde/incubator-calcite/tree/chi). What can
> we do
> >>> about that?
> >>>
> >>> Milinda
> >>>
> >>> On Wed, Feb 11, 2015 at 12:30 PM, Chris Riccomini <
> criccom...@apache.org>
> >>> wrote:
> >>>
> >>>> Hey Milinda,
> >>>>
> >>>> That'd be great. Do you have any dependency on SAMZA-484? I haven't
> had
> >>> a
> >>>> chance to commit that one yet.
> >>>>
> >>>> Cheers,
> >>>> Chris
> >>>>
> >>>> On Wed, Feb 11, 2015 at 9:23 AM, Milinda Pathirage <
> >>> mpath...@umail.iu.edu>
> >>>> wrote:
> >>>>
> >>>>> /need/want
> >>>>>
> >>>>> Milinda
> >>>>>
> >>>>> On Wed, Feb 11, 2015 at 12:22 PM, Milinda Pathirage <
> >>>> mpath...@umail.iu.edu
> >>>>>>
> >>>>> wrote:
> >>>>>
> >>>>>> Hi Chris,
> >>>>>>
> >>>>>> Do you need me to create the SAMZA-483 patch against latest master
> >>> with
> >>>>>> SAMZA-482 patch? I think that will make it easier to review the
> >>> patch?
> >>>>>>
> >>>>>> Thanks
> >>>>>> Milinda
> >>>>>>
> >>>>>> On Mon, Feb 9, 2015 at 11:39 AM, Chris Riccomini <
> >>>> criccom...@apache.org>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Cool. Looks like we've got consensus. I'll move forward with RTC on
> >>>> some
> >>>>>>> of
> >>>>>>> the early SAMZA-390 tickets. (SAMZA-482, SAMZA-483, SAMZA-484)
> >>>>>>>
> >>>>>>> On Mon, Feb 9, 2015 at 8:35 AM, Yan Fang <yanfang...@gmail.com>
> >>>> wrote:
> >>>>>>>
> >>>>>>>> +1 on this.
> >>>>>>>>
> >>>>>>>> Fang, Yan
> >>>>>>>> yanfang...@gmail.com
> >>>>>>>> +1 (206) 849-4108
> >>>>>>>>
> >>>>>>>> On Fri, Feb 6, 2015 at 4:38 PM, Chris Riccomini <
> >>>>> criccom...@apache.org>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Hey all,
> >>>>>>>>>
> >>>>>>>>> Are we +1 on this? I think Jakob was the only one who was
> >>> curious
> >>>>>>> about
> >>>>>>>> it.
> >>>>>>>>>
> >>>>>>>>> Cheers,
> >>>>>>>>> Chris
> >>>>>>>>>
> >>>>>>>>> On Thu, Feb 5, 2015 at 1:22 PM, Yi Pan <nickpa...@gmail.com>
> >>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi, Jakob,
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>> Eh? Not sure what this means...
> >>>>>>>>>>>
> >>>>>>>>>>> I mean SAMZA-484 depends on SAMZA-482, and neither are
> >>>>> committed.
> >>>>>>> So
> >>>>>>>>>> Navina
> >>>>>>>>>>> is having to post Yi's patch, as well as her own, on the
> >>> JIRA.
> >>>>> It
> >>>>>>>> makes
> >>>>>>>>>> it
> >>>>>>>>>>> really hard to do code reviews because you can't tell
> >>> whether
> >>>> Yi
> >>>>>>> made
> >>>>>>>>> the
> >>>>>>>>>>> changes or Navina did.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>> Just to add to the point. It is also difficult to always see
> >>> a
> >>>>> long
> >>>>>>>> list
> >>>>>>>>> of
> >>>>>>>>>> changed files if the RB request is always based on the
> >>> master.
> >>>> It
> >>>>> is
> >>>>>>>>>> possible to have RB request based on another RB request (I
> >>> have
> >>>>>>> tried
> >>>>>>>> it
> >>>>>>>>>> before). But what happens if the base RB request is
> >>>>>>>> cancelled/discarded?
> >>>>>>>>> RB
> >>>>>>>>>> is not designed to track the revision changes in a dependency
> >>>>> chain.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> Cheers,
> >>>>>>>>>>> Chris
> >>>>>>>>>>>
> >>>>>>>>>>> On Thu, Feb 5, 2015 at 12:16 PM, Jakob Homan <
> >>>> jgho...@gmail.com
> >>>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>>> I want to avoid branches,
> >>>>>>>>>>>> Just curious, any reason for this?
> >>>>>>>>>>>>
> >>>>>>>>>>>>> and I also want to avoid revision control over JIRA
> >>>>>>>>>>>> Eh? Not sure what this means...
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks,
> >>>>>>>>>>>> jg
> >>>>>>>>>>>>
> >>>>>>>>>>>> On 4 February 2015 at 17:11, Chris Riccomini <
> >>>>>>>> criccom...@apache.org>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> @Jakob, yeah I was thinking we'll follow our normal
> >>> flow.
> >>>>>>> RTC. I
> >>>>>>>>> just
> >>>>>>>>>>>>> wanted to set expectation that the code committed
> >>> might be
> >>>>>>> not up
> >>>>>>>>> to
> >>>>>>>>>>> our
> >>>>>>>>>>>>> normal quality initially (missing docs, no tests, etc).
> >>>>> Until
> >>>>>>> the
> >>>>>>>>>>> quality
> >>>>>>>>>>>>> is raised, we should think of this module as
> >>> experimental.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> @Milinda, awesome! Thanks. :)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>> Chris
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Wed, Feb 4, 2015 at 11:57 AM, Milinda Pathirage <
> >>>>>>>>>>>> mpath...@umail.iu.edu>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi Chris,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hope we no longer need the SQL API. I'll create a RB
> >>> for
> >>>>>>> Calcite
> >>>>>>>>>>>>>> integration.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks
> >>>>>>>>>>>>>> Milinda
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Wed, Feb 4, 2015 at 1:31 PM, Chris Riccomini <
> >>>>>>>>>>> criccom...@apache.org>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I think so. There was some RB downtime, but it just
> >>> got
> >>>>>>> fixed.
> >>>>>>>>> Yi,
> >>>>>>>>>>>>>> Navina,
> >>>>>>>>>>>>>>> Milinda, can you make sure your JIRAs have up to
> >>> date
> >>>>> RBs?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Wed, Feb 4, 2015 at 10:24 AM, sriram <
> >>>>>>> sriram....@gmail.com
> >>>>>>>>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Can we have updated RBs for all the three sub
> >>> tasks
> >>>>>>> before
> >>>>>>>> we
> >>>>>>>>>>>> commit?
> >>>>>>>>>>>>>>> This
> >>>>>>>>>>>>>>>> would help us to review even after we commit.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Wed, Feb 4, 2015 at 10:15 AM, Chris Riccomini <
> >>>>>>>>>>>>>> criccom...@apache.org>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Yi, Navina, and Milinda have been working on
> >>>>> SAMZA-390
> >>>>>>>>>>> sub-tickets
> >>>>>>>>>>>>>>>> related
> >>>>>>>>>>>>>>>>> to SQL operators. We're getting to the point
> >>> where
> >>>>> the
> >>>>>>>>> amount
> >>>>>>>>>> of
> >>>>>>>>>>>> work
> >>>>>>>>>>>>>>>>> floating around is quite large, and some tickets
> >>>>> build
> >>>>>>> off
> >>>>>>>>> of
> >>>>>>>>>>>> others.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I'm proposing that we commit this work into a
> >>>>> samza-sql
> >>>>>>>>>>> submodule
> >>>>>>>>>>>> on
> >>>>>>>>>>>>>>>>> master, and treat this module as experimental. I
> >>>> want
> >>>>>>> to
> >>>>>>>>> avoid
> >>>>>>>>>>>>>>> branches,
> >>>>>>>>>>>>>>>>> and I also want to avoid revision control over
> >>>> JIRA.
> >>>>>>> This
> >>>>>>>>>> means
> >>>>>>>>>>>> that
> >>>>>>>>>>>>>>>> there
> >>>>>>>>>>>>>>>>> will probably be a fair amount of commits/JIRAs
> >>> on
> >>>>> this
> >>>>>>>>> module
> >>>>>>>>>>> as
> >>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>> iterate, but I think that's OK.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Does this sound good to everyone?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>> Chris
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> --
> >>>>>>>>>>>>>> Milinda Pathirage
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> PhD Student | Research Assistant
> >>>>>>>>>>>>>> School of Informatics and Computing | Data to Insight
> >>>>> Center
> >>>>>>>>>>>>>> Indiana University
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> twitter: milindalakmal
> >>>>>>>>>>>>>> skype: milinda.pathirage
> >>>>>>>>>>>>>> blog: http://milinda.pathirage.org
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Milinda Pathirage
> >>>>>>
> >>>>>> PhD Student | Research Assistant
> >>>>>> School of Informatics and Computing | Data to Insight Center
> >>>>>> Indiana University
> >>>>>>
> >>>>>> twitter: milindalakmal
> >>>>>> skype: milinda.pathirage
> >>>>>> blog: http://milinda.pathirage.org
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Milinda Pathirage
> >>>>>
> >>>>> PhD Student | Research Assistant
> >>>>> School of Informatics and Computing | Data to Insight Center
> >>>>> Indiana University
> >>>>>
> >>>>> twitter: milindalakmal
> >>>>> skype: milinda.pathirage
> >>>>> blog: http://milinda.pathirage.org
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Milinda Pathirage
> >>>
> >>> PhD Student | Research Assistant
> >>> School of Informatics and Computing | Data to Insight Center
> >>> Indiana University
> >>>
> >>> twitter: milindalakmal
> >>> skype: milinda.pathirage
> >>> blog: http://milinda.pathirage.org
> >>>
> >>
> >>
>
>
