Stefan Richter found the following problem with HA: https://issues.apache.org/jira/browse/FLINK-4150
I think we should fix it for the 1.1 release. On Mon, Jul 4, 2016 at 9:05 PM, Robert Metzger <rmetz...@apache.org> wrote: > +1 to do a RC0 this week, but the master-forking with RC1. I would like to > reduce the time we need to apply patches to multiple branches. > > @Aljoscha: I was running into the same issue on EMR when I used Flink w/ > RocksDB recently, so I agree ;) > > > > On Mon, Jul 4, 2016 at 3:35 PM, Aljoscha Krettek <aljos...@apache.org> > wrote: > > > IMHO, the fix for this should also go in: > > https://issues.apache.org/jira/browse/FLINK-4115. This is blocking for > > users that want to use the RocksDB backend or FsStateBackend on Amazon > EMR > > with S3. > > > > There is already an open PR that I'm hoping to get in this week. > > > > On Mon, 4 Jul 2016 at 13:48 Ufuk Celebi <u...@apache.org> wrote: > > > > > Thanks for the feedback. I would like to create a preview RC0 this > > > week like we did for the last releases, too. In past major releases, > > > we needed to create many release candidates, often for fixing just > > > some small issues. I would like to speed up the release process by > > > collecting as many issues as possible now with the RC0. Once these > > > issues are resolved, we can start voting with the RC1. This > > > essentially means that we have a feature freeze. I would create the > > > release-1.1 branch with RC1. > > > > > > Regarding the open issues: > > > > > > - The savepoint fixes are not yet in. There is a PR for the savepoint > > > headers (#2194) and the savepoint disposal PR needs addressing of > > > comments. > > > > > > - The Cassandra connector fixes are either merged or very close to be > > > merged. > > > > > > - Metrics docs are not a blocker since the online docs are updated > > > after the release. Regarding the renaming, we should decide soon. Any > > > opinions here? > > > > > > - The YARN issues have been resolved. > > > > > > I will also make a pass over JIRA and our PRs to check whether we've > > > missed something. > > > > > > @Greg: You are right, the hash-based combine PR has been extensively > > > reviewed. Unfortunately, I'm not familiar with the code as I didn't > > > look into it and cannot tell whether we should merge it now. Is the > > > hash-based combine strategy activated by default or does the user has > > > to activate it manually? The later case would make me feel more > > > comfortable merging it for the release. > > > > > > > > > On Fri, Jul 1, 2016 at 7:14 PM, Maximilian Michels <m...@apache.org> > > wrote: > > > > Yet another one for the release: FLINK-4144 > > > > https://github.com/apache/flink/pull/2191 > > > > > > > > On Fri, Jul 1, 2016 at 5:18 PM, Maximilian Michels <m...@apache.org> > > > wrote: > > > >> There is also FLINK-4141. We just found this during testing. PR is > > > >> waiting to be merged here: > https://github.com/apache/flink/pull/2190 > > > >> > > > >> On Fri, Jul 1, 2016 at 3:26 PM, Maximilian Michels <m...@apache.org> > > > wrote: > > > >>> FLINK-3904 is not Yarn related. Not pressing to fix for this > release > > > >>> and actually a bit tricky to fix. I've addressed the other issues > and > > > >>> merged all pending pull requests. Good to go from my side. > > > >>> > > > >>> On Fri, Jul 1, 2016 at 11:29 AM, Chesnay Schepler < > > ches...@apache.org> > > > wrote: > > > >>>> There are also 2 fixes for Cassandra that should be included: > > > >>>> https://github.com/apache/flink/pull/2167 > > > >>>> https://github.com/apache/flink/pull/2183 > > > >>>> > > > >>>> We should also include the documentation for the metrics stuff > > > (hopefully > > > >>>> merged today) > > > >>>> https://github.com/apache/flink/pull/2158 > > > >>>> > > > >>>> In regards to metrics: To add a counter metric a user currently > has > > > to call > > > >>>> "counter(...)" on > > > >>>> a MetricGroup. The point was raised in the documentation PR that > we > > > may want > > > >>>> to give > > > >>>> them a more descriptive name like "addCounter(...)". > > > >>>> > > > >>>> I would be in favor of changing them but would like others to > weigh > > > in on > > > >>>> this. IMO we > > > >>>> should nail this down before 1.1 . > > > >>>> > > > >>>> Regards, > > > >>>> Chesnay > > > >>>> > > > >>>> > > > >>>> On 30.06.2016 22:59, Greg Hogan wrote: > > > >>>>> > > > >>>>> It would be great if hash-based combine (FLINK-3477) could make > it > > > in to > > > >>>>> be > > > >>>>> tested for this release. We've seen impressive improvements in > > > performance > > > >>>>> (though, admittedly, some sort-based enhancements are yet to be > > > worked > > > >>>>> on). > > > >>>>> This PR looks to be ripe. > > > >>>>> > > > >>>>> Also, as we tidy up a few things with Gelly and documentation, > what > > > is the > > > >>>>> schedule for a feature freeze and creating a 1.1 branch off > master? > > > >>>>> > > > >>>>> Thanks, > > > >>>>> Greg > > > >>>>> > > > >>>>> On Mon, Jun 27, 2016 at 7:23 AM, Robert Metzger < > > rmetz...@apache.org > > > > > > > >>>>> wrote: > > > >>>>> > > > >>>>>> Sure Ufuk! Thanks a lot for taking care of the release > management. > > > >>>>>> I'll be on vacation in three weeks, for three weeks and I'm not > > > sure if > > > >>>>>> we > > > >>>>>> get the release done until then. > > > >>>>>> > > > >>>>>> On Mon, Jun 27, 2016 at 12:08 PM, Ufuk Celebi <u...@apache.org> > > > wrote: > > > >>>>>> > > > >>>>>>> I would like to do it if that's OK with you Robert. I would > > follow > > > >>>>>>> your suggestion and wait a few days until the following > important > > > >>>>>>> fixes are in: > > > >>>>>>> - Savepoint headers and proper disposal (FLINK-4067 and > > > >>>>>>> https://github.com/apache/flink/pull/2083) > > > >>>>>>> - Metrics (https://github.com/apache/flink/pull/2146) > > > >>>>>>> - Table API time support ( > > > https://github.com/apache/flink/pull/2150) > > > >>>>>>> - Kafka at-least-once Producer ( > > > >>>>>> > > > >>>>>> https://github.com/apache/flink/pull/2108) > > > >>>>>>> > > > >>>>>>> - Cassandra connector fixes ( > > > https://github.com/apache/flink/pull/2163) > > > >>>>>>> - YARN client fixes (FLINK-3675, FLINK-3904 @Max: is there > > > something > > > >>>>>> > > > >>>>>> else?) > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> On Thu, Jun 23, 2016 at 1:33 PM, Robert Metzger < > > > rmetz...@apache.org> > > > >>>>>>> wrote: > > > >>>>>>>> > > > >>>>>>>> Hi, > > > >>>>>>>> it doesn't seem that there are volunteers for the RM, so I'll > > > probably > > > >>>>>> > > > >>>>>> do > > > >>>>>>>> > > > >>>>>>>> it. > > > >>>>>>>> > > > >>>>>>>> I try to do the first release candidate (mostly for testing) > > next > > > week > > > >>>>>>> > > > >>>>>>> (it > > > >>>>>>>> > > > >>>>>>>> depends on the JIRAs fixed by then) > > > >>>>>>>> > > > >>>>>>>> On Thu, Jun 16, 2016 at 10:56 PM, Henry Saputra < > > > >>>>>> > > > >>>>>> henry.sapu...@gmail.com > > > >>>>>>>> > > > >>>>>>>> wrote: > > > >>>>>>>> > > > >>>>>>>>> Thanks for the reply, @Max. I was not aware it was about > > dynamic > > > >>>>>>> > > > >>>>>>> scaling, > > > >>>>>>>>> > > > >>>>>>>>> which I think also asked for YARN support. > > > >>>>>>>>> I agree to list all related half merge JIRA for the > > > ResourceManager. > > > >>>>>>>>> > > > >>>>>>>>> Looking forward for the Apache Mesos integration design for > > sure > > > =) > > > >>>>>>>>> > > > >>>>>>>>> - Henry > > > >>>>>>>>> > > > >>>>>>>>> On Thu, Jun 16, 2016 at 2:12 AM, Maximilian Michels < > > > m...@apache.org> > > > >>>>>>>>> wrote: > > > >>>>>>>>> > > > >>>>>>>>>> Hi Robert, hi Henry, > > > >>>>>>>>>> > > > >>>>>>>>>> +1 for a 1.1.0 release soon! We have enough new features > that > > > >>>>>> > > > >>>>>> justify > > > >>>>>>>>>> > > > >>>>>>>>>> a major release. > > > >>>>>>>>>> > > > >>>>>>>>>> @Henry We have plans to extend the ResourceManager to > interact > > > with > > > >>>>>>>>>> the Scheduler which will be a prerequisite for dynamic > > scaling. > > > I > > > >>>>>>>>>> think this is out of scope for 1.1.0. The upcoming Mesos > > > integration > > > >>>>>>>>>> won't require additional refactoring of the ResourceManager. > > > >>>>>> > > > >>>>>> Instead, > > > >>>>>>>>>> > > > >>>>>>>>>> we will create a new "Dispatcher" component that takes care > of > > > >>>>>>>>>> bootstrapping the initial node with the > > > JobManager/ResourceManager. > > > >>>>>>>>>> From there on, everything will be handled by the Mesos > > > >>>>>>>>>> ResourceManager. I recently discussed this with Eron (CC) > who > > > came > > > >>>>>> > > > >>>>>> up > > > >>>>>>>>>> > > > >>>>>>>>>> with this design and he plans to publish it to the mailing > > list > > > >>>>>> > > > >>>>>> soon. > > > >>>>>>>>>> > > > >>>>>>>>>> How about listing relevant JIRA issues here? "Half Merged" > is > > > kind > > > >>>>>> > > > >>>>>> of > > > >>>>>>>>>> > > > >>>>>>>>>> hard to get for people who are not involved in the different > > > >>>>>>>>>> components. > > > >>>>>>>>>> > > > >>>>>>>>>> The Cassandra adapter seems like a pretty important thing to > > > have > > > >>>>>> > > > >>>>>> for > > > >>>>>>>>>> > > > >>>>>>>>>> the next release. In addition, I would like to merge > > FLINK-3667 > > > and > > > >>>>>>>>>> FLINK-3937. Robert is doing a review at the moment :) Those > > are > > > a) > > > >>>>>>>>>> refactoring of the command-line and client classes b) adding > > > >>>>>>>>>> capability to resume cluster programmatically. > > > >>>>>>>>>> > > > >>>>>>>>>> Then we should also have a look at any other critical/major > > bugs > > > >>>>>>> > > > >>>>>>> listed > > > >>>>>>>>> > > > >>>>>>>>> in > > > >>>>>>>>>> > > > >>>>>>>>>> JIRA. > > > >>>>>>>>>> > > > >>>>>>>>>> Cheers, > > > >>>>>>>>>> Max > > > >>>>>>>>>> > > > >>>>>>>>>> On Wed, Jun 15, 2016 at 10:50 PM, Henry Saputra < > > > >>>>>>> > > > >>>>>>> henry.sapu...@gmail.com > > > >>>>>>>>>> > > > >>>>>>>>>> wrote: > > > >>>>>>>>>>> > > > >>>>>>>>>>> Hi Robert, > > > >>>>>>>>>>> > > > >>>>>>>>>>> Thanks for staying the discussion. > > > >>>>>>>>>>> > > > >>>>>>>>>>> Do you know if there any open tasks for the Resource > Manager > > > left? > > > >>>>>>>>>>> > > > >>>>>>>>>>> That is probably needed for Mesos integration? > > > >>>>>>>>>>> > > > >>>>>>>>>>> - Henry > > > >>>>>>>>>>> > > > >>>>>>>>>>> On Wed, Jun 15, 2016 at 12:55 PM, Robert Metzger < > > > >>>>>>> > > > >>>>>>> rmetz...@apache.org> > > > >>>>>>>>>>> > > > >>>>>>>>>>> wrote: > > > >>>>>>>>>>> > > > >>>>>>>>>>>> Hi, > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Flink 1.0.0 was released early March, so three months have > > > passed > > > >>>>>>> > > > >>>>>>> and > > > >>>>>>>>> > > > >>>>>>>>> I > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> think we should start discussing the scope of the next > major > > > >>>>>>> > > > >>>>>>> release > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> (1.1.0). > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> From a high level point of view, we've added the > following > > > new > > > >>>>>>>>> > > > >>>>>>>>> features: > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> in master: > > > >>>>>>>>>>>> - Table API Refactoring, SQL, StreamSQL > > > >>>>>>>>>>>> - The metrics system > > > >>>>>>>>>>>> - Kinesis Connector > > > >>>>>>>>>>>> - Persistent file sources for streaming > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Half merged: > > > >>>>>>>>>>>> - Resource manager refactoring > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Unmerged features: > > > >>>>>>>>>>>> - Cassandra connector > > > >>>>>>>>>>>> - Key groups ("rescaling from savepoints") > > > >>>>>>>>>>>> - Queryable state > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> I'm pretty sure I forgot many other features / pull > > requests, > > > >>>>>>> > > > >>>>>>> please > > > >>>>>>>>>> > > > >>>>>>>>>> post > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> them to this thread. I'll collect them and create a Wiki > > page > > > out > > > >>>>>>> > > > >>>>>>> of > > > >>>>>>>>> > > > >>>>>>>>> it. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Some immediate TODOs for us: > > > >>>>>>>>>>>> - Which of the unmerged features are we going to add to > the > > > >>>>>>> > > > >>>>>>> release? > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> - Which blockers do we need to address before releasing? > > > >>>>>>>>>>>> - Are there any volunteers for the release manager? > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Regards, > > > >>>>>>>>>>>> Robert > > > >>>>>>>>>>>> > > > >>>> > > > > > >
