Great that we are all on the same page :-) Thanks for pointing out the two issues Aljoscha and Till. I agree with you and I've updated them to blockers ;-)
The FsStateBackend looks like it will be done soon. @Till: do you have time to look into FLINK-4150 this week? I can also do it after I've addressed your comments for the savepoint PRs today. On Tue, Jul 5, 2016 at 9:55 AM, Till Rohrmann <trohrm...@apache.org> wrote: > Stefan Richter found the following problem with HA: > https://issues.apache.org/jira/browse/FLINK-4150 > > I think we should fix it for the 1.1 release. > > On Mon, Jul 4, 2016 at 9:05 PM, Robert Metzger <rmetz...@apache.org> wrote: > >> +1 to do a RC0 this week, but the master-forking with RC1. I would like to >> reduce the time we need to apply patches to multiple branches. >> >> @Aljoscha: I was running into the same issue on EMR when I used Flink w/ >> RocksDB recently, so I agree ;) >> >> >> >> On Mon, Jul 4, 2016 at 3:35 PM, Aljoscha Krettek <aljos...@apache.org> >> wrote: >> >> > IMHO, the fix for this should also go in: >> > https://issues.apache.org/jira/browse/FLINK-4115. This is blocking for >> > users that want to use the RocksDB backend or FsStateBackend on Amazon >> EMR >> > with S3. >> > >> > There is already an open PR that I'm hoping to get in this week. >> > >> > On Mon, 4 Jul 2016 at 13:48 Ufuk Celebi <u...@apache.org> wrote: >> > >> > > Thanks for the feedback. I would like to create a preview RC0 this >> > > week like we did for the last releases, too. In past major releases, >> > > we needed to create many release candidates, often for fixing just >> > > some small issues. I would like to speed up the release process by >> > > collecting as many issues as possible now with the RC0. Once these >> > > issues are resolved, we can start voting with the RC1. This >> > > essentially means that we have a feature freeze. I would create the >> > > release-1.1 branch with RC1. >> > > >> > > Regarding the open issues: >> > > >> > > - The savepoint fixes are not yet in. There is a PR for the savepoint >> > > headers (#2194) and the savepoint disposal PR needs addressing of >> > > comments. >> > > >> > > - The Cassandra connector fixes are either merged or very close to be >> > > merged. >> > > >> > > - Metrics docs are not a blocker since the online docs are updated >> > > after the release. Regarding the renaming, we should decide soon. Any >> > > opinions here? >> > > >> > > - The YARN issues have been resolved. >> > > >> > > I will also make a pass over JIRA and our PRs to check whether we've >> > > missed something. >> > > >> > > @Greg: You are right, the hash-based combine PR has been extensively >> > > reviewed. Unfortunately, I'm not familiar with the code as I didn't >> > > look into it and cannot tell whether we should merge it now. Is the >> > > hash-based combine strategy activated by default or does the user has >> > > to activate it manually? The later case would make me feel more >> > > comfortable merging it for the release. >> > > >> > > >> > > On Fri, Jul 1, 2016 at 7:14 PM, Maximilian Michels <m...@apache.org> >> > wrote: >> > > > Yet another one for the release: FLINK-4144 >> > > > https://github.com/apache/flink/pull/2191 >> > > > >> > > > On Fri, Jul 1, 2016 at 5:18 PM, Maximilian Michels <m...@apache.org> >> > > wrote: >> > > >> There is also FLINK-4141. We just found this during testing. PR is >> > > >> waiting to be merged here: >> https://github.com/apache/flink/pull/2190 >> > > >> >> > > >> On Fri, Jul 1, 2016 at 3:26 PM, Maximilian Michels <m...@apache.org> >> > > wrote: >> > > >>> FLINK-3904 is not Yarn related. Not pressing to fix for this >> release >> > > >>> and actually a bit tricky to fix. I've addressed the other issues >> and >> > > >>> merged all pending pull requests. Good to go from my side. >> > > >>> >> > > >>> On Fri, Jul 1, 2016 at 11:29 AM, Chesnay Schepler < >> > ches...@apache.org> >> > > wrote: >> > > >>>> There are also 2 fixes for Cassandra that should be included: >> > > >>>> https://github.com/apache/flink/pull/2167 >> > > >>>> https://github.com/apache/flink/pull/2183 >> > > >>>> >> > > >>>> We should also include the documentation for the metrics stuff >> > > (hopefully >> > > >>>> merged today) >> > > >>>> https://github.com/apache/flink/pull/2158 >> > > >>>> >> > > >>>> In regards to metrics: To add a counter metric a user currently >> has >> > > to call >> > > >>>> "counter(...)" on >> > > >>>> a MetricGroup. The point was raised in the documentation PR that >> we >> > > may want >> > > >>>> to give >> > > >>>> them a more descriptive name like "addCounter(...)". >> > > >>>> >> > > >>>> I would be in favor of changing them but would like others to >> weigh >> > > in on >> > > >>>> this. IMO we >> > > >>>> should nail this down before 1.1 . >> > > >>>> >> > > >>>> Regards, >> > > >>>> Chesnay >> > > >>>> >> > > >>>> >> > > >>>> On 30.06.2016 22:59, Greg Hogan wrote: >> > > >>>>> >> > > >>>>> It would be great if hash-based combine (FLINK-3477) could make >> it >> > > in to >> > > >>>>> be >> > > >>>>> tested for this release. We've seen impressive improvements in >> > > performance >> > > >>>>> (though, admittedly, some sort-based enhancements are yet to be >> > > worked >> > > >>>>> on). >> > > >>>>> This PR looks to be ripe. >> > > >>>>> >> > > >>>>> Also, as we tidy up a few things with Gelly and documentation, >> what >> > > is the >> > > >>>>> schedule for a feature freeze and creating a 1.1 branch off >> master? >> > > >>>>> >> > > >>>>> Thanks, >> > > >>>>> Greg >> > > >>>>> >> > > >>>>> On Mon, Jun 27, 2016 at 7:23 AM, Robert Metzger < >> > rmetz...@apache.org >> > > > >> > > >>>>> wrote: >> > > >>>>> >> > > >>>>>> Sure Ufuk! Thanks a lot for taking care of the release >> management. >> > > >>>>>> I'll be on vacation in three weeks, for three weeks and I'm not >> > > sure if >> > > >>>>>> we >> > > >>>>>> get the release done until then. >> > > >>>>>> >> > > >>>>>> On Mon, Jun 27, 2016 at 12:08 PM, Ufuk Celebi <u...@apache.org> >> > > wrote: >> > > >>>>>> >> > > >>>>>>> I would like to do it if that's OK with you Robert. I would >> > follow >> > > >>>>>>> your suggestion and wait a few days until the following >> important >> > > >>>>>>> fixes are in: >> > > >>>>>>> - Savepoint headers and proper disposal (FLINK-4067 and >> > > >>>>>>> https://github.com/apache/flink/pull/2083) >> > > >>>>>>> - Metrics (https://github.com/apache/flink/pull/2146) >> > > >>>>>>> - Table API time support ( >> > > https://github.com/apache/flink/pull/2150) >> > > >>>>>>> - Kafka at-least-once Producer ( >> > > >>>>>> >> > > >>>>>> https://github.com/apache/flink/pull/2108) >> > > >>>>>>> >> > > >>>>>>> - Cassandra connector fixes ( >> > > https://github.com/apache/flink/pull/2163) >> > > >>>>>>> - YARN client fixes (FLINK-3675, FLINK-3904 @Max: is there >> > > something >> > > >>>>>> >> > > >>>>>> else?) >> > > >>>>>>> >> > > >>>>>>> >> > > >>>>>>> >> > > >>>>>>> >> > > >>>>>>> On Thu, Jun 23, 2016 at 1:33 PM, Robert Metzger < >> > > rmetz...@apache.org> >> > > >>>>>>> wrote: >> > > >>>>>>>> >> > > >>>>>>>> Hi, >> > > >>>>>>>> it doesn't seem that there are volunteers for the RM, so I'll >> > > probably >> > > >>>>>> >> > > >>>>>> do >> > > >>>>>>>> >> > > >>>>>>>> it. >> > > >>>>>>>> >> > > >>>>>>>> I try to do the first release candidate (mostly for testing) >> > next >> > > week >> > > >>>>>>> >> > > >>>>>>> (it >> > > >>>>>>>> >> > > >>>>>>>> depends on the JIRAs fixed by then) >> > > >>>>>>>> >> > > >>>>>>>> On Thu, Jun 16, 2016 at 10:56 PM, Henry Saputra < >> > > >>>>>> >> > > >>>>>> henry.sapu...@gmail.com >> > > >>>>>>>> >> > > >>>>>>>> wrote: >> > > >>>>>>>> >> > > >>>>>>>>> Thanks for the reply, @Max. I was not aware it was about >> > dynamic >> > > >>>>>>> >> > > >>>>>>> scaling, >> > > >>>>>>>>> >> > > >>>>>>>>> which I think also asked for YARN support. >> > > >>>>>>>>> I agree to list all related half merge JIRA for the >> > > ResourceManager. >> > > >>>>>>>>> >> > > >>>>>>>>> Looking forward for the Apache Mesos integration design for >> > sure >> > > =) >> > > >>>>>>>>> >> > > >>>>>>>>> - Henry >> > > >>>>>>>>> >> > > >>>>>>>>> On Thu, Jun 16, 2016 at 2:12 AM, Maximilian Michels < >> > > m...@apache.org> >> > > >>>>>>>>> wrote: >> > > >>>>>>>>> >> > > >>>>>>>>>> Hi Robert, hi Henry, >> > > >>>>>>>>>> >> > > >>>>>>>>>> +1 for a 1.1.0 release soon! We have enough new features >> that >> > > >>>>>> >> > > >>>>>> justify >> > > >>>>>>>>>> >> > > >>>>>>>>>> a major release. >> > > >>>>>>>>>> >> > > >>>>>>>>>> @Henry We have plans to extend the ResourceManager to >> interact >> > > with >> > > >>>>>>>>>> the Scheduler which will be a prerequisite for dynamic >> > scaling. >> > > I >> > > >>>>>>>>>> think this is out of scope for 1.1.0. The upcoming Mesos >> > > integration >> > > >>>>>>>>>> won't require additional refactoring of the ResourceManager. >> > > >>>>>> >> > > >>>>>> Instead, >> > > >>>>>>>>>> >> > > >>>>>>>>>> we will create a new "Dispatcher" component that takes care >> of >> > > >>>>>>>>>> bootstrapping the initial node with the >> > > JobManager/ResourceManager. >> > > >>>>>>>>>> From there on, everything will be handled by the Mesos >> > > >>>>>>>>>> ResourceManager. I recently discussed this with Eron (CC) >> who >> > > came >> > > >>>>>> >> > > >>>>>> up >> > > >>>>>>>>>> >> > > >>>>>>>>>> with this design and he plans to publish it to the mailing >> > list >> > > >>>>>> >> > > >>>>>> soon. >> > > >>>>>>>>>> >> > > >>>>>>>>>> How about listing relevant JIRA issues here? "Half Merged" >> is >> > > kind >> > > >>>>>> >> > > >>>>>> of >> > > >>>>>>>>>> >> > > >>>>>>>>>> hard to get for people who are not involved in the different >> > > >>>>>>>>>> components. >> > > >>>>>>>>>> >> > > >>>>>>>>>> The Cassandra adapter seems like a pretty important thing to >> > > have >> > > >>>>>> >> > > >>>>>> for >> > > >>>>>>>>>> >> > > >>>>>>>>>> the next release. In addition, I would like to merge >> > FLINK-3667 >> > > and >> > > >>>>>>>>>> FLINK-3937. Robert is doing a review at the moment :) Those >> > are >> > > a) >> > > >>>>>>>>>> refactoring of the command-line and client classes b) adding >> > > >>>>>>>>>> capability to resume cluster programmatically. >> > > >>>>>>>>>> >> > > >>>>>>>>>> Then we should also have a look at any other critical/major >> > bugs >> > > >>>>>>> >> > > >>>>>>> listed >> > > >>>>>>>>> >> > > >>>>>>>>> in >> > > >>>>>>>>>> >> > > >>>>>>>>>> JIRA. >> > > >>>>>>>>>> >> > > >>>>>>>>>> Cheers, >> > > >>>>>>>>>> Max >> > > >>>>>>>>>> >> > > >>>>>>>>>> On Wed, Jun 15, 2016 at 10:50 PM, Henry Saputra < >> > > >>>>>>> >> > > >>>>>>> henry.sapu...@gmail.com >> > > >>>>>>>>>> >> > > >>>>>>>>>> wrote: >> > > >>>>>>>>>>> >> > > >>>>>>>>>>> Hi Robert, >> > > >>>>>>>>>>> >> > > >>>>>>>>>>> Thanks for staying the discussion. >> > > >>>>>>>>>>> >> > > >>>>>>>>>>> Do you know if there any open tasks for the Resource >> Manager >> > > left? >> > > >>>>>>>>>>> >> > > >>>>>>>>>>> That is probably needed for Mesos integration? >> > > >>>>>>>>>>> >> > > >>>>>>>>>>> - Henry >> > > >>>>>>>>>>> >> > > >>>>>>>>>>> On Wed, Jun 15, 2016 at 12:55 PM, Robert Metzger < >> > > >>>>>>> >> > > >>>>>>> rmetz...@apache.org> >> > > >>>>>>>>>>> >> > > >>>>>>>>>>> wrote: >> > > >>>>>>>>>>> >> > > >>>>>>>>>>>> Hi, >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> Flink 1.0.0 was released early March, so three months have >> > > passed >> > > >>>>>>> >> > > >>>>>>> and >> > > >>>>>>>>> >> > > >>>>>>>>> I >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> think we should start discussing the scope of the next >> major >> > > >>>>>>> >> > > >>>>>>> release >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> (1.1.0). >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> From a high level point of view, we've added the >> following >> > > new >> > > >>>>>>>>> >> > > >>>>>>>>> features: >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> in master: >> > > >>>>>>>>>>>> - Table API Refactoring, SQL, StreamSQL >> > > >>>>>>>>>>>> - The metrics system >> > > >>>>>>>>>>>> - Kinesis Connector >> > > >>>>>>>>>>>> - Persistent file sources for streaming >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> Half merged: >> > > >>>>>>>>>>>> - Resource manager refactoring >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> Unmerged features: >> > > >>>>>>>>>>>> - Cassandra connector >> > > >>>>>>>>>>>> - Key groups ("rescaling from savepoints") >> > > >>>>>>>>>>>> - Queryable state >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> I'm pretty sure I forgot many other features / pull >> > requests, >> > > >>>>>>> >> > > >>>>>>> please >> > > >>>>>>>>>> >> > > >>>>>>>>>> post >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> them to this thread. I'll collect them and create a Wiki >> > page >> > > out >> > > >>>>>>> >> > > >>>>>>> of >> > > >>>>>>>>> >> > > >>>>>>>>> it. >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> Some immediate TODOs for us: >> > > >>>>>>>>>>>> - Which of the unmerged features are we going to add to >> the >> > > >>>>>>> >> > > >>>>>>> release? >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> - Which blockers do we need to address before releasing? >> > > >>>>>>>>>>>> - Are there any volunteers for the release manager? >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> Regards, >> > > >>>>>>>>>>>> Robert >> > > >>>>>>>>>>>> >> > > >>>> >> > > >> > >>
