I'm a bit concerned about end of October, because it means we have Flink
forward, which usually means at least 1 week of little-to-no activity,
and then 1 week until feature-freeze.
On 05/08/2020 11:56, jincheng sun wrote:
+1 for end of October from me as well.
Best,
Jincheng
Kostas Kloudas <kklou...@gmail.com> 于2020年8月5日周三 下午4:59写道：
+1 for end of October from me as well.
Cheers,
Kostas
On Wed, Aug 5, 2020 at 9:59 AM Till Rohrmann <trohrm...@apache.org> wrote:
+1 for end of October from my side as well.
Cheers,
Till
On Tue, Aug 4, 2020 at 9:46 PM Stephan Ewen <se...@apache.org> wrote:
The end of October sounds good from my side, unless it collides with
some
holidays that affect many committers.
Feature-wise, I believe we can definitely make good use of the time to
wrap
up some critical threads (like finishing the FLIP-27 source efforts).
So +1 to the end of October from my side.
Best,
Stephan
On Tue, Aug 4, 2020 at 8:59 AM Robert Metzger <rmetz...@apache.org>
wrote:
Thanks a lot for commenting on the feature freeze date.
You are raising a few good points on the timing.
If we have already (2 months before) concerns regarding the deadline,
then
I agree that we should move it till the end of October.
We then just need to be careful not to run into the Christmas season
at
the
end of December.
If nobody objects within a few days, I'll update the feature freeze
date
in
the Wiki.
On Tue, Aug 4, 2020 at 7:52 AM Kurt Young <ykt...@gmail.com> wrote:
Regarding setting the feature freeze date to late September, I have
some
concern that it might make
the development time of 1.12 too short.
One reason for this is we took too much time (about 1.5 month, from
mid
of
May to beginning of July)
for testing 1.11. It's not ideal but further squeeze the
development
time
of 1.12 won't make this better.
Besides, AFAIK July & August is also a popular vacation season for
European. Given the fact most
committers of Flink come from Europe, I think we should also take
this
into consideration.
It's also true that the first week of October is the national
holiday
of
China, so I'm wondering whether the
end of October could be a candidate feature freeze date.
Best,
Kurt
On Tue, Jul 28, 2020 at 2:41 AM Robert Metzger <
rmetz...@apache.org>
wrote:
Hi all,
Thanks a lot for the responses so far. I've put them into this
Wiki
page:
https://cwiki.apache.org/confluence/display/FLINK/1.12+Release
to
keep
track of them. Ideally, post JIRA tickets for your feature, then
the
status
will update automatically in the wiki :)
Please keep posting features here, or add them to the Wiki
yourself
🙏
@Prasanna kumar <prasannakumarram...@gmail.com>: Dynamic Auto
Scaling
is a
feature request the community is well-aware of. Till has posted
"Reactive-scaling mode" as a feature he's working on for the 1.12
release.
This work will introduce the basic building blocks and partial
support
for
the feature you are requesting.
Proper support for dynamic scaling, while maintaining Flink's
high
performance (throughout, low latency) and correctness is a
difficult
task
that needs a lot of work. It will probably take a little bit of
time
till
this is fully available.
Cheers,
Robert
On Thu, Jul 23, 2020 at 2:27 PM Till Rohrmann <
trohrm...@apache.org>
wrote:
Thanks for being our release managers for the 1.12 release
Dian &
Robert!
Here are some features I would like to work on for this
release:
# Features
## Finishing pipelined region scheduling (
https://issues.apache.org/jira/browse/FLINK-16430)
With the pipelined region scheduler we want to implement a
scheduler
which
can serve streaming as well as batch workloads alike while
being
able
to
run jobs under constrained resources. The latter is
particularly
important
for bounded streaming jobs which, currently, are not well
supported.
## Reactive-scaling mode
Being able to react to newly available resources and rescaling
a
running
job accordingly will make Flink's operation much easier because
resources
can then be controlled by an external tool (e.g. GCP
autoscaling,
K8s
horizontal pod scaler, etc.). In this release we want to make a
big
step
towards this direction. As a first step we want to support the
execution
of
jobs with a parallelism which is lower than the specified
parallelism
in
case that Flink lost a TaskManager or could not acquire enough
resources.
# Maintenance/Stability
## JM / TM finished task reconciliation (
https://issues.apache.org/jira/browse/FLINK-17075)
This prevents the system from going out of sync if a task state
change
from
the TM to the JM is lost.
## Make metrics services work with Kubernetes deployments (
https://issues.apache.org/jira/browse/FLINK-11127)
Invert the direction in which the MetricFetcher connects to the
MetricQueryFetchers. That way it will no longer be necessary to
expose
on
K8s for every TaskManager a port on which the
MetricQueryFetcher
runs.
This
will then make the deployment of Flink clusters on K8s easier.
## Handle long-blocking operations during job submission
(savepoint
restore) (https://issues.apache.org/jira/browse/FLINK-16866)
Submitting a Flink job can involve the interaction with
external
systems
(blocking operations). Depending on the job the interactions
can
take
so
long that it exceeds the submission timeout which reports a
failure
on
the
client side even though the actual submission succeeded. By
decoupling
the
creation of the ExecutionGraph from the job submission, we can
make
the
job
submission non-blocking which will solve this problem.
## Make IDs more intuitive to ease debugging (FLIP-118) (
https://issues.apache.org/jira/browse/FLINK-15679)
By making the internal Flink IDs compositional or logging how
they
belong
together, we can make the debugging of Flink's operations much
easier.
Cheers,
Till
On Thu, Jul 23, 2020 at 7:48 AM Canbin Zheng <
felixzhen...@gmail.com
wrote:
Hi All,
Thanks for bring-up this discussion, Robert!
Congratulations on becoming the release manager of 1.12， Dian
and
Robert
!
----------
Here are some of my thoughts of the features for native
integration
with
Kubernetes in Flink 1.12:
1. Support user-specified pod templates
Description:
The current approach of introducing new configuration
options
for
each
aspect of pod specification a user might wish is becoming
unwieldy,
we
have
to maintain more and more Flink side Kubernetes configuration
options
and
users have to learn the gap between the declarative model
used
by
Kubernetes and the configuration model used by Flink. It's a
great
improvement to allow users to specify pod templates as
central
places
for
all customization needs for the jobmanager and taskmanager
pods.
Benefits:
Users can leverage many of the advanced K8s features that
the
Flink
community does not support explicitly, such as volume
mounting,
DNS
configuration, pod affinity/anti-affinity setting, etc.
2. Support running PyFlink on Kubernetes
Description:
Support running PyFlink on Kubernetes, including session
cluster
and
application cluster.
Benefits:
Running python application in a containerized
environment.
3. Support built-in init-Container
Description:
We need a built-in init-Container to help solve
dependency
management
in a containerized environment, especially in the application
mode.
Benefits:
Separate the base Flink image from dynamic dependencies.
4. Support accessing secured services via K8s secrets
Description:
Kubernetes Secrets
<https://kubernetes.io/docs/concepts/configuration/secret/>
can
be
used
to
provide credentials for a Flink application to access secured
services.
It
helps people who want to use a user-specified K8s Secret
through
an
environment variable.
Benefits:
Improve user experience.
5. Support configuring replica of JobManager Deployment in
ZooKeeper
HA
setups
Description:
Make the *replica* of Deployment configurable in the
ZooKeeper
HA
setups.
Benefits:
Achieve faster failover.
6. Support to configure limit for CPU requirement
Description:
To leverage the Kubernetes feature of container
request/limit
CPU.
Benefits:
Reduce cost.
Regards,
Canbin Zheng
Harold.Miao <miaohong...@gmail.com> 于2020年7月23日周四 下午12:44写道：
I'm excited to hear about this feature, very, very, very
highly
encouraged
Prasanna kumar <prasannakumarram...@gmail.com>
于2020年7月23日周四
上午12:10写道：
Hi Flink Dev Team,
Dynamic AutoScaling Based on the incoming data load would
be
a
great
feature.
We should be able have some rule say If the load
increased
by
20% ,
add
extra resource should be added.
Or time based say during these peak hours the pipeline
should
scale
automatically by 50%.
This will help a lot in cost reduction.
EMR cluster provides a similar feature for SPARK based
application.
Thanks,
Prasanna.
On Wed, Jul 22, 2020 at 5:40 PM Robert Metzger <
rmetz...@apache.org>
wrote:
Hi all,
Now that the 1.11 release is out, it is time to plan
for
the
next
major
Flink release.
Some items:
1.
Dian Fu and me volunteer to be the release managers
for
Flink
1.12.
1.
Timeline: We propose to stick to our approximate 4
month
release
cycle,
thus the release should be done by late October.
Given
that
there’s
a
holiday week in China at the beginning of October, I
propose
to
do
the
feature freeze on master by late September.
2.
Collecting features: It would be good to have a
rough
overview
of
the
features that will likely be ready to be merged by
late
September,
and
that
we want in the release.
Based on the discussion, we will update the Roadmap
on
the
Flink
website
again!
1.
Test instabilities and blockers: I would like to
avoid a
situation
where
we have many blocking issues or build instabilities
at
the
time
of
the
feature freeze. To achieve that, we will try to
check
every
build
instability within a week, to decide if it is a
blocker
(make
sure
to
use
the “test-stability” label for those tickets!)
Blocker issues will need to have somebody assigned
(responsible)
within
a week, and we want to see progress on all blocker
issues
(downgrade,
resolution, a good plan how to proceed if it is more
complicated)
2.
Quality and stability of new features: In order to
have
a
short
feature
freeze phase, we encourage developers to only merge
well-tested
and
documented features. In our experience, the feature
freeze
works
best
if
new features are complete, and the community can
focus
fully
on
addressing
newly found bugs and voting the release.
By having a smooth release process, the next
merge-window
for
the
next
release will come sooner.
Let me know what you think about our items, and share
which
features
you
want in Flink 1.12.
Best,
Robert & Dian
--
Best Regards,
Harold Miao
