Re: [ANNOUNCE] New Apache Flink Committer - Xintong Song
Congratulations Xintong! Best, Jark On Fri, 5 Jun 2020 at 14:32, Danny Chan wrote: > Congratulations Xintong ! > > Best, > Danny Chan > 在 2020年6月5日 +0800 PM2:20，dev@flink.apache.org，写道： > > > > Congratulations Xintong >
[jira] [Created] (FLINK-18146) The note of StreamingJobGraphGenerator#createJobGraph has a misspelling
ZhuShang created FLINK-18146:
Summary: The note of StreamingJobGraphGenerator#createJobGraph
has a misspelling
Key: FLINK-18146
URL: https://issues.apache.org/jira/browse/FLINK-18146
Project: Flink
Issue Type: Bug
Components: Documentation
Affects Versions: 1.10.0
Reporter: ZhuShang
The note of
StreamingJobGraphGenerator#createJobGraph in line 160 has a misspelling.
{code:java}
private JobGraph createJobGraph() {
...
// Generate deterministic hashes for the nodes in order to identify them across
// submission iff they didn't change.
Map hashes =
defaultStreamGraphHasher.traverseStreamGraphAndGenerateHashes(streamGraph);
...
}
{code}
Is 'iff' should be 'if'?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18147) Orc document display is disordered
Shuai Xia created FLINK-18147: - Summary: Orc document display is disordered Key: FLINK-18147 URL: https://issues.apache.org/jira/browse/FLINK-18147 Project: Flink Issue Type: Bug Components: Documentation, FileSystems Affects Versions: 1.11.0 Reporter: Shuai Xia Fix For: 1.11.0 Official documents show that there is a problem. link: [https://ci.apache.org/projects/flink/flink-docs-master/dev/connectors/streamfile_sink.html#tab_java_2] -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [ANNOUNCE] New Apache Flink Committer - Xintong Song
Congrats, Xintong! Best, tison. Jark Wu 于2020年6月5日周五 下午3:00写道： > Congratulations Xintong! > > Best, > Jark > > On Fri, 5 Jun 2020 at 14:32, Danny Chan wrote: > > > Congratulations Xintong ! > > > > Best, > > Danny Chan > > 在 2020年6月5日 +0800 PM2:20，dev@flink.apache.org，写道： > > > > > > Congratulations Xintong > > >
Re: [ANNOUNCE] New Apache Flink Committer - Xintong Song
Congratulations! Xintong! Best, Congxian Jark Wu 于2020年6月5日周五 下午3:00写道： > Congratulations Xintong! > > Best, > Jark > > On Fri, 5 Jun 2020 at 14:32, Danny Chan wrote: > > > Congratulations Xintong ! > > > > Best, > > Danny Chan > > 在 2020年6月5日 +0800 PM2:20，dev@flink.apache.org，写道： > > > > > > Congratulations Xintong > > >
[jira] [Created] (FLINK-18148) "Resuming Savepoint" e2e fails with TimeoutException in CliFrontend.stop()
Robert Metzger created FLINK-18148:
--
Summary: "Resuming Savepoint" e2e fails with TimeoutException in
CliFrontend.stop()
Key: FLINK-18148
URL: https://issues.apache.org/jira/browse/FLINK-18148
Project: Flink
Issue Type: Bug
Components: Command Line Client
Affects Versions: 1.11.0
Reporter: Robert Metzger
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=2759&view=logs&j=c88eea3b-64a0-564d-0031-9fdcd7b8abee&t=1e2bbe5b-4657-50be-1f07-d84bfce5b1f5
{code}
The program finished with the following exception:
org.apache.flink.util.FlinkException: Could not stop with a savepoint job
"081bda854bc250e01055ed1ba9d43178".
at
org.apache.flink.client.cli.CliFrontend.lambda$stop$5(CliFrontend.java:495)
at
org.apache.flink.client.cli.CliFrontend.runClusterAction(CliFrontend.java:864)
at org.apache.flink.client.cli.CliFrontend.stop(CliFrontend.java:487)
at
org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:931)
at
org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:992)
at
org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:992)
Caused by: java.util.concurrent.TimeoutException
at
java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1784)
at
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
at
org.apache.flink.client.cli.CliFrontend.lambda$stop$5(CliFrontend.java:493)
... 6 more
Waiting for job (081bda854bc250e01055ed1ba9d43178) to reach terminal state
FINISHED ...
Job (081bda854bc250e01055ed1ba9d43178) reached terminal state FINISHED
Savepoint location was empty. This may mean that the stop-with-savepoint failed.
[FAIL] Test script contains errors.
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [Discussion] flink elasticsearch connector supports
hi Jacky Lau :
I agree with jark's point of view, the use of es is not just to read data,
more use is to group query, aggregate these.
Best,
Forward
Jacky Lau 于2020年6月5日周五 下午2:47写道：
> hi Etienne Chauchot:
> you can read here https://www.jianshu.com/p/d32e17dab90c, which is
> chinese.But you can konw that slice api has poor performance in es-hadoop
> project .
>
> And i found that es-hadoop has removed this and disable sliced scrolls by
> default. you can see below, which i found in the lastest es-hadoop release
> version
> Configuration Changes
> `es.input.use.sliced.partitions` is deprecated in 6.5.0, and will be
> removed
> in 7.0.0. The default value for `es.input.max.docs.per.partition` (10)
> will also be removed in 7.0.0, thus disabling sliced scrolls by default,
> and
> switching them to be an explicitly opt-in feature.
>
> added[5.0.0]
> `es.input.max.docs.per.partition` ::
> When reading from an {es} cluster that supports scroll slicing ({es} v5.0.0
> and above), this parameter advises the
> connector on what the maximum number of documents per input partition
> should
> be. The connector will sample and estimate
> the number of documents on each shard to be read and divides each shard
> into
> input slices using the value supplied by
> this property. This property is a suggestion, not a guarantee. The final
> number of documents per partition is not
> guaranteed to be below this number, but rather, they will be close to this
> number. This property is ignored if you are
> reading from an {es} cluster that does not support scroll slicing ({es} any
> version below v5.0.0). By default, this
> value is unset, and the input partitions are calculated based on the number
> of shards in the indices being read.
>
>
>
> Jacky Lau wrote
> > hi Etienne Chauchot:
> > thanks for your discussion.
> > for 1) we do not supprt es unbouded source currently
> >
> > for 2) RichParallelSourceFunction is used for streaming ,InputFormat is
> > for
> > batch
> >
> > for 3) i downloaded beam just now. and the beam es connector is also
> > using
> > es-hadoop. i have read the code of es-hadoop(inputsplit contains shard
> and
> > slice. And i think it is better when diffirent shard has diffirent number
> > of
> > docs), which you can seed here
> > .https://github.com/elastic/elasticsearch-hadoop. But the code is not
> > good.
> > so we do not want to reference . and you can see presto, there is also
> > just
> > using inputsplit with shard not contains slice
> >
> > for 4) because flink es connectro has alreay using diffrent client (es 5
> > for
> > tranport client, es 6,7 for highlevelrest), we just reuse it,which will
> > not
> > change too much code
> >
> >
> >
> > --
> > Sent from:
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/
>
>
>
>
>
> --
> Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/
>
Re: [DISCUSS] Releasing "fat" and "slim" Flink distributions
+1 for Jingsong’s proposal to put flink-csv, flink-json and flink-avro under lib/ directory. I have heard many SQL users(most of newbies) complaint the out-of-box experience in mail list. Best, Leonard Xu > 在 2020年6月5日，14:39，Benchao Li 写道： > > +1 to include them for sql-client by default; > +0 to put into lib and exposed to all kinds of jobs, including DataStream. > > Danny Chan 于2020年6月5日周五 下午2:31写道： > >> +1, at least, we should keep an out of the box SQL-CLI, it’s very poor >> experience to add such required format jars for SQL users. >> >> Best, >> Danny Chan >> 在 2020年6月5日 +0800 AM11:14，Jingsong Li ，写道： >>> Hi all, >>> >>> Considering that 1.11 will be released soon, what about my previous >>> proposal? Put flink-csv, flink-json and flink-avro under lib. >>> These three formats are very small and no third party dependence, and >> they >>> are widely used by table users. >>> >>> Best, >>> Jingsong Lee >>> >>> On Tue, May 12, 2020 at 4:19 PM Jingsong Li >> wrote: >>> Thanks for your discussion. Sorry to start discussing another thing: The biggest problem I see is the variety of problems caused by users' >> lack of format dependency. As Aljoscha said, these three formats are very small and no third party dependence, and they are widely used by table users. Actually, we don't have any other built-in table formats now... In >> total 151K... 73K flink-avro-1.10.0.jar 36K flink-csv-1.10.0.jar 42K flink-json-1.10.0.jar So, Can we just put them into "lib/" or flink-table-uber? It not solve all problems and maybe it is independent of "fat" and >> "slim". But also improve usability. What do you think? Any objections? Best, Jingsong Lee On Mon, May 11, 2020 at 5:48 PM Chesnay Schepler wrote: > One downside would be that we're shipping more stuff when running on > YARN for example, since the entire plugins directory is shiped by >> default. > > On 17/04/2020 16:38, Stephan Ewen wrote: >> @Aljoscha I think that is an interesting line of thinking. the >> swift-fs > may >> be rarely enough used to move it to an optional download. >> >> I would still drop two more thoughts: >> >> (1) Now that we have plugins support, is there a reason to have a > metrics >> reporter or file system in /opt instead of /plugins? They don't >> spoil > the >> class path any more. >> >> (2) I can imagine there still being a desire to have a "minimal" >> docker >> file, for users that want to keep the container images as small as >> possible, to speed up deployment. It is fine if that would not be >> the >> default, though. >> >> >> On Fri, Apr 17, 2020 at 12:16 PM Aljoscha Krettek < >> aljos...@apache.org> >> wrote: >> >>> I think having such tools and/or tailor-made distributions can >> be nice >>> but I also think the discussion is missing the main point: The >> initial >>> observation/motivation is that apparently a lot of users (Kurt >> and I >>> talked about this) on the chinese DingTalk support groups, and >> other >>> support channels have problems when first using the SQL client >> because >>> of these missing connectors/formats. For these, having >> additional tools >>> would not solve anything because they would also not take that >> extra >>> step. I think that even tiny friction should be avoided because >> the >>> annoyance from it accumulates of the (hopefully) many users that >> we > want >>> to have. >>> >>> Maybe we should take a step back from discussing the >> "fat"/"slim" idea >>> and instead think about the composition of the current dist. As >>> mentioned we have these jars in opt/: >>> >>> 17M flink-azure-fs-hadoop-1.10.0.jar >>> 52K flink-cep-scala_2.11-1.10.0.jar >>> 180K flink-cep_2.11-1.10.0.jar >>> 746K flink-gelly-scala_2.11-1.10.0.jar >>> 626K flink-gelly_2.11-1.10.0.jar >>> 512K flink-metrics-datadog-1.10.0.jar >>> 159K flink-metrics-graphite-1.10.0.jar >>> 1.0M flink-metrics-influxdb-1.10.0.jar >>> 102K flink-metrics-prometheus-1.10.0.jar >>> 10K flink-metrics-slf4j-1.10.0.jar >>> 12K flink-metrics-statsd-1.10.0.jar >>> 36M flink-oss-fs-hadoop-1.10.0.jar >>> 28M flink-python_2.11-1.10.0.jar >>> 22K flink-queryable-state-runtime_2.11-1.10.0.jar >>> 18M flink-s3-fs-hadoop-1.10.0.jar >>> 31M flink-s3-fs-presto-1.10.0.jar >>> 196K flink-shaded-netty-tcnative-dynamic-2.0.25.Final-9.0.jar >>> 518K flink-sql-client_2.11-1.10.0.jar >>> 99K flink-state-processor-api_2.11-1.10.0.jar >>> 25M flink-swift-fs-hadoop-1.10.0.jar >>> 160M opt >>> >>> The "filesystem" connectors ar ethe heavy hitters, there. >>> >>> I downloaded most of the SQL connectors/formats and this is what >> I got: >>> >>>
Re: [ANNOUNCE] New Apache Flink Committer - Xintong Song
Congratulations Xintong, well deserved! Best Regards, Yu On Fri, 5 Jun 2020 at 15:12, Congxian Qiu wrote: > Congratulations! Xintong! > Best, > Congxian > > > Jark Wu 于2020年6月5日周五 下午3:00写道： > > > Congratulations Xintong! > > > > Best, > > Jark > > > > On Fri, 5 Jun 2020 at 14:32, Danny Chan wrote: > > > > > Congratulations Xintong ! > > > > > > Best, > > > Danny Chan > > > 在 2020年6月5日 +0800 PM2:20，dev@flink.apache.org，写道： > > > > > > > > Congratulations Xintong > > > > > >
[jira] [Created] (FLINK-18149) Taskmanager logs could not show up in native K8s deployment
Yang Wang created FLINK-18149:
-
Summary: Taskmanager logs could not show up in native K8s
deployment
Key: FLINK-18149
URL: https://issues.apache.org/jira/browse/FLINK-18149
Project: Flink
Issue Type: Bug
Components: Deployment / Kubernetes
Affects Versions: 1.11.0, 1.12.0
Reporter: Yang Wang
Fix For: 1.11.0, 1.12.0
In FLINK-17935, we use {{flinkConfig.get(DeploymentOptionsInternal.CONF_DIR)}}
to replace {{CliFrontend.getConfigurationDirectoryFromEnv}}. It will cause
problem in native K8s integration. The root cause we set the
{{DeploymentOptionsInternal.CONF_DIR}} in flink-conf.yaml to a local path.
However, it does not exist in JobManager pod.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [ANNOUNCE] New Apache Flink Committer - Xintong Song
Congrats Xintong! On Fri, Jun 5, 2020 at 3:12 PM Congxian Qiu wrote: > Congratulations! Xintong! > Best, > Congxian > > > Jark Wu 于2020年6月5日周五 下午3:00写道： > > > Congratulations Xintong! > > > > Best, > > Jark > > > > On Fri, 5 Jun 2020 at 14:32, Danny Chan wrote: > > > > > Congratulations Xintong ! > > > > > > Best, > > > Danny Chan > > > 在 2020年6月5日 +0800 PM2:20，dev@flink.apache.org，写道： > > > > > > > > Congratulations Xintong > > > > > > -- Best regards! Rui Li
Re: [DISCUSS] Releasing "fat" and "slim" Flink distributions
+1 to add light-weighted formats into the lib On Fri, Jun 5, 2020 at 3:28 PM Leonard Xu wrote: > +1 for Jingsong’s proposal to put flink-csv, flink-json and flink-avro > under lib/ directory. > I have heard many SQL users(most of newbies) complaint the out-of-box > experience in mail list. > > Best, > Leonard Xu > > > > 在 2020年6月5日，14:39，Benchao Li 写道： > > > > +1 to include them for sql-client by default; > > +0 to put into lib and exposed to all kinds of jobs, including > DataStream. > > > > Danny Chan 于2020年6月5日周五 下午2:31写道： > > > >> +1, at least, we should keep an out of the box SQL-CLI, it’s very poor > >> experience to add such required format jars for SQL users. > >> > >> Best, > >> Danny Chan > >> 在 2020年6月5日 +0800 AM11:14，Jingsong Li ，写道： > >>> Hi all, > >>> > >>> Considering that 1.11 will be released soon, what about my previous > >>> proposal? Put flink-csv, flink-json and flink-avro under lib. > >>> These three formats are very small and no third party dependence, and > >> they > >>> are widely used by table users. > >>> > >>> Best, > >>> Jingsong Lee > >>> > >>> On Tue, May 12, 2020 at 4:19 PM Jingsong Li > >> wrote: > >>> > Thanks for your discussion. > > Sorry to start discussing another thing: > > The biggest problem I see is the variety of problems caused by users' > >> lack > of format dependency. > As Aljoscha said, these three formats are very small and no third > party > dependence, and they are widely used by table users. > Actually, we don't have any other built-in table formats now... In > >> total > 151K... > > 73K flink-avro-1.10.0.jar > 36K flink-csv-1.10.0.jar > 42K flink-json-1.10.0.jar > > So, Can we just put them into "lib/" or flink-table-uber? > It not solve all problems and maybe it is independent of "fat" and > >> "slim". > But also improve usability. > What do you think? Any objections? > > Best, > Jingsong Lee > > On Mon, May 11, 2020 at 5:48 PM Chesnay Schepler > wrote: > > > One downside would be that we're shipping more stuff when running on > > YARN for example, since the entire plugins directory is shiped by > >> default. > > > > On 17/04/2020 16:38, Stephan Ewen wrote: > >> @Aljoscha I think that is an interesting line of thinking. the > >> swift-fs > > may > >> be rarely enough used to move it to an optional download. > >> > >> I would still drop two more thoughts: > >> > >> (1) Now that we have plugins support, is there a reason to have a > > metrics > >> reporter or file system in /opt instead of /plugins? They don't > >> spoil > > the > >> class path any more. > >> > >> (2) I can imagine there still being a desire to have a "minimal" > >> docker > >> file, for users that want to keep the container images as small as > >> possible, to speed up deployment. It is fine if that would not be > >> the > >> default, though. > >> > >> > >> On Fri, Apr 17, 2020 at 12:16 PM Aljoscha Krettek < > >> aljos...@apache.org> > >> wrote: > >> > >>> I think having such tools and/or tailor-made distributions can > >> be nice > >>> but I also think the discussion is missing the main point: The > >> initial > >>> observation/motivation is that apparently a lot of users (Kurt > >> and I > >>> talked about this) on the chinese DingTalk support groups, and > >> other > >>> support channels have problems when first using the SQL client > >> because > >>> of these missing connectors/formats. For these, having > >> additional tools > >>> would not solve anything because they would also not take that > >> extra > >>> step. I think that even tiny friction should be avoided because > >> the > >>> annoyance from it accumulates of the (hopefully) many users that > >> we > > want > >>> to have. > >>> > >>> Maybe we should take a step back from discussing the > >> "fat"/"slim" idea > >>> and instead think about the composition of the current dist. As > >>> mentioned we have these jars in opt/: > >>> > >>> 17M flink-azure-fs-hadoop-1.10.0.jar > >>> 52K flink-cep-scala_2.11-1.10.0.jar > >>> 180K flink-cep_2.11-1.10.0.jar > >>> 746K flink-gelly-scala_2.11-1.10.0.jar > >>> 626K flink-gelly_2.11-1.10.0.jar > >>> 512K flink-metrics-datadog-1.10.0.jar > >>> 159K flink-metrics-graphite-1.10.0.jar > >>> 1.0M flink-metrics-influxdb-1.10.0.jar > >>> 102K flink-metrics-prometheus-1.10.0.jar > >>> 10K flink-metrics-slf4j-1.10.0.jar > >>> 12K flink-metrics-statsd-1.10.0.jar > >>> 36M flink-oss-fs-hadoop-1.10.0.jar > >>> 28M flink-python_2.11-1.10.0.jar > >>> 22K flink-queryable-state-runtime_2.11-1.10.0.jar > >>> 18M flink-s3-fs-hadoop-1.10.0.jar > >>> 31M flink-s3-fs-presto-1.10.0.jar > >>> 196K flink-shaded-netty-tcnative-dynamic-2.0.25.Final-9.0
Re: [ANNOUNCE] New Apache Flink Committer - Xintong Song
Congratulations! Best, Dawid On 05/06/2020 09:10, tison wrote: > Congrats, Xintong! > > Best, > tison. > > > Jark Wu 于2020年6月5日周五 下午3:00写道： > >> Congratulations Xintong! >> >> Best, >> Jark >> >> On Fri, 5 Jun 2020 at 14:32, Danny Chan wrote: >> >>> Congratulations Xintong ! >>> >>> Best, >>> Danny Chan >>> 在 2020年6月5日 +0800 PM2:20，dev@flink.apache.org，写道： Congratulations Xintong signature.asc Description: OpenPGP digital signature
[jira] [Created] (FLINK-18150) A single failing Kafka broker may cause jobs to fail indefinitely with TimeoutException: Timeout expired while fetching topic metadata
Julius Michaelis created FLINK-18150:
Summary: A single failing Kafka broker may cause jobs to fail
indefinitely with TimeoutException: Timeout expired while fetching topic
metadata
Key: FLINK-18150
URL: https://issues.apache.org/jira/browse/FLINK-18150
Project: Flink
Issue Type: Bug
Components: Connectors / Kafka
Affects Versions: 1.10.1
Environment: It is a bit unclear to me under what circumstances this
can be reproduced. I created a "minimum" non-working example at
https://github.com/jcaesar/flink-kafka-ha-failure. Note that this deploys the
minimum number of Kafka brokers, but it works just as well with replication
factor 3 and 8 brokers, e.g.
I run this with
{code:bash}
docker-compose kill; and docker-compose rm -vf; and docker-compose up
--abort-on-container-exit --build
{code}
The exception should appear on the webui after 5~6 minutes.
You verify that the Kafka cluster is running "normally" e.g. with:
{code:bash}
kafkacat -b localhost,localhost:9093 -L
{code}
So far, I only know that
* {{flink.partition-discovery.interval-millis}} must be set.
* The broker that failed must be part of the {{bootstrap.servers}}
* There needs to be a certain amount of topics or producers, but I'm unsure
which is crucial
* Changing the values of {{metadata.request.timeout.ms}} or
{{flink.partition-discovery.interval-millis}} does not seem to have any effect.
Reporter: Julius Michaelis
When a Kafka broker fails that is listed among the bootstrap servers and
partition discovery is active, the Flink job reading from that Kafka may enter
a failing loop.
At first, the job seems to react normally without failure with only a short
latency spike when switching Kafka leaders.
Then, it fails with a
{code:plain}
org.apache.flink.streaming.connectors.kafka.internal.Handover$ClosedException
at
org.apache.flink.streaming.connectors.kafka.internal.Handover.close(Handover.java:182)
at
org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher.cancel(KafkaFetcher.java:175)
at
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.cancel(FlinkKafkaConsumerBase.java:821)
at
org.apache.flink.streaming.api.operators.StreamSource.cancel(StreamSource.java:147)
at
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.cancelTask(SourceStreamTask.java:136)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.cancel(StreamTask.java:602)
at
org.apache.flink.runtime.taskmanager.Task$TaskCanceler.run(Task.java:1355)
at java.lang.Thread.run(Thread.java:748)
{code}
It recovers, but processes fewer than the expected amount of records.
Finally, the job fails with
{code:plain}
2020-06-05 13:59:37
org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching
topic metadata
{code}
and repeats doing so while not processing any records.
I have also observed this behavior with partition-discovery turned off, but
only when the Flink job failed (after a broker failure) and had to run
checkpoint recovery for some other reason.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [ANNOUNCE] New Apache Flink Committer - Xintong Song
Congratulations, Xintong Best Yun Tang From: Dawid Wysakowicz Sent: Friday, June 05, 2020 16:00 To: dev@flink.apache.org Subject: Re: [ANNOUNCE] New Apache Flink Committer - Xintong Song Congratulations! Best, Dawid On 05/06/2020 09:10, tison wrote: > Congrats, Xintong! > > Best, > tison. > > > Jark Wu 于2020年6月5日周五 下午3:00写道： > >> Congratulations Xintong! >> >> Best, >> Jark >> >> On Fri, 5 Jun 2020 at 14:32, Danny Chan wrote: >> >>> Congratulations Xintong ! >>> >>> Best, >>> Danny Chan >>> 在 2020年6月5日 +0800 PM2:20，dev@flink.apache.org，写道： Congratulations Xintong
Re: [VOTE] Apache Flink Stateful Functions 2.1.0, release candidate #1
Thanks a lot for creating this release Gordon! +1 (binding) - maven staging repo looks fine (version tags, license files) - source archive looks good (no binaries, no unexpected files, pom has right version) - quickly checked diff: https://github.com/apache/flink-statefun/compare/release-2.0.0-rc6...release-2.1.0-rc1 On Fri, Jun 5, 2020 at 5:05 AM Congxian Qiu wrote: > @Tzu-Li (Gordon) Tai Thanks for the info. `mvn > clean > install -Prun=e2e-tests` works for me. before verified demo on a clean > source directory. > > Best, > Congxian > > > Tzu-Li (Gordon) Tai 于2020年6月4日周四 下午6:35写道： > > > +1 (binding) > > > > Legal > > > > - Verified signatures and hashes of staged Maven artifacts, source > > distribution and Python SDK distribution > > - Checked NOTICE file of statefun-flink-distribution and > > statefun-ridesharing-example-simulator > > > > Functional > > > > - Full build with end-to-end-tests, JDK 8: mvn clean install > > -Prun-e2e-tests > > - Manually verified state TTL for remote functions > > - Manually verified checkpointing with failure recovery > > - Manually verified savepointing + manual restore > > - Generated quickstart project from archetype works > > > > > > On Thu, Jun 4, 2020 at 3:10 PM Hequn Cheng wrote: > > > > > +1 (binding) > > > > > > - Signatures and hash are correct. > > > - All artifacts to be released to Maven in the staging Nexus > repository. > > > - Verify that the source archives do not contain any binaries. > > > - Go through all commits from the last release. No license problem > > spotted. > > > - Check end-to-end tests. All tests have been passed on Travis(both for > > JDK > > > 1.8 and 1.11). > > > > > > Best, > > > Hequn > > > > > > On Thu, Jun 4, 2020 at 12:50 PM Tzu-Li (Gordon) Tai < > tzuli...@apache.org > > > > > > wrote: > > > > > > > Hi Hequn, > > > > > > > > Sorry, I mis-tagged the wrong commit. > > > > Just fixed this, the tag [1] [2] should now be pointing to the > correct > > > > commit that contains the updated version. > > > > > > > > Gordon > > > > > > > > [1] > > > > > > > > > > > > > > https://gitbox.apache.org/repos/asf?p=flink-statefun.git;a=tag;h=c08c9850147d818fc8fed877a01ff87021f3cf21 > > > > [2] https://github.com/apache/flink-statefun/tree/release-2.1.0-rc1 > > > > > > > > On Thu, Jun 4, 2020 at 12:10 PM Hequn Cheng > wrote: > > > > > > > > > It seems the release tag is not correct? The version in the poms > > should > > > > > be 2.1.0 instead of 2.1-SNAPSHOT. > > > > > > > > > > Best, > > > > > Hequn > > > > > > > > > > > > > > > On Thu, Jun 4, 2020 at 10:33 AM Congxian Qiu < > qcx978132...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > +1 (non-binding) > > > > > > > > > > > > maybe there is something that needs to be updated in > > > > README.md(currently > > > > > > the official docs link points to the master instead of 2.1) > > > > > > > > > > > > and have another question: do we need to add the command used to > > > build > > > > > the > > > > > > base image locally(which was on the README.md in release-2.0.0)? > > > > > > > > > > > > checked > > > > > > - sha & gpg, ok > > > > > > - mvn clean install -Prun-e2e-test on 1.8.0_252, ok > > > > > > - source archives do not contains any binaries > > > > > > - maven clean install -Papache-release, ok (this step need a gpg > > > secret > > > > > > key) > > > > > > - check all pom files, dockerfiles, examples point to the same > > > version, > > > > > ok > > > > > > - check READM.md, nothing unexpected. > > > > > > - but the official docs link points to the master instead of > > 2.1 > > > > > > - run greeter&ride-share demo, ok > > > > > > > > > > > > Best, > > > > > > Congxian > > > > > > > > > > > > > > > > > > Tzu-Li (Gordon) Tai 于2020年6月1日周一 下午3:25写道： > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > Please review and vote on the *release candidate #1* for the > > > version > > > > > > 2.1.0 > > > > > > > of > > > > > > > Apache Flink Stateful Functions, > > > > > > > as follows: > > > > > > > [ ] +1, Approve the release > > > > > > > [ ] -1, Do not approve the release (please provide specific > > > comments) > > > > > > > > > > > > > > ***Testing Guideline*** > > > > > > > > > > > > > > You can find here [1] a page in the project wiki on > instructions > > > for > > > > > > > testing. > > > > > > > To cast a vote, it is not necessary to perform all listed > checks, > > > > > > > but please mention which checks you have performed when voting. > > > > > > > > > > > > > > ***Release Overview*** > > > > > > > > > > > > > > As an overview, the release consists of the following: > > > > > > > a) Stateful Functions canonical source distribution, to be > > deployed > > > > to > > > > > > the > > > > > > > release repository at dist.apache.org > > > > > > > b) Stateful Functions Python SDK distributions to be deployed > to > > > PyPI > > > > > > > c) Maven artifacts to be deployed to the Maven Central > Repository > > > > > > > > > > > > > > ***Staging Areas to Rev
Re: [Discussion] flink elasticsearch connector supports
Hi Jacky Lau, 1) yes I saw that 2) I saw sources like IntegerSource which are bounded and which extend RichParallelSourceFunction. This is why I mentioned it. 3) True, there is an hadoop ES connector in Beam but it is more of a side connector, the main one is ElasticsearchIO here: https://github.com/apache/beam/blob/095589c28f5c427bf99fc0330af91c859bb2ad6b/sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java#L156 and it does not use hadoop. 4) Yes but using the same client could simplify the code in the end, but I agree it needs more change in the current code. Etienne On 05/06/2020 05:50, Jacky Lau wrote: hi Etienne Chauchot: thanks for your discussion. for 1) we do not supprt es unbouded source currently for 2) RichParallelSourceFunction is used for streaming ,InputFormat is for batch for 3) i downloaded beam just now. and the beam es connector is also using es-hadoop. i have read the code of es-hadoop(inputsplit contains shard and slice. And i think it is better when diffirent shard has diffirent number of docs), which you can seed here .https://github.com/elastic/elasticsearch-hadoop. But the code is not good. so we do not want to reference . and you can see presto, there is also just using inputsplit with shard not contains slice for 4) because flink es connectro has alreay using diffrent client (es 5 for tranport client, es 6,7 for highlevelrest), we just reuse it,which will not change too much code -- Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/
[jira] [Created] (FLINK-18151) Resolve CWE22 problems in pyflink_gateway_server.py
Hequn Cheng created FLINK-18151: --- Summary: Resolve CWE22 problems in pyflink_gateway_server.py Key: FLINK-18151 URL: https://issues.apache.org/jira/browse/FLINK-18151 Project: Flink Issue Type: Bug Components: API / Python Affects Versions: 1.10.1, 1.11.0, 1.12.0 Reporter: Hequn Cheng For example, the code `if os.path.isfile(flink_conf_path):` contains CWE22 problem that calling "os.path.isfile" with the tainted value in argument 1. This constructs a path or URI using the tainted value and may thus allow an attacker to access, modify, or test the existence of critical or sensitive files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18152) Master starts despite IllegalConfigurationException
Chesnay Schepler created FLINK-18152:
Summary: Master starts despite IllegalConfigurationException
Key: FLINK-18152
URL: https://issues.apache.org/jira/browse/FLINK-18152
Project: Flink
Issue Type: Bug
Components: Runtime / Coordination
Affects Versions: 1.11.0
Reporter: Chesnay Schepler
When no memory configuration parameters are set for the Master process then
this exception is logged by the BashJavaUtils:
{code}
Exception in thread "main"
org.apache.flink.configuration.IllegalConfigurationException: Either required
fine-grained memory (jobmanager.memory.heap.size), or Total Flink Memory size
(Key: 'jobmanager.memory.flink.size' , default: null $at
org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils.failBecauseRequiredOptionsNotConfigured(ProcessMemoryUtils.java:111)
at
org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils.memoryProcessSpecFromConfig(ProcessMemoryUtils.java:81)
at
org.apache.flink.runtime.jobmanager.JobManagerProcessUtils.processSpecFromConfig(JobManagerProcessUtils.java:76)
at
org.apache.flink.runtime.jobmanager.JobManagerProcessUtils.processSpecFromConfigWithNewOptionToInterpretLegacyHeap(JobManagerProcessUtils.java:71)
at
org.apache.flink.runtime.util.bash.BashJavaUtils.getJmResourceParams(BashJavaUtils.java:92)
at
org.apache.flink.runtime.util.bash.BashJavaUtils.runCommand(BashJavaUtils.java:66)
at
org.apache.flink.runtime.util.bash.BashJavaUtils.main(BashJavaUtils.java:54)
{code}
Afterwards, the Master process however continues to start just fine.
TaskManagers will not be started given the same situation, which is
inconsistent behavior, and it seems dangerous to start a process despite an
illegal configuration.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18153) flink-daemon does not detect errors if JVM crashes immediately
Chesnay Schepler created FLINK-18153:
Summary: flink-daemon does not detect errors if JVM crashes
immediately
Key: FLINK-18153
URL: https://issues.apache.org/jira/browse/FLINK-18153
Project: Flink
Issue Type: Bug
Components: Runtime / Coordination
Affects Versions: 1.11.0
Reporter: Chesnay Schepler
Configuring obscenely large amounts of memory that exceed the machine, which
prevents the JMV startup, does not result in any error to be printed to the
console. The logs are empty, and the only hint for an error is found in the
.out file.
{code}
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x7aaf7000,
5241403604992, 0) failed; error='Not enough space' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 5241403604992 bytes for
committing reserved memory.
{code}
This behavior is just a bit too subtle.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18155) Disable Log4j's JMX for (integration)tests
Robert Metzger created FLINK-18155: -- Summary: Disable Log4j's JMX for (integration)tests Key: FLINK-18155 URL: https://issues.apache.org/jira/browse/FLINK-18155 Project: Flink Issue Type: Improvement Components: Build System Reporter: Robert Metzger There were reports about Log4j2 initializing slowly when executing tests locally. The root cause of this has not been understood yet, but we know that Log4j is starting a JXM server, which could be responsible for the slow startup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18154) Unhelpful error message if heap.size takes up signficant portion of process.size
Chesnay Schepler created FLINK-18154:
Summary: Unhelpful error message if heap.size takes up signficant
portion of process.size
Key: FLINK-18154
URL: https://issues.apache.org/jira/browse/FLINK-18154
Project: Flink
Issue Type: Bug
Components: Runtime / Coordination
Affects Versions: 1.11.0
Reporter: Chesnay Schepler
The process still starts fine though, similar to FLINK-18152.
{code}
[] - Loading configuration property: jobmanager.memory.process.size, 2000m
[] - Loading configuration property: jobmanager.memory.heap.size, 1999m
Exception in thread "main" java.lang.IllegalArgumentException: bytes must be >= 0
at
org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:139)
at org.apache.flink.configuration.MemorySize.(MemorySize.java:82)
at
org.apache.flink.configuration.MemorySize.subtract(MemorySize.java:216)
at
org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils.deriveJvmMetaspaceAndOverheadFromTotalFlinkMemory(ProcessMemoryUtils.java:147)
at
org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils.deriveProcessSpecWithExplicitInternalMemory(ProcessMemoryUtils.java:86)
at
org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils.memoryProcessSpecFromConfig(ProcessMemoryUtils.java:71)
at
org.apache.flink.runtime.jobmanager.JobManagerProcessUtils.processSpecFromConfig(JobManagerProcessUtils.java:76)
at
org.apache.flink.runtime.jobmanager.JobManagerProcessUtils.processSpecFromConfigWithNewOptionToInterpretLegacyHeap(JobManagerProcessUtils.java:71)
at
org.apache.flink.runtime.util.bash.BashJavaUtils.getJmResourceParams(BashJavaUtils.java:92)
at
org.apache.flink.runtime.util.bash.BashJavaUtils.runCommand(BashJavaUtils.java:66)
at
org.apache.flink.runtime.util.bash.BashJavaUtils.main(BashJavaUtils.java:54)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18156) Misleading error message if derived JVM overhead is too small
Chesnay Schepler created FLINK-18156:
Summary: Misleading error message if derived JVM overhead is too
small
Key: FLINK-18156
URL: https://issues.apache.org/jira/browse/FLINK-18156
Project: Flink
Issue Type: Bug
Components: Runtime / Coordination
Affects Versions: 1.11.0
Reporter: Chesnay Schepler
The error message says that the derived size is lower then the configured
overhead range, but now such property was configured by the user.
{code}
[] - Loading configuration property: jobmanager.memory.process.size, 2000m
[] - Loading configuration property: jobmanager.memory.heap.size, 1500m
Exception in thread "main"
org.apache.flink.configuration.IllegalConfigurationException: Derived JVM
Overhead size (116.000mb (121634816 bytes)) is not in configured JVM Overhead
range [192.000mb (201326592 bytes), 1024.000mb (107374182$at
org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils.sanityCheckJvmOverhead(ProcessMemoryUtils.java:171)
at
org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils.deriveJvmMetaspaceAndOverheadFromTotalFlinkMemory(ProcessMemoryUtils.java:148)
at
org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils.deriveProcessSpecWithExplicitInternalMemory(ProcessMemoryUtils.java:86)
at
org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils.memoryProcessSpecFromConfig(ProcessMemoryUtils.java:71)
at
org.apache.flink.runtime.jobmanager.JobManagerProcessUtils.processSpecFromConfig(JobManagerProcessUtils.java:76)
at
org.apache.flink.runtime.jobmanager.JobManagerProcessUtils.processSpecFromConfigWithNewOptionToInterpretLegacyHeap(JobManagerProcessUtils.java:71)
at
org.apache.flink.runtime.util.bash.BashJavaUtils.getJmResourceParams(BashJavaUtils.java:92)
at
org.apache.flink.runtime.util.bash.BashJavaUtils.runCommand(BashJavaUtils.java:66)
at
org.apache.flink.runtime.util.bash.BashJavaUtils.main(BashJavaUtils.java:54)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18157) Jobstore size check compares against offHeapMemory
Chesnay Schepler created FLINK-18157:
Summary: Jobstore size check compares against offHeapMemory
Key: FLINK-18157
URL: https://issues.apache.org/jira/browse/FLINK-18157
Project: Flink
Issue Type: Bug
Components: Runtime / Coordination
Affects Versions: 1.11.0
Reporter: Chesnay Schepler
Setting {{jobmanager.memory.off-heap.size}} to 0 results in this confusing
error:
{code}
[] - Loading configuration property: jobmanager.memory.process.size, 2000m
[] - Loading configuration property: jobmanager.memory.heap.size, 1500m
[] - Loading configuration property: jobmanager.memory.jvm-overhead.min, 100m
[] - Loading configuration property: jobmanager.memory.jvm-overhead.max, 350m
[] - Loading configuration property: jobmanager.memory.off-heap.size, 0m
[] - The configured or derived JVM heap memory size (jobstore.cache-size: 0
bytes) is less than the configured or default size of the job store cache
(jobmanager.memory.heap.size: 50.000mb (52428800 bytes))
{code}
According to the documentation the jobstore uses the heap though.
{code}
private static JobManagerFlinkMemory createJobManagerFlinkMemory(
Configuration config,
MemorySize jvmHeap,
MemorySize offHeapMemory) {
verifyJvmHeapSize(jvmHeap);
verifyJobStoreCacheSize(config, offHeapMemory);
return new JobManagerFlinkMemory(jvmHeap, offHeapMemory);
}
private static void verifyJvmHeapSize(MemorySize jvmHeapSize) {
if (jvmHeapSize.compareTo(JobManagerOptions.MIN_JVM_HEAP_SIZE) < 1) {
LOG.warn(
"The configured or derived JVM heap memory size ({}) is
less than its recommended minimum value ({})",
jvmHeapSize.toHumanReadableString(),
JobManagerOptions.MIN_JVM_HEAP_SIZE);
}
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18158) Add a utility to create a DDL statement from avro schema
Dawid Wysakowicz created FLINK-18158: Summary: Add a utility to create a DDL statement from avro schema Key: FLINK-18158 URL: https://issues.apache.org/jira/browse/FLINK-18158 Project: Flink Issue Type: Improvement Components: Table SQL / API Reporter: Dawid Wysakowicz User asked if there is a way to create a TableSchema/Table originating from avro schema. https://lists.apache.org/thread.html/r9bd43449314230fad0b627a170db05284c9727371092fc275fc05b74%40%3Cuser.flink.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18159) Add support for trimming stack traces in user-facing components
Chesnay Schepler created FLINK-18159:
Summary: Add support for trimming stack traces in user-facing
components
Key: FLINK-18159
URL: https://issues.apache.org/jira/browse/FLINK-18159
Project: Flink
Issue Type: New Feature
Components: Command Line Client, Runtime / REST
Reporter: Chesnay Schepler
Fix For: 1.12.0
Add a verbosity query parameter / flag(==config option) into the REST API / CLI
to trim the exception stack traces, which means excluding the location
information.
The result would be something like this:
{code}
org.apache.flink.runtime.client.JobSubmissionException: Failed to submit job.
Caused by: java.lang.RuntimeException:
org.apache.flink.runtime.client.JobExecutionException: Could not set up
JobManager
Caused by: org.apache.flink.runtime.client.JobExecutionException: Could not set
up JobManager
Caused by: java.io.FileNotFoundException: Cannot find checkpoint or savepoint
file/directory 'ashudasd' on file system 'file'.
{code}
This approach renders even the biggest stack traces fairly readable, and is
rather convenient since it only requires changes in the actual user-facing
components.
Logging would not be impacted by this.
The trimming was already implemented in this
[PR|https://github.com/apache/flink/pull/12392], but the flags are missing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [ANNOUNCE] New Apache Flink Committer - Xintong Song
Congratulations! Cheers, Till On Fri, Jun 5, 2020 at 10:00 AM Dawid Wysakowicz wrote: > Congratulations! > > Best, > > Dawid > > On 05/06/2020 09:10, tison wrote: > > Congrats, Xintong! > > > > Best, > > tison. > > > > > > Jark Wu 于2020年6月5日周五 下午3:00写道： > > > >> Congratulations Xintong! > >> > >> Best, > >> Jark > >> > >> On Fri, 5 Jun 2020 at 14:32, Danny Chan wrote: > >> > >>> Congratulations Xintong ! > >>> > >>> Best, > >>> Danny Chan > >>> 在 2020年6月5日 +0800 PM2:20，dev@flink.apache.org，写道： > Congratulations Xintong >
[jira] [Created] (FLINK-18160) YARN session logs about HADOOP_CONF_DIR even though HADOOP_CLASSPATH containing a config is set
Robert Metzger created FLINK-18160:
--
Summary: YARN session logs about HADOOP_CONF_DIR even though
HADOOP_CLASSPATH containing a config is set
Key: FLINK-18160
URL: https://issues.apache.org/jira/browse/FLINK-18160
Project: Flink
Issue Type: Improvement
Components: Deployment / YARN
Affects Versions: 1.11.0
Reporter: Robert Metzger
Flink prints
{code}
Setting HADOOP_CONF_DIR=/etc/hadoop/conf because no HADOOP_CONF_DIR was set.
{code}
When running Flink on YARN with the HADOOP_CLASSPATH set. "hadoop classpath"
also returns the configuration directory, so HADOOP_CONF_DIR is not needed
anymore.
I suggest to revisit this log message / behavior as it is misleading
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18161) Changing parallelism is not possible in sql-client.sh
Robert Metzger created FLINK-18161:
--
Summary: Changing parallelism is not possible in sql-client.sh
Key: FLINK-18161
URL: https://issues.apache.org/jira/browse/FLINK-18161
Project: Flink
Issue Type: Bug
Components: Table SQL / Client
Affects Versions: 1.11.0
Reporter: Robert Metzger
I tried using
{code}
SET execution.parallelism=12
{code}
and changing the parallelism in the configuration file.
My SQL queries were always running with p=1 for all operators.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [ANNOUNCE] New Apache Flink Committer - Xintong Song
Welcome to committers and congrats, Xintong! Cheers, Andrey On Fri, Jun 5, 2020 at 4:22 PM Till Rohrmann wrote: > Congratulations! > > Cheers, > Till > > On Fri, Jun 5, 2020 at 10:00 AM Dawid Wysakowicz > wrote: > > > Congratulations! > > > > Best, > > > > Dawid > > > > On 05/06/2020 09:10, tison wrote: > > > Congrats, Xintong! > > > > > > Best, > > > tison. > > > > > > > > > Jark Wu 于2020年6月5日周五 下午3:00写道： > > > > > >> Congratulations Xintong! > > >> > > >> Best, > > >> Jark > > >> > > >> On Fri, 5 Jun 2020 at 14:32, Danny Chan wrote: > > >> > > >>> Congratulations Xintong ! > > >>> > > >>> Best, > > >>> Danny Chan > > >>> 在 2020年6月5日 +0800 PM2:20，dev@flink.apache.org，写道： > > Congratulations Xintong > > >
[jira] [Created] (FLINK-18162) AddSplitEvents should serialize the splits into bytes.
Jiangjie Qin created FLINK-18162:
Summary: AddSplitEvents should serialize the splits into bytes.
Key: FLINK-18162
URL: https://issues.apache.org/jira/browse/FLINK-18162
Project: Flink
Issue Type: Bug
Components: Connectors / Common
Reporter: Jiangjie Qin
{{AddSplitsEvent}} is a serializable at the moment which contains a list of
splits. However, the {{SourceSplit}} is not a subclass of {{Serializable}}. We
need to serialize the splits in the {{AddSplitsEvent}} using SplitSerializer
before sending it over the wire and deserialize the splits on the reader side.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [ANNOUNCE] New Apache Flink Committer - Xintong Song
Congratulations! Best, Zhijiang -- From:Andrey Zagrebin Send Time:2020年6月5日(星期五) 22:34 To:dev Subject:Re: [ANNOUNCE] New Apache Flink Committer - Xintong Song Welcome to committers and congrats, Xintong! Cheers, Andrey On Fri, Jun 5, 2020 at 4:22 PM Till Rohrmann wrote: > Congratulations! > > Cheers, > Till > > On Fri, Jun 5, 2020 at 10:00 AM Dawid Wysakowicz > wrote: > > > Congratulations! > > > > Best, > > > > Dawid > > > > On 05/06/2020 09:10, tison wrote: > > > Congrats, Xintong! > > > > > > Best, > > > tison. > > > > > > > > > Jark Wu 于2020年6月5日周五 下午3:00写道： > > > > > >> Congratulations Xintong! > > >> > > >> Best, > > >> Jark > > >> > > >> On Fri, 5 Jun 2020 at 14:32, Danny Chan wrote: > > >> > > >>> Congratulations Xintong ! > > >>> > > >>> Best, > > >>> Danny Chan > > >>> 在 2020年6月5日 +0800 PM2:20，dev@flink.apache.org，写道： > > Congratulations Xintong > > >
[jira] [Created] (FLINK-18163) Should be volatile: network.api.writer.RecordWriter.flusherException
Roman Khachatryan created FLINK-18163: - Summary: Should be volatile: network.api.writer.RecordWriter.flusherException Key: FLINK-18163 URL: https://issues.apache.org/jira/browse/FLINK-18163 Project: Flink Issue Type: Bug Components: Runtime / Task Affects Versions: 1.11.0 Reporter: Roman Khachatryan Assignee: Roman Khachatryan Fix For: 1.12.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18164) null <> 'str' should be true
Benchao Li created FLINK-18164: -- Summary: null <> 'str' should be true Key: FLINK-18164 URL: https://issues.apache.org/jira/browse/FLINK-18164 Project: Flink Issue Type: Bug Components: Table SQL / API Reporter: Benchao Li Currently, if we compare null with other literals, the result will always be false. It's because the code gen always gives a default value (false) for the result. And I think it's a bug if `null <> 'str'` is false. It's reported from user-zh: http://apache-flink.147419.n8.nabble.com/flink-sql-null-false-td3640.html CC [~jark] [~ykt836] -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [ANNOUNCE] New Apache Flink Committer - Xintong Song
Congratulations! --- Best, Matt Wang On 06/5/2020 22:34，Andrey Zagrebin wrote： Welcome to committers and congrats, Xintong! Cheers, Andrey On Fri, Jun 5, 2020 at 4:22 PM Till Rohrmann wrote: Congratulations! Cheers, Till On Fri, Jun 5, 2020 at 10:00 AM Dawid Wysakowicz wrote: Congratulations! Best, Dawid On 05/06/2020 09:10, tison wrote: Congrats, Xintong! Best, tison. Jark Wu 于2020年6月5日周五 下午3:00写道： Congratulations Xintong! Best, Jark On Fri, 5 Jun 2020 at 14:32, Danny Chan wrote: Congratulations Xintong ! Best, Danny Chan 在 2020年6月5日 +0800 PM2:20，dev@flink.apache.org，写道： Congratulations Xintong
Re: [DISCUSS] (Document) Backwards Compatibility of Savepoints
Sorry for jumping in late. Currently, we only have a forward-compatible guarantee and do not have the backward-compatible guarantee. And as this may take a large effort to support the backward-compatible guarantee. so I agree that we should write this down explicitly. For the given scenario, I have a little question: Why do we want to restore from the savepoint taken the new Flink version instead of the previous savepoint, is that we want to minimize the source rewind? Best, Congxian Steven Wu 于2020年6月3日周三 上午9:08写道： > Current Flink documentation is actually pretty clear about no guarantees > for backward compatibility. > > https://ci.apache.org/projects/flink/flink-docs-stable/ops/upgrading.html#compatibility-table > > On Tue, Jun 2, 2020 at 3:20 AM Yun Tang wrote: > > > Since Flink lacks of such kind of experiments to ensure the backwards > > compatibility of savepoints before, especially those built-in operators > > with their own operator state. > > I am afraid we need huge energy to cover all cases to give the most > > correct result. > > > > I prefer to just point out this in documentation to say explicitly Flink > > does not guarantee such kind of backwards compatibility. > > > > Best > > Yun Tang > > > > From: Ufuk Celebi > > Sent: Wednesday, May 27, 2020 16:42 > > To: dev@flink.apache.org > > Subject: Re: [DISCUSS] (Document) Backwards Compatibility of Savepoints > > > > I agree with Konstantin and Steven that it makes sense to point this out > > explicitly. > > > > I think that the following would be helpful: > > > > 1/ Mention breaking compatibility in release notes > > > > 2/ Update the linked table to reflect compatibilities while pointing out > > what the community commits to maintain going forward (e.g. "happens to > > work" vs. "guaranteed to work") > > > > In general, the table is quite large. Would it make sense to order the > > releases in reverse order (assuming that the table is more relevant for > > recent releases)? > > > > – Ufuk > > > > On Tue, May 26, 2020 at 8:36 PM Steven Wu wrote: > > > > > > A use case for this might be when you want to rollback a framework > > > upgrade (after some time) due to e.g. a performance > > > or stability issue. > > > > > > Downgrade (that Konstantin called out) is an important and realistic > > > scenario. It will be great to support backward compatibility for > > savepoint > > > or at least document any breaking change. > > > > > > On Tue, May 26, 2020 at 4:39 AM Piotr Nowojski > > > wrote: > > > > > > > Hi, > > > > > > > > It might have been implicit choice, but so far we were not supporting > > the > > > > scenario that you are asking for. It has never been tested and we > have > > > > lot’s of state migration code sprinkled among our code base (for > > example > > > > upgrading state fields of the operators like [1]), that only supports > > > > upgrades, not downgrades. > > > > > > > > Also we do not have testing infrastructure for checking the > downgrades. > > > We > > > > would need to check if save points taken from master branch, are > > readable > > > > by previous releases (not release branch!). > > > > > > > > So all in all, I don’t think it can be easily done. It would require > > some > > > > effort to start maintaining backward compatibility. > > > > > > > > Piotrek > > > > > > > > [1] > > > > > > > > > > org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011#migrateNextTransactionalIdHindState > > > > > > > > > On 26 May 2020, at 13:18, Konstantin Knauf > > wrote: > > > > > > > > > > Hi everyone, > > > > > > > > > > I recently stumbled across the fact that Savepoints created with > > Flink > > > > 1.11 > > > > > can not be read by Flink 1.10. A use case for this might be when > you > > > want > > > > > to rollback a framework upgrade (after some time) due to e.g. a > > > > performance > > > > > or stability issue. > > > > > > > > > > From the documentation [1] it seems as if the Savepoint format is > > > > generally > > > > > only forward-compatible although in many cases it is actually also > > > > > backwards compatible (e.g. Savepoint taken in Flink 1.10, restored > > with > > > > > Flink 1.9). > > > > > > > > > > Was it a deliberate choice not to document any backwards > > compatibility? > > > > If > > > > > not, should we add the missing entries in the compatibility table? > > > > > > > > > > Thanks, > > > > > > > > > > Konstantin > > > > > > > > > > [1] > > > > > > > > > > > > > > > https://ci.apache.org/projects/flink/flink-docs-master/ops/upgrading.html#compatibility-table > > > > > > > > > > -- > > > > > > > > > > Konstantin Knauf > > > > > > > > > > https://twitter.com/snntrable > > > > > > > > > > https://github.com/knaufk > > > > > > > > > > > > > >
